Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change yurthub's protocol from http to https #386

Merged
merged 1 commit into from
Jul 30, 2021
Merged

change yurthub's protocol from http to https #386

merged 1 commit into from
Jul 30, 2021

Conversation

luckymrwang
Copy link
Member

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #361

Special notes for your reviewer:

Does this PR introduce a user-facing change?


other Note

@openyurt-bot
Copy link
Collaborator

@luckymrwang: GitHub didn't allow me to assign the following users: your_reviewer.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:
/kind bug
/kind documentation
/kind enhancement
/kind good-first-issue
/kind feature
/kind question
/kind design
/sig ai
/sig iot
/sig network
/sig storage
/sig storage

/kind feature

What this PR does / why we need it:

Which issue(s) this PR fixes:

Fixes #361

Special notes for your reviewer:

Does this PR introduce a user-facing change?


other Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openyurt-bot openyurt-bot added the kind/feature kind/feature label Jul 20, 2021
@openyurt-bot openyurt-bot added the size/XL size/XL: 500-999 label Jul 20, 2021
@luckymrwang
Copy link
Member Author

luckymrwang commented Jul 20, 2021

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:00:47Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
  • OS (e.g: cat /etc/os-release):
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
  • Startup log (hubself mode):
I0610 15:37:30.432488       1 start.go:58] FLAG: --add_dir_header="false"
I0610 15:37:30.432521       1 start.go:58] FLAG: --alsologtostderr="false"
I0610 15:37:30.432524       1 start.go:58] FLAG: --bind-address="127.0.0.1"
I0610 15:37:30.432528       1 start.go:58] FLAG: --cert-mgr-mode="hubself"
I0610 15:37:30.432531       1 start.go:58] FLAG: --disk-cache-path="/etc/kubernetes/cache/"
I0610 15:37:30.432534       1 start.go:58] FLAG: --dummy-if-ip="169.254.2.1"
I0610 15:37:30.432536       1 start.go:58] FLAG: --dummy-if-name="yurthub-dummy0"
I0610 15:37:30.432539       1 start.go:58] FLAG: --enable-dummy-if="true"
I0610 15:37:30.432542       1 start.go:58] FLAG: --enable-iptables="true"
yurthub version: projectinfo.Info{GitVersion:"v0.0.0", GitCommit:"unknown", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
I0610 15:37:30.432545       1 start.go:58] FLAG: --gc-frequency="120"
I0610 15:37:30.432552       1 start.go:58] FLAG: --heartbeat-failed-retry="3"
I0610 15:37:30.432554       1 start.go:58] FLAG: --heartbeat-healthy-threshold="2"
I0610 15:37:30.432557       1 start.go:58] FLAG: --heartbeat-timeout-seconds="2"
I0610 15:37:30.432559       1 start.go:58] FLAG: --help="false"
I0610 15:37:30.432561       1 start.go:58] FLAG: --join-token=""
I0610 15:37:30.432564       1 start.go:58] FLAG: --kubelet-ca-file="/etc/kubernetes/pki/ca.crt"
I0610 15:37:30.432567       1 start.go:58] FLAG: --kubelet-client-certificate="/var/lib/kubelet/pki/kubelet-client-current.pem"
I0610 15:37:30.432571       1 start.go:58] FLAG: --lb-mode="rr"
I0610 15:37:30.432573       1 start.go:58] FLAG: --log-flush-frequency="5s"
I0610 15:37:30.432577       1 start.go:58] FLAG: --log_backtrace_at=":0"
I0610 15:37:30.432581       1 start.go:58] FLAG: --log_dir=""
I0610 15:37:30.432583       1 start.go:58] FLAG: --log_file=""
I0610 15:37:30.432586       1 start.go:58] FLAG: --log_file_max_size="1800"
I0610 15:37:30.432588       1 start.go:58] FLAG: --logtostderr="true"
I0610 15:37:30.432591       1 start.go:58] FLAG: --max-requests-in-flight="250"
I0610 15:37:30.432594       1 start.go:58] FLAG: --node-name="k8s-node3"
I0610 15:37:30.432596       1 start.go:58] FLAG: --profiling="true"
I0610 15:37:30.432599       1 start.go:58] FLAG: --proxy-port="10261"
I0610 15:37:30.432601       1 start.go:58] FLAG: --proxy-secure-port="10268"
I0610 15:37:30.432604       1 start.go:58] FLAG: --root-dir="/var/lib/yurthub"
I0610 15:37:30.432607       1 start.go:58] FLAG: --serve-port="10267"
I0610 15:37:30.432609       1 start.go:58] FLAG: --server-addr="https://10.211.55.18:6443"
I0610 15:37:30.432612       1 start.go:58] FLAG: --skip_headers="false"
I0610 15:37:30.432614       1 start.go:58] FLAG: --skip_log_headers="false"
I0610 15:37:30.432617       1 start.go:58] FLAG: --stderrthreshold="2"
I0610 15:37:30.432619       1 start.go:58] FLAG: --v="4"
I0610 15:37:30.432623       1 start.go:58] FLAG: --version="false"
I0610 15:37:30.432626       1 start.go:58] FLAG: --vmodule=""
I0610 15:37:30.432647       1 config.go:136] yurthub would connect remote servers: https://10.211.55.18:6443
I0610 15:37:30.432950       1 start.go:68] yurthub cfg: &config.YurtHubConfiguration{LBMode:"rr", RemoteServers:[]*url.URL{(*url.URL)(0xc00034e1b0)}, YurtHubServerAddr:"127.0.0.1:10267", YurtHubProxyServerAddr:"127.0.0.1:10261", YurtHubProxyServerSecureAddr:"127.0.0.1:10268", YurtHubProxyServerDummyAddr:"169.254.2.1:10261", YurtHubProxyServerSecureDummyAddr:"169.254.2.1:10268", GCFrequency:120, CertMgrMode:"hubself", KubeletRootCAFilePath:"/etc/kubernetes/pki/ca.crt", KubeletPairFilePath:"/var/lib/kubelet/pki/kubelet-client-current.pem", NodeName:"k8s-node3", HeartbeatFailedRetry:3, HeartbeatHealthyThreshold:2, HeartbeatTimeoutSeconds:2, MaxRequestInFlight:250, JoinToken:"", RootDir:"/var/lib/yurthub", EnableProfiling:true, EnableDummyIf:true, EnableIptables:true, HubAgentDummyIfName:"yurthub-dummy0", StorageWrapper:(*cachemanager.storageWrapper)(0xc00003ad80), SerializerManager:(*serializer.SerializerManager)(0xc00003ae00), TLSConfig:(*tls.Config)(nil)}
I0610 15:37:30.433007       1 start.go:83] 1. register cert managers
I0610 15:37:30.433024       1 certificate.go:60] Registered certificate manager kubelet
I0610 15:37:30.433029       1 certificate.go:60] Registered certificate manager hubself
I0610 15:37:30.433033       1 start.go:89] 2. create cert manager with hubself mode
I0610 15:37:30.433076       1 cert_mgr.go:220] /var/lib/yurthub/pki/ca.crt file not exists, so create it
I0610 15:37:30.440271       1 cert_mgr.go:127] use /var/lib/yurthub/pki/ca.crt ca file to bootstrap yurthub
I0610 15:37:30.440340       1 cert_mgr.go:280] yurthub bootstrap conf file does not exist, so create it
I0610 15:37:30.441984       1 certificate_manager.go:282] Certificate rotation is enabled.
I0610 15:37:30.442013       1 cert_mgr.go:418] /var/lib/yurthub/yurthub.conf file not exists, so create it
I0610 15:37:30.442674       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:30.442708       1 certificate_manager.go:409] Rotating certificates
I0610 15:37:35.444099       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:40.446269       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:45.445038       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:50.443761       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:55.443295       1 certificate.go:83] waiting for preparing client certificate
I0610 15:38:00.443689       1 certificate.go:83] waiting for preparing client certificate
I0610 15:38:00.451122       1 cert_mgr.go:378] use bootstrap client config to create csr client
I0610 15:38:00.451609       1 cert_mgr.go:346] no join token, so use kubelet config to bootstrap hub
I0610 15:38:00.452523       1 cert_mgr.go:390] bootstrap client config: &rest.Config{Host:"https://10.211.55.18:6443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/var/lib/kubelet/pki/kubelet-client-current.pem", KeyFile:"/var/lib/kubelet/pki/kubelet-client-current.pem", CAFile:"/etc/kubernetes/pki/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(nil), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil)}
I0610 15:38:00.452686       1 cert_mgr.go:398] avoid tcp conn leak, close old tcp conn that used to rotate certificate
I0610 15:38:00.452709       1 connrotation.go:110] forcibly close 0 connections on 10.211.55.18:6443 for hub certificate manager dialer
I0610 15:38:00.454279       1 cert_rotation.go:137] Starting client certificate rotation controller
I0610 15:38:00.455453       1 connrotation.go:145] create a connection from 10.211.55.20:52634 to 10.211.55.18:6443, total 1 connections in hub certificate manager dialer
I0610 15:38:00.464106       1 reflector.go:175] Starting reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:00.464137       1 reflector.go:211] Listing and watching *v1beta1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:00.477572       1 csr.go:124] certificate signing request csr-c4bxd is approved, waiting to be issued
I0610 15:38:00.491725       1 csr.go:121] certificate signing request csr-c4bxd is issued
I0610 15:38:00.491806       1 reflector.go:181] Stopping reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:01.495075       1 certificate_manager.go:553] Certificate expiration is 2022-04-28 09:13:41 +0000 UTC, rotation deadline is 2022-03-15 11:16:15.359179016 +0000 UTC
I0610 15:38:01.495211       1 certificate_manager.go:288] Waiting 6667h38m13.863974479s for next certificate rotation
I0610 15:38:05.443850       1 start.go:97] 3. new transport manager
I0610 15:38:05.443906       1 transport.go:57] use /var/lib/yurthub/pki/ca.crt ca cert file to access remote server
I0610 15:38:05.444139       1 start.go:105] 4. create health checker for remote servers 
I0610 15:38:05.444958       1 connrotation.go:145] create a connection from 10.211.55.20:52636 to 10.211.55.18:6443, total 1 connections in transport manager dialer
I0610 15:38:05.464615       1 start.go:114] 5. new restConfig manager for hubself mode
I0610 15:38:05.464638       1 start.go:122] 6. create tls config for secure servers 
I0610 15:38:05.465171       1 config.go:107] re-fix hub rest config host successfully with server https://10.211.55.18:6443
I0610 15:38:05.465547       1 cert_rotation.go:137] Starting client certificate rotation controller
I0610 15:38:05.466450       1 certmanager.go:47] subject of yurthub server certificate
I0610 15:38:05.466493       1 certificate_store.go:130] Loading cert/key pair from "/var/lib/yurthub/pki/yurthub-current.pem".
I0610 15:38:05.466655       1 certificate_manager.go:282] Certificate rotation is enabled.
I0610 15:38:05.466701       1 certificate_manager.go:488] Current certificate CN (system:node:k8s-node3) does not match requested CN (kube-apiserver-kubelet-client)
I0610 15:38:05.466712       1 certificate_manager.go:409] Rotating certificates
I0610 15:38:05.466750       1 certificate_manager.go:488] Current certificate CN (system:node:k8s-node3) does not match requested CN (kube-apiserver-kubelet-client)
I0610 15:38:05.473559       1 reflector.go:175] Starting reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:05.473581       1 reflector.go:211] Listing and watching *v1beta1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:05.481582       1 csr.go:124] certificate signing request csr-lz9vj is approved, waiting to be issued
I0610 15:38:05.488916       1 csr.go:121] certificate signing request csr-lz9vj is issued
I0610 15:38:05.488983       1 reflector.go:181] Stopping reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:06.490588       1 certificate_manager.go:553] Certificate expiration is 2022-04-28 09:13:46 +0000 UTC, rotation deadline is 2022-03-18 10:39:42.549807508 +0000 UTC
I0610 15:38:06.490636       1 certificate_manager.go:288] Waiting 6739h1m36.059175259s for next certificate rotation
I0610 15:38:07.492108       1 certificate_manager.go:553] Certificate expiration is 2022-04-28 09:13:46 +0000 UTC, rotation deadline is 2022-01-09 07:57:00.670505088 +0000 UTC
I0610 15:38:07.492147       1 certificate_manager.go:288] Waiting 5104h18m53.178360717s for next certificate rotation
I0610 15:38:10.467331       1 start.go:130] 7. new cache manager with storage wrapper and serializer manager
I0610 15:38:10.467383       1 cache_agent.go:68] reset cache agents to [kubelet kube-proxy flanneld coredns yurttunnel-agent]
I0610 15:38:10.468595       1 start.go:138] 8. new gc manager for node k8s-node3, and gc frequency is a random time between 120 min and 360 min
I0610 15:38:10.468620       1 start.go:147] 9. new reverse proxy handler for remote servers
I0610 15:38:10.468646       1 start.go:156] 10. create dummy network interface yurthub-dummy0 and init iptables manager
I0610 15:38:10.468978       1 gc.go:74] start gc events after waiting 336.493µs from previous gc
I0610 15:38:10.469487       1 config.go:107] re-fix hub rest config host successfully with server https://10.211.55.18:6443
I0610 15:38:10.470462       1 gc.go:160] no kubelet events in local storage, skip kubelet events gc
I0610 15:38:10.470474       1 gc.go:160] no kube-proxy events in local storage, skip kube-proxy events gc
I0610 15:38:10.474838       1 iptables.go:671] couldn't get iptables-restore version; assuming it doesn't support --wait
I0610 15:38:10.566008       1 start.go:164] 11. new yurthub server and begin to serve, dummy proxy server: 169.254.2.1:10261, secure dummy proxy server: 169.254.2.1:10268
I0610 15:38:10.566025       1 start.go:167] 11. new yurthub server and begin to serve, proxy server: 127.0.0.1:10261, secure proxy server: 127.0.0.1:10268, hub server: 127.0.0.1:10267
[root@k8s-node3 manifests]# docker logs 87eb429f5aea
I0610 15:37:30.432488       1 start.go:58] FLAG: --add_dir_header="false"
I0610 15:37:30.432521       1 start.go:58] FLAG: --alsologtostderr="false"
I0610 15:37:30.432524       1 start.go:58] FLAG: --bind-address="127.0.0.1"
I0610 15:37:30.432528       1 start.go:58] FLAG: --cert-mgr-mode="hubself"
I0610 15:37:30.432531       1 start.go:58] FLAG: --disk-cache-path="/etc/kubernetes/cache/"
I0610 15:37:30.432534       1 start.go:58] FLAG: --dummy-if-ip="169.254.2.1"
I0610 15:37:30.432536       1 start.go:58] FLAG: --dummy-if-name="yurthub-dummy0"
I0610 15:37:30.432539       1 start.go:58] FLAG: --enable-dummy-if="true"
I0610 15:37:30.432542       1 start.go:58] FLAG: --enable-iptables="true"
yurthub version: projectinfo.Info{GitVersion:"v0.0.0", GitCommit:"unknown", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
I0610 15:37:30.432545       1 start.go:58] FLAG: --gc-frequency="120"
I0610 15:37:30.432552       1 start.go:58] FLAG: --heartbeat-failed-retry="3"
I0610 15:37:30.432554       1 start.go:58] FLAG: --heartbeat-healthy-threshold="2"
I0610 15:37:30.432557       1 start.go:58] FLAG: --heartbeat-timeout-seconds="2"
I0610 15:37:30.432559       1 start.go:58] FLAG: --help="false"
I0610 15:37:30.432561       1 start.go:58] FLAG: --join-token=""
I0610 15:37:30.432564       1 start.go:58] FLAG: --kubelet-ca-file="/etc/kubernetes/pki/ca.crt"
I0610 15:37:30.432567       1 start.go:58] FLAG: --kubelet-client-certificate="/var/lib/kubelet/pki/kubelet-client-current.pem"
I0610 15:37:30.432571       1 start.go:58] FLAG: --lb-mode="rr"
I0610 15:37:30.432573       1 start.go:58] FLAG: --log-flush-frequency="5s"
I0610 15:37:30.432577       1 start.go:58] FLAG: --log_backtrace_at=":0"
I0610 15:37:30.432581       1 start.go:58] FLAG: --log_dir=""
I0610 15:37:30.432583       1 start.go:58] FLAG: --log_file=""
I0610 15:37:30.432586       1 start.go:58] FLAG: --log_file_max_size="1800"
I0610 15:37:30.432588       1 start.go:58] FLAG: --logtostderr="true"
I0610 15:37:30.432591       1 start.go:58] FLAG: --max-requests-in-flight="250"
I0610 15:37:30.432594       1 start.go:58] FLAG: --node-name="k8s-node3"
I0610 15:37:30.432596       1 start.go:58] FLAG: --profiling="true"
I0610 15:37:30.432599       1 start.go:58] FLAG: --proxy-port="10261"
I0610 15:37:30.432601       1 start.go:58] FLAG: --proxy-secure-port="10268"
I0610 15:37:30.432604       1 start.go:58] FLAG: --root-dir="/var/lib/yurthub"
I0610 15:37:30.432607       1 start.go:58] FLAG: --serve-port="10267"
I0610 15:37:30.432609       1 start.go:58] FLAG: --server-addr="https://10.211.55.18:6443"
I0610 15:37:30.432612       1 start.go:58] FLAG: --skip_headers="false"
I0610 15:37:30.432614       1 start.go:58] FLAG: --skip_log_headers="false"
I0610 15:37:30.432617       1 start.go:58] FLAG: --stderrthreshold="2"
I0610 15:37:30.432619       1 start.go:58] FLAG: --v="4"
I0610 15:37:30.432623       1 start.go:58] FLAG: --version="false"
I0610 15:37:30.432626       1 start.go:58] FLAG: --vmodule=""
I0610 15:37:30.432647       1 config.go:136] yurthub would connect remote servers: https://10.211.55.18:6443
I0610 15:37:30.432950       1 start.go:68] yurthub cfg: &config.YurtHubConfiguration{LBMode:"rr", RemoteServers:[]*url.URL{(*url.URL)(0xc00034e1b0)}, YurtHubServerAddr:"127.0.0.1:10267", YurtHubProxyServerAddr:"127.0.0.1:10261", YurtHubProxyServerSecureAddr:"127.0.0.1:10268", YurtHubProxyServerDummyAddr:"169.254.2.1:10261", YurtHubProxyServerSecureDummyAddr:"169.254.2.1:10268", GCFrequency:120, CertMgrMode:"hubself", KubeletRootCAFilePath:"/etc/kubernetes/pki/ca.crt", KubeletPairFilePath:"/var/lib/kubelet/pki/kubelet-client-current.pem", NodeName:"k8s-node3", HeartbeatFailedRetry:3, HeartbeatHealthyThreshold:2, HeartbeatTimeoutSeconds:2, MaxRequestInFlight:250, JoinToken:"", RootDir:"/var/lib/yurthub", EnableProfiling:true, EnableDummyIf:true, EnableIptables:true, HubAgentDummyIfName:"yurthub-dummy0", StorageWrapper:(*cachemanager.storageWrapper)(0xc00003ad80), SerializerManager:(*serializer.SerializerManager)(0xc00003ae00), TLSConfig:(*tls.Config)(nil)}
I0610 15:37:30.433007       1 start.go:83] 1. register cert managers
I0610 15:37:30.433024       1 certificate.go:60] Registered certificate manager kubelet
I0610 15:37:30.433029       1 certificate.go:60] Registered certificate manager hubself
I0610 15:37:30.433033       1 start.go:89] 2. create cert manager with hubself mode
I0610 15:37:30.433076       1 cert_mgr.go:220] /var/lib/yurthub/pki/ca.crt file not exists, so create it
I0610 15:37:30.440271       1 cert_mgr.go:127] use /var/lib/yurthub/pki/ca.crt ca file to bootstrap yurthub
I0610 15:37:30.440340       1 cert_mgr.go:280] yurthub bootstrap conf file does not exist, so create it
I0610 15:37:30.441984       1 certificate_manager.go:282] Certificate rotation is enabled.
I0610 15:37:30.442013       1 cert_mgr.go:418] /var/lib/yurthub/yurthub.conf file not exists, so create it
I0610 15:37:30.442674       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:30.442708       1 certificate_manager.go:409] Rotating certificates
I0610 15:37:35.444099       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:40.446269       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:45.445038       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:50.443761       1 certificate.go:83] waiting for preparing client certificate
I0610 15:37:55.443295       1 certificate.go:83] waiting for preparing client certificate
I0610 15:38:00.443689       1 certificate.go:83] waiting for preparing client certificate
I0610 15:38:00.451122       1 cert_mgr.go:378] use bootstrap client config to create csr client
I0610 15:38:00.451609       1 cert_mgr.go:346] no join token, so use kubelet config to bootstrap hub
I0610 15:38:00.452523       1 cert_mgr.go:390] bootstrap client config: &rest.Config{Host:"https://10.211.55.18:6443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/var/lib/kubelet/pki/kubelet-client-current.pem", KeyFile:"/var/lib/kubelet/pki/kubelet-client-current.pem", CAFile:"/etc/kubernetes/pki/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(nil), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil)}
I0610 15:38:00.452686       1 cert_mgr.go:398] avoid tcp conn leak, close old tcp conn that used to rotate certificate
I0610 15:38:00.452709       1 connrotation.go:110] forcibly close 0 connections on 10.211.55.18:6443 for hub certificate manager dialer
I0610 15:38:00.454279       1 cert_rotation.go:137] Starting client certificate rotation controller
I0610 15:38:00.455453       1 connrotation.go:145] create a connection from 10.211.55.20:52634 to 10.211.55.18:6443, total 1 connections in hub certificate manager dialer
I0610 15:38:00.464106       1 reflector.go:175] Starting reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:00.464137       1 reflector.go:211] Listing and watching *v1beta1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:00.477572       1 csr.go:124] certificate signing request csr-c4bxd is approved, waiting to be issued
I0610 15:38:00.491725       1 csr.go:121] certificate signing request csr-c4bxd is issued
I0610 15:38:00.491806       1 reflector.go:181] Stopping reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:01.495075       1 certificate_manager.go:553] Certificate expiration is 2022-04-28 09:13:41 +0000 UTC, rotation deadline is 2022-03-15 11:16:15.359179016 +0000 UTC
I0610 15:38:01.495211       1 certificate_manager.go:288] Waiting 6667h38m13.863974479s for next certificate rotation
I0610 15:38:05.443850       1 start.go:97] 3. new transport manager
I0610 15:38:05.443906       1 transport.go:57] use /var/lib/yurthub/pki/ca.crt ca cert file to access remote server
I0610 15:38:05.444139       1 start.go:105] 4. create health checker for remote servers 
I0610 15:38:05.444958       1 connrotation.go:145] create a connection from 10.211.55.20:52636 to 10.211.55.18:6443, total 1 connections in transport manager dialer
I0610 15:38:05.464615       1 start.go:114] 5. new restConfig manager for hubself mode
I0610 15:38:05.464638       1 start.go:122] 6. create tls config for secure servers 
I0610 15:38:05.465171       1 config.go:107] re-fix hub rest config host successfully with server https://10.211.55.18:6443
I0610 15:38:05.465547       1 cert_rotation.go:137] Starting client certificate rotation controller
I0610 15:38:05.466450       1 certmanager.go:47] subject of yurthub server certificate
I0610 15:38:05.466493       1 certificate_store.go:130] Loading cert/key pair from "/var/lib/yurthub/pki/yurthub-current.pem".
I0610 15:38:05.466655       1 certificate_manager.go:282] Certificate rotation is enabled.
I0610 15:38:05.466701       1 certificate_manager.go:488] Current certificate CN (system:node:k8s-node3) does not match requested CN (kube-apiserver-kubelet-client)
I0610 15:38:05.466712       1 certificate_manager.go:409] Rotating certificates
I0610 15:38:05.466750       1 certificate_manager.go:488] Current certificate CN (system:node:k8s-node3) does not match requested CN (kube-apiserver-kubelet-client)
I0610 15:38:05.473559       1 reflector.go:175] Starting reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:05.473581       1 reflector.go:211] Listing and watching *v1beta1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:05.481582       1 csr.go:124] certificate signing request csr-lz9vj is approved, waiting to be issued
I0610 15:38:05.488916       1 csr.go:121] certificate signing request csr-lz9vj is issued
I0610 15:38:05.488983       1 reflector.go:181] Stopping reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0610 15:38:06.490588       1 certificate_manager.go:553] Certificate expiration is 2022-04-28 09:13:46 +0000 UTC, rotation deadline is 2022-03-18 10:39:42.549807508 +0000 UTC
I0610 15:38:06.490636       1 certificate_manager.go:288] Waiting 6739h1m36.059175259s for next certificate rotation
I0610 15:38:07.492108       1 certificate_manager.go:553] Certificate expiration is 2022-04-28 09:13:46 +0000 UTC, rotation deadline is 2022-01-09 07:57:00.670505088 +0000 UTC
I0610 15:38:07.492147       1 certificate_manager.go:288] Waiting 5104h18m53.178360717s for next certificate rotation
I0610 15:38:10.467331       1 start.go:130] 7. new cache manager with storage wrapper and serializer manager
I0610 15:38:10.467383       1 cache_agent.go:68] reset cache agents to [kubelet kube-proxy flanneld coredns yurttunnel-agent]
I0610 15:38:10.468595       1 start.go:138] 8. new gc manager for node k8s-node3, and gc frequency is a random time between 120 min and 360 min
I0610 15:38:10.468620       1 start.go:147] 9. new reverse proxy handler for remote servers
I0610 15:38:10.468646       1 start.go:156] 10. create dummy network interface yurthub-dummy0 and init iptables manager
I0610 15:38:10.468978       1 gc.go:74] start gc events after waiting 336.493µs from previous gc
I0610 15:38:10.469487       1 config.go:107] re-fix hub rest config host successfully with server https://10.211.55.18:6443
I0610 15:38:10.470462       1 gc.go:160] no kubelet events in local storage, skip kubelet events gc
I0610 15:38:10.470474       1 gc.go:160] no kube-proxy events in local storage, skip kube-proxy events gc
I0610 15:38:10.474838       1 iptables.go:671] couldn't get iptables-restore version; assuming it doesn't support --wait
I0610 15:38:10.566008       1 start.go:164] 11. new yurthub server and begin to serve, dummy proxy server: 169.254.2.1:10261, secure dummy proxy server: 169.254.2.1:10268
I0610 15:38:10.566025       1 start.go:167] 11. new yurthub server and begin to serve, proxy server: 127.0.0.1:10261, secure proxy server: 127.0.0.1:10268, hub server: 127.0.0.1:10267


func startYurtHubCSRApproverController(ctx ControllerContext) (http.Handler, bool, error) {
clientSet := ctx.ClientBuilder.ClientOrDie("node-controller")
sharedInformerFactory := informers.NewSharedInformerFactory(clientSet, 10*time.Second)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not need to new sharedInformerFactory, we can use InformerFactory in ctx parameter.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest code has been commited.Using InformerFactory in ctx parameter.

}

// NewYurtHubServer creates a Server object
func NewYurtHubServer(cfg *config.YurtHubConfiguration,
certificateMgr interfaces.YurtCertificateManager,
proxyHandler http.Handler) (Server, error) {
proxyHandler http.Handler, stopCh <-chan struct{}) (Server, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

stopCh is not used?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest code has been commited. stopCh parameter has been removed.

limitations under the License.
*/

package yurthub
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

package name yurthub is component name and maybe makes user confused, so use the function of this package will be more understandable. so how about rename the package name to certificates?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and file name has a typo error. crsapprover.go --> csrapprover.go

"k8s.io/client-go/kubernetes"
clicert "k8s.io/client-go/kubernetes/typed/certificates/v1beta1"
"k8s.io/client-go/util/certificate"
"k8s.io/klog/v2"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yurthub uses k8s.io/klog instead of k8s.io/klog/v2

typev1beta1 "k8s.io/client-go/kubernetes/typed/certificates/v1beta1"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/util/workqueue"
"k8s.io/klog/v2"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yurt-controller-manager uses k8s.io/klog instead of k8s.io/klog/v2

certificates.CertificateSigningRequestCondition{
Type: certificates.CertificateApproved,
Reason: "AutoApproved",
Message: fmt.Sprintf("self-approving %s csr", projectinfo.GetTunnelName()),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

projectinfo.GetTunnelName() --> projectinfo.GetHubName()

}

if !isYurtHubCSR(csr) {
klog.Infof("csr(%s) is not %s csr", csr.GetName(), projectinfo.GetTunnelName())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

projectinfo.GetTunnelName() --> projectinfo.GetHubName()

klog.Errorf("failed to approve %s csr(%s), %v", projectinfo.GetTunnelName(), csr.GetName(), err)
return err
}
klog.Infof("successfully approve %s csr(%s)", projectinfo.GetTunnelName(), result.Name)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

projectinfo.GetTunnelName() --> projectinfo.GetHubName()


result, err := csrClient.UpdateApproval(context.Background(), csr, metav1.UpdateOptions{})
if err != nil {
klog.Errorf("failed to approve %s csr(%s), %v", projectinfo.GetTunnelName(), csr.GetName(), err)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

projectinfo.GetTunnelName() --> projectinfo.GetHubName()

runtime.HandleError(err)
return
}
wq.AddRateLimited(key)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to reduce the overhead of csr controller, before add into worker queue, we need to verify the object, like the object is csr, or csr object is yurthub csr, or csr is approved or denied.

you can reference the following link: https://github.com/openyurtio/openyurt/blob/master/pkg/yurttunnel/pki/certmanager/csrapprover.go#L105

@@ -53,3 +54,11 @@ func startNodeLifecycleController(ctx ControllerContext) (http.Handler, bool, er
go lifecycleController.Run(ctx.Stop)
return nil, true, nil
}

func startYurtHubCSRApproverController(ctx ControllerContext) (http.Handler, bool, error) {
clientSet := ctx.ClientBuilder.ClientOrDie("node-controller")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node-controller --> csr-controller?

klog.Infof("subject of yurthub server certificate")
return newCertManager(
clientset,
projectinfo.GetHubName(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make sure the name of yurthub server certificate is different from yurthub client certificate, so componentName should not use projectinfo.GetHubName(). maybe ``projectinfo.GetHubName()-server` is more reasonable.

// Can't use TLSv1.1 because of RC4 cipher usage
MinVersion: tls.VersionTLS12,
RootCAs: root,
ClientAuth: tls.RequireAnyClientCert,
Copy link
Member

@rambohe-ch rambohe-ch Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it's not need to verify the client certificate by yurthub server because pods with InClusterConfig will not send certificate to server, so ClientAuth can be setup with tls. NoClientCert . meanwhile, tlsConfig.RootCAs setting is not needed.

@luckymrwang
Copy link
Member Author

/assign @kadisi

@luckymrwang
Copy link
Member Author

yurthub version: projectinfo.Info{GitVersion:"v0.0.0", GitCommit:"unknown", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.16.3", Compiler:"gc", Platform:"linux/amd64"}
I0611 15:04:23.093242       1 start.go:59] FLAG: --add_dir_header="false"
I0611 15:04:23.093276       1 start.go:59] FLAG: --alsologtostderr="false"
I0611 15:04:23.093280       1 start.go:59] FLAG: --bind-address="127.0.0.1"
I0611 15:04:23.093284       1 start.go:59] FLAG: --cert-mgr-mode="hubself"
I0611 15:04:23.093286       1 start.go:59] FLAG: --disk-cache-path="/etc/kubernetes/cache/"
I0611 15:04:23.093289       1 start.go:59] FLAG: --dummy-if-ip="169.254.2.1"
I0611 15:04:23.093292       1 start.go:59] FLAG: --dummy-if-name="yurthub-dummy0"
I0611 15:04:23.093294       1 start.go:59] FLAG: --enable-dummy-if="true"
I0611 15:04:23.093298       1 start.go:59] FLAG: --enable-iptables="true"
I0611 15:04:23.093300       1 start.go:59] FLAG: --gc-frequency="120"
I0611 15:04:23.093304       1 start.go:59] FLAG: --heartbeat-failed-retry="3"
I0611 15:04:23.093306       1 start.go:59] FLAG: --heartbeat-healthy-threshold="2"
I0611 15:04:23.093309       1 start.go:59] FLAG: --heartbeat-timeout-seconds="2"
I0611 15:04:23.093311       1 start.go:59] FLAG: --help="false"
I0611 15:04:23.093313       1 start.go:59] FLAG: --join-token=""
I0611 15:04:23.093316       1 start.go:59] FLAG: --kubelet-ca-file="/etc/kubernetes/pki/ca.crt"
I0611 15:04:23.093319       1 start.go:59] FLAG: --kubelet-client-certificate="/var/lib/kubelet/pki/kubelet-client-current.pem"
I0611 15:04:23.093322       1 start.go:59] FLAG: --lb-mode="rr"
I0611 15:04:23.093324       1 start.go:59] FLAG: --log-flush-frequency="5s"
I0611 15:04:23.093328       1 start.go:59] FLAG: --log_backtrace_at=":0"
I0611 15:04:23.093332       1 start.go:59] FLAG: --log_dir=""
I0611 15:04:23.093335       1 start.go:59] FLAG: --log_file=""
I0611 15:04:23.093337       1 start.go:59] FLAG: --log_file_max_size="1800"
I0611 15:04:23.093340       1 start.go:59] FLAG: --logtostderr="true"
I0611 15:04:23.093342       1 start.go:59] FLAG: --max-requests-in-flight="250"
I0611 15:04:23.093345       1 start.go:59] FLAG: --node-name="k8s-node3"
I0611 15:04:23.093348       1 start.go:59] FLAG: --profiling="true"
I0611 15:04:23.093350       1 start.go:59] FLAG: --proxy-port="10261"
I0611 15:04:23.093353       1 start.go:59] FLAG: --proxy-secure-port="10268"
I0611 15:04:23.093355       1 start.go:59] FLAG: --root-dir="/var/lib/yurthub"
I0611 15:04:23.093358       1 start.go:59] FLAG: --serve-port="10267"
I0611 15:04:23.093360       1 start.go:59] FLAG: --server-addr="https://10.211.55.18:6443"
I0611 15:04:23.093363       1 start.go:59] FLAG: --skip_headers="false"
I0611 15:04:23.093366       1 start.go:59] FLAG: --skip_log_headers="false"
I0611 15:04:23.093368       1 start.go:59] FLAG: --stderrthreshold="2"
I0611 15:04:23.093371       1 start.go:59] FLAG: --v="4"
I0611 15:04:23.093373       1 start.go:59] FLAG: --version="false"
I0611 15:04:23.093376       1 start.go:59] FLAG: --vmodule=""
I0611 15:04:23.093398       1 config.go:136] yurthub would connect remote servers: https://10.211.55.18:6443
I0611 15:04:23.093755       1 start.go:69] yurthub cfg: &config.YurtHubConfiguration{LBMode:"rr", RemoteServers:[]*url.URL{(*url.URL)(0xc0006c8870)}, YurtHubServerAddr:"127.0.0.1:10267", YurtHubProxyServerAddr:"127.0.0.1:10261", YurtHubProxyServerSecureAddr:"127.0.0.1:10268", YurtHubProxyServerDummyAddr:"169.254.2.1:10261", YurtHubProxyServerSecureDummyAddr:"169.254.2.1:10268", GCFrequency:120, CertMgrMode:"hubself", KubeletRootCAFilePath:"/etc/kubernetes/pki/ca.crt", KubeletPairFilePath:"/var/lib/kubelet/pki/kubelet-client-current.pem", NodeName:"k8s-node3", HeartbeatFailedRetry:3, HeartbeatHealthyThreshold:2, HeartbeatTimeoutSeconds:2, MaxRequestInFlight:250, JoinToken:"", RootDir:"/var/lib/yurthub", EnableProfiling:true, EnableDummyIf:true, EnableIptables:true, HubAgentDummyIfName:"yurthub-dummy0", StorageWrapper:(*cachemanager.storageWrapper)(0xc00003ba00), SerializerManager:(*serializer.SerializerManager)(0xc00003ba40), TLSConfig:(*tls.Config)(nil)}
I0611 15:04:23.093806       1 start.go:84] 1. register cert managers
I0611 15:04:23.093823       1 certificate.go:60] Registered certificate manager kubelet
I0611 15:04:23.093830       1 certificate.go:60] Registered certificate manager hubself
I0611 15:04:23.093833       1 start.go:90] 2. create cert manager with hubself mode
I0611 15:04:23.093860       1 cert_mgr.go:220] /var/lib/yurthub/pki/ca.crt file not exists, so create it
I0611 15:04:23.101716       1 cert_mgr.go:127] use /var/lib/yurthub/pki/ca.crt ca file to bootstrap yurthub
I0611 15:04:23.101781       1 cert_mgr.go:280] yurthub bootstrap conf file does not exist, so create it
I0611 15:04:23.105633       1 certificate_manager.go:282] Certificate rotation is enabled.
I0611 15:04:23.105659       1 cert_mgr.go:418] /var/lib/yurthub/yurthub.conf file not exists, so create it
I0611 15:04:23.106380       1 certificate.go:83] waiting for preparing client certificate
I0611 15:04:23.106420       1 certificate_manager.go:409] Rotating certificates
I0611 15:04:28.106601       1 certificate.go:83] waiting for preparing client certificate
I0611 15:04:33.106830       1 certificate.go:83] waiting for preparing client certificate
I0611 15:04:38.107542       1 certificate.go:83] waiting for preparing client certificate
I0611 15:04:43.107370       1 certificate.go:83] waiting for preparing client certificate
I0611 15:04:48.107344       1 certificate.go:83] waiting for preparing client certificate
I0611 15:04:53.107371       1 certificate.go:83] waiting for preparing client certificate
I0611 15:04:53.114709       1 cert_mgr.go:378] use bootstrap client config to create csr client
I0611 15:04:53.115404       1 cert_mgr.go:346] no join token, so use kubelet config to bootstrap hub
I0611 15:04:53.115575       1 cert_mgr.go:390] bootstrap client config: &rest.Config{Host:"https://10.211.55.18:6443", APIPath:"", ContentConfig:rest.ContentConfig{AcceptContentTypes:"", ContentType:"", GroupVersion:(*schema.GroupVersion)(nil), NegotiatedSerializer:runtime.NegotiatedSerializer(nil)}, Username:"", Password:"", BearerToken:"", BearerTokenFile:"", Impersonate:rest.ImpersonationConfig{UserName:"", Groups:[]string(nil), Extra:map[string][]string(nil)}, AuthProvider:<nil>, AuthConfigPersister:rest.AuthProviderConfigPersister(nil), ExecProvider:<nil>, TLSClientConfig:rest.sanitizedTLSClientConfig{Insecure:false, ServerName:"", CertFile:"/var/lib/kubelet/pki/kubelet-client-current.pem", KeyFile:"/var/lib/kubelet/pki/kubelet-client-current.pem", CAFile:"/etc/kubernetes/pki/ca.crt", CertData:[]uint8(nil), KeyData:[]uint8(nil), CAData:[]uint8(nil), NextProtos:[]string(nil)}, UserAgent:"", DisableCompression:false, Transport:http.RoundTripper(nil), WrapTransport:(transport.WrapperFunc)(nil), QPS:0, Burst:0, RateLimiter:flowcontrol.RateLimiter(nil), Timeout:0, Dial:(func(context.Context, string, string) (net.Conn, error))(nil)}
I0611 15:04:53.115711       1 cert_mgr.go:398] avoid tcp conn leak, close old tcp conn that used to rotate certificate
I0611 15:04:53.115733       1 connrotation.go:110] forcibly close 0 connections on 10.211.55.18:6443 for hub certificate manager dialer
I0611 15:04:53.117120       1 cert_rotation.go:137] Starting client certificate rotation controller
I0611 15:04:53.118550       1 connrotation.go:145] create a connection from 10.211.55.20:55976 to 10.211.55.18:6443, total 1 connections in hub certificate manager dialer
I0611 15:04:53.133557       1 reflector.go:175] Starting reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0611 15:04:53.133583       1 reflector.go:211] Listing and watching *v1beta1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0611 15:04:53.149111       1 csr.go:124] certificate signing request csr-szwst is approved, waiting to be issued
I0611 15:04:53.160139       1 csr.go:121] certificate signing request csr-szwst is issued
I0611 15:04:53.160232       1 reflector.go:181] Stopping reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0611 15:04:54.162883       1 certificate_manager.go:553] Certificate expiration is 2022-04-30 17:32:29 +0000 UTC, rotation deadline is 2022-03-18 20:39:55.651788693 +0000 UTC
I0611 15:04:54.163047       1 certificate_manager.go:288] Waiting 6725h35m1.488748731s for next certificate rotation
I0611 15:04:58.107552       1 start.go:98] 3. new transport manager
I0611 15:04:58.107612       1 transport.go:57] use /var/lib/yurthub/pki/ca.crt ca cert file to access remote server
I0611 15:04:58.107799       1 start.go:106] 4. create health checker for remote servers 
I0611 15:04:58.108635       1 connrotation.go:145] create a connection from 10.211.55.20:55994 to 10.211.55.18:6443, total 1 connections in transport manager dialer
I0611 15:04:58.121106       1 start.go:115] 5. new restConfig manager for hubself mode
I0611 15:04:58.121123       1 start.go:123] 6. create tls config for secure servers 
I0611 15:04:58.121636       1 config.go:107] re-fix hub rest config host successfully with server https://10.211.55.18:6443
I0611 15:04:58.122161       1 cert_rotation.go:137] Starting client certificate rotation controller
I0611 15:04:58.123063       1 certmanager.go:47] subject of yurthub server certificate
I0611 15:04:58.123119       1 certificate_manager.go:282] Certificate rotation is enabled.
I0611 15:04:58.123231       1 certificate_manager.go:409] Rotating certificates
I0611 15:04:58.130895       1 reflector.go:175] Starting reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0611 15:04:58.130922       1 reflector.go:211] Listing and watching *v1beta1.CertificateSigningRequest from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0611 15:04:58.140028       1 csr.go:124] certificate signing request csr-bzlk5 is approved, waiting to be issued
I0611 15:04:58.145046       1 csr.go:121] certificate signing request csr-bzlk5 is issued
I0611 15:04:58.145120       1 reflector.go:181] Stopping reflector *v1beta1.CertificateSigningRequest (0s) from pkg/mod/k8s.io/client-go@v0.18.8/tools/cache/reflector.go:125
I0611 15:04:59.146350       1 certificate_manager.go:553] Certificate expiration is 2022-04-30 17:32:34 +0000 UTC, rotation deadline is 2022-02-02 08:33:27.521678983 +0000 UTC
I0611 15:04:59.146410       1 certificate_manager.go:288] Waiting 5657h28m28.37527533s for next certificate rotation
I0611 15:05:00.148155       1 certificate_manager.go:553] Certificate expiration is 2022-04-30 17:32:34 +0000 UTC, rotation deadline is 2022-03-08 21:35:54.435205712 +0000 UTC
I0611 15:05:00.148206       1 certificate_manager.go:288] Waiting 6486h30m54.287003942s for next certificate rotation
I0611 15:05:03.123590       1 start.go:131] 7. new cache manager with storage wrapper and serializer manager
I0611 15:05:03.123698       1 cache_agent.go:68] reset cache agents to [kubelet kube-proxy flanneld coredns yurttunnel-agent]
I0611 15:05:03.125449       1 start.go:139] 8. new gc manager for node k8s-node3, and gc frequency is a random time between 120 min and 360 min
I0611 15:05:03.125562       1 gc.go:97] list pod keys from storage, total: 4
I0611 15:05:03.126137       1 config.go:107] re-fix hub rest config host successfully with server https://10.211.55.18:6443
I0611 15:05:03.141481       1 gc.go:125] list all of pod that on the node: total: 4
I0611 15:05:03.141512       1 start.go:148] 9. new reverse proxy handler for remote servers
I0611 15:05:03.141534       1 start.go:157] 10. create dummy network interface yurthub-dummy0 and init iptables manager
I0611 15:05:03.141572       1 gc.go:74] start gc events after waiting 53.194µs from previous gc
I0611 15:05:03.141989       1 config.go:107] re-fix hub rest config host successfully with server https://10.211.55.18:6443
I0611 15:05:03.143319       1 gc.go:163] list kubelet event keys from storage, total: 14
I0611 15:05:03.143795       1 iptables.go:671] couldn't get iptables-restore version; assuming it doesn't support --wait
E0611 15:05:03.144959       1 gc.go:177] could not get kubelet kubelet/events/kube-system/coredns-7f89b7bc75-928mn.168782152224151f event for node(k8s-node3), events "coredns-7f89b7bc75-928mn.168782152224151f" is forbidden: User "system:node:k8s-node3" cannot get resource "events" in API group "" in the namespace "kube-system"
I0611 15:05:03.144994       1 gc.go:160] no kube-proxy events in local storage, skip kube-proxy events gc
I0611 15:05:03.173010       1 start.go:165] 11. new yurthub server and begin to serve, dummy proxy server: 169.254.2.1:10261, secure dummy proxy server: 169.254.2.1:10268
I0611 15:05:03.173029       1 start.go:168] 11. new yurthub server and begin to serve, proxy server: 127.0.0.1:10261, secure proxy server: 127.0.0.1:10268, hub server: 127.0.0.1:10267
I0611 15:05:05.784985       1 util.go:232] start proxying: patch /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.168788934c1a2cff, in flight requests: 1
I0611 15:05:05.785068       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet patch events: /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.168788934c1a2cff
I0611 15:05:05.793975       1 util.go:215] kubelet patch events: /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.168788934c1a2cff with status code 200, spent 8.917934ms
I0611 15:05:05.794451       1 util.go:232] start proxying: patch /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.168788934dea2a9d, in flight requests: 1
I0611 15:05:05.794496       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet patch events: /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.168788934dea2a9d
I0611 15:05:05.798153       1 util.go:215] kubelet patch events: /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.168788934dea2a9d with status code 200, spent 3.675587ms
I0611 15:05:05.798683       1 util.go:232] start proxying: patch /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.1687889353d5473b, in flight requests: 1
I0611 15:05:05.798720       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet patch events: /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.1687889353d5473b
I0611 15:05:05.802262       1 util.go:215] kubelet patch events: /api/v1/namespaces/kube-system/events/yurt-hub-k8s-node3.1687889353d5473b with status code 200, spent 3.547694ms
I0611 15:05:08.150840       1 util.go:232] start proxying: get /api/v1/nodes/k8s-node3?resourceVersion=0&timeout=10s, in flight requests: 1
I0611 15:05:08.150910       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet get nodes: /api/v1/nodes/k8s-node3?resourceVersion=0&timeout=10s
I0611 15:05:08.152904       1 util.go:215] kubelet get nodes: /api/v1/nodes/k8s-node3?resourceVersion=0&timeout=10s with status code 200, spent 2.000746ms
I0611 15:05:09.190122       1 util.go:232] start proxying: get /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dcoredns&resourceVersion=142175, in flight requests: 1
I0611 15:05:09.190258       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet list configmaps: /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dcoredns&resourceVersion=142175
I0611 15:05:09.193856       1 util.go:215] kubelet list configmaps: /api/v1/namespaces/kube-system/configmaps?fieldSelector=metadata.name%3Dcoredns&resourceVersion=142175 with status code 200, spent 3.608479ms
I0611 15:05:09.194838       1 util.go:232] start proxying: get /api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcoredns&resourceVersion=142175&timeout=7m53s&timeoutSeconds=473&watch=true, in flight requests: 1
I0611 15:05:09.194904       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet watch configmaps: /api/v1/namespaces/kube-system/configmaps?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dcoredns&resourceVersion=142175&timeout=7m53s&timeoutSeconds=473&watch=true
I0611 15:05:10.042286       1 util.go:232] start proxying: get /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-node3?timeout=10s, in flight requests: 2
I0611 15:05:10.042352       1 local.go:66] go into local proxy for request kubelet get leases: /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-node3?timeout=10s
I0611 15:05:10.042585       1 util.go:215] kubelet get leases: /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-node3?timeout=10s with status code 200, spent 249.807µs
I0611 15:05:10.043137       1 util.go:232] start proxying: put /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-node3?timeout=10s, in flight requests: 2
I0611 15:05:10.043185       1 local.go:66] go into local proxy for request kubelet update leases: /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-node3?timeout=10s
I0611 15:05:10.043219       1 util.go:215] kubelet update leases: /apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/k8s-node3?timeout=10s with status code 200, spent 39.625µs
I0611 15:05:11.017154       1 util.go:232] start proxying: get /api/v1/namespaces/kube-system/pods/yurt-hub-k8s-node3, in flight requests: 2
I0611 15:05:11.017202       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet get pods: /api/v1/namespaces/kube-system/pods/yurt-hub-k8s-node3
I0611 15:05:11.022369       1 util.go:215] kubelet get pods: /api/v1/namespaces/kube-system/pods/yurt-hub-k8s-node3 with status code 200, spent 5.173062ms
I0611 15:05:11.024004       1 util.go:232] start proxying: patch /api/v1/namespaces/kube-system/pods/yurt-hub-k8s-node3/status, in flight requests: 2
I0611 15:05:11.024038       1 loadbalancer.go:184] picked backend https://10.211.55.18:6443 by rr algorithm for request kubelet patch pods: /api/v1/namespaces/kube-system/pods/yurt-hub-k8s-node3/status
I0611 15:05:11.030290       1 util.go:215] kubelet patch pods: /api/v1/namespaces/kube-system/pods/yurt-hub-k8s-node3/status with status code 200, spent 6.255397ms

@rambohe-ch
Copy link
Member

/lgtm

@@ -164,7 +173,7 @@ func Run(cfg *config.YurtHubConfiguration, stopCh <-chan struct{}) error {
}
networkMgr.Run(stopCh)
trace++
klog.Infof("%d. new %s server and begin to serve, dummy proxy server: %s", trace, projectinfo.GetHubName(), cfg.YurtHubProxyServerDummyAddr)
klog.Infof("%d. new %s server and begin to serve, dummy proxy server: %s, secure dummy proxy server: %s", trace, projectinfo.GetHubName(), cfg.YurtHubProxyServerDummyAddr, cfg.YurtHubProxyServerSecureDummyAddr)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add log info for YurtHubProxyServerSecureAddr field.

Copy link
Member Author

@luckymrwang luckymrwang Jul 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's just dummy proxy log info here. The log info of YurtHubProxyServerSecureAddr is on line 185.

@rambohe-ch
Copy link
Member

@luckymrwang please merge two commits into one commit.

@openyurt-bot openyurt-bot removed the lgtm lgtm label Jul 28, 2021
@luckymrwang
Copy link
Member Author

@luckymrwang please merge two commits into one commit.

I have merged two commits into one commit.

@rambohe-ch
Copy link
Member

@luckymrwang Would you be able be upload the detail logs of yurthub that pod uses InClusterConfig to access kube-apiserver through yurthub.

@rambohe-ch
Copy link
Member

/lgtm
/approve

@openyurt-bot openyurt-bot added the lgtm lgtm label Jul 30, 2021
@openyurt-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: luckymrwang, rambohe-ch

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openyurt-bot openyurt-bot added the approved approved label Jul 30, 2021
@openyurt-bot openyurt-bot merged commit 01c1834 into openyurtio:master Jul 30, 2021
SataQiu added a commit to SataQiu/openyurt that referenced this pull request Aug 6, 2021
MrGirl pushed a commit to MrGirl/openyurt that referenced this pull request Mar 29, 2022
MrGirl pushed a commit to MrGirl/openyurt that referenced this pull request Mar 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved approved kind/feature kind/feature lgtm lgtm size/XL size/XL: 500-999
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Question]Listen protocol of yurthub is http, why not https
4 participants