update yurt-tunnel-server certificate automatically #525

YRXING · 2021-10-15T14:05:47Z

What type of PR is this?

/kind enhancement
/kind good-first-issue
/kind feature
/kind design

What this PR does / why we need it:

improve yurt-tunnel-server to automatically update server certificates when LB service address changed.

Which issue(s) this PR fixes:

Fixes #411

Special notes for your reviewer:

the test is needed.
and is this idea feasible?

Does this PR introduce a user-facing change?

other Note

openyurt-bot · 2021-10-15T14:05:52Z

@YRXING: GitHub didn't allow me to assign the following users: your_reviewer.

Note that only openyurtio members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

What type of PR is this?

/kind enhancement
/kind good-first-issue
/kind feature
/kind design

What this PR does / why we need it:

improve yurt-tunnel-server to automatically update server certificates when LB service address changed.

Which issue(s) this PR fixes:

Fixes #411

Special notes for your reviewer:

the test is needed.
and is this idea feasible?

Does this PR introduce a user-facing change?
other Note

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

pkg/controller/service/tunnel_service_controller.go

pkg/yurttunnel/pki/certmanager/certmanager.go

cmd/yurt-tunnel-server/app/start.go

pkg/yurttunnel/constants/constants.go

pkg/yurttunnel/server/tunnel_service_controller.go

rambohe-ch · 2021-10-19T03:01:38Z

@YRXING would you be able to add an e2e test case for server certificate updating?

rambohe-ch · 2021-10-22T08:30:20Z

@YRXING Code is not readable if server certificate coupling with IPs resolving. how about decouple server certificate and IPs resolving?

pkg/yurttunnel/dns/dns.go

pkg/yurttunnel/constants/constants.go

pkg/yurttunnel/server/serveraddr/addr.go

pkg/yurttunnel/pki/certmanager/certmanager.go

pkg/yurttunnel/server/serveraddr/addr.go

pkg/yurttunnel/pki/certmanager/certmanager.go

rambohe-ch · 2021-11-01T11:50:49Z

@YRXING please fix e2e test error.

hack/lib/release-images.sh

pkg/yurttunnel/pki/certmanager/certmanager.go

pkg/projectinfo/projectinfo.go

pkg/yurttunnel/server/serveraddr/addr_test.go

hack/lib/release-images.sh

YRXING · 2021-11-09T02:26:17Z

Here is the yurttunnel-server's log. Firstly, it will get the dnsNames and ips through api-server directly and generate the certificate. Then it begins to watch the related resources to regenerate the certificate automatically. When I recreate the x-tunnel-server-svc resources, the ip changed, and you can see a new certificate at the directory /var/lib/yurttunnel-server/pki

I1108 09:56:14.373548 1 start.go:53] yurttunnel-server version: projectinfo.Info{GitVersion:"v0.5.0", GitCommit:"58587a2", BuildDate:"2021-11-08T09:47:55Z", GoVersion:"go1.15.15", Compiler:"gc", Platform:"linux/amd64"}
W1108 09:56:14.374525 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I1108 09:56:14.378621 1 options.go:155] yurttunnel server config: &config.Config{EgressSelectorEnabled:false, EnableIptables:true, EnableDNSController:true, IptablesSyncPeriod:60, DNSSyncPeriod:1800, CertDNSNames:[]string{}, CertIPs:[]net.IP{}, ListenAddrForAgent:"172.18.0.3:10262", ListenAddrForMaster:"172.18.0.3:10263", ListenInsecureAddrForMaster:"172.18.0.3:10264", ListenMetaAddr:"172.18.0.3:10265", RootCert:(*x509.CertPool)(0xc0001b2150), Client:(*kubernetes.Clientset)(0xc00052e2c0), SharedInformerFactory:(*informers.sharedInformerFactory)(0xc000184fa0), ServerCount:1, ProxyStrategy:"destHost", InterceptorServerUDSFile:""}
I1108 09:56:14.379496 1 leaderelection.go:242] attempting to acquire leader lease kube-system/tunnel-dns-controller...
E1108 09:56:14.391434 1 iptables.go:200] failed to delete rule that nat chain OUTPUT jumps to TUNNEL-PORT: error checking rule: exit status 2: iptables v1.6.0: Couldn't load target `TUNNEL-PORT':No such file or directory

Try `iptables I1108 09:56:14.427397 I1108 09:56:14.427772 I1108 09:56:14.427798 I1108 09:56:19.427766 W1108 09:56:19.428083 I1108 09:56:19.450782 I1108 09:56:19.452619 I1108 09:56:19.452693 I1108 09:56:19.458894 I1108 09:56:19.458942 I1108 09:56:19.528734 I1108 09:56:19.600667 I1108 09:56:19.600699 I1108 09:56:19.600891 I1108 09:56:19.733518 E1108 09:56:19.741585 E1108 09:56:19.743633 I1108 09:56:19.743688 I1108 09:56:20.608794 I1108 09:56:21.200157 I1108 09:56:21.602066 I1108 09:56:24.429200 I1108 09:56:24.429288 I1108 09:56:24.429446 I1108 09:56:24.429479 I1108 09:56:24.429612 I1108 09:56:24.429945 I1108 09:56:24.430119 I1108 09:56:24.430802 I1108 09:56:31.634010 I1108 09:56:31.634097 I1108 09:56:31.634137 I1108 10:11:32.544041 I1108 10:17:50.520728 -h' or 'iptables --help' for more information.
1 leaderelection.go:252] successfully acquired lease kube-system/tunnel-dns-controller
1 dns.go:197] starting tunnel dns controller
1 shared_informer.go:223] Waiting for caches to sync for tunnel-dns-controller
1 certmanager.go:92] subject of tunnel server certificate, ips=[]net.IP{net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xac, 0x12, 0x0, 0x3}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x60, 0x24, 0x81}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0x7f, 0x0, 0x0, 0x1}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xac, 0x12, 0x0, 0x3}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x60, 0x4a, 0x9b}}, dnsNames=[]string{"openyurt-e2e-test-control-plane", "x-tunnel-server-svc", "x-tunnel-server-svc.kube-system", "x-tunnel-server-svc.kube-system.svc", "x-tunnel-server-svc.kube-system.svc.cluster.local", "x-tunnel-server-internal-svc", "x-tunnel-server-internal-svc.kube-system", "x-tunnel-server-internal-svc.kube-system.svc", "x-tunnel-server-internal-svc.kube-system.svc.cluster.local"}
1 filestore_wrapper.go:49] unexpected error occurred when loading the certificate: no cert/key files read at "/var/lib/yurttunnel-server/pki/yurttunnel-server-current.pem", ("", "") or ("/var/lib/yurttunnel-server/pki", "/var/lib/yurttunnel-server/pki"), will regenerate it
1 handler.go:140] enqueue service add event for kube-system/x-tunnel-server-internal-svc
1 handler.go:173] handle configmap add event for kube-system/yurt-tunnel-server-cfg to update localhost ports
1 handler.go:92] enqueue configmap add event for kube-system/yurt-tunnel-server-cfg
1 handler.go:42] enqueue node add event for openyurt-e2e-test-worker
1 handler.go:42] enqueue node add event for openyurt-e2e-test-control-plane
1 shared_informer.go:230] Caches are synced for tunnel-dns-controller
1 dns.go:317] sync tunnel server service as whole
1 dns.go:326] sync dns record as whole
1 dns.go:326] sync dns record as whole
1 iptables.go:456] clear conntrack entries for ports ["10250" "10255"] and nodes ["172.18.0.3"]
1 iptables.go:473] clear conntrack for 172.18.0.3:10250 failed: "conntrack v1.4.4 (conntrack-tools): 0 flow entries have been deleted.\n", error message: exit status 1
1 iptables.go:473] clear conntrack for 172.18.0.3:10255 failed: "conntrack v1.4.4 (conntrack-tools): 0 flow entries have been deleted.\n", error message: exit status 1
1 iptables.go:525] directly access nodes changed, [172.18.0.3] for ports [10250 10255]
1 dns.go:317] sync tunnel server service as whole
1 handler.go:165] adding node dns record for openyurt-e2e-test-worker
1 handler.go:165] adding node dns record for openyurt-e2e-test-control-plane
1 anpserver.go:106] start handling request from interceptor
1 wraphandler.go:67] add localHostProxyMiddleware into wrap handler
1 tracereq.go:63] 1 informer synced in traceReqMiddleware
1 wraphandler.go:67] add TraceReqMiddleware into wrap handler
1 anpserver.go:142] start handling https request from master at 172.18.0.3:10263
1 anpserver.go:194] start handling connection from agents
1 anpserver.go:156] start handling http request from master at 172.18.0.3:10264
1 util.go:71] "start handling meta requests(metrics/pprof)" server endpoint="172.18.0.3:10265"
1 server.go:616] "Connect request from agent" agentID="openyurt-e2e-test-worker"
1 backend_manager.go:184] "Register backend for agent" connection=&agent.agentServiceConnectServer{ServerStream:(*grpc.serverStream)(0xc0001a8240)} agentID="172.18.0.2"
1 backend_manager.go:184] "Register backend for agent" connection=&agent.agentServiceConnectServer{ServerStream:(*grpc.serverStream)(0xc0001a8240)} agentID="openyurt-e2e-test-worker"
1 certmanager.go:170] ip changed, the latest tunnel server's ips=[]net.IP{net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x60, 0x4a, 0x9b}}
1 certmanager.go:170] ip changed, the latest tunnel server's ips=[]net.IP{net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xac, 0x12, 0x0, 0x3}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x60, 0xa, 0x27}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0x7f, 0x0, 0x0, 0x1}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xac, 0x12, 0x0, 0x3}, net.IP{0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xff, 0xff, 0xa, 0x60, 0x4a, 0x9b}}

rambohe-ch · 2021-11-09T08:06:59Z

/lgtm
/approve

openyurt-bot · 2021-11-09T08:07:10Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rambohe-ch, YRXING

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [rambohe-ch]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openyurt-bot requested review from SataQiu and yixingjia October 15, 2021 14:05

openyurt-bot added the size/L size/L: 100-499 label Oct 15, 2021

SataQiu reviewed Oct 18, 2021

View reviewed changes

pkg/controller/service/tunnel_service_controller.go Outdated Show resolved Hide resolved

pkg/controller/service/tunnel_service_controller.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Oct 19, 2021

View reviewed changes

pkg/controller/service/tunnel_service_controller.go Outdated Show resolved Hide resolved

YRXING force-pushed the master branch from aeacf37 to 2f6bee9 Compare October 19, 2021 02:40

rambohe-ch reviewed Oct 19, 2021

View reviewed changes

pkg/yurttunnel/pki/certmanager/certmanager.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Oct 19, 2021

View reviewed changes

cmd/yurt-tunnel-server/app/start.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Oct 19, 2021

View reviewed changes

pkg/yurttunnel/constants/constants.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Oct 19, 2021

View reviewed changes

pkg/yurttunnel/server/tunnel_service_controller.go Outdated Show resolved Hide resolved

YRXING force-pushed the master branch from 2f6bee9 to c53dcd7 Compare October 21, 2021 13:13

YRXING force-pushed the master branch 2 times, most recently from 2a9e9db to 9589f51 Compare October 27, 2021 02:22

rambohe-ch reviewed Oct 27, 2021

View reviewed changes

pkg/yurttunnel/dns/dns.go Show resolved Hide resolved

rambohe-ch reviewed Oct 27, 2021

View reviewed changes

pkg/yurttunnel/constants/constants.go Outdated Show resolved Hide resolved

YRXING force-pushed the master branch from 9589f51 to 7a4bd43 Compare October 28, 2021 06:47

rambohe-ch reviewed Oct 28, 2021

View reviewed changes

pkg/yurttunnel/server/serveraddr/addr.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Oct 28, 2021

View reviewed changes

pkg/yurttunnel/pki/certmanager/certmanager.go Outdated Show resolved Hide resolved

rambohe-ch mentioned this pull request Oct 28, 2021

[BUG] yurt-tunnel v0.5.0: x509 certificate is invalid #535

Closed

YRXING force-pushed the master branch from 7a4bd43 to ad72a56 Compare October 28, 2021 13:04

rambohe-ch reviewed Oct 29, 2021

View reviewed changes

pkg/yurttunnel/pki/certmanager/certmanager.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Oct 29, 2021

View reviewed changes

pkg/yurttunnel/pki/certmanager/certmanager.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Oct 29, 2021

View reviewed changes

pkg/yurttunnel/server/serveraddr/addr.go Show resolved Hide resolved

rambohe-ch reviewed Oct 29, 2021

View reviewed changes

pkg/yurttunnel/pki/certmanager/certmanager.go Outdated Show resolved Hide resolved

rambohe-ch mentioned this pull request Oct 29, 2021

[feature request] How about using emptyDir volume to store certificate files instead of hostPath for tunnel server/agent? #537

Closed

YRXING force-pushed the master branch from ad72a56 to f2be17f Compare November 1, 2021 06:32

rambohe-ch reviewed Nov 1, 2021

View reviewed changes

pkg/yurttunnel/pki/certmanager/certmanager.go Outdated Show resolved Hide resolved

YRXING force-pushed the master branch from f2be17f to 2523459 Compare November 1, 2021 11:48

YRXING force-pushed the master branch from 2523459 to 120007b Compare November 4, 2021 09:54

rambohe-ch reviewed Nov 5, 2021

View reviewed changes

hack/lib/release-images.sh Outdated Show resolved Hide resolved

rambohe-ch reviewed Nov 5, 2021

View reviewed changes

pkg/yurttunnel/pki/certmanager/certmanager.go Outdated Show resolved Hide resolved

YRXING force-pushed the master branch 2 times, most recently from 21efefb to f964d51 Compare November 5, 2021 01:46

rambohe-ch reviewed Nov 5, 2021

View reviewed changes

pkg/projectinfo/projectinfo.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Nov 5, 2021

View reviewed changes

pkg/yurttunnel/server/serveraddr/addr_test.go Outdated Show resolved Hide resolved

rambohe-ch reviewed Nov 5, 2021

View reviewed changes

pkg/yurttunnel/server/serveraddr/addr_test.go Outdated Show resolved Hide resolved

YRXING force-pushed the master branch from f964d51 to 58587a2 Compare November 5, 2021 08:05

openyurt-bot added size/XL size/XL: 500-999 kind/good-first-issue kind/good-first-issue kind/feature kind/feature and removed size/L size/L: 100-499 labels Nov 5, 2021

rambohe-ch reviewed Nov 6, 2021

View reviewed changes

hack/lib/release-images.sh Show resolved Hide resolved

YRXING force-pushed the master branch from 58587a2 to 9f6fb41 Compare November 9, 2021 06:40

update certificate automatically

177ee3c

YRXING force-pushed the master branch from 9f6fb41 to 177ee3c Compare November 9, 2021 07:12

rambohe-ch approved these changes Nov 9, 2021

View reviewed changes

openyurt-bot assigned rambohe-ch Nov 9, 2021

openyurt-bot added approved approved lgtm lgtm labels Nov 9, 2021

openyurt-bot merged commit f4df656 into openyurtio:master Nov 9, 2021

MrGirl pushed a commit to MrGirl/openyurt that referenced this pull request Mar 29, 2022

update certificate automatically (openyurtio#525)

f178944

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update yurt-tunnel-server certificate automatically #525

update yurt-tunnel-server certificate automatically #525

YRXING commented Oct 15, 2021 •

edited

Loading

openyurt-bot commented Oct 15, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

rambohe-ch commented Oct 19, 2021

rambohe-ch commented Oct 22, 2021

rambohe-ch commented Nov 1, 2021

YRXING commented Nov 9, 2021

rambohe-ch commented Nov 9, 2021

openyurt-bot commented Nov 9, 2021

update yurt-tunnel-server certificate automatically #525

update yurt-tunnel-server certificate automatically #525

Conversation

YRXING commented Oct 15, 2021 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

openyurt-bot commented Oct 15, 2021

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

other Note

rambohe-ch commented Oct 19, 2021

rambohe-ch commented Oct 22, 2021

rambohe-ch commented Nov 1, 2021

YRXING commented Nov 9, 2021

rambohe-ch commented Nov 9, 2021

openyurt-bot commented Nov 9, 2021

YRXING commented Oct 15, 2021 •

edited

Loading