gRPC connection fail when `--join` with TLS enabled #1682

AstroProfundis · 2019-08-14T10:06:23Z

Please answer these questions before submitting your issue. Thanks!

What did you do?

When trying to add TLS support to tidb-operator (#750), we found that --join is not working correctly with TLS enabled.

Assume there're pd-0, pd-1 and pd-2, starting sequently. pd-0 starts as normal with --initial-cluster=pd-0=https://pd-0:2380 and TLS enabled, and pd-1 starts with --join=https://pd-0:2380, then pd-1 is not able to connect to pd-0 and exits.

The errors from pd-1 are:

[2019/08/14 05:50:43.611 -04:00] [ERROR] [join.go:180] ["failed to open directory"] [error="open /home/tidb/deploy/pd-server/member: no such file or directory"]
2019/08/14 05:50:43.612 grpclog.go:45: [info] parsed scheme: "endpoint"
2019/08/14 05:50:43.612 grpclog.go:45: [info] ccResolverWrapper: sending new addresses to cc: [{https://172.16.5.146:2380 0  <nil>}]
2019/08/14 05:50:43.613 grpclog.go:60: [warning] grpc: addrConn.createTransport failed to connect to {https://172.16.5.146:2380 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.5.146:2380: connect: connection refused". Reconnecting...
2019/08/14 05:50:44.613 grpclog.go:60: [warning] grpc: addrConn.createTransport failed to connect to {https://172.16.5.146:2380 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.5.146:2380: connect: connection refused". Reconnecting...
2019/08/14 05:50:46.138 grpclog.go:60: [warning] grpc: addrConn.createTransport failed to connect to {https://172.16.5.146:2380 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.5.146:2380: connect: connection refused". Reconnecting...
2019/08/14 05:50:48.301 grpclog.go:60: [warning] grpc: addrConn.createTransport failed to connect to {https://172.16.5.146:2380 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.5.146:2380: connect: connection refused". Reconnecting...
2019/08/14 05:50:52.802 grpclog.go:60: [warning] grpc: addrConn.createTransport failed to connect to {https://172.16.5.146:2380 0  <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 172.16.5.146:2380: connect: connection refused". Reconnecting...
[2019/08/14 05:50:53.613 -04:00] [FATAL] [main.go:85] ["join meet error"] [error="context deadline exceeded"] [stack="github.com/pingcap/log.Fatal\n\t/home/jenkins/workspace/release_tidb_3.0/go/pkg/mod/github.com/pingcap/log@v0.0.0-20190715063458-479153f07ebd/global.go:59\nmain.main\n\t/home/jenkins/workspace/release_tidb_3.0/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:85\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"]

And there errors on pd-0 shows:

[2019/08/14 05:57:27.602 -04:00] [WARN] [grpclog.go:60] ["grpc: Server.Serve failed to complete security handshake from \"172.16.5.147:59118\": tls: first record does not look like a TLS handshake"]

So this might because pd-1 uses a plain connection to try to connect to pd-0, instead of a TLS encrypted connection.

We initially found this issue on a Kubernetes deployment, but it's also reproducible with binary deployment, the procedure are as above:

deploy a cluster using tidb-ansible with enable_tls = True in inventory.ini, but do not start it
login to pd-0, change pd-0:/path/to/deploy/script/run_pd.sh and set --initial-cluster=pd-0=https://pd-0:2380 (remove other nodes), and start it with sudo systemctl start pd-2379.service
login to pd-1, change pd-1:/path/to/deploy/script/run_pd.sh and delete --initial-cluster line, add a --join=https://pd-0:2380 line, then start it with sudo systemctl start pd-2379.service
watch pd-0 and pd-1's log

What did you expect to see?

The process work normally

What did you see instead?

pd-1 produce errors and exits

What version of PD are you using (pd-server -V)?

Release Version: v3.0.1
Git Commit Hash: 811ce0b9a1335d1b2a049fd97ef9e186f1c9efc1
Git Branch: HEAD
UTC Build Time:  2019-07-16 01:02:23

This issue also exists with v3.0.2 docker image.

The text was updated successfully, but these errors were encountered:

rleungx · 2019-09-05T05:21:00Z

I think the join related parameter should be --join=https://pd-0:2379 instead of --join=https://pd-0:2380 @AstroProfundis

AstroProfundis · 2019-09-05T13:29:56Z

The problem is, when TLS is disabled, --join works normally with port 2380... This parameter comes from "peer-url", and is the same as tidb-ansible does (but ansible don't use --join)

rleungx · 2019-09-06T08:51:36Z

@AstroProfundis The join parameter is used to create etcd client, the actual address for join is still peer-url.

AstroProfundis · 2019-09-06T12:52:23Z

I tried to verify the fix with:

Release Version: v4.0.0-alpha-50-gb66ba448
Git Commit Hash: b66ba4482c5dfb3d976461544b5df3b8442d4d37
Git Branch: master
UTC Build Time:  2019-09-06 08:57:10

With the cmdline parameters unchanged:

/pd-server --data-dir=/var/lib/pd --name=tls-pd-1 --peer-urls=https://0.0.0.0:2380 --advertise-peer-urls=https://tls-pd-1.tls-pd-peer.allenz.svc:2380 --client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://tls-pd-1.tls-pd-peer.allenz.svc:2379 --config=/etc/pd/pd.toml --join=https://tls-pd-0.tls-pd-peer.allenz.svc:2380

The second PD instance still fails to join:

[2019/09/06 11:53:44.943 +00:00] [INFO] [util.go:56] ["Welcome to Placement Driver (PD)"]
[2019/09/06 11:53:44.943 +00:00] [INFO] [util.go:57] [PD] [release-version=v4.0.0-alpha-50-gb66ba448]
[2019/09/06 11:53:44.943 +00:00] [INFO] [util.go:58] [PD] [git-hash=b66ba4482c5dfb3d976461544b5df3b8442d4d37]
[2019/09/06 11:53:44.943 +00:00] [INFO] [util.go:59] [PD] [git-branch=master]
[2019/09/06 11:53:44.943 +00:00] [INFO] [util.go:60] [PD] [utc-build-time="2019-09-06 08:57:10"]
[2019/09/06 11:53:44.943 +00:00] [INFO] [metricutil.go:80] ["disable Prometheus push client"]
[2019/09/06 11:53:44.943 +00:00] [ERROR] [join.go:213] ["failed to open directory"] [error="open /var/lib/pd/member: no such file or directory"]
2019/09/06 11:53:44.945 grpclog.go:45: [info] parsed scheme: "endpoint"
2019/09/06 11:53:44.948 grpclog.go:45: [info] ccResolverWrapper: sending new addresses to cc: [{https://tls-pd-0.tls-pd-peer.allenz.svc:2380 0  <nil>}]
{"level":"warn","ts":"2019-09-06T11:53:44.966Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-2ffaae40-3df0-4cc9-bc3f-f41f81843e6c/tls-pd-0.tls-pd-peer.allenz.svc:2380","attempt":0,"error":"rpc error: code = Unavailable desc = transport is closing"}
{"level":"warn","ts":"2019-09-06T11:53:45.967Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-2ffaae40-3df0-4cc9-bc3f-f41f81843e6c/tls-pd-0.tls-pd-peer.allenz.svc:2380","attempt":1,"error":"rpc error: code = Unavailable desc = transport is closing"}
{"level":"warn","ts":"2019-09-06T11:53:47.686Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-2ffaae40-3df0-4cc9-bc3f-f41f81843e6c/tls-pd-0.tls-pd-peer.allenz.svc:2380","attempt":2,"error":"rpc error: code = Unavailable desc = transport is closing"}
{"level":"warn","ts":"2019-09-06T11:53:50.423Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-2ffaae40-3df0-4cc9-bc3f-f41f81843e6c/tls-pd-0.tls-pd-peer.allenz.svc:2380","attempt":3,"error":"rpc error: code = Unavailable desc = transport is closing"}
{"level":"warn","ts":"2019-09-06T11:53:54.342Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-2ffaae40-3df0-4cc9-bc3f-f41f81843e6c/tls-pd-0.tls-pd-peer.allenz.svc:2380","attempt":4,"error":"rpc error: code = Unavailable desc = transport is closing"}
{"level":"warn","ts":"2019-09-06T11:53:54.948Z","caller":"clientv3/retry_interceptor.go:60","msg":"retrying of unary invoker failed","target":"endpoint://client-2ffaae40-3df0-4cc9-bc3f-f41f81843e6c/tls-pd-0.tls-pd-peer.allenz.svc:2380","attempt":5,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
[2019/09/06 11:53:54.948 +00:00] [FATAL] [main.go:93] ["join meet error"] [error="context deadline exceeded"] [stack="github.com/pingcap/log.Fatal\n\t/home/jenkins/workspace/build_pd_master/go/pkg/mod/github.com/pingcap/log@v0.0.0-20190715063458-479153f07ebd/global.go:59\nmain.main\n\t/home/jenkins/workspace/build_pd_master/go/src/github.com/pingcap/pd/cmd/pd-server/main.go:93\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"]

The error seems unchanged as before

With --join changed to port 2379:

/pd-server --data-dir=/var/lib/pd --name=tls-pd-1 --peer-urls=https://0.0.0.0:2380 --advertise-peer-urls=https://tls-pd-1.tls-pd-peer.allenz.svc:2380 --client-urls=https://0.0.0.0:2379 --advertise-client-urls=https://tls-pd-1.tls-pd-peer.allenz.svc:2379 --config=/etc/pd/pd.toml --join=https://tls-pd-0.tls-pd-peer.allenz.svc:2380:2379

The etcd server started sucessfully:

2019/09/06 12:44:24.861 grpclog.go:45: [info] parsed scheme: "endpoint"
2019/09/06 12:44:24.861 grpclog.go:45: [info] ccResolverWrapper: sending new addresses to cc: [{https://tls-pd-0.tls-pd-peer.allenz.svc:2379 0  <nil>}]
[2019/09/06 12:44:24.897 +00:00] [INFO] [server.go:108] ["PD Config"] [config="{\"client-urls\":\"https://0.0.0.0:2379\",\"peer-urls\":\"https://0.0.0.0:2380\",\"advertise-client-urls\":\"https://tls-pd-1.tls-pd-peer.allenz.svc:2379\",\"advertise-peer-urls\":\"https://tls-pd-1.tls-pd-peer.allenz.svc:2380\",\"name\":\"tls-pd-1\",\"data-dir\":\"/var/lib/pd\",\"force-new-cluster\":false,\"enable-grpc-gateway\":true,\"initial-cluster\":\"tls-pd-0=https://tls-pd-0.tls-pd-peer.allenz.svc:2380,tls-pd-1=https://tls-pd-1.tls-pd-peer.allenz.svc:2380\",\"initial-cluster-state\":\"existing\",\"join\":\"https://tls-pd-0.tls-pd-peer.allenz.svc:2379\",\"lease\":3,\"log\":{\"level\":\"info\",\"format\":\"text\",\"disable-timestamp\":false,\"file\":{\"filename\":\"\",\"log-rotate\":true,\"max-size\":0,\"max-days\":0,\"max-backups\":0},\"development\":false,\"disable-caller\":false,\"disable-stacktrace\":false,\"disable-error-verbose\":true,\"sampling\":null},\"log-file\":\"\",\"log-level\":\"\",\"tso-save-interval\":\"3s\",\"metric\":{\"job\":\"tls-pd-1\",\"address\":\"\",\"interval\":\"15s\"},\"schedule\":{\"max-snapshot-count\":3,\"max-pending-peer-count\":16,\"max-merge-region-size\":20,\"max-merge-region-keys\":200000,\"split-merge-interval\":\"1h0m0s\",\"enable-one-way-merge\":\"false\",\"patrol-region-interval\":\"100ms\",\"max-store-down-time\":\"30m0s\",\"leader-schedule-limit\":4,\"region-schedule-limit\":64,\"replica-schedule-limit\":64,\"merge-schedule-limit\":8,\"hot-region-schedule-limit\":4,\"hot-region-cache-hits-threshold\":3,\"store-balance-rate\":15,\"tolerant-size-ratio\":0,\"low-space-ratio\":0.8,\"high-space-ratio\":0.6,\"scheduler-max-waiting-operator\":3,\"disable-raft-learner\":\"false\",\"disable-remove-down-replica\":\"false\",\"disable-replace-offline-replica\":\"false\",\"disable-make-up-replica\":\"false\",\"disable-remove-extra-replica\":\"false\",\"disable-location-replacement\":\"false\",\"disable-namespace-relocation\":\"false\",\"schedulers-v2\":[{\"type\":\"balance-region\",\"args\":null,\"disable\":false},{\"type\":\"balance-leader\",\"args\":null,\"disable\":false},{\"type\":\"hot-region\",\"args\":null,\"disable\":false},{\"type\":\"label\",\"args\":null,\"disable\":false}]},\"replication\":{\"max-replicas\":3,\"location-labels\":\"region,zone,rack,host\",\"strictly-match-label\":\"false\"},\"namespace\":{},\"pd-server\":{\"use-region-storage\":\"true\"},\"cluster-version\":\"0.0.0\",\"quota-backend-bytes\":\"0 B\",\"auto-compaction-mode\":\"periodic\",\"auto-compaction-retention-v2\":\"1h\",\"TickInterval\":\"500ms\",\"ElectionInterval\":\"3s\",\"PreVote\":true,\"security\":{\"cacert-path\":\"/var/run/secrets/kubernetes.io/serviceaccount/ca.crt\",\"cert-path\":\"/var/lib/pd-tls/cert\",\"key-path\":\"/var/lib/pd-tls/key\"},\"label-property\":null,\"WarningMsgs\":null,\"namespace-classifier\":\"table\",\"DisableStrictReconfigCheck\":false,\"HeartbeatStreamBindInterval\":\"1m0s\",\"LeaderPriorityCheckInterval\":\"1m0s\"}"]
[2019/09/06 12:44:24.912 +00:00] [INFO] [server.go:143] ["start embed etcd"]
[2019/09/06 12:44:24.912 +00:00] [INFO] [etcd.go:117] ["configuring peer listeners"] [listen-peer-urls="[https://0.0.0.0:2380]"]
[2019/09/06 12:44:24.912 +00:00] [INFO] [etcd.go:463] ["starting with peer TLS"] [tls-info="cert = /var/lib/pd-tls/cert, key = /var/lib/pd-tls/key, trusted-ca = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, client-cert-auth = false, crl-file = "] [cipher-suites="[]"]
[2019/09/06 12:44:24.914 +00:00] [INFO] [systime_mon.go:25] ["start system time monitor"]
[2019/09/06 12:44:24.915 +00:00] [INFO] [etcd.go:127] ["configuring client listeners"] [listen-client-urls="[https://0.0.0.0:2379]"]
[2019/09/06 12:44:24.915 +00:00] [INFO] [etcd.go:600] ["pprof is enabled"] [path=/debug/pprof]
[2019/09/06 12:44:24.915 +00:00] [INFO] [etcd.go:297] ["starting an etcd server"] [etcd-version=3.3.0+git] [git-sha="Not provided (use ./build instead of go build)"] [go-version=go1.12] [go-os=linux] [go-arch=amd64] [max-cpu-set=4] [max-cpu-available=4] [member-initialized=false] [name=tls-pd-1] [data-dir=/var/lib/pd] [wal-dir=] [wal-dir-dedicated=] [member-dir=/var/lib/pd/member] [force-new-cluster=false] [heartbeat-interval=500ms] [election-timeout=3s] [initial-election-tick-advance=true] [snapshot-count=100000] [snapshot-catchup-entries=5000] [initial-advertise-peer-urls="[https://tls-pd-1.tls-pd-peer.allenz.svc:2380]"] [listen-peer-urls="[https://0.0.0.0:2380]"] [advertise-client-urls="[https://tls-pd-1.tls-pd-peer.allenz.svc:2379]"] [listen-client-urls="[https://0.0.0.0:2379]"] [listen-metrics-urls="[]"] [cors="[*]"] [host-whitelist="[*]"] [initial-cluster="tls-pd-0=https://tls-pd-0.tls-pd-peer.allenz.svc:2380,tls-pd-1=https://tls-pd-1.tls-pd-peer.allenz.svc:2380"] [initial-cluster-state=existing] [initial-cluster-token=etcd-cluster] [quota-size-bytes=2147483648] [pre-vote=true] [initial-corrupt-check=false] [corrupt-check-time-interval=0s] [auto-compaction-mode=periodic] [auto-compaction-retention=1h0m0s] [auto-compaction-interval=1h0m0s] [discovery-url=] [discovery-proxy=]

But still not fully working:

[2019/09/06 12:44:24.992 +00:00] [INFO] [server.go:662] ["starting initial election tick advance"] [election-ticks=6]
[2019/09/06 12:44:24.992 +00:00] [INFO] [etcd.go:574] ["serving peer traffic"] [address="[::]:2380"]
[2019/09/06 12:44:25.025 +00:00] [INFO] [peer_status.go:51] ["peer became active"] [peer-id=385576565d01bba]
[2019/09/06 12:44:25.025 +00:00] [INFO] [stream.go:424] ["established TCP streaming connection with remote peer"] [stream-reader-type="stream MsgApp v2"] [local-member-id=f70e014ad1908002] [remote-peer-id=385576565d01bba]
[2019/09/06 12:44:25.041 +00:00] [INFO] [stream.go:249] ["set message encoder"] [from=f70e014ad1908002] [to=f70e014ad1908002] [stream-type="stream MsgApp v2"]
[2019/09/06 12:44:25.041 +00:00] [WARN] [stream.go:276] ["established TCP streaming connection with remote peer"] [stream-writer-type="stream MsgApp v2"] [local-member-id=f70e014ad1908002] [remote-peer-id=385576565d01bba]
[2019/09/06 12:44:25.042 +00:00] [INFO] [stream.go:249] ["set message encoder"] [from=f70e014ad1908002] [to=f70e014ad1908002] [stream-type="stream Message"]
[2019/09/06 12:44:25.042 +00:00] [WARN] [stream.go:276] ["established TCP streaming connection with remote peer"] [stream-writer-type="stream Message"] [local-member-id=f70e014ad1908002] [remote-peer-id=385576565d01bba]
[2019/09/06 12:44:25.043 +00:00] [INFO] [stream.go:424] ["established TCP streaming connection with remote peer"] [stream-reader-type="stream Message"] [local-member-id=f70e014ad1908002] [remote-peer-id=385576565d01bba]
[2019/09/06 12:44:25.043 +00:00] [INFO] [raft.go:862] ["f70e014ad1908002 [term: 1] received a MsgApp message with higher term from 385576565d01bba [term: 2]"]
[2019/09/06 12:44:25.043 +00:00] [INFO] [raft.go:712] ["f70e014ad1908002 became follower at term 2"]
[2019/09/06 12:44:25.043 +00:00] [INFO] [node.go:330] ["raft.node: f70e014ad1908002 elected leader 385576565d01bba at term 2"]
[2019/09/06 12:44:25.052 +00:00] [INFO] [cluster.go:344] ["added member"] [cluster-id=fb349a4f363de893] [local-member-id=f70e014ad1908002] [added-peer-id=385576565d01bba] [added-peer-peer-urls="[https://tls-pd-0.tls-pd-peer.allenz.svc:2380]"]
[2019/09/06 12:44:25.053 +00:00] [INFO] [cluster.go:486] ["set initial cluster version"] [cluster-id=fb349a4f363de893] [local-member-id=f70e014ad1908002] [cluster-version=3.3]
[2019/09/06 12:44:25.054 +00:00] [INFO] [capability.go:75] ["enabled capabilities for version"] [cluster-version=3.3]
[2019/09/06 12:44:25.061 +00:00] [INFO] [cluster.go:344] ["added member"] [cluster-id=fb349a4f363de893] [local-member-id=f70e014ad1908002] [added-peer-id=f70e014ad1908002] [added-peer-peer-urls="[https://tls-pd-1.tls-pd-peer.allenz.svc:2380]"]
[2019/09/06 12:44:25.063 +00:00] [INFO] [server.go:1824] ["published local member to cluster through raft"] [local-member-id=f70e014ad1908002] [local-member-attributes="{Name:tls-pd-1 ClientURLs:[https://tls-pd-1.tls-pd-peer.allenz.svc:2379]}"] [request-path=/0/members/f70e014ad1908002/attributes] [cluster-id=fb349a4f363de893] [publish-timeout=11s]
[2019/09/06 12:44:25.067 +00:00] [INFO] [serve.go:191] ["serving client traffic insecurely"] [address="[::]:2379"]
[2019/09/06 12:44:25.070 +00:00] [INFO] [server.go:173] ["create etcd v3 client"] [endpoints="[https://tls-pd-1.tls-pd-peer.allenz.svc:2379]"]
[2019/09/06 12:44:25.089 +00:00] [INFO] [server.go:210] ["init cluster id"] [cluster-id=6733536034233686735]
[2019/09/06 12:44:25.098 +00:00] [WARN] [history_buffer.go:138] ["load history index failed"] [error="leveldb: not found"]
[2019/09/06 12:44:25.098 +00:00] [INFO] [history_buffer.go:146] ["start from history index"] [start-index=0]
[2019/09/06 12:44:25.100 +00:00] [INFO] [namespace_classifier.go:461] ["load namespaces information"] [namespace-count=0] [cost=2.26595ms]
[2019/09/06 12:44:25.104 +00:00] [INFO] [server.go:938] ["server enable region storage"]
[2019/09/06 12:44:25.105 +00:00] [INFO] [server.go:820] ["start watch leader"] [leader="name:\"tls-pd-0\" member_id:253705047027751866 peer_urls:\"https://tls-pd-0.tls-pd-peer.allenz.svc:2380\" client_urls:\"https://tls-pd-0.tls-pd-peer.allenz.svc:2379\" "]
[2019/09/06 12:44:25.118 +00:00] [INFO] [client.go:106] ["server starts to synchronize with leader"] [server=tls-pd-1] [leader=tls-pd-0] [request-index=0]
[2019/09/06 12:44:41.106 +00:00] [ERROR] [redirector.go:98] ["request failed"] [error="Get https://tls-pd-0.tls-pd-peer.allenz.svc:2379/pd/ping: x509: certificate signed by unknown authority"]
[2019/09/06 12:44:41.174 +00:00] [ERROR] [redirector.go:98] ["request failed"] [error="Get https://tls-pd-0.tls-pd-peer.allenz.svc:2379/pd/ping: x509: certificate signed by unknown authority"]
[2019/09/06 12:44:50.773 +00:00] [ERROR] [redirector.go:98] ["request failed"] [error="Get https://tls-pd-0.tls-pd-peer.allenz.svc:2379/pd/ping: x509: certificate signed by unknown authority"]

Missed CA cert somewhere? I can see PD server itself and etcd server are using the correct CA cert trusted-ca = /var/run/secrets/kubernetes.io/serviceaccount/ca.crt.

rleungx · 2019-09-09T02:44:35Z

How do you generate the cacert file and the key file? And how about your config file?

AstroProfundis · 2019-09-09T08:53:02Z

# cat /etc/pd/pd.toml 
[log]
level = "info"
[replication]
location-labels = ["region", "zone", "rack", "host"]

[security]
cacert-path = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
cert-path = "/var/lib/pd-tls/cert"
key-path = "/var/lib/pd-tls/key"

Certs are generated with Kubernetes' certificate management infrastructures, all other certs used (for TiDB, TiKV and for clients) are generated the same way. Other components are also able to connect to PD (if only one PD instance is running, this issue only occurs when trying to run multiple PD instances) without problem.

ca.crt:

pd.crt:

pd.key:

rleungx · 2019-09-09T09:35:19Z

@AstroProfundis #1740 has fixed the redirect problem.

AstroProfundis · 2019-09-09T09:43:38Z

@rleungx Thanks, I'll update once verified.

disksing · 2019-09-23T04:04:45Z

Hi @AstroProfundis does the fix work out?

AstroProfundis · 2019-09-24T10:58:35Z

Fix confirmed with release-version=v4.0.0-alpha-67-g5c648dc3, thanks!

tennix added the type/bug The issue is confirmed as a bug. label Aug 18, 2019

disksing assigned nolouch Sep 3, 2019

AstroProfundis mentioned this issue Sep 4, 2019

Auto generate and sign certificates for TLS enabled cluster pingcap/tidb-operator#782

Merged

rleungx mentioned this issue Sep 5, 2019

syncer: support TLS for region syncer #1728

Merged

rleungx assigned rleungx and unassigned nolouch Sep 5, 2019

sre-bot closed this as completed in #1728 Sep 5, 2019

AstroProfundis mentioned this issue Sep 9, 2019

discovery: use port 2379 for cmdline parameters pingcap/tidb-operator#892

Merged

AstroProfundis reopened this Sep 9, 2019

AstroProfundis closed this as completed Sep 24, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gRPC connection fail when `--join` with TLS enabled #1682

gRPC connection fail when `--join` with TLS enabled #1682

AstroProfundis commented Aug 14, 2019

rleungx commented Sep 5, 2019

AstroProfundis commented Sep 5, 2019

rleungx commented Sep 6, 2019

AstroProfundis commented Sep 6, 2019

rleungx commented Sep 9, 2019

AstroProfundis commented Sep 9, 2019

rleungx commented Sep 9, 2019

AstroProfundis commented Sep 9, 2019

disksing commented Sep 23, 2019

AstroProfundis commented Sep 24, 2019

gRPC connection fail when --join with TLS enabled #1682

gRPC connection fail when --join with TLS enabled #1682

Comments

AstroProfundis commented Aug 14, 2019

rleungx commented Sep 5, 2019

AstroProfundis commented Sep 5, 2019

rleungx commented Sep 6, 2019

AstroProfundis commented Sep 6, 2019

rleungx commented Sep 9, 2019

AstroProfundis commented Sep 9, 2019

rleungx commented Sep 9, 2019

AstroProfundis commented Sep 9, 2019

disksing commented Sep 23, 2019

AstroProfundis commented Sep 24, 2019

gRPC connection fail when `--join` with TLS enabled #1682

gRPC connection fail when `--join` with TLS enabled #1682