clientv3: fix secure endpoint failover, refactor with gRPC 1.22 upgrade #10911

gyuho · 2019-07-19T15:43:52Z

ref. clientconn: set dial target "Authority" with target address grpc/grpc-go#2650
Fix etcd go client fails when querying a cluster with a down node #9949
Override gRPC upgrade vendor: update grpc-go to v1.20.0 #10624

Manually tested, and works:

Server 1 certificate:

X509v3 extensions:
    X509v3 Key Usage: critical
        Digital Signature, Key Encipherment
    X509v3 Extended Key Usage:
        TLS Web Server Authentication, TLS Web Client Authentication
    X509v3 Basic Constraints: critical
        CA:FALSE
    X509v3 Subject Alternative Name:
        DNS:localhost, IP Address:127.0.0.1, IP Address:192.168.223.156

Server 2 certificate:

X509v3 extensions:
    X509v3 Key Usage: critical
        Digital Signature, Key Encipherment
    X509v3 Extended Key Usage:
        TLS Web Server Authentication, TLS Web Client Authentication
    X509v3 Basic Constraints: critical
        CA:FALSE
    X509v3 Subject Alternative Name:
        DNS:localhost, IP Address:127.0.0.1, IP Address:192.168.121.180

Server 3 certificate:

X509v3 extensions:
    X509v3 Key Usage: critical
        Digital Signature, Key Encipherment
    X509v3 Extended Key Usage:
        TLS Web Server Authentication, TLS Web Client Authentication
    X509v3 Basic Constraints: critical
        CA:FALSE
    X509v3 Subject Alternative Name:
        DNS:localhost, IP Address:127.0.0.1, IP Address:192.168.150.136

set up 3 instances 192.168.223.156, 192.168.121.180, 192.168.150.136
start etcd cluster with TLS enabled
shut down the first node 192.168.223.156
send client requests with --endpoints 192.168.223.156:2379,192.168.121.180:2379,192.168.150.136:2379
client requests may fail on first endpoint 192.168.223.156 but client balancer should fali-over to other endpoints

$ ETCD_CLIENT_DEBUG=1 ETCDCTL_API=3 /usr/local/bin/etcdctl   --endpoints 192.168.223.156:2379,192.168.121.180:2379,192.168.150.136:2379   --cacert ${HOME}/certs/etcd-root-ca.pem   --cert ${HOME}/certs/s1.pem   --key ${HOME}/certs/s1-key.pem   put foo bar

{"level":"info","ts":1563550687.373153,"caller":"balancer/balancer.go:79","msg":"built balancer","balancer-id":"bvncglqjjf4a","policy":"etcd-client-roundrobin-balanced","resolver-target":"endpoint://client-7ba31ebc-ac8e-4fdf-b973-d03190a7b920/192.168.223.156:2379"}
{"level":"info","ts":1563550687.3732452,"caller":"balancer/balancer.go:131","msg":"resolved","balancer-id":"bvncglqjjf4a","addresses":["192.168.121.180:2379","192.168.150.136:2379","192.168.223.156:2379"]}
{"level":"info","ts":1563550687.373312,"caller":"balancer/balancer.go:189","msg":"state changed","balancer-id":"bvncglqjjf4a","connected":false,"subconn":"0xc000350aa0","address":"192.168.223.156:2379","old-state":"IDLE","new-state":"CONNECTING"}
{"level":"info","ts":1563550687.3733385,"caller":"balancer/balancer.go:189","msg":"state changed","balancer-id":"bvncglqjjf4a","connected":false,"subconn":"0xc000350ac0","address":"192.168.121.180:2379","old-state":"IDLE","new-state":"CONNECTING"}
{"level":"info","ts":1563550687.3733535,"caller":"balancer/balancer.go:189","msg":"state changed","balancer-id":"bvncglqjjf4a","connected":false,"subconn":"0xc000350ae0","address":"192.168.150.136:2379","old-state":"IDLE","new-state":"CONNECTING"}
{"level":"info","ts":1563550687.3736932,"caller":"balancer/balancer.go:189","msg":"state changed","balancer-id":"bvncglqjjf4a","connected":false,"subconn":"0xc000350aa0","address":"192.168.223.156:2379","old-state":"CONNECTING","new-state":"TRANSIENT_FAILURE"}
{"level":"info","ts":1563550687.3815699,"caller":"balancer/balancer.go:189","msg":"state changed","balancer-id":"bvncglqjjf4a","connected":true,"subconn":"0xc000350ac0","address":"192.168.121.180:2379","old-state":"CONNECTING","new-state":"READY"}
{"level":"info","ts":1563550687.3816028,"caller":"balancer/balancer.go:257","msg":"generated picker","balancer-id":"bvncglqjjf4a","policy":"etcd-client-roundrobin-balanced","subconn-ready":["192.168.121.180:2379 (0xc000350ac0)"],"subconn-size":1}
{"level":"info","ts":1563550687.3828685,"caller":"balancer/balancer.go:189","msg":"state changed","balancer-id":"bvncglqjjf4a","connected":true,"subconn":"0xc000350ae0","address":"192.168.150.136:2379","old-state":"CONNECTING","new-state":"READY"}
{"level":"info","ts":1563550687.3828938,"caller":"balancer/balancer.go:257","msg":"generated picker","balancer-id":"bvncglqjjf4a","policy":"etcd-client-roundrobin-balanced","subconn-ready":["192.168.121.180:2379 (0xc000350ac0)","192.168.150.136:2379 (0xc000350ae0)"],"subconn-size":2}
OK

Without this patch, it fails:

{"level":"info","ts":1563549545.0493703,"caller":"balancer/balancer.go:193","msg":"state changed","balancer-id":"bvnc1znj92bp","connected":false,"subconn":"0xc00038ad00","address":"192.168.121.180:2379","old-state":"CONNECTING","new-state":"TRANSIENT_FAILURE"}
{"level":"warn","ts":"2019-07-19T15:19:07.203Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-1ac3c2d5-25e2-45af-ac0c-3b71b11dce83/192.168.223.156:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate is valid for 127.0.0.1, 192.168.121.180, not 192.168.223.156\""}

Action Items

Replace authTokenCredential with custom credential bundle
Improve logging
CI testing
Enable client balancer testing in functional tests? Functional tests are running with secure endpoints. This could have been caught.

/cc @jpbetz @xiang90

gyuho · 2019-07-19T15:47:22Z

This is the initial proposal. I will improve the logging and docs around this change.
CI tests may fail. Will look into this.

@hexfusion @jingyih @wenjiaswe @spzala Can you take a look as well?

This should be the last major change before 3.4 code freeze.

dims · 2019-07-19T17:03:53Z

@gyuho is the crux of the change this bit? ~~dd73d08~~ db61ee1 or is there another change in grpc that is needed along with ~~dd73d08~~ db61ee1?

gyuho · 2019-07-19T18:02:46Z

@dims ~~dd73d08~~ db61ee1 is the only change needed in etcd. Other than upgrading gRPC in etcd, we don't need any change in gRPC. Newer gRPC is needed because older version does not support credential bundle.

dims · 2019-07-19T18:27:35Z

excellent. thanks @gyuho

jingyih · 2019-07-23T21:59:00Z

Thanks for implementing the fix. Do we have a test to verify the original bug was fixed?

gyuho · 2019-07-23T22:37:13Z

@jingyih

See

Manually tested, and works:

We can only test it manually, because the fix is made in our TLS SAN field authentication :)

I manually confirmed that this fixes the original bug.

jingyih · 2019-07-23T23:21:19Z

clientv3/credentials/credentials.go

+	return nil, nil
+}
+
+// transportCredential implements "grpccredentials.PerRPCCredentials" interface.


nit: "grpccredentials.TransportCredentials"

xiang90 · 2019-07-25T18:57:03Z

clientv3/credentials/credentials.go

@@ -71,6 +71,19 @@ func newTransportCredential(cfg *tls.Config) *transportCredential {
 }

 func (tc *transportCredential) ClientHandshake(ctx context.Context, authority string, rawConn net.Conn) (net.Conn, grpccredentials.AuthInfo, error) {
+	target := rawConn.RemoteAddr().String()
+	if authority != target {


do we need to check authority is an IP address rather than a DNS name?

xiang90 · 2019-07-25T18:57:20Z

the core idea looks good to me.

gyuho · 2019-07-25T21:44:26Z

@xiang90 Good point. I made a change to overwrite authority only when the target is IP. Manually tested with SRV records:

diff --git a/tests/docker-dns-srv/certs/server-ca-csr.json b/tests/docker-dns-srv/certs/server-ca-csr.json
index 661de3799..f3a23c656 100644
--- a/tests/docker-dns-srv/certs/server-ca-csr.json
+++ b/tests/docker-dns-srv/certs/server-ca-csr.json
@@ -19,8 +19,6 @@
     "m4.etcd.local",
     "m5.etcd.local",
     "m6.etcd.local",
-    "etcd.local",
-    "127.0.0.1",
-    "localhost"
+    "etcd.local"
   ]
 }

Before (if we overwrite authority)

{"level":"warn","ts":"2019-07-25T21:14:38.454Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-0d92363c-2ca8-42a9-ae36-ddb1700a016e/m5.etcd.local.:23791","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = "transport: authentication handshake failed: x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs""}

After (if we overwrite only when it's an IP)

No error

codecov-io · 2019-07-25T23:04:04Z

Codecov Report

Merging #10911 into master will decrease coverage by 0.23%.
The diff coverage is 81.76%.

@@            Coverage Diff             @@
##           master   #10911      +/-   ##
==========================================
- Coverage    63.9%   63.66%   -0.24%     
==========================================
  Files         400      400              
  Lines       37688    37415     -273     
==========================================
- Hits        24085    23821     -264     
- Misses      11966    11976      +10     
+ Partials     1637     1618      -19

Impacted Files	Coverage Δ
embed/serve.go	`37.38% <0%> (ø)`	⬆️
clientv3/balancer/picker/roundrobin_balanced.go	`100% <100%> (ø)`	⬆️
clientv3/balancer/utils.go	`100% <100%> (ø)`	⬆️
clientv3/balancer/picker/err.go	`100% <100%> (ø)`	⬆️
clientv3/balancer/connectivity/connectivity.go	`100% <100%> (ø)`
etcdserver/api/v3rpc/grpc.go	`100% <100%> (ø)`	⬆️
clientv3/balancer/picker/picker.go	`50% <50%> (ø)`
clientv3/client.go	`75.95% <70.58%> (-2.58%)`	⬇️
clientv3/credentials/credentials.go	`77.27% <77.27%> (ø)`
clientv3/balancer/balancer.go	`85.07% <90.47%> (+0.94%)`	⬆️
... and 35 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50babc1...a7b8034. Read the comment docs.

jpbetz

Did a first pass review. I'll do a second pass tomorrow to make sure I fully understand some of the details.

jpbetz · 2019-07-25T23:26:49Z

clientv3/balancer/balancer.go

-	oldAggrState := bb.currentState
-	bb.currentState = bb.csEvltr.recordTransition(old, s)
+	oldAggrState := bb.connectivityRecorder.GetCurrentState()
+	bb.connectivityRecorder.RecordTransition(old, s)

 	// Regenerate picker when one of the following happens:


Update comment to reflect that we're calling updatedPicker now instead regeneratePicker?

jpbetz · 2019-07-25T23:35:26Z

clientv3/balancer/connectivity/connectivity.go

+	rc.mu.RLock()
+	state = rc.cur
+	rc.mu.RUnlock()
+	return state


nit: Can we just do:

rc.mu.RLock() defer rc.mu.RUnlock() return rc.cur

?

jpbetz · 2019-07-25T23:39:35Z

clientv3/balancer/connectivity/connectivity.go

+			rc.numConnecting += updateVal
+		case connectivity.TransientFailure:
+			rc.numTransientFailure += updateVal
+		}


nit: Log a warn or something if none of the cases match?

jpbetz · 2019-07-25T23:46:41Z

clientv3/balancer/picker/picker.go

+	// Error is error picker policy.
+	Error Policy = iota
+
+	// RoundrobinBalanced balance loads over multiple endpoints


jpbetz · 2019-07-25T23:52:01Z

clientv3/balancer/balancer.go

+	// Picker implements gRPC picker.
+	// Leave empty if "Policy" field is not custom.
+	// TODO: currently custom policy is not supported.
+	// Picker picker.Picker


Remove comments about picker configuration and Custom picker enum in this PR and instead open an issue about it? It's a adds quite a bit of noise in this PR, might be better to just leave it out of the code until we decide to implement something. Same for custom balancers.

jpbetz · 2019-07-26T00:08:40Z

clientv3/credentials/credentials.go

+}
+
+func (tc *transportCredential) Clone() grpccredentials.TransportCredentials {
+	return tc.gtc.Clone()


return &transportCredential{ gtc: tc.gtc.Clone() }

? (I realize we're not using it for anything, but just to future proof..)

jpbetz · 2019-07-26T00:14:31Z

clientv3/credentials/credentials.go

+// perRPCCredential implements "grpccredentials.PerRPCCredentials" interface.
+type perRPCCredential struct {
+	authToken   string
+	authTokenMu *sync.RWMutex


pointer needed? just do authTokenMu sync.RWMutex to eliminate authTokenMu: new(sync.RWMutex) below?

jpbetz · 2019-07-26T00:31:19Z

clientv3/credentials/credentials.go

+	}
+}
+
+func (tc *transportCredential) ClientHandshake(ctx context.Context, authority string, rawConn net.Conn) (net.Conn, grpccredentials.AuthInfo, error) {


This looks right to me.

@wenjiaswe Would you double check this function as well? Docs for ClientHandshake here: https://godoc.org/google.golang.org/grpc/credentials#TransportCredentials

I checked the function, lgtm, thanks!

jpbetz · 2019-07-26T21:47:15Z

@xiang90 @gyuho Will this fix be back ported to release-3.3 branch? (hopefully a v3.3.14)?

I don't think 3.3 had the problem this fixes. This is a fix we need a result of the transition to the grpc load balancer that was introduced on master for 3.4.

jingyih · 2019-07-26T21:57:14Z

The original issue from Kubernetes repo was reported with etcd version 3.2.24. etcd 3.2 and 3.3 do not use the grpc load balancer, but still have the same issue which needs to be fixed (primarily by 13e26a4.)

I think we should consider backporting (part of) this.

dims · 2019-07-26T22:01:31Z

@jingyih correct. and as i said before in #10911 (comment) we tried patching up the ClientHandshake in grpc directly (the vendored copy in k/k) to confirm that the patch works in manual testing

gyuho · 2019-07-27T05:38:12Z

@dims @jpbetz We did something similar for etcd 3.2 (ref. kubernetes/kubernetes#57480) to backport client balancer fixes from 3.3 to 3.2. We can look into backporting this after 3.4 release.

frittentheke · 2019-07-28T06:03:42Z

@gyuho massive thanks for fixing this.
Regarding fixing this for older versions, what would keep Kubernetes or rather kubeadm from straight installing 3.4 of etcd inatead of a point release of 3.2 or 3.3 containing this kind of bugfix?

dims · 2019-07-29T12:51:18Z

@frittentheke the fix is in the etcd client code that is vendored into k/k and hence part of api server. So just updating the etcd "server" will not help in any way to mitigate the problem

frittentheke · 2019-07-29T14:15:04Z

@dims urgh, yeah, my bad.

But since this solely is a client issue, there is no need for any backporting required, is there?
The only issue that might needs raising with them, is to pull in a more recent version of the etcd client lib for their next point releases of all still supported K8s versions right?

dims · 2019-07-29T14:23:32Z

@frittentheke "etcd client lib" with the fix in this PR, note that this drags in new grpc as well. Note that this combination has not landed in k/k master, so there is no testing at all at this point :) Also see my note in the k/k issue - kubernetes/kubernetes#72102 (comment)

pariminaresh · 2019-08-22T22:18:45Z

We are experiencing the same issue as API Server is crashing out every time the first etcd server goes down. Earlier using etcd v3.3.13 and upgraded this to v3.3.15 after coming across this PR and kubernetes/kubernetes#72102.

But I still see that apiserver is crashing out.After spending some time to make sure I'm using the proper versions, decided to take help from other folks here. I believe I might be missing something. Please correct me if so. Thanks!

Below are the details:

3 node cluster : 10.66.0.162,10.66.0.166,10.66.0.168
shutdown 10.66.0.166 & running test on 10.66.0.162

1# etcdctl -version
etcdctl version: 3.3.15
API version: 2

2# etcd -version
etcd Version: 3.3.15
Git SHA: 94745a4ee
Go Version: go1.12.9
Go OS/Arch: linux/amd64

3# ETCDCTL_API=3 etcdctl --endpoints=10.66.0.166:2379,10.66.0.162:2379,10.66.0.168:2379 --cacert /etc/kubernetes/pki/etcd/ca.pem --cert /etc/kubernetes/pki/etcd/client.pem put --key /etc/kubernetes/pki/etcd/client-key.pem foo bar
{"level":"warn","ts":"2019-08-22T21:37:36.613Z","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-3fc8c37e-a22f-4452-a372-9c98664d645e/10.66.0.166:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = \"transport: authentication handshake failed: x509: certificate is valid for 10.66.0.162, not 10.66.0.166\""}

4# ETCDCTL_API=3 etcdctl --endpoints=10.66.0.162:2379,10.66.0.166:2379,10.66.0.168:2379 --cacert /etc/kubernetes/pki/etcd/ca.pem --cert /etc/kubernetes/pki/etcd/client.pem put --key /etc/kubernetes/pki/etcd/client-key.pem foo bar
OK

As you notice here, I expected 'OK' response in the 3rd step as well. Do I need re-generate the certificates in any specific way? the same certs works fine when the first server is up. Please point me in right direction.

zhangxinZKCD · 2019-09-17T11:53:41Z

I use a domain name to issue a certificate,the error still exists!

{"level":"warn","ts":"2019-09-17T19:23:17.183+0800","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-8728de64-f8ea-4354-8151-4365e42b3acd/e2341-1.etcd.dpool.com.cn:2341","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest connection error: connection error: desc = "transport: authentication handshake failed: x509: certificate is valid for e2341-4.etcd.dpool.com.cn, not e2341-1.etcd.dpool.com.cn""}

Contains an important fix in clientv3 that allows vault to successfully failover to another etcdv3 endpoint in the event that the current active connection becomes unavailable. See also: * etcd-io/etcd#9949 * etcd-io/etcd#10911 * https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3314-2019-08-16 Fixes hashicorp#4961

Specifically, we are looking for this fix etcd-io/etcd#10911 Which, was included in v3.3.14 according to the changelog: https://github.com/etcd-io/etcd/blob/master/CHANGELOG-3.3.md#v3314-2019-08-16

gyuho added the WIP label Jul 19, 2019

gyuho mentioned this pull request Jul 19, 2019

kube-apiserver 1.13.x refuses to work when first etcd-server is not available. kubernetes/kubernetes#72102

Closed

gyuho added the Release-Note label Jul 19, 2019

gyuho force-pushed the balancer branch from a082996 to 0b1369c Compare July 19, 2019 16:03

jingyih added this to the etcd-v3.4 milestone Jul 19, 2019

gyuho force-pushed the balancer branch from 0b1369c to aa2e046 Compare July 22, 2019 06:32

gyuho changed the title ~~clientv3: fix secure endpoint failover, upgrade gRPC to 1.22~~ clientv3: fix secure endpoint failover, refactor with gRPC 1.22 upgrade Jul 22, 2019

gyuho force-pushed the balancer branch 2 times, most recently from 874f1bf to 010e6d7 Compare July 23, 2019 15:37

gyuho removed the WIP label Jul 23, 2019

gyuho requested a review from jpbetz July 23, 2019 17:55

gyuho self-assigned this Jul 23, 2019

gyuho force-pushed the balancer branch 2 times, most recently from 2e3d59b to 1b4a401 Compare July 25, 2019 17:32

jingyih reviewed Jul 25, 2019

View reviewed changes

xiang90 reviewed Jul 25, 2019

View reviewed changes

gyuho force-pushed the balancer branch 3 times, most recently from 8c64b9f to 3b39c57 Compare July 25, 2019 22:12

etcd-io deleted a comment from codecov-io Jul 25, 2019

jpbetz reviewed Jul 26, 2019

View reviewed changes

gyuho merged commit 89d4002 into etcd-io:master Jul 26, 2019

gyuho deleted the balancer branch July 27, 2019 05:38

gyuho mentioned this pull request Jul 27, 2019

etcd go client fails when querying a cluster with a down node #9949

Closed

jingyih mentioned this pull request Jul 29, 2019

3.3.7 panic: send on closed channel #9956

Closed

jsok mentioned this pull request Jul 30, 2019

Vault is inaccessible if an etcd unit is lost hashicorp/vault#4961

Closed

gyuho mentioned this pull request Aug 5, 2019

Backport client secure endpoint failover fix to 3.3 with gRPC 1.23 #10980

Closed

5 tasks

dims mentioned this pull request Aug 19, 2019

Updating to etcd v3.3.15 kubernetes/kubernetes#81434

Merged

pariminaresh mentioned this pull request Aug 26, 2019

upgrading etcd version to 3.3.15, doesn't fix API Server crash issue #11078

Closed

jsok mentioned this pull request Sep 2, 2019

Upgrade etcd dependency to v3.3.15 hashicorp/vault#7403

Closed

liggitt mentioned this pull request Sep 23, 2019

kube-apiserver: failover on multi-member etcd cluster fails certificate check on DNS mismatch kubernetes/kubernetes#83028

Closed

lining2020x mentioned this pull request Sep 24, 2019

The latest etcdctl seems buggy: etcdctl will request fail when the first etcd cluster endpoint down #11176

Closed

jpbetz mentioned this pull request Oct 3, 2019

Add integration TLS tests exercising SAN certs #11203

Closed

hexfusion mentioned this pull request Oct 23, 2019

[openshift-4.3]: bump etcd to v3.3.17 openshift/etcd#20

Merged

caseydavenport mentioned this pull request Dec 17, 2019

Update etcd version to pull in bug fixes projectcalico/libcalico-go#1184

Merged

3 tasks

liggitt mentioned this pull request Mar 10, 2022

Add failover test cases #13737

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clientv3: fix secure endpoint failover, refactor with gRPC 1.22 upgrade #10911

clientv3: fix secure endpoint failover, refactor with gRPC 1.22 upgrade #10911

gyuho commented Jul 19, 2019 •

edited

Loading

gyuho commented Jul 19, 2019

dims commented Jul 19, 2019 •

edited by gyuho

Loading

gyuho commented Jul 19, 2019 •

edited

Loading

dims commented Jul 19, 2019

jingyih commented Jul 23, 2019

gyuho commented Jul 23, 2019 •

edited

Loading

jingyih Jul 23, 2019

xiang90 Jul 25, 2019

xiang90 commented Jul 25, 2019

gyuho commented Jul 25, 2019

codecov-io commented Jul 25, 2019 •

edited

Loading

jpbetz left a comment

jpbetz Jul 25, 2019

jpbetz Jul 25, 2019

jpbetz Jul 25, 2019

jpbetz Jul 25, 2019

jpbetz Jul 25, 2019

jpbetz Jul 26, 2019

jpbetz Jul 26, 2019

jpbetz Jul 26, 2019

wenjiaswe Jul 26, 2019

jpbetz commented Jul 26, 2019

jingyih commented Jul 26, 2019 •

edited

Loading

dims commented Jul 26, 2019

gyuho commented Jul 27, 2019

frittentheke commented Jul 28, 2019

dims commented Jul 29, 2019

frittentheke commented Jul 29, 2019 •

edited

Loading

dims commented Jul 29, 2019

pariminaresh commented Aug 22, 2019 •

edited

Loading

zhangxinZKCD commented Sep 17, 2019

clientv3: fix secure endpoint failover, refactor with gRPC 1.22 upgrade #10911

clientv3: fix secure endpoint failover, refactor with gRPC 1.22 upgrade #10911

Conversation

gyuho commented Jul 19, 2019 • edited Loading

gyuho commented Jul 19, 2019

dims commented Jul 19, 2019 • edited by gyuho Loading

gyuho commented Jul 19, 2019 • edited Loading

dims commented Jul 19, 2019

jingyih commented Jul 23, 2019

gyuho commented Jul 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiang90 commented Jul 25, 2019

gyuho commented Jul 25, 2019

codecov-io commented Jul 25, 2019 • edited Loading

Codecov Report

jpbetz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpbetz commented Jul 26, 2019

jingyih commented Jul 26, 2019 • edited Loading

dims commented Jul 26, 2019

gyuho commented Jul 27, 2019

frittentheke commented Jul 28, 2019

dims commented Jul 29, 2019

frittentheke commented Jul 29, 2019 • edited Loading

dims commented Jul 29, 2019

pariminaresh commented Aug 22, 2019 • edited Loading

zhangxinZKCD commented Sep 17, 2019

gyuho commented Jul 19, 2019 •

edited

Loading

dims commented Jul 19, 2019 •

edited by gyuho

Loading

gyuho commented Jul 19, 2019 •

edited

Loading

gyuho commented Jul 23, 2019 •

edited

Loading

codecov-io commented Jul 25, 2019 •

edited

Loading

jingyih commented Jul 26, 2019 •

edited

Loading

frittentheke commented Jul 29, 2019 •

edited

Loading

pariminaresh commented Aug 22, 2019 •

edited

Loading