clientv3/ordering TestUnresolvableOrderViolation fails

[gyuho]

TestUnresolvableOrderViolation goes into infinite leader elections...

https://semaphoreci.com/coreos/etcd/branches/master/builds/2379

[xiang90]

is this already fixed?

[gyuho]

Let me try to reproduce.

[gyuho]

Still happens https://semaphoreci.com/coreos/etcd/branches/master.

log-ordering.txt

[xiang90]

/cc @lorneli would you like to take a look at this test failure?

[lorneli]

@xiang90 Yeah. I'll follow up this weekend.

[lorneli]

I haven't reproduced this failure locally, so have to depend on log in semaphore.

Based on log-ordering.txt, TestUnresolvableOrderViolation blocks on putting k-v pair to first member of the cluster, actually not related to clientv3/ordering code. Looks like a integration/etcd-server bug.

cfg := clientv3.Config{
    // ...
}
cli, err := clientv3.New(cfg)
    // ....
cli.SetEndpoints(clus.Members[0].GRPCAddr())
time.Sleep(1 * time.Second)
_, err = cli.Put(ctx, "foo", "bar")  // block here, server doesn't resp

I see many grpc warning lines about connection closing, printed by grpclog . For example, line 5163-5168 in log-ordering.txt(See below). Is there a guarantee that log generated by capnslog and grpclog happends in sequence? Log shows three nodes can't be dialed after etcd server is published, which is a little confusing...

2017-10-05 11:41:31.385241 I | etcdserver: published {Name:3998243288755946524 ClientURLs:[unix://127.0.0.1:2105415012]} to cluster 77a8bf0cb4e3ab13
2017-10-05 11:41:31.389455 I | etcdserver: published {Name:337497985577860612 ClientURLs:[unix://127.0.0.1:2104815012]} to cluster 77a8bf0cb4e3ab13
2017-10-05 11:41:31.393393 I | etcdserver: published {Name:3210302211912965408 ClientURLs:[unix://127.0.0.1:2105015012]} to cluster 77a8bf0cb4e3ab13
2017-10-05 11:41:31.393679 I | etcdserver: published {Name:3123809328254117252 ClientURLs:[unix://127.0.0.1:2105215012]} to cluster 77a8bf0cb4e3ab13
2017-10-05 11:41:31.399699 I | etcdserver: published {Name:1904611785361370535 ClientURLs:[unix://127.0.0.1:2105615012]} to cluster 77a8bf0cb4e3ab13
2017-10-05 11:41:31.414984 I | etcdserver: setting up the initial cluster version to 3.2
2017-10-05 11:41:31.423877 N | etcdserver/membership: set the initial cluster version to 3.2
2017-10-05 11:41:31.425623 N | etcdserver/membership: set the initial cluster version to 3.2
2017-10-05 11:41:31.430211 N | etcdserver/membership: set the initial cluster version to 3.2
2017-10-05 11:41:31.431297 N | etcdserver/membership: set the initial cluster version to 3.2
2017-10-05 11:41:31.432081 N | etcdserver/membership: set the initial cluster version to 3.2
WARNING: 2017/10/05 11:41:31 Failed to dial localhost:31238093282541172520: grpc: the connection is closing; please retry.
WARNING: 2017/10/05 11:41:31 Failed to dial localhost:31238093282541172520: grpc: the connection is closing; please retry.
WARNING: 2017/10/05 11:41:31 Failed to dial localhost:39982432887559465240: grpc: the connection is closing; please retry.
WARNING: 2017/10/05 11:41:31 Failed to dial localhost:39982432887559465240: grpc: the connection is closing; please retry.
WARNING: 2017/10/05 11:41:31 Failed to dial localhost:19046117853613705350: grpc: the connection is closing; please retry.
WARNING: 2017/10/05 11:41:31 Failed to dial localhost:19046117853613705350: grpc: the connection is closing; please retry.

[xiang90]

probably we need to investigate why there is the 1 second sleep. it seems pretty random and arbitrary.

[xiang90]

/cc @mangoslicer

[mangoslicer]

The 1 second sleep was to ensure that the endpoint was set to the first member. I'll try to reproduce the error locally and see if changing the 1 second delay or not setting the endpoint to the first member changes anything.

[mkumatag]

This is failing consistently in ppc64le platform - https://jenkins-etcd-public.prod.coreos.systems/job/etcd-ci-ppc64/

[mkumatag]

@gyuho I still see tests are failing, any idea why this issue is closed.?

[gyuho]

@mkumatag client integration tests aren't stable yet.
We are fixing those failures now with highest priority (#8678 and #8677).

Sorry!