lease: force leader to apply its pending committed index for lease op… #7015

fanminshi · 2016-12-15T20:44:43Z

suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending committed index before looking up lease.

FIX #6978

gyuho · 2016-12-15T20:52:37Z

etcdserver/api/v2http/peer.go

-		lh = leasehttp.NewHandler(l)
+	l := s.Lessor()
+	w := s.ApplyWait()
+	if l != nil && w != nil {


when do we get applyWait == nil?

we will have a nil pointer exception in leaseHTTP. Probably need to add an extra check.

ignore my previous comment, i mis-read it. applyWait == nil happens if we don't call EtcdServer.Start() from here https://github.com/coreos/etcd/blob/master/etcdserver/server.go#L526

heyitsanthony · 2016-12-15T20:52:58Z

integration/v3_lease_test.go

+
+	for i := 0; i < 30; i++ {
+		if err := <-errc; err != nil {
+			cancel()


this cancel isn't necessary; the Fatal will panic and the deferred cancel will run

heyitsanthony · 2016-12-15T20:55:03Z

integration/v3_lease_test.go

+	errc := make(chan error)
+
+	for i := 0; i < 10; i++ {
+		go func() {


for j := 0; j < 3; j++ { go func() { errc <- stressLeaseRenew(ctx, clus.Client(j) }() }

heyitsanthony · 2016-12-15T20:55:48Z

integration/v3_lease_test.go

@@ -22,6 +22,7 @@ import (
 	"golang.org/x/net/context"
 	"google.golang.org/grpc/metadata"

+	"github.com/coreos/etcd/clientv3"


integration/ tests don't use clientv3

heyitsanthony · 2016-12-15T20:57:36Z

integration/v3_lease_test.go

+	}
+}
+
+func stressLeaseRenew(ctx context.Context, cli *clientv3.Client) error {


if ctx is passed in, the rpcs usually should respect that ctx instead of using a fresh TODO() context

heyitsanthony · 2016-12-15T21:00:54Z

integration/v3_lease_test.go

+// TestV3LeaseTimeToLiveStress keeps creating lease and retriving it immediately to ensure the lease can be retrived.
+// it was oberserved that the immediate lease retrival after granting a lease from follower resulted lease not found.
+// related issue https://github.com/coreos/etcd/issues/6978
+func TestV3LeaseTimeToLiveStress(t *testing.T) {


this is nearly the same thing as the other test, the only difference is whether it calls stressLeaseTimeToLive or stressLeaseRenew; this can be simplified into a generic test func testLeaseStress(stresser func(context.Context, leaseClient)) which would be called by both TestV3Lease functions

what is the reproducibility of this test? can this reproduce the issue every time?

@xiang90 it is quite reproducible. the test fails very fast within one second or two even if i give it a 5 seconds timeout.

awesome. thanks.

heyitsanthony · 2016-12-15T21:13:02Z

etcdserver/server.go

@@ -1613,6 +1615,10 @@ func (s *EtcdServer) getCommittedIndex() uint64 {
 	return atomic.LoadUint64(&s.committedIndex)
 }

+func (s *EtcdServer) GetCommittedIndex() uint64 {


exporting internal details (e.g., indexes) like this really should be avoided; it leads high coupling and poor cohesion

heyitsanthony · 2016-12-15T21:14:00Z

etcdserver/server.go

@@ -565,6 +565,8 @@ func (s *EtcdServer) RaftHandler() http.Handler { return s.r.transport.Handler()

 func (s *EtcdServer) Lessor() lease.Lessor { return s.lessor }

+func (s *EtcdServer) ApplyWait() wait.WaitTime { return s.applyWait }


this shouldn't be exported to the outside world if possible, it's an internal implementation detail. Try

func (s* EtcdServer) ApplyNotify() <-chan { return s.applyWait.Wait(atomic.LoadUint64(&s.committedIndex)) } }

that's is much better. will change.

heyitsanthony · 2016-12-15T21:17:14Z

lease/leasehttp/http.go

 )

 // NewHandler returns an http Handler for lease renewals
-func NewHandler(l lease.Lessor) http.Handler {
-	return &leaseHandler{l}
+func NewHandler(l lease.Lessor, w wait.WaitTime, i Indexer) http.Handler {


This is a bad interface; it forces the handler to reason/know about indexes and WaitTime, then introduces a lot of interfaces to pretend it's an appropriate abstraction. Having func NewHandler(l lease.Lessor, waitch func() <-chan) instead would be so much simpler; it can be called like NewHandler(l, func() <-chan { return s.ApplyNotify()}).

heyitsanthony · 2016-12-15T21:19:38Z

lease/leasehttp/http.go

+		select {
+		case <-h.w.Wait(cci):
+		case <-time.After(applyTimeout):
+			http.Error(w, "waiting for node to catch up its applied index has timed out", http.StatusInternalServerError)


should be an actual error code like the other errors; otherwise, this will cause the client lease to shutdown despite this most likely being a transient error

heyitsanthony · 2016-12-15T21:23:05Z

lease/leasehttp/http_test.go

@@ -76,3 +122,32 @@ func TestTimeToLiveHTTP(t *testing.T) {
 		t.Fatalf("granted TTL expected 5, got %d", resp.LeaseTimeToLiveResponse.GrantedTTL)
 	}
 }
+
+func TestTimeToLiveHTTPTimeout(t *testing.T) {


this is very similar to TestRenewHTTPTimeout both should probably call into a common function that has Renew/TimeToLive passed in

fanminshi · 2016-12-16T02:22:55Z

integration/v3_lease_test.go

+
+func stressLeaseTimeToLive(tctx context.Context, lc pb.LeaseClient) error {
+	for tctx.Err() == nil {
+		resp, gerr := lc.LeaseGrant(context.Background(), &pb.LeaseGrantRequest{TTL: 60})


I couldn't use tctx because tctx is associated with a timeout. i don't want LeaseGrant and LeaseTimeToLive to throw err due to timed out context.

fanminshi · 2016-12-16T02:23:48Z

integration/v3_lease_test.go

+}
+
+func stressLeaseRenew(tctx context.Context, lc pb.LeaseClient) error {
+	cctx, cancel := context.WithCancel(context.Background())


I couldn't use tctx because tctx is associated with a timeout. i don't want LeaseGrant, Send(), Recv()and to throw err due to timed out context.

func stressLeaseRenew(tctx context.Context, lc pb.LeaseClient) (reterr error) { defer func() { if tctx.Err() != nil { reterr = nil } }() ... }

that's nice. good to know.

fanminshi · 2016-12-16T02:25:01Z

All fixed. PLTAL @heyitsanthony

heyitsanthony · 2016-12-16T18:49:25Z

lease/leasehttp/http_test.go

 	le.Promote(time.Second)
 	l, err := le.Grant(1, int64(5))
 	if err != nil {
 		t.Fatalf("failed to create lease: %v", err)
 	}

-	ts := httptest.NewServer(NewHandler(le))
+	ts := httptest.NewServer(NewHandler(le, func() <-chan struct{} { return w.Wait(2) }))


No need for a time list; it's not being used for anything aside from getting a closed channel:

func waitReady() <-chan { ch := make(chan struct) close(ch) return ch }

ts := httptest.NewServer(NewHandler(le, waitReady))

heyitsanthony · 2016-12-16T18:49:44Z

lease/leasehttp/http_test.go

+	w := wait.NewTimeList()
+	// trigger committed index 2 to simulate the case that node has catched up to the latest CommittedIndex 2 returned by indexerMock
+	w.Trigger(2)
+	ts := httptest.NewServer(NewHandler(le, func() <-chan struct{} { return w.Wait(2) }))


ts := httptest.NewServer(NewHandler(le, waitReady))

heyitsanthony · 2016-12-16T18:50:34Z

lease/leasehttp/http_test.go

+		t.Fatalf("failed to create lease: %v", err)
+	}
+
+	ts := httptest.NewServer(NewHandler(le, func() <-chan struct{} { return w.Wait(2) }))


again, no need for a time list, it only needs to simulate a timeout:

ts := httptest.NewServer(NewHandler(le, func() <-chan struct{} { return nil }))

heyitsanthony · 2016-12-16T18:57:33Z

integration/v3_lease_test.go

+}
+
+func stressLeaseRenew(tctx context.Context, lc pb.LeaseClient) error {
+	cctx, cancel := context.WithCancel(context.Background())


func stressLeaseRenew(tctx context.Context, lc pb.LeaseClient) (reterr error) { defer func() { if tctx.Err() != nil { reterr = nil } }() ... }

heyitsanthony · 2016-12-16T18:59:15Z

lease/leasehttp/http_test.go

@@ -76,3 +85,42 @@ func TestTimeToLiveHTTP(t *testing.T) {
 		t.Fatalf("granted TTL expected 5, got %d", resp.LeaseTimeToLiveResponse.GrantedTTL)
 	}
 }
+
+func testTimeout(t *testing.T, f func(*lease.Lease, *httptest.Server) error) {


func testApplyTimeout(t *testing.T, f func(*lease.Lease, string) error)

heyitsanthony · 2016-12-16T19:05:52Z

etcdserver/api/v3rpc/rpctypes/error.go

@@ -31,6 +31,8 @@ var (
 	ErrGRPCLeaseNotFound = grpc.Errorf(codes.NotFound, "etcdserver: requested lease not found")
 	ErrGRPCLeaseExist    = grpc.Errorf(codes.FailedPrecondition, "etcdserver: lease already exists")

+	ErrGRPCLeaseHTTPTimeout = grpc.Errorf(codes.Unavailable, "etcdserver: lease http request timed out")


After looking through EtcdServer.LeaseRenew and EtcdServer.LeaseTimeToLive again, it seems like this error can't possibly be returned to the client, so I was wrong about the client shutdown problem. Probably better not to define it as an rpc error if it won't leave the etcd server.

if the LeaseHTTPTimeout do happen, how should the error being handled? right now, i have it set up so that the error will propagate to client as a grpc error with codes.Unavailable. then the client should retry the request on this error.

Huh? How is propagating to the grpc client? It goes through LeaseRenew/LeaseTimeToLive in EtcdServer which will retry so long as the request context hasn't timed out. If the context times out, EtcdServer will return ErrTimeout back to the client.

@heyitsanthony, you are right. i didn't see the loop logic in EtcdServer LeaseRenew. I'll get rid of the grpc error then.

fanminshi · 2016-12-16T21:17:45Z

All Fixed. PTAL @heyitsanthony

…erations suppose a lease granting request from a follower goes through and followed by a lease look up or renewal, the leader might not apply the lease grant request locally. So the leader might not find the lease from the lease look up or renewal request which will result lease not found error. To fix this issue, we force the leader to apply its pending commited index before looking up lease. FIX etcd-io#6978

fanminshi · 2016-12-22T23:01:45Z

All fixed. PTAL @xiang90 @heyitsanthony. CI passed.

xiang90 · 2016-12-26T03:18:27Z

etcdserver/raft.go

@@ -188,6 +188,8 @@ func (r *raftNode) start(rh *raftReadyHandler) {
 					return
 				}

+				updateCommittedIndex(&ap, rh)


move this before line 184? or the etcd routine might apply entries before we update commit index. then applied index might be greater than commit index.

good catch! will change

xiang90 · 2016-12-26T03:19:03Z

etcdserver/server.go

@@ -596,7 +598,8 @@ type etcdProgress struct {
 // and helps decouple state machine logic from Raft algorithms.
 // TODO: add a state machine interface to apply the commit entries and do snapshot/recover
 type raftReadyHandler struct {
-	leadershipUpdate func()
+	leadershipUpdate   func()
+	committedIdxUpdate func(uint64)


updateCommittedIndex.

that's better. I also like to keep a consistent naming convention. should I also make leadershipUpdate -> updateLeadership?

probably. we should do it in another pr though.

make sense. will do!

fanminshi · 2016-12-27T18:13:52Z

All fixed. PTAL @xiang90

xiang90 · 2016-12-27T18:40:14Z

defer to @heyitsanthony

heyitsanthony · 2016-12-27T19:25:12Z

lgtm

fanminshi requested review from xiang90, heyitsanthony and gyuho December 15, 2016 20:44

fanminshi added type/bug area/testing and removed type/bug labels Dec 15, 2016

gyuho reviewed Dec 15, 2016

View reviewed changes

heyitsanthony suggested changes Dec 15, 2016

View reviewed changes

fanminshi force-pushed the fix_lease_expired_too_soon branch 2 times, most recently from 27b7560 to d8849cb Compare December 16, 2016 02:18

fanminshi commented Dec 16, 2016

View reviewed changes

heyitsanthony suggested changes Dec 16, 2016

View reviewed changes

fanminshi force-pushed the fix_lease_expired_too_soon branch 2 times, most recently from b7aa0b3 to a727aa0 Compare December 16, 2016 21:17

fanminshi force-pushed the fix_lease_expired_too_soon branch 2 times, most recently from 7bf2163 to 0c64dcc Compare December 22, 2016 21:43

fanminshi added 2 commits December 22, 2016 14:24

leasehttp: buffer error channel to prevent goroutine leak

7b7feb4

fanminshi force-pushed the fix_lease_expired_too_soon branch from 0c64dcc to c81d108 Compare December 22, 2016 22:42

xiang90 reviewed Dec 26, 2016

View reviewed changes

etcdserver: rework update committed index logic

2faf72f

fanminshi force-pushed the fix_lease_expired_too_soon branch from c81d108 to 2faf72f Compare December 27, 2016 18:12

heyitsanthony approved these changes Dec 27, 2016

View reviewed changes

fanminshi merged commit 89b18ff into etcd-io:master Dec 27, 2016

lzhfromustc mentioned this pull request Oct 29, 2019

agent: add 1 buffer to errc to prevent goroutine leak #11302

Closed

ahrtr mentioned this pull request Feb 12, 2022

Leases wait for entries to be applied #13690

Merged

ahrtr mentioned this pull request Apr 9, 2022

Make leases linearizable (excluding TTL) #13915

Open

		@@ -565,6 +565,8 @@ func (s *EtcdServer) RaftHandler() http.Handler { return s.r.transport.Handler()

		func (s *EtcdServer) Lessor() lease.Lessor { return s.lessor }

		func (s *EtcdServer) ApplyWait() wait.WaitTime { return s.applyWait }

lease: force leader to apply its pending committed index for lease op… #7015

lease: force leader to apply its pending committed index for lease op… #7015

Conversation

fanminshi commented Dec 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanminshi Dec 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heyitsanthony Dec 15, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanminshi commented Dec 16, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanminshi commented Dec 16, 2016

fanminshi commented Dec 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fanminshi commented Dec 27, 2016

xiang90 commented Dec 27, 2016

heyitsanthony commented Dec 27, 2016

fanminshi Dec 15, 2016 •

edited

Loading

heyitsanthony Dec 15, 2016 •

edited

Loading