Skip to content

Commit

Permalink
embed: wait up to 10s for pending RPCs when closing
Browse files Browse the repository at this point in the history
Both grpc.Server.Stop and grpc.Server.GracefulStop close the listeners
first, to stop accepting the new connections. GracefulStop blocks until
all clients close their open transports(connections), when unary RPCs
only take a few seconds to finish. Stream RPCs like watch might never
close the connections in client side, thus making gRPC server wait
forever.

This patch still calls GracefulStop, but waits up to 10s before manually
closing the open transports.

Address #8224.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
  • Loading branch information
gyuho committed Jul 14, 2017
1 parent 511f4d5 commit 08f55eb
Showing 1 changed file with 17 additions and 3 deletions.
20 changes: 17 additions & 3 deletions embed/etcd.go
Original file line number Diff line number Diff line change
Expand Up @@ -189,11 +189,25 @@ func (e *Etcd) Config() Config {
func (e *Etcd) Close() {
e.closeOnce.Do(func() { close(e.stopc) })

// (gRPC server) stops accepting new connections,
// RPCs, and blocks until all pending RPCs are finished
for _, sctx := range e.sctxs {
for gs := range sctx.grpcServerC {
gs.GracefulStop()
ch := make(chan struct{})
go func() {
defer close(ch)
// close listeners to stop accepting new connections,
// will block on any existing transports
gs.GracefulStop()
}()
// wait until all pending RPCs are finished
select {
case <-ch:
case <-time.After(10 * time.Second):
// took too long, manually close open transports
// e.g. watch streams
gs.Stop()
// concurrent GracefulStop should be interrupted
<-ch
}
}
}

Expand Down

0 comments on commit 08f55eb

Please sign in to comment.