Gracefully shutdown when receiving SIGTERM

[aylei]

Feature Request

Is your feature request related to a problem? Please describe:

tidb-server listen for SIGQUIT to perform gracefully shutdown, however, most of the process managers send SIGTERM by default and expect the process exits gracefully, e.g. systemd, kubernetes.

#8711 introduce a gracefully shutdown with timeout for all signals, however, that did not guarantee all the ongoing requests could get finished.

Describe the feature you'd like:

Gracefully shutdown when tidb-server receiving SIGTERM.

Describe alternatives you've considered:

Making tryGracefullyShutdownTimeout configurable, which could satisfy most of the scenarios. However, a better default behavior is waiting for shutting down gracefully forever if request keeps processing and leave the decision to operators or automation systems to shutdown the server forcefully, e.g. sending SIGKILL. So I propose to change the default behavior of receiving SIGTERM, which should wait forever to be graceful by default. A configurable timeout is useful but is non goal of this issue.

Sending SIGQUIT in process manager, which is possible but requires customized work on different platforms, e.g. ansible and kubernetes.

Teachability, Documentation, Adoption, Migration Strategy:

This changes a long standing behavior, so though the fix itself is tiny, we should target it in the next major release.

ref: pingcap/tidb-ansible#506 #8711

[ekalinin]

BTW, it seems like it's very easy to implement.
Just add a new signal in two places:

• tidb/util/signal/signal_posix.go

  Line 54 in 3ca8661

  shudownFunc(sig == syscall.SIGQUIT)

• tidb/util/signal/signal_windows.go

  Line 39 in 3ca8661

  shudownFunc(sig == syscall.SIGQUIT)

[aylei]

@ekalinin Exactly.

And I am still investigating whether tidb-ansible and tidb-operator need to be modified accordingly.