Gracefully terminate an operator on SIGINT/SIGTERM

[nolar]

Part of an effort to make Kopf-based operators more stable and resilient.

Issue: #142, partially replaces #143.

Description

First of all, react to SIGINT/SIGTERM at all. Previously, the process was terminated on the OS level, but not via the intended reaction.

Second, when exiting for any reason — e.g. SIGINT/SIGTERM, or explicit cancellation, or any exception — do this as gracefully as possible. For this, first cancel the core ever-running tasks of the framework; give them some time to finish; then cancel all running tasks & sub-tasks if left; give it some time again; and only then actually exit.

Third, add a check for all tests on whether any pending tasks are left unattended (i.e. scheduled but not executed) — which is especially important for the e2e tests.

There are no behaviour changes except of a better termination.

Types of Changes

• Bug fix (non-breaking change which fixes an issue)
• Refactor/improvements

[zincr]

🤖 zincr found 0 problems , 0 warnings

✅ Large Commits
✅ Approvals
✅ Specification
✅ Dependency Licensing

[codecov-io]

Codecov Report

Merging #147 into master will decrease coverage by 0.98%.
The diff coverage is 21.62%.

@@            Coverage Diff             @@
##           master     #147      +/-   ##
==========================================
- Coverage   83.55%   82.56%   -0.99%     
==========================================
  Files          30       30              
  Lines        1545     1572      +27     
  Branches      227      229       +2     
==========================================
+ Hits         1291     1298       +7     
- Misses        209      229      +20     
  Partials       45       45

Impacted Files	Coverage Δ
kopf/reactor/queueing.py	56% <21.62%> (-8.29%)	⬇️