Reinforce the watchers's shields from cancellations #628
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The shielding has been introduced in e0ae2ea for #147 for general stability of tests and for graceful termination of operators.
However, the shielding was only done for one operation in the
finally:
block — the scheduler closing — and that shield could itself be cancelled.When the watcher was double-cancelled (with the second cancellation hitting somewhere in the
finally:
block), it left several tasks running in the background:asyncio.shield()
).At that time, the operators/tests were already going forward and not going to wait for these tasks to finish. This led to system resource (asyncio tasks) leakage in some tests; it was also probable in operators (though, they do not cancel the watchers often).
With this change, the anti-cancellation shield is reinforced to:
This ensures that under no circumstances any worker is left unattended after the watcher exits — all workers are now guaranteed to finish or be cancelled as part of the watcher's cancellation/finalisation.