Commits on Jan 2, 2021

1. Reinforce the watchers's shields from cancellations …

   The shielding has been introduced in e0ae2ea for #147 for general stability of tests and for graceful termination of operators.
   
   However, the shielding was only done for one operation in the `finally:` block — the scheduler closing — and that shield could itself be cancelled.
   
   When the watcher was double-cancelled (with the second cancellation hitting somewhere in the `finally:` block), it left several tasks running in the background:
   
   * The cancellation task itself (implicitly converted to a future/task by `asyncio.shield()`).
   * The scheduler's internal job closing tasks.
   * The scheduler's task for processing the job failures (implementation details).
   
   At that time, the operators/tests were already going forward and not going to wait for these tasks to finish. This led to system resource (asyncio tasks) leakage in some tests; it was also probable in operators (though, they do not cancel the watchers often).
   
   With this change, the anti-cancellation shield is reinforced to:
   
   1. Shield the streams depletion (limited by an "exit timeout") too; previously, it was ignored and remained cancellable without getting to the scheduler closing.
   2. Wait until the real completion of both the streams depletion and the scheduler closure; previously, the wait could be cancelled.
   
   This ensures that under no circumstances any worker is left unattended after the watcher exits — all workers are now guaranteed to finish or be cancelled as part of the watcher's cancellation/finalisation.

   

   nolar committed Jan 2, 2021
   Configuration menu

   • View commit details
   • Copy full SHA for 142c414
   • Browse repository at this point

   Copy the full SHA

   142c414 View commit details

   Browse the repository at this point in the history