Pause JobProcess
when transport task falls through exponential backoff
#1903
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1835
All transport tasks for the
JobProcess
are wrapped in the exponentialbackoff retry coroutine utility, which when an exception occurs during
the transport task, will reschedule the task with an exponential backoff.
However, the backoff has a maximum number of retries, which when hit would
bubble up the exception and causing the process to except. With the new
pausing functionality in place, instead we can catch the
TransportTaskException
and pause the process. The user then has the chanceto investigate the logs to determine the problem. If the problem was just
of a temporary nature, the user can then resume the process. If instead
the failure was of unrecoverable nature, the user can always decide to kill
the process.