Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fail in-progress jobs when the worker running them exits abnormally #277

Merged
merged 5 commits into from
Aug 21, 2024
Merged

Fail in-progress jobs when the worker running them exits abnormally #277

merged 5 commits into from
Aug 21, 2024

Commits on Aug 21, 2024

  1. Add new column name to processes 

    So we can uniquely identify processes by supervisor and name, without
having to rely on the PID, that can be duplicated across processes.
    @rosa
    rosa committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    84cb6e4 View commit details
    Browse the repository at this point in the history

  2. Create always new Worker and Dispatcher instances before starting 

    We were reusing the instances of Worker and Dispatcher from the initial
configuration all the time, which could bring some problems with stopped
pools. Now that we need a name to be generated and be unique per process
instance, we really need to instantiate new processes every time they're
started.
    @rosa
    rosa committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    3e78fb2 View commit details
    Browse the repository at this point in the history

  3. Remove process's name from metadata and add it to instrumentation events

    @rosa
    rosa committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    cb5669f View commit details
    Browse the repository at this point in the history

  4. Fail in-progress jobs when the worker running them exits abnormally 

    This applies to:
- Killed workers that the supervisor detects as dead.
- Reaped workers without a clear exit status.
- Orphaned executions that somehow lost their worker.
- Workers whose heartbeat expired.
    @rosa
    rosa committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    9fb89f1 View commit details
    Browse the repository at this point in the history

  5. Split processes' name migration into two 

    As it won't be possible to start new processes after the column
is made NOT NULL and before deploying the code that uses that column.
    @rosa
    rosa committed Aug 21, 2024
    Configuration menu
    Copy the full SHA
    76d2c0f View commit details
    Browse the repository at this point in the history