alls-green

A check for whether the dependency jobs are all green.

Why?

Do you have more than one job in your GitHub Actions CI/CD workflows setup? Do you use branch protection? Are you annoyed that you have to manually update the required checks in the repository settings hoping that you don't forget something on each improvement of the test matrix structure?

Yeah.. me too! But there's a solution — you can make a single check job listing all of the other jobs in its needs list. How about that? 🤯 Now you can add only that single job to your branch protection settings and you won't have to maintain that list manually anymore.

Right..? Wrong 🙁 — apparently, in case when something fails, the check job's result is set to skipped and not failed as one would expect. This is problematic and requires extra work to get it right. Some of it is iteration over the needed jobs and checking that all their results are set to success but it's no fun to maintain this sort of thing in many different repositories. Also, it is mandatory to make the check job run always, not only when all of the needed jobs succeed.

This is why I decided to make this action to simplify the maintenance.

--@webknjaz

:wq

Usage

To use the action add a check job to your workflow file (e.g. .github/workflows/ci-cd.yml) following the example below:

---
on:
  pull_request:
  push:

jobs:
  build:
    ...

  docs:
    ...

  linters:
    ...

  package-linters:
    needs:
    - build
    ...

  tests:
    needs:
    - build
    ...

  check:
    if: always()

    needs:
    - docs
    - linters
    - package-linters
    - tests

    runs-on: Ubuntu-latest

    steps:
    - name: Decide whether the needed jobs succeeded or failed
      uses: re-actors/alls-green@release/v1
      with:
        allowed-failures: docs, linters
        allowed-skips: non-voting-flaky-job
        jobs: ${{ toJSON(needs) }}
...

Options

There are three options — allowed-failures, allowed-skips and jobs. The first two are optional but jobs is mandatory. allowed-failures tells the action which jobs should not affect the outcome if they don't succeed, by default all the jobs will be "voting". Same goes for allowed-skips — it won't allow the listed jobs to affect the outcome if they are skipped but are still "voting" in case they run. jobs is an object representing the jobs that should affect the decision of whether the pipeline failed or not, it is important to pass a JSON-serialized needs context to this argument.

Important: For this to work properly, it is a must to have the job always run, otherwise GitHub will make it skipped when any of the dependencies fail. In some contexts, skipped is interpreted as success which may lead to undersired, unobvious and even dangerous (as in security breach "dangerous") side-effects.

Outputs

failure

Whether this check decided that the job matrix failed.

result

Failure or success result of the job matrix.

success

Whether this check decided that the job matrix succeeded.

Gotchas

An attentive reader may have noticed that there is no clear way to allow failures for specific job generated by matrixes in a simple manner, through action inputs. This is due to the fact that those individual matrix job statuses are not exposed by the GitHub platform.

Although, there is a solution — use a continue-on-error job-level setting with a dynamic matrix-dependent value. In this case, the matching jobs will not influence the resulting outcome of the whole matrix and this action will "see" that as a success (provided that the jobs that are not allowed to fail succeed, of course).

Here's a simplified example of what testing against an unstable Python 3.11 release that is allowed to fail might look like:

---

...  # Some sections have been removed from the example to simplify it

jobs:
  tests:
    runs-on: ubuntu-latest

    matrix:
       python-version:
       - >-
         3.10
       - ~3.11.0-0

    continue-on-error: >-
      ${{ contains(matrix.python-version, '~') && true || false }}

    steps:
    - ...

  check:
    if: always()

    needs:
    - tests

    runs-on: ubuntu-latest

    steps:
    - name: Decide whether the needed jobs succeeded or failed
      uses: re-actors/alls-green@release/v1
      with:
        jobs: ${{ toJSON(needs) }}

...

Does anybody actually use this?

We've seen it being integrated into some projects of @aio-libs (notably, aiohttp), Ansible Collections Community, attrs, @CherryPy, conda, coveragepy, Open edX, pip-tools, setuptools, structlog, spaceship-prompt, Towncrier, @PyCA, @PyPA and @pytest ecosystems. And here's more: https://github.com/re-actors/alls-green/network/dependents.

Whose idea is this?

My inspiration came from Zuul — a project gating system that practices having one "check-gate" that depends on multiple individual checks. Later when I started implementing this practice across the projects that I maintain, I discovered that I was not the only one who got the same idea @graingert posted a similar solution on the GitHub Community Forum. At some point I noticed that GitHub's auto-merge merges Dependabot PRs that are broken despite having branch protection enabled in the aiohttp repository, it wasn't obvious why. I stumbled on the solution accidentally, it was implemented in the PyCA/cryptography repository — the credit for that goes to @reaperhulk. He listed all of the needed jobs manually in the check. With those findings I've spent some time on experimentation and composing a more generic solution — this is how this GitHub Action came to be.

License

The contents of this project is released under the BSD 3-clause license.