How do I prevent automerge from merging PRs when required checks are skipped due to needs: dependencies failing?

[JasonGross]

In JasonGross/coq-tools#130, automerge merged my PR into master despite the fact that check-all-docker and check-all are required to succeed by branch protection and did not succeed on this branch. If I'm requiring a check to succeed, then all of its transitive dependencies should also be required to succeed. It should not be possible to bypass branch protection by failing one of the transitive dependencies of a job that is required to succeed.

[jesseditson]

The way I dealt with this (which is convoluted, it would be nice if GH had a more official solution):

• Create a required_checks job that generates a matrix from a needs list.
• Have the required_checks job fail if the job it is for is not successful.
• Run this job if: ${{ always() }}

At this point, there will be one failure per failed OR skipped (or any other than success) status job in the needs list of required_checks.

However, unless you want to also keep your github required jobs up to date in the UI, you'll need one more job, which has the required_checks job as a needs, and does the same thing, also if: ${{ always() }}, but without the matrix and with a hardcoded dependency on required_checks. That way you can configure this as the only required check in the github UI.

Example config:

name: CI

on:
  push:
    branches:
      - main

jobs:
  job_one:
    strategy:
      matrix:
        index: ["0", "1", "2"]
    name: Test ${{ matrix.index }}
    runs-on: ubuntu-latest
    steps:
      - name: Do Stuff
        run: echo "example"
  job_two:
    name: Job Two
    runs-on: ubuntu-latest
    steps:
      - name: More Stuff
        run: echo "more"

  required_checks:
    if: ${{ always() }}
    name: Verify Required Checks
    needs: [job_one, job_two]
    strategy:
      matrix:
        required: [job_one, job_two]
    runs-on: ubuntu-latest
    steps:
      - name: Fail unless ${{matrix.required}} pass
        if: ${{ needs[matrix.required].result != 'success' }}
        run: exit 1

  all_passed:
      if: ${{ always() }}
      needs: [required_checks]
      name: All Checks Passed.
      runs-on: ubuntu-latest
      steps:
        - name: Fail unless required checks pass
          if: ${{ needs.required_checks.result != 'success' }}
          run: exit 1
        - run: 'echo "✅ All required jobs have passed!"'