Research: ./wpt test-jobs and ./wpt affected-tests stats

[foolip]

Travis does not support making decisions in one job that affects another. Instead we start 10 jobs (example) and exit early in each if there's nothing to do. Both Azure Pipelines and Taskcluster supports making decisions, with build-wide variables and decision tasks respectively.

This issue is about how to set up such dependent jobs. Considerations:

• Starting a VM and cloning the repo takes time, ~30s on Azure Pipelines.
• Avoid waste by starting jobs that end up doing nothing.
• Keep overall latency low. (In tension with above, because with enough parallelism speculatively starting jobs lowers latency.)
• Keep latency to test results in particular low, since those will feed into @lukebjerring's wpt.fyi check that combines results from CIs.

To inform the decision, I've looked into what ./wpt test-jobs and ./wpt affected-tests has returned in recent history, see script at end.

For ./wpt test-jobs (fast) I compared 1410 merge_pr_* to their preceding tags as an approximation of what would happen on the PRs, and got these numbers of "PRs" triggering each job:

   1410 manifest_upload (100%)
   1410 lint (100%)
   1151 stability (82%)
    317 build_css (22%)
    227 resources_unittest (16%)
    192 wptrunner_infrastructure (14%)
    185 wpt_integration (13%)
    185 tools_unittest (13%)
    181 update_built (13%)
     97 wptrunner_unittest (7%)

For ./wpt test-jobs (fast) I looked at 243 merge_pr_* tags. Of the 190 that would triggered the stability job, I made a spreadsheet, and it's clear (and unsurprising) that most PRs don't change many tests. The 50th percentile is 1, and the 90th percentile is 12. Only 6 "PRs" affected >100 tests.

(Among the 53 that didn't trigger the stability job, 3 still had affected tests. Will file bug.)

Putting this together, conclusions are:

• Lint always runs, so whatever job it's in should be unconditional. (No change.)
• Prioritize running affected tests, since there will usually (82%) be some, and getting those results fast is very useful.
• The rest are rare enough that the best trade-off is probably to spin up parallel job for them from the "decision task".

shell script used

#!/bin/bash

mkdir -p ../tests-affected
prevtag=""
# manifest version bumped in merge_pr_12563
# interfaces/*.idl affected tests logic fixed in merge_pr_13392
git tag --list --contains=merge_pr_13392 --sort=committerdate | while read tag; do
    if [[ -z "$prevtag" ]]; then
        prevtag=$tag
        continue
    fi

    echo "Listing $tag tests-affected"
    git checkout -q $tag
    ./wpt manifest
    ./wpt tests-affected $prevtag > ../tests-affected/$tag.txt
    prevtag=$tag
done

mkdir -p ../test-jobs
prevtag=""
# tools/ci/jobs.py changed in merge_pr_12174
git tag --list --contains=merge_pr_12174 --sort=committerdate | while read tag; do
    if [[ -z "$prevtag" ]]; then
        prevtag=$tag
        continue
    fi

    echo "Listing $tag test-jobs"
    git checkout -q $tag
    ./wpt test-jobs $prevtag > ../test-jobs/$tag.txt
    prevtag=$tag
done

Possibly interested parties: @jugglinmike and @web-platform-tests/admins

[foolip]

The research has been done, and I've used my own recommendation in #14156 and #14459, closing.