feat(tasks): shard linter benchmarks in CI

[overlookmotel]

Follow-on from #2751. Further shards linter benchmarks so each fixture runs in its own job.

This reduces total time to run benchmarks by another ~75 secs. So approx 2.5 mins shaved off in total.

[overlookmotel]

• feat(tasks): shard linter benchmarks in CI #2752 👈
• feat(tasks): shard benchmarks in CI #2751
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @overlookmotel and the rest of your teammates on Graphite

[codspeed-hq]

CodSpeed Performance Report

Merging #2752 will not alter performance

_{Comparing 03-18-feat_tasks_shard_linter_benchmarks_in_CI (6126b9f) with main (93e1ea4)}

Summary

✅ 29 untouched benchmarks

[overlookmotel]

What's taking most of the time is building the benchmarks, not running them.

@Boshen Is there any way to get Github Actions to cache the builds, so e.g. if a PR only makes changes to oxc_transformer, then the oxc_linter benchmarks don't need to be re-built? It feels like this should be possible, but I haven't got a clue how. Do you?

[Boshen]

Merge activity

• Mar 17, 10:45 PM EDT: @Boshen started a stack merge that includes this pull request via Graphite.
• Mar 17, 10:46 PM EDT: Graphite rebased this pull request as part of a merge.
• Mar 17, 10:50 PM EDT: @Boshen merged this pull request with Graphite.

[Boshen]

What's taking most of the time is building the benchmarks, not running them.

@Boshen Is there any way to get Github Actions to cache the builds, so e.g. if a PR only makes changes to oxc_transformer, then the oxc_linter benchmarks don't need to be re-built? It feels like this should be possible, but I haven't got a clue how. Do you?

Feels weird that argo build --release -p oxc_benchmark --features codspeed --bench linter builds all the crates, may need to look into it in the future.

[overlookmotel]

@Boshen oxc_benchmark does have dependencies on all the crates, so I guess they need to be built, even though they'll be "tree shaken" out during linking.

FYI, I did try out making sure only the crates which are needed for each benchmark get compiled, by disabling the rest with optional features:

overlookmotel@e753b6b

But this made no real difference to the build time. Almost all the time is spent building oxc_benchmark itself.