Proposal for Optimizing Parallel CI/CD Jobs

[JoSeBu1]

Issue Summary:

Given the current setup where multiple parallel jobs are running during PR checks, there's a notable observation that concurrent PRs can lead to longer PR check times due to job overlaps.

Problem talked here: #2751 (comment)

Proposed Solution:

Sequential Execution of Short-lived CI Jobs: Consider running some short-lived CI jobs sequentially instead of in parallel, especially during PR checks, to optimize resource utilization and mitigate job overlaps. This would effectively maintain the same total execution time while potentially freeing up resources for other PRs.

By addressing these points, we aim to streamline the CI/CD pipeline, reduce PR check times, optimize resource allocation, ultimately enhancing the development workflow for all contributors.

[Boshen]

I'll look into reducing the benchmarks first. It's always tradeoffs.

[Boshen]

I removed 3 linter benchmarks. I'll optimize these jobs if queueing jobs becomes a bottleneck in the future. Closing for now.