From c7967b194b41cb16907eed718b78d89120089f6a Mon Sep 17 00:00:00 2001 From: Sebastian Silbermann Date: Mon, 20 Feb 2023 22:16:23 +0100 Subject: [PATCH] Distribute bundles more evenly into CI shards (#26208) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary Previously, we distributed bundles into shards and then checked if we need to actually build that bundle. This can under-utilize shards heavily (e.g. https://app.circleci.com/pipelines/github/facebook/react/38611/workflows/df9e56e7-d476-49ee-9392-d8b37c81aa66/jobs/630545/parallel-runs/28?filterBy=ALL only building a single bundle). This won't result in an optimal distribution but, if we're lucky, we might end up with shard #26 not taking 7mins anymore. The slowest shard ultimately decicdes when we can start with testing builds. ## How did you test this change? - [x] `CIRCLE_NODE_INDEX=28 CIRCLE_NODE_TOTAL=40 yarn build` building more than 1 bundle - [x] Check timings of `yarn_build` so that we don't up with an over-stuffed shard (e.g. a shard having to build all the expensive bundles). Casually dropping 60min idle time 🎉: - Before: https://app.circleci.com/pipelines/github/facebook/react/38683/workflows/a41533d7-811c-439d-9751-214ba06035c5/jobs/632230/timing - After: https://app.circleci.com/pipelines/github/facebook/react/38686/workflows/8a770df6-5b3e-41ea-b3b5-10abeae703e7/jobs/632247/timing --- scripts/rollup/build.js | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/scripts/rollup/build.js b/scripts/rollup/build.js index f95fbe12ac398..4c8ed5d9e4dff 100644 --- a/scripts/rollup/build.js +++ b/scripts/rollup/build.js @@ -527,10 +527,6 @@ function resolveEntryFork(resolvedEntry, isFBBundle) { } async function createBundle(bundle, bundleType) { - if (shouldSkipBundle(bundle, bundleType)) { - return; - } - const filename = getFilename(bundle, bundleType); const logKey = chalk.white.bold(filename) + chalk.dim(` (${bundleType.toLowerCase()})`); @@ -766,6 +762,10 @@ async function buildEverything() { ); } + bundles = bundles.filter(([bundle, bundleType]) => { + return !shouldSkipBundle(bundle, bundleType); + }); + if (process.env.CIRCLE_NODE_TOTAL) { // In CI, parallelize bundles across multiple tasks. const nodeTotal = parseInt(process.env.CIRCLE_NODE_TOTAL, 10);