Distribute bundles more evenly into CI shards (#26208)

eps1lon · web-flow · commit c7967b194b41 · 2023-02-20T22:16:23.000+01:00
## Summary Previously, we distributed bundles into shards and then checked if we need to actually build that bundle. This can under-utilize shards heavily (e.g. https://app.circleci.com/pipelines/github/facebook/react/38611/workflows/df9e56e7-d476-49ee-9392-d8b37c81aa66/jobs/630545/parallel-runs/28?filterBy=ALL only building a single bundle). This won't result in an optimal distribution but, if we're lucky, we might end up with shard #26 not taking 7mins anymore. The slowest shard ultimately decicdes when we can start with testing builds. ## How did you test this change? - [x] `CIRCLE_NODE_INDEX=28 CIRCLE_NODE_TOTAL=40 yarn build` building more than 1 bundle - [x] Check timings of `yarn_build` so that we don't up with an over-stuffed shard (e.g. a shard having to build all the expensive bundles). Casually dropping 60min idle time 🎉: - Before: https://app.circleci.com/pipelines/github/facebook/react/38683/workflows/a41533d7-811c-439d-9751-214ba06035c5/jobs/632230/timing - After: https://app.circleci.com/pipelines/github/facebook/react/38686/workflows/8a770df6-5b3e-41ea-b3b5-10abeae703e7/jobs/632247/timing
diff --git a/scripts/rollup/build.js b/scripts/rollup/build.js
@@ -527,10 +527,6 @@ function resolveEntryFork(resolvedEntry, isFBBundle) {
 }
 
 async function createBundle(bundle, bundleType) {
-  if (shouldSkipBundle(bundle, bundleType)) {
-    return;
-  }
-
   const filename = getFilename(bundle, bundleType);
   const logKey =
     chalk.white.bold(filename) + chalk.dim(` (${bundleType.toLowerCase()})`);
@@ -766,6 +762,10 @@ async function buildEverything() {
     );
   }
 
+  bundles = bundles.filter(([bundle, bundleType]) => {
+    return !shouldSkipBundle(bundle, bundleType);
+  });
+
   if (process.env.CIRCLE_NODE_TOTAL) {
     // In CI, parallelize bundles across multiple tasks.
     const nodeTotal = parseInt(process.env.CIRCLE_NODE_TOTAL, 10);
