From 7ce11474dba9f1bd7218e9033995c9b91d455bb1 Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Wed, 29 Jul 2026 17:59:02 -0700 Subject: [PATCH 1/4] =?UTF-8?q?exp(ci):=20run=20A=20=E2=80=94=20cold=20sti?= =?UTF-8?q?cky=20disk,=20Turbopack=20FS=20build=20cache=20ON?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Do not merge. Adds --force so Turbo cannot replay a cached log and reprint a stale compile time, which a .github-only commit would otherwise trigger. --- .github/workflows/test-build.yml | 8 ++++- EXPERIMENT.md | 60 ++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 1 deletion(-) create mode 100644 EXPERIMENT.md diff --git a/.github/workflows/test-build.yml b/.github/workflows/test-build.yml index 192337458aa..b5561b07385 100644 --- a/.github/workflows/test-build.yml +++ b/.github/workflows/test-build.yml @@ -322,8 +322,14 @@ jobs: # build only. #5869's 105s-cold/22s-warm was measured locally and has # never reproduced in CI (compile has ranged 3.5-17.8 min) — local # numbers, not a CI target. + # EXPERIMENT VARIABLE — flipped across runs on this branch. NEXT_TURBOPACK_BUILD_CACHE: '1' - run: bunx turbo run build --filter=sim + # EXPERIMENT ONLY, never merge: `--force` bypasses Turbo's task cache. + # Without it a commit touching only .github/** leaves Turbo's task + # inputs unchanged, so Turbo replays a cached log and reprints a STALE + # "Compiled successfully in Xmin" from an unrelated run. That is not a + # build, and it silently fabricates the exact number being measured. + run: bunx turbo run build --filter=sim --force - name: Report Next.js cache size (post-build) if: always() diff --git a/EXPERIMENT.md b/EXPERIMENT.md new file mode 100644 index 00000000000..4dfb01b3a97 --- /dev/null +++ b/EXPERIMENT.md @@ -0,0 +1,60 @@ +# Experiment: is the Turbopack FS build cache a net win? + +**Do not merge.** This branch exists to produce three measurements and then be closed. + +## Question + +`NEXT_TURBOPACK_BUILD_CACHE=1` was added in #5869 on the premise that a warm Turbopack +filesystem cache makes `next build` faster. Observational data from 2026-07-29 suggests the +opposite: + +| cache state | pre-build disk | compile | +|---|---|---| +| genuinely cold, FS cache on | 4.0K | **1.8 min** | +| perfect-warm, FS cache on (written 50s earlier) | 5.1G | **4.0 min** | +| mismatched-warm, FS cache on (old event-only key) | — | **11.3 min** | +| FS cache absent entirely (13 runs before #5869) | n/a | **2.2–3.6 min, median 2.7** | + +Cold < perfect-warm < mismatched-warm. If that ordering is real, the cache is costing wall +time and the correct action is to turn it off. + +All four rows are observational, from runs that differed in commit as well as cache state. +This branch holds the tree constant and varies only the cache. + +## Method + +Three sequential runs on this one branch. Each commit touches **only** `.github/**` + +this file, so the Next module graph is byte-identical across all three — cache-match +quality is held constant and only cache *state* varies. + +| run | commit | `NEXT_TURBOPACK_BUILD_CACHE` | expected disk state | +|---|---|---|---| +| A | first push | `'1'` | **cold** (brand-new per-branch sticky key) | +| B | trivial edit | `'1'` | **warm**, written by A, identical module graph | +| C | flip the env | `'0'` | irrelevant — Turbopack ignores the on-disk cache | + +Read from each Build App log: `Report Next.js cache size (pre-build)`, +`Compiled successfully in X`, `Report Next.js cache size (post-build)`. + +## Two traps this design defuses + +1. **Turbo task-cache replay.** A `.github`-only commit leaves Turbo's task inputs + unchanged, so Turbo would replay a cached log and reprint a *stale* compile time from an + unrelated run — fabricating the number being measured. `--force` makes every run a real + build. Five such phantom runs had to be discarded from the observational data. +2. **Cancelled-run confusion.** `ci.yml` sets `cancel-in-progress` for pull_request events, + so pushing before a run finishes cancels it — and a cancelled run still writes its + partial cache to the sticky disk. A prior conclusion ("a brand-new sticky key is not + cold") was wrong for exactly this reason: the "first" run had a cancelled sibling 3 + minutes earlier that had already committed 5.1 GB. **Each run here must reach + `completed` before the next push.** + +## Result + +Filled in as runs land. + +| run | pre-build | compile | post-build | run id | +|---|---|---|---|---| +| A — cold, cache on | | | | | +| B — warm, cache on | | | | | +| C — cache off | | | | | From cc30aff92aba17775b03d689e66e8382f40ba6cf Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Wed, 29 Jul 2026 18:04:49 -0700 Subject: [PATCH 2/4] =?UTF-8?q?exp(ci):=20run=20B=20=E2=80=94=20warm=20sti?= =?UTF-8?q?cky=20disk=20(5.1G=20from=20run=20A),=20FS=20cache=20still=20ON?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Module graph byte-identical to run A; only the disk state differs. --- EXPERIMENT.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/EXPERIMENT.md b/EXPERIMENT.md index 4dfb01b3a97..984af10e71b 100644 --- a/EXPERIMENT.md +++ b/EXPERIMENT.md @@ -53,8 +53,14 @@ Read from each Build App log: `Report Next.js cache size (pre-build)`, Filled in as runs land. -| run | pre-build | compile | post-build | run id | -|---|---|---|---|---| -| A — cold, cache on | | | | | -| B — warm, cache on | | | | | -| C — cache off | | | | | +| run | pre-build | compile | post-build | job total | run id | +|---|---|---|---|---|---| +| A — cold, cache on | **4.0K** | **2.7 min** | 5.1G | 3m54s | 30504331243 | +| B — warm, cache on | | | | | | +| C — cache off | | | | | | + +Run A notes: single CI run on the branch, no cancelled sibling, `--force` confirmed in the +log so this was a real build and not a Turbo replay. 4.0K pre-build settles a separate +question — a brand-new sticky-disk key is **cold**; Blacksmith does not hydrate it. 2.7 min +also lands exactly on the pre-#5869 no-FS-cache median (2.7 min), which is the first hint +that the cache buys nothing on a cold run. From 5746314530270ccbd0330ed3b39c583e088d3f7f Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Wed, 29 Jul 2026 18:16:17 -0700 Subject: [PATCH 3/4] =?UTF-8?q?exp(ci):=20run=20C=20=E2=80=94=20FS=20build?= =?UTF-8?q?=20cache=20OFF?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit NEXT_TURBOPACK_BUILD_CACHE '1' -> '0'. Module graph still byte-identical to runs A and B; the 12G on the sticky disk is now ignored by Turbopack. Records run B: warm was 2.2x SLOWER than cold and the cache grew 5.1G -> 12G. --- .github/workflows/test-build.yml | 2 +- EXPERIMENT.md | 10 +++++++++- 2 files changed, 10 insertions(+), 2 deletions(-) diff --git a/.github/workflows/test-build.yml b/.github/workflows/test-build.yml index b5561b07385..a943a8ecf69 100644 --- a/.github/workflows/test-build.yml +++ b/.github/workflows/test-build.yml @@ -323,7 +323,7 @@ jobs: # never reproduced in CI (compile has ranged 3.5-17.8 min) — local # numbers, not a CI target. # EXPERIMENT VARIABLE — flipped across runs on this branch. - NEXT_TURBOPACK_BUILD_CACHE: '1' + NEXT_TURBOPACK_BUILD_CACHE: '0' # EXPERIMENT ONLY, never merge: `--force` bypasses Turbo's task cache. # Without it a commit touching only .github/** leaves Turbo's task # inputs unchanged, so Turbo replays a cached log and reprints a STALE diff --git a/EXPERIMENT.md b/EXPERIMENT.md index 984af10e71b..9584d469110 100644 --- a/EXPERIMENT.md +++ b/EXPERIMENT.md @@ -56,7 +56,7 @@ Filled in as runs land. | run | pre-build | compile | post-build | job total | run id | |---|---|---|---|---|---| | A — cold, cache on | **4.0K** | **2.7 min** | 5.1G | 3m54s | 30504331243 | -| B — warm, cache on | | | | | | +| B — warm, cache on | **5.1G** | **6.0 min** | **12G** | 8m18s | 30504621611 | | C — cache off | | | | | | Run A notes: single CI run on the branch, no cancelled sibling, `--force` confirmed in the @@ -64,3 +64,11 @@ log so this was a real build and not a Turbo replay. 4.0K pre-build settles a se question — a brand-new sticky-disk key is **cold**; Blacksmith does not hydrate it. 2.7 min also lands exactly on the pre-#5869 no-FS-cache median (2.7 min), which is the first hint that the cache buys nothing on a cold run. + +Run B notes: pre-build 5.1G written by run A, module graph byte-identical, `--force` again +confirmed. Compile **2.2x slower than cold** (6.0 vs 2.7 min); job 8m18s vs 3m54s. The cache +also **grew 5.1G -> 12G on an identical tree**, which is the likely mechanism behind the +progressive degradation seen on older shared disks (up to 11.7 min): the more a disk is +written, the more there is to read and revalidate. Run B's CI run shows `failure`, but Build +App passed - the failing job was `Lint and Test` on a 1ms real-clock flake in +`pi-lifetime.test.ts` (`expected 5399999 to be 5400000`), unrelated to this experiment. From 0439fcb4bfad79382475dfc898d2405feecae1cf Mon Sep 17 00:00:00 2001 From: Waleed Latif Date: Wed, 29 Jul 2026 18:23:04 -0700 Subject: [PATCH 4/4] =?UTF-8?q?exp(ci):=20record=20run=20C=20=E2=80=94=20c?= =?UTF-8?q?ache=20OFF=20is=20fastest=20(113s=20vs=20162s=20cold,=20360s=20?= =?UTF-8?q?warm)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- EXPERIMENT.md | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/EXPERIMENT.md b/EXPERIMENT.md index 9584d469110..489aa44d270 100644 --- a/EXPERIMENT.md +++ b/EXPERIMENT.md @@ -57,7 +57,7 @@ Filled in as runs land. |---|---|---|---|---|---| | A — cold, cache on | **4.0K** | **2.7 min** | 5.1G | 3m54s | 30504331243 | | B — warm, cache on | **5.1G** | **6.0 min** | **12G** | 8m18s | 30504621611 | -| C — cache off | | | | | | +| C — cache off | 5.1G (ignored) | **113s = 1.9 min** | 5.1G | **2m53s** | 30505173423 | Run A notes: single CI run on the branch, no cancelled sibling, `--force` confirmed in the log so this was a real build and not a Turbo replay. 4.0K pre-build settles a separate @@ -72,3 +72,28 @@ progressive degradation seen on older shared disks (up to 11.7 min): the more a written, the more there is to read and revalidate. Run B's CI run shows `failure`, but Build App passed - the failing job was `Lint and Test` on a 1ms real-clock flake in `pi-lifetime.test.ts` (`expected 5399999 to be 5400000`), unrelated to this experiment. + +Run C notes: flag manipulation verified in the log — run A prints +`✓ turbopackFileSystemCacheForBuild`, run C omits it entirely. Fastest of the three: +**3.2x faster than warm** (113s vs 360s) and still 1.4x faster than cold-with-cache, because +it neither reads nor writes the 5.1G. + +## Conclusion + +The Turbopack FS build cache is a net negative for this app at its current size. Ranked: + +| config | compile | Build App job | +|---|---|---| +| cache OFF | **113s** | **2m53s** | +| cache ON, cold | 162s | 3m54s | +| cache ON, warm | 360s | 8m18s | + +Recommendation: disable `NEXT_TURBOPACK_BUILD_CACHE`. #5869 enabled it on locally-measured +numbers (105s cold -> 22s warm) that never reproduced in CI and are inverted here. + +Caveat: n=1 per cell. The effect size (3.2x) and agreement with ~15 prior observations make +it convincing, but this is three runs, not a distribution. + +One unexplained anomaly: run B committed 12G, yet run C mounted 5.1G. Does not affect the +conclusion (C ignores the disk either way), but it means sticky-disk commit semantics are not +fully understood — do not build anything on assumptions about them.