Perf: add benchmark regression gates and historical evidence

[chrisbbreuer]

Parent: #44
Related: #45, #53, #62, #97

Goal

Keep performance work measurable and prevent accepted wins from silently regressing without turning ordinary CI into a noisy hardware contest.

Landed controls

• Versioned raw TSV samples and rendered reports retain exact date, host/CPU, OS, Zig, zig-js/zig-gc/zig-regex revisions, system JavaScriptCore version, and power metadata.
• The comparison driver validates the complete symmetric matrix, exact sample indexes/counts, stable checksums, cross-engine equal work, runner alternation, a 50 ms full-run timing floor, and per-row min/max/RSD.
• Publishing raw/report evidence refuses a dirty tracked worktree and captures metadata before generated outputs modify the tree.
• zig build benchmark-comparison-test exercises matrix and publication guards in ordinary CI without compiling or running benchmarks.
• The manual-only Performance workflow, added in c7e52fb1, runs the full macOS/JSC matrix with configurable sample/lane counts and retains its artifacts for 90 days. It never runs on pushes or pull requests and never gates ordinary CI.
• 0ff34466 adds the controlled history/publication tool. It reruns the full raw-matrix validation, reproduces the paired Markdown report byte-for-byte, and then generates the marker-delimited README scorecard.
• History comparison requires exact host, OS, Zig, zig-gc, zig-regex, JavaScriptCore, matrix, jobs, and sample-count matches. Volatile battery details are normalized while power source and charging state remain part of the identity.
• Every engine row is retained with baseline/current medians, delta, and both RSDs. A zig-js row gates only when median wall time worsens by more than 10% and both RSDs are at most 5%; JSC rows remain visible environmental controls. The exact baseline/current report paths and zig-js revisions are recorded in the artifact.
• The manual workflow always generates a README scorecard artifact and optionally accepts a committed controlled baseline pair to produce the per-row history artifact or reject an incompatible run. Failed validation still uploads available evidence.

Current accepted evidence

• measured source: 12aa217c2f66c0fe7fa95e38ec414ea5ec15a993
• report: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.md
• raw 1,540 samples: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.tsv
• 10/10 direct wins (2.67x geometric-mean zig-js throughput)
• 10/10 eight-lane warmed-independent wins (2.95x; 5.14x zig-js scaling vs 4.77x JSC)
• 10/10 eight-lane cold-lifecycle wins (2.91x; 5.18x vs 4.76x)
• 3.97x shared-realm geometric-mean eight-lane scaling; Perf: remove shared-realm object-churn GC scaling collapse #97 owns the remaining object-churn deficit

The generated calculation corrected the previously hand-copied 2.63x / 2.94x / 2.94x aggregate summary to the raw-derived 2.67x / 2.95x / 2.91x, demonstrating the drift this issue was intended to eliminate.

Acceptance criteria

• Store versioned raw samples and rendered reports with exact commit/environment/power metadata.
• Add deterministic standalone harness/unit checks to ordinary CI.
• Add a manually triggered full performance workflow that does not gate ordinary CI.
• Compare only like-for-like host/runtime configurations and retain explicit historical baselines.
• Define a minimum timing floor, expose repeated-sample dispersion, and apply a conservative noise-aware policy only to controlled runs.
• Fail publication when the tracked worktree is dirty and capture metadata before generated outputs change the tree.
• Link optimization commits and resulting benchmark deltas from their workload issues.
• Keep README headline numbers generated from the latest accepted report rather than manually copied.

[chrisbbreuer]

Progress in e70eeff and 6fb3b28: publication now refuses a dirty tracked worktree, captures metadata before writing tracked outputs, and retains the exact source commit plus 224 raw samples. #50 remains open for CI harness tests, controlled performance runners/history, derived regression thresholds, and generated README tables.

[chrisbbreuer]

The publication guard and dated evidence passed the final combined gate (31 success, 0 failure): https://github.com/zig-utils/zig-js/actions/runs/29280193748. Remaining CI/history/generation work stays open.

[chrisbbreuer]

Latest accepted evidence is now published:

• measured source: 42d0ec26
• 616-sample TSV + generated report: ccee8842
• README + benchmark guide: 09d543c3

The clean full matrix passed every checksum and 50 ms floor guard. zig-js wins all 36 symmetric single/steady/cold rows. README headline values are still manually summarized, so the generated-headline/workflow acceptance items remain open.

[chrisbbreuer]

Final validation for the published comparison is green on main at 889611de:

• CI run: https://github.com/zig-utils/zig-js/actions/runs/29305518043
• 32/32 runnable jobs passed; the single nightly-only entry was correctly skipped
• Includes the docs build, benchmark comparison harness, all unit shards, thread/reference/no-GIL checks, thread-fuzz profiles, TSan unit coverage, and all four TSan no-GIL corpus shards
• The two follow-up CI fixes are included: isolate property quickening mode in its unit test (e8f98e49) and honor the committed Bun lockfile for docs (889611de)

The measured report (ccee8842) and README summary (09d543c3) are therefore validated end-to-end. I am leaving #50 open only for its remaining automation work: generate/check the README summary from the committed report so future benchmark updates cannot drift.

[chrisbbreuer]

Pushed 0e22e4bb (bench: align comparison allocators (#50)). Every measured zig-js context now uses the same process-wide libc allocator, matching JSC's process-scoped cached allocator model instead of mixing the runner allocator into direct/shared rows. The generator and benchmark guide document the current method while preserving the July 13 report as immutable historical evidence.

Validation:

• ReleaseFast comparison runners built successfully
• benchmark harness contract: 7/7 passed
• exact array checksum stayed 23508000000 at one lane and 188190000000 at eight shared lanes
• 7-sample medians were neutral within noise: direct arrays ~61.10 -> ~61.30 ms; shared one lane ~63.15 -> ~63.49 ms; shared eight lanes ~157.59 -> ~156.23 ms

This is a fairness/alignment change, not a claimed speedup. The next accepted full matrix will measure the unified allocator rather than rewriting the saved report.