Skip to content

test(bench): inject fake CPU regression to validate detection pipeline#66

Closed
sedghi wants to merge 2 commits into
mainfrom
regression-fake
Closed

test(bench): inject fake CPU regression to validate detection pipeline#66
sedghi wants to merge 2 commits into
mainfrom
regression-fake

Conversation

@sedghi
Copy link
Copy Markdown
Member

@sedghi sedghi commented May 20, 2026

Summary

  • Adds an identical FAKE_REGRESSION CPU-burn block at the top of every codec's decode() / encode() — 5 wasm packages (charls, openjpeg, openjphjs, libjpeg-turbo-8bit, libjpeg-turbo-12bit) and 2 pure-JS packages (little-endian, big-endian).
  • Purpose: exercise the benchmark PR-comment pipeline introduced in test: migrate to vitest with parallel per-package CI + benchmark PR comments #64 by producing a known, measurable slowdown vs main for every codec. Do not merge — this is a one-off validation branch.

What the burn does

  • C++ wasm: 1M iterations of volatile double burn = (burn + i) * 1.0000001. volatile forces a load/store per iter so the compiler can't fold it; expect a few ms of added CPU per call.
  • JS: 100K iterations of the same shape, with the result published to globalThis.__fakeRegressionBurn so V8 can't DCE it. Sanity-checked locally — adds ~0.1 ms per decode() on top of a sub-0.05 ms baseline.
  • dicom-codec dispatch picks up the regression transitively through the underlying codecs; no separate change needed there.

Test plan

  • CI matrix rebuilds every wasm package and runs vitest bench
  • Sticky PR comment renders before/after/% deltas with the regression highlighted on every row
  • dicom-codec dispatch rows show the transitive slowdown
  • After confirming, revert with grep -rln "FAKE_REGRESSION" packages and delete each block (begin/end markers make this a one-shot cleanup)

🤖 Generated with Claude Code

Introduced a block of code across multiple decoder and encoder implementations to create a controlled CPU load, allowing for the detection of performance regressions in the benchmark pipeline. This block is marked for removal before merging.
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 20, 2026

Merging this PR will degrade performance by 63.87%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 28 regressed benchmarks
✅ 22 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
JPEG-LS Near-Lossless (.81) 649.8 ms 808.5 ms -19.63%
JPEG Baseline 8-bit (.50) 344.7 ms 515.7 ms -33.16%
JPEG-LS Lossless (.80) 553.6 ms 712.8 ms -22.33%
16-bit unsigned + swap, 512x512 73.1 ms 79.8 ms -8.42%
JPEG 2000 Lossy (.91) 1.5 s 1.6 s -10.49%
8-bit passthrough, 512x512 53.6 µs 6,774.7 µs -99.21%
decode CT-512x512-near-lossless.JLS (.81 near-lossless) — cold 19.8 ms 22.7 ms -12.98%
decode CT-512x512-near-lossless.JLS (.81 near-lossless) — warm 19.8 ms 22.7 ms -12.97%
16-bit signed, 512x512 56.7 µs 6,778.5 µs -99.16%
decode CT1.JLS (.80 lossless, 512x512x16bit) — cold 17.2 ms 20.1 ms -14.64%
16-bit unsigned, 512x512 49.6 µs 6,781.9 µs -99.27%
decode CT1.JLS (.80 lossless, 512x512x16bit) — warm 17.2 ms 20.1 ms -14.66%
16-bit signed + swap, 512x512 77.5 ms 84.2 ms -7.98%
32-bit float, 512x512 54.7 µs 6,779.1 µs -99.19%
8-bit passthrough, 512x512 54.9 µs 6,776.5 µs -99.19%
decode CT1.j2c (.201 lossless, 512x512x16bit) — warm 36.8 ms 42.5 ms -13.39%
decode CT2.j2c (.201 lossless, 512x512x16bit) — cold 36.6 ms 42.3 ms -13.46%
decode CT2.JLS (.80 lossless, 512x512x16bit) — cold 15.8 ms 18.7 ms -15.76%
decode CT2.j2c (.201 lossless, 512x512x16bit) — warm 36.6 ms 42.3 ms -13.47%
encode CT1.RAW (HTJ2K lossless) — cold 36.7 ms 68.2 ms -46.27%
... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing regression-fake (2c1ff1c) with main (04c3e87)

Open in CodSpeed

#67 landed on main and produced the first CodSpeed baseline (run
26167789689). Re-running codspeed-bench on this PR so the sticky
comment can flip from "Congrats!" to a real before/after table with
the fake-regression deltas highlighted on every codec.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@sedghi sedghi closed this May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant