From 5863315a2911b5e33b616fab19b9b83725a6ec58 Mon Sep 17 00:00:00 2001 From: Eric Proulx Date: Mon, 7 Sep 2026 18:26:51 +0200 Subject: [PATCH] Bench 4.0.0 alongside master in the version throughput benchmark 4.0.0 shipped today, so the version benchmark's last released data point was 3.3.5 and every `master` delta was measured against a release two months and one major behind it. With 4.0.0 in DEFAULT_VERSIONS the table separates what the major actually delivered (+82.1% no-YJIT, +72.2% YJIT over 3.3.5) from what has landed on master since it was cut. Right now that is nothing: master differs from v4.0.0 only in lib/grape/version.rb, so the two rows bench identical code and their -0.8% / +0.6% spread is a direct read of the benchmark's own noise floor. README.md gains a bullet under "Interpreting results" saying so, since that reading recurs every time the benchmark is run just after a release. RESULTS.md is regenerated on the same Ruby 4.0.6 as the previous run, so the existing rows move only by run-to-run noise. README.md's usage example follows the new default list. Co-authored-by: Claude Opus 5 --- CHANGELOG.md | 1 + benchmark/version_throughput/README.md | 3 ++- benchmark/version_throughput/RESULTS.md | 15 ++++++++------- benchmark/version_throughput/run.rb | 2 +- 4 files changed, 12 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e1cd9b943..ad63567b2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,7 @@ #### Features +* [#2914](https://github.com/ruby-grape/grape/pull/2914): Bench 4.0.0 in the version throughput benchmark - [@ericproulx](https://github.com/ericproulx). * Your contribution here. #### Fixes diff --git a/benchmark/version_throughput/README.md b/benchmark/version_throughput/README.md index b7bc60305..dcf8e7100 100644 --- a/benchmark/version_throughput/README.md +++ b/benchmark/version_throughput/README.md @@ -14,7 +14,7 @@ Cross-version throughput benchmark for Grape. Measures `BenchAPI.call(env)` requ ## Usage ```sh -# default: 3.0.1, 3.1.1, 3.2.1, 3.3.5, master +# default: 3.0.1, 3.1.1, 3.2.1, 3.3.5, 4.0.0, master ruby benchmark/version_throughput/run.rb # subset @@ -41,6 +41,7 @@ YJIT columns are only emitted if the running Ruby was built with YJIT support (` ## Interpreting results - **Noise floor is ~5-8%.** A single 5s window on macOS can easily move a few percent under thermal throttling or background load. Rerun before drawing conclusions on small deltas. +- **`master` right after a release benches the same code twice.** When nothing has landed on `master` since the newest benched release, the two rows exercise identical code and their spread is a direct read of that session's noise floor — not a regression or a win. - **Run on a quiet machine.** Close other apps, plug in the laptop, don't touch the keyboard during the run. Each version takes ~14s of wall-clock measurement plus bundle install on first use. - **`master` vs released gems is not apples-to-apples for code paths that changed.** If a refactor moved code between files, both numbers still measure the same `app.rb` request — that's the point — but interpret deltas as "end-to-end request cost" rather than per-method. - **YJIT speedup is `(yjit_ips - no_yjit_ips) / no_yjit_ips`.** Both passes share the same Ruby binary; only the `--yjit` flag differs. diff --git a/benchmark/version_throughput/RESULTS.md b/benchmark/version_throughput/RESULTS.md index 32c64c6a1..71cd04842 100644 --- a/benchmark/version_throughput/RESULTS.md +++ b/benchmark/version_throughput/RESULTS.md @@ -1,6 +1,6 @@ # Grape throughput by version -Generated: 2026-09-07 17:13:53 CEST +Generated: 2026-09-07 18:16:50 CEST Ruby: ruby 4.0.6 (2026-07-14 revision 03b6d3f889) +PRISM [arm64-darwin25] Host: Darwin 25.6.0 arm64 YJIT available: true @@ -9,13 +9,14 @@ Single-threaded `Benchmark.ips`, 2s warmup + 5s measure, `BenchAPI.call(env)` ag | Version | No-YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT speedup | | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | -| 3.0.1 | 34,512 | 28.98 | — | — | 52,487 | 19.05 | — | — | +52.1% | -| 3.1.1 | 45,656 | 21.90 | +32.3% | +32.3% | 85,284 | 11.73 | +62.5% | +62.5% | +86.8% | -| 3.2.1 | 47,440 | 21.08 | +3.9% | +37.5% | 90,745 | 11.02 | +6.4% | +72.9% | +91.3% | -| 3.3.5 | 67,834 | 14.74 | +43.0% | +96.6% | 137,675 | 7.26 | +51.7% | +162.3% | +103.0% | -| master | 120,877 | 8.27 | +78.2% | +250.2% | 230,153 | 4.34 | +67.2% | +338.5% | +90.4% | +| 3.0.1 | 34,255 | 29.19 | — | — | 54,997 | 18.18 | — | — | +60.6% | +| 3.1.1 | 43,711 | 22.88 | +27.6% | +27.6% | 84,743 | 11.80 | +54.1% | +54.1% | +93.9% | +| 3.2.1 | 45,470 | 21.99 | +4.0% | +32.7% | 89,697 | 11.15 | +5.8% | +63.1% | +97.3% | +| 3.3.5 | 67,602 | 14.79 | +48.7% | +97.3% | 135,813 | 7.36 | +51.4% | +146.9% | +100.9% | +| 4.0.0 | 123,107 | 8.12 | +82.1% | +259.4% | 233,841 | 4.28 | +72.2% | +325.2% | +89.9% | +| master | 122,134 | 8.19 | -0.8% | +256.5% | 235,134 | 4.25 | +0.6% | +327.5% | +92.5% | -Over time, 3.0.1 → master: **+250.2%** without YJIT, **+338.5%** with YJIT. +Over time, 3.0.1 → master: **+256.5%** without YJIT, **+327.5%** with YJIT. ## Notes - All versions exercised through the same `BenchAPI` definition (kept stable in `app.rb`). diff --git a/benchmark/version_throughput/run.rb b/benchmark/version_throughput/run.rb index b901b4b7f..c7042f641 100644 --- a/benchmark/version_throughput/run.rb +++ b/benchmark/version_throughput/run.rb @@ -30,7 +30,7 @@ HERE = __dir__ TMP = File.join(ROOT, 'tmp', 'bench-versions') -DEFAULT_VERSIONS = %w[3.0.1 3.1.1 3.2.1 3.3.5 master].freeze +DEFAULT_VERSIONS = %w[3.0.1 3.1.1 3.2.1 3.3.5 4.0.0 master].freeze versions = (ENV['GRAPE_VERSIONS']&.split(',')&.map(&:strip) || DEFAULT_VERSIONS).freeze def gemfile_for(version)