ruby-grape · ericproulx · Sep 7, 2026 · Sep 7, 2026
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,6 +3,7 @@
 #### Features
 
 * [#2914](https://github.com/ruby-grape/grape/pull/2914): Bench 4.0.0 in the version throughput benchmark - [@ericproulx](https://github.com/ericproulx).
+* [#2913](https://github.com/ruby-grape/grape/pull/2913): Bench ZJIT alongside YJIT in the version benchmark - [@ericproulx](https://github.com/ericproulx).
 * Your contribution here.
 
 #### Fixes

diff --git a/benchmark/version_throughput/README.md b/benchmark/version_throughput/README.md
@@ -7,8 +7,8 @@ Cross-version throughput benchmark for Grape. Measures `BenchAPI.call(env)` requ
 | File | Role |
 |---|---|
 | `app.rb` | The API under test. Kept deliberately small and version-agnostic — only DSL surface stable across Grape 3.x, so the same file can run against `3.0.1 … master` without edits. |
-| `bench.rb` | Single-version benchmark. Loads `app.rb`, sanity-checks the response, runs `Benchmark.ips` (2s warmup + 5s measure), prints one `RESULT,<ips>,<μs>,<stddev>,<yjit>` line for the orchestrator. |
-| `run.rb` | Orchestrator. For each version: writes a `Gemfile`, runs `bundle install` under `tmp/bench-versions/<version>/`, exec's `bench.rb` once without YJIT and once with `--yjit` (if available), parses results, writes `RESULTS.md`. |
+| `bench.rb` | Single-version benchmark. Loads `app.rb`, sanity-checks the response, runs `Benchmark.ips` (2s warmup + 5s measure), prints one `RESULT,<ips>,<μs>,<stddev>,<jit>` line for the orchestrator. The trailing field names the JIT that was actually enabled, so the orchestrator can tell a flag that took effect from one that was silently ignored. |
+| `run.rb` | Orchestrator. For each version: writes a `Gemfile`, runs `bundle install` under `tmp/bench-versions/<version>/`, exec's `bench.rb` once per available JIT mode — plain, `--yjit`, `--zjit` — parses results, writes `RESULTS.md`. |
 | `RESULTS.md` | Generated report — overwritten on every run. |
 
 ## Usage
@@ -20,7 +20,7 @@ ruby benchmark/version_throughput/run.rb
 # subset
 GRAPE_VERSIONS="3.3.5,master" ruby benchmark/version_throughput/run.rb
 
-# different Ruby (e.g. one built with YJIT)
+# different Ruby (e.g. one built with YJIT/ZJIT)
 RBENV_VERSION=4.0.6 ruby benchmark/version_throughput/run.rb
 ```
 
@@ -30,24 +30,54 @@ RBENV_VERSION=4.0.6 ruby benchmark/version_throughput/run.rb
 
 Each version produces a row in `RESULTS.md`, listed oldest to newest so the table reads as a timeline:
 
-| Version | No-YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT speedup |
-|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
-| … | … | … | … | … | … | … | … | … | … |
+| Version | No JIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT speedup | ZJIT (i/s) | μs/req | vs prev | vs 3.0.1 | ZJIT speedup |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| … | … | … | … | … | … | … | … | … | … | … | … | … | … | … |
 
 The two delta columns per pass are the point of the report: `vs prev` is the release-over-release change, `vs <first>` the cumulative change since the oldest benched version (the header names it — `vs 3.0.1` with the default list, whatever comes first in `GRAPE_VERSIONS` otherwise). A one-line summary under the table restates first → last for both passes.
 
-YJIT columns are only emitted if the running Ruby was built with YJIT support (`run.rb` probes via `ruby --yjit -e 'exit(defined?(RubyVM::YJIT) ? 0 : 1)'`). Without YJIT the table keeps its `± stddev` column and still carries both deltas.
+A JIT's columns are only emitted if the running Ruby can actually run it: `run.rb` boots `ruby <flag>` and checks `RubyVM::YJIT.enabled?` / `RubyVM::ZJIT.enabled?`, not merely that the constant exists — both are defined on a supporting build whether or not the flag was given. With no JIT available the table falls back to the compact layout, keeping its `± stddev` column and both deltas.
 
 ## Interpreting results
 
 - **Noise floor is ~5-8%.** A single 5s window on macOS can easily move a few percent under thermal throttling or background load. Rerun before drawing conclusions on small deltas.
 - **`master` right after a release benches the same code twice.** When nothing has landed on `master` since the newest benched release, the two rows exercise identical code and their spread is a direct read of that session's noise floor — not a regression or a win.
 - **Run on a quiet machine.** Close other apps, plug in the laptop, don't touch the keyboard during the run. Each version takes ~14s of wall-clock measurement plus bundle install on first use.
 - **`master` vs released gems is not apples-to-apples for code paths that changed.** If a refactor moved code between files, both numbers still measure the same `app.rb` request — that's the point — but interpret deltas as "end-to-end request cost" rather than per-method.
-- **YJIT speedup is `(yjit_ips - no_yjit_ips) / no_yjit_ips`.** Both passes share the same Ruby binary; only the `--yjit` flag differs.
+- **A JIT's speedup is measured against the No JIT pass on the same row.** Every pass shares the same Ruby binary; only the JIT flag differs. Ruby refuses to boot with both `--yjit` and `--zjit`, so each JIT gets its own pass rather than being stacked.
 - **Deltas are computed per pass, on i/s.** A version that failed to bench is skipped as a reference, so `vs prev` always points at the closest version that actually produced a number — an `error:` row never breaks the chain.
 - **Cumulative deltas compound the noise floor.** Read `vs 3.0.1` for the shape of the trend, not as a precise figure.
 
+## Why ZJIT trails YJIT
+
+ZJIT comes out well ahead of the interpreter but well behind YJIT — roughly +20% against +86% on `master`. That gap is ZJIT's codegen, not the harness. Measured on Ruby 4.0.6; re-check before assuming it still holds, because the answer is expected to move as ZJIT matures.
+
+What was ruled out:
+
+- **Warmup.** Throughput is flat from a 2s to a 20s warmup in both JITs. Both trigger at 30 calls (`--yjit-call-threshold`, `--zjit-call-threshold`), which a 5s measure passes in the first millisecond.
+- **Compilation.** `compiled_iseq_count: 306` with `failed_iseq_count: 0` — ZJIT compiled the whole request path.
+- **Deoptimisation.** `guard_type_exit_ratio: 0.0%`, `guard_shape_exit_ratio: 2.7%`, ~4 side exits per request.
+- **Grape.** A small non-Grape JSON workload on the same Ruby shows the same ordering, just with narrower margins.
+
+What `--zjit-stats` does show is about 30 uninlined C method calls per request, and the list is the set YJIT emits inline codegen for:
+
+```
+String#include?  Array#any?     Array#to_a   Array#include?
+Regexp#match     Kernel#dup     Hash#fetch   String#to_sym
+```
+
+Only ~36% of sends take an inlined cfunc path, and each uninlined one syncs interpreter state before the call — `vm_write_pc_count` and `vm_write_sp_count` both land near 167 per request. Grape's request path is dispatch-heavy and leans on exactly those core methods, so the gap shows up wider here than on C-bound code.
+
+Reproduce with a fixed call count rather than `Benchmark.ips`, so the counters are per-request:
+
+```sh
+BUNDLE_GEMFILE=tmp/bench-versions/master/Gemfile \
+  bundle exec ruby --zjit --zjit-stats -e '
+    $LOAD_PATH.unshift("benchmark/version_throughput"); require "app"
+    env = Rack::MockRequest.env_for("/api/v1/hello", method: Rack::GET).freeze
+    300_000.times { BenchAPI.call(env.dup) }'
+```
+
 ## Adding a version
 
 Edit `DEFAULT_VERSIONS` in `run.rb`, keeping it in release order — the delta columns assume the list runs oldest to newest. The orchestrator handles `bundle install` and gemfile generation; nothing else needs to change as long as the new version exposes the DSL `app.rb` uses (`prefix`, `format`, `version 'v1', using: :path`, `get`).
diff --git a/benchmark/version_throughput/RESULTS.md b/benchmark/version_throughput/RESULTS.md
@@ -1,26 +1,26 @@
 # Grape throughput by version
 
-Generated: 2026-09-07 18:16:50 CEST  
+Generated: 2026-09-07 18:53:55 CEST  
 Ruby: ruby 4.0.6 (2026-07-14 revision 03b6d3f889) +PRISM [arm64-darwin25]  
 Host: Darwin 25.6.0 arm64  
-YJIT available: true
+JIT modes benched: No JIT, YJIT, ZJIT
 
 Single-threaded `Benchmark.ips`, 2s warmup + 5s measure, `BenchAPI.call(env)` against `/api/v1/hello` returning a small JSON object. Reproduce with `ruby benchmark/version_throughput/run.rb`.
 
-| Version | No-YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT speedup |
-| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
-| 3.0.1 | 34,255 | 29.19 | — | — | 54,997 | 18.18 | — | — | +60.6% |
-| 3.1.1 | 43,711 | 22.88 | +27.6% | +27.6% | 84,743 | 11.80 | +54.1% | +54.1% | +93.9% |
-| 3.2.1 | 45,470 | 21.99 | +4.0% | +32.7% | 89,697 | 11.15 | +5.8% | +63.1% | +97.3% |
-| 3.3.5 | 67,602 | 14.79 | +48.7% | +97.3% | 135,813 | 7.36 | +51.4% | +146.9% | +100.9% |
-| 4.0.0 | 123,107 | 8.12 | +82.1% | +259.4% | 233,841 | 4.28 | +72.2% | +325.2% | +89.9% |
-| master | 122,134 | 8.19 | -0.8% | +256.5% | 235,134 | 4.25 | +0.6% | +327.5% | +92.5% |
+| Version | No JIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT (i/s) | μs/req | vs prev | vs 3.0.1 | YJIT speedup | ZJIT (i/s) | μs/req | vs prev | vs 3.0.1 | ZJIT speedup |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| 3.0.1 | 34,645 | 28.86 | — | — | 54,054 | 18.50 | — | — | +56.0% | 40,893 | 24.45 | — | — | +18.0% |
+| 3.1.1 | 44,818 | 22.31 | +29.4% | +29.4% | 84,212 | 11.87 | +55.8% | +55.8% | +87.9% | 53,249 | 18.78 | +30.2% | +30.2% | +18.8% |
+| 3.2.1 | 44,075 | 22.69 | -1.7% | +27.2% | 88,309 | 11.32 | +4.9% | +63.4% | +100.4% | 55,028 | 18.17 | +3.3% | +34.6% | +24.8% |
+| 3.3.5 | 68,050 | 14.70 | +54.4% | +96.4% | 133,833 | 7.47 | +51.6% | +147.6% | +96.7% | 88,472 | 11.30 | +60.8% | +116.3% | +30.0% |
+| 4.0.0 | 122,414 | 8.17 | +79.9% | +253.3% | 228,069 | 4.38 | +70.4% | +321.9% | +86.3% | 153,183 | 6.53 | +73.1% | +274.6% | +25.1% |
+| master | 121,325 | 8.24 | -0.9% | +250.2% | 225,823 | 4.43 | -1.0% | +317.8% | +86.1% | 145,129 | 6.89 | -5.3% | +254.9% | +19.6% |
 
-Over time, 3.0.1 → master: **+256.5%** without YJIT, **+327.5%** with YJIT.
+Over time, 3.0.1 → master: **+250.2%** without a JIT, **+317.8%** with YJIT, **+254.9%** with ZJIT.
 
 ## Notes
 - All versions exercised through the same `BenchAPI` definition (kept stable in `app.rb`).
 - `vs prev` compares throughput against the previous benched version and `vs 3.0.1` against the first one; read those columns for improvement over time.
 - Results are noisy at this scale (±5-8%); rerun if a number looks off.
-- `YJIT speedup` is `(yjit_ips - no_yjit_ips) / no_yjit_ips`.
-- YJIT pass uses `ruby --yjit`; both passes share the same Ruby binary.
+- `<JIT> speedup` is that JIT's throughput against the No JIT pass on the same row.
+- JIT passes use `ruby --yjit` and `ruby --zjit`; every pass shares the same Ruby binary. Only one JIT can be enabled at a time, so each gets its own pass.
diff --git a/benchmark/version_throughput/bench.rb b/benchmark/version_throughput/bench.rb
@@ -23,5 +23,9 @@
 end
 
 entry = report.entries.first
-yjit = (defined?(RubyVM::YJIT) && RubyVM::YJIT.enabled?) ? 'on' : 'off'
-puts format('RESULT,%.2f,%.4f,%.2f,%s', entry.ips, 1_000_000.0 / entry.ips, entry.error_percentage, yjit)
+
+# Which JIT actually ran, so the orchestrator can confirm the flag it passed
+# took effect rather than trusting a silently ignored one. Only one can be
+# enabled at a time -- Ruby refuses to boot with both flags.
+jit = %w[YJIT ZJIT].find { |mod| RubyVM.const_defined?(mod) && RubyVM.const_get(mod).enabled? } || 'none'
+puts format('RESULT,%.2f,%.4f,%.2f,%s', entry.ips, 1_000_000.0 / entry.ips, entry.error_percentage, jit.downcase)