Bench ZJIT alongside YJIT in the version benchmark

[ericproulx]

Summary

run.rb was structured around a with_yjit boolean: one pass without a JIT, one with --yjit, and a report shape chosen by that flag. It now walks a list of JIT modes — interpreter, --yjit, --zjit — so each gets its own throughput, both delta columns, and a speedup against the interpreter on the same row. Ruby refuses to boot with both JIT flags (Only one JIT can be enabled at the same time. Exiting), so they cannot share a pass. With no JIT available the report falls back to the compact layout it always used.

Availability probe

The old probe was ruby --yjit -e 'exit(defined?(RubyVM::YJIT) ? 0 : 1)'. That constant is defined on any build supporting YJIT whether or not the flag was given, so it would have reported YJIT available on a build where --yjit did nothing — and there is no way to write the ZJIT equivalent, since RubyVM::ZJIT is defined under --yjit too:

$ ruby --yjit -e 'p [defined?(RubyVM::ZJIT), RubyVM::ZJIT.enabled?, RubyVM::YJIT.enabled?]'
["constant", false, true]

It now boots ruby <flag> and asks the JIT whether it is enabled?. bench.rb additionally reports which JIT actually ran instead of a yjit on/off flag, and a pass whose JIT does not match the one asked for is recorded as an error rather than published as a column quietly repeating the interpreter's number.

Results

Ruby 4.0.6, arm64-darwin25. ZJIT lands between the interpreter and YJIT throughout.

Version	No JIT (i/s)	YJIT (i/s)	YJIT speedup	ZJIT (i/s)	ZJIT speedup
3.0.1	34,645	54,054	+56.0%	40,893	+18.0%
3.1.1	44,818	84,212	+87.9%	53,249	+18.8%
3.2.1	44,075	88,309	+100.4%	55,028	+24.8%
3.3.5	68,050	133,833	+96.7%	88,472	+30.0%
4.0.0	122,414	228,069	+86.3%	153,183	+25.1%
master	121,325	225,823	+86.1%	145,129	+19.6%

Rebased onto master, so the table picks up the 4.0.0 row added by #2914. Those last two rows bench the same code — master differs from v4.0.0 only in lib/grape/version.rb — which makes their spread a free reading of each pass's noise floor: -0.9% interpreter, -1.0% YJIT, -5.3% ZJIT. Read the ZJIT column with that in mind; it is the noisiest of the three.

Full table with μs/req and both delta columns per pass is in RESULTS.md.

Why ZJIT trails YJIT

That ordering is the first question the table raises, so README.md gains a section recording the answer rather than leaving it to be re-derived. Ruled out on Ruby 4.0.6: warmup (throughput flat from a 2s to a 20s warmup; both JITs trigger at 30 calls), compilation (compiled_iseq_count: 306, failed_iseq_count: 0), deoptimisation (guard_type_exit_ratio: 0.0%, guard_shape_exit_ratio: 2.7%, ~4 side exits per request), and Grape itself (a non-Grape JSON control shows the same ordering).

What --zjit-stats does show is ~30 uninlined C method calls per request — String#include?, Array#any?, Array#to_a, Array#include?, Regexp#match, Kernel#dup, Hash#fetch, String#to_sym — the set YJIT emits inline. Only ~36% of sends take an inlined cfunc path, and each uninlined one syncs interpreter state first: vm_write_pc_count and vm_write_sp_count both land near 167 per request. The section carries the --zjit-stats command that reproduces those counters, and is dated to the Ruby it was measured on.

Test plan

• Full run: all 6 versions × 3 passes produced numbers, no JIT mismatches.
• Availability probe returns false for an unsupported flag rather than raising.
• Warmup sweep (2s/10s/20s) confirms both JITs are at steady state under the benchmark's 2s warmup.
• The --zjit-stats command documented in the README runs verbatim from the repo root and reproduces every counter cited.
• ruby -c clean on both scripts (benchmark/**/* is excluded from RuboCop).

CHANGELOG entry added under 4.1.0 Features, matching #2914.

🤖 Generated with Claude Code

[github-actions]

Danger Report

No issues found.

View run

[dblock]

Impressive improvements! Since YJIT speedup is bigger than ZGIT probably swapping the columns in the report would make it more readable.