Bench 4.0.0 alongside master in the version throughput benchmark - #2914
Merged
Merged
Conversation
Danger ReportNo issues found. |
ericproulx
marked this pull request as ready for review
September 7, 2026 16:27
ericproulx
force-pushed
the
chore/bench-4-0-0-default
branch
from
September 7, 2026 16:29
7a31162 to
b58d160
Compare
4.0.0 shipped today, so the version benchmark's last released data point was 3.3.5 and every `master` delta was measured against a release two months and one major behind it. With 4.0.0 in DEFAULT_VERSIONS the table separates what the major actually delivered (+82.1% no-YJIT, +72.2% YJIT over 3.3.5) from what has landed on master since it was cut. Right now that is nothing: master differs from v4.0.0 only in lib/grape/version.rb, so the two rows bench identical code and their -0.8% / +0.6% spread is a direct read of the benchmark's own noise floor. README.md gains a bullet under "Interpreting results" saying so, since that reading recurs every time the benchmark is run just after a release. RESULTS.md is regenerated on the same Ruby 4.0.6 as the previous run, so the existing rows move only by run-to-run noise. README.md's usage example follows the new default list. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
ericproulx
force-pushed
the
chore/bench-4-0-0-default
branch
from
September 7, 2026 16:30
b58d160 to
5863315
Compare
ericproulx
added a commit
that referenced
this pull request
Sep 7, 2026
run.rb was structured around a `with_yjit` boolean: one pass without a
JIT, one with `--yjit`, and a report shape chosen by that flag. It now
walks a list of JIT modes -- interpreter, `--yjit`, `--zjit` -- so each
gets its own throughput, both delta columns and a speedup against the
interpreter on the same row. Ruby refuses to boot with both JIT flags
("Only one JIT can be enabled at the same time"), so they cannot share a
pass. With no JIT available the report falls back to the compact layout
it always used.
Availability is now probed by booting `ruby <flag>` and asking the JIT
whether it is `enabled?`. The old probe only checked that RubyVM::YJIT
was defined, which is true on any build that supports YJIT whether or
not the flag was given -- it would have reported YJIT available on a
build where `--yjit` did nothing, and there is no way to write the ZJIT
equivalent, since RubyVM::ZJIT is defined under `--yjit` too.
bench.rb reports which JIT actually ran rather than a yjit on/off flag,
and a pass whose JIT does not match the one asked for is recorded as an
error instead of being published as a column quietly repeating the
interpreter's number.
RESULTS.md regenerated on Ruby 4.0.6, arm64-darwin25, now including the
4.0.0 row that landed in #2914. ZJIT lands between the interpreter and
YJIT throughout: on master, 145,129 i/s against 121,325 without a JIT
and 225,823 with YJIT. Since that ordering is the first question the
table raises, README.md records where the gap comes from -- ~30
uninlined C method calls per request, each syncing interpreter state
first -- along with what was ruled out (warmup, compilation,
deoptimisation, Grape itself) and the --zjit-stats command that
reproduces the counters.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
5 tasks
ericproulx
added a commit
that referenced
this pull request
Sep 7, 2026
run.rb was structured around a `with_yjit` boolean: one pass without a
JIT, one with `--yjit`, and a report shape chosen by that flag. It now
walks a list of JIT modes -- interpreter, `--yjit`, `--zjit` -- so each
gets its own throughput, both delta columns and a speedup against the
interpreter on the same row. Ruby refuses to boot with both JIT flags
("Only one JIT can be enabled at the same time"), so they cannot share a
pass. With no JIT available the report falls back to the compact layout
it always used.
Availability is now probed by booting `ruby <flag>` and asking the JIT
whether it is `enabled?`. The old probe only checked that RubyVM::YJIT
was defined, which is true on any build that supports YJIT whether or
not the flag was given -- it would have reported YJIT available on a
build where `--yjit` did nothing, and there is no way to write the ZJIT
equivalent, since RubyVM::ZJIT is defined under `--yjit` too.
bench.rb reports which JIT actually ran rather than a yjit on/off flag,
and a pass whose JIT does not match the one asked for is recorded as an
error instead of being published as a column quietly repeating the
interpreter's number.
RESULTS.md regenerated on Ruby 4.0.6, arm64-darwin25, now including the
4.0.0 row that landed in #2914. ZJIT lands between the interpreter and
YJIT throughout: on master, 145,129 i/s against 121,325 without a JIT
and 225,823 with YJIT. Since that ordering is the first question the
table raises, README.md records where the gap comes from -- ~30
uninlined C method calls per request, each syncing interpreter state
first -- along with what was ruled out (warmup, compilation,
deoptimisation, Grape itself) and the --zjit-stats command that
reproduces the counters.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
ericproulx
added a commit
that referenced
this pull request
Sep 7, 2026
run.rb was structured around a `with_yjit` boolean: one pass without a
JIT, one with `--yjit`, and a report shape chosen by that flag. It now
walks a list of JIT modes -- interpreter, `--yjit`, `--zjit` -- so each
gets its own throughput, both delta columns and a speedup against the
interpreter on the same row. Ruby refuses to boot with both JIT flags
("Only one JIT can be enabled at the same time"), so they cannot share a
pass. With no JIT available the report falls back to the compact layout
it always used.
Availability is now probed by booting `ruby <flag>` and asking the JIT
whether it is `enabled?`. The old probe only checked that RubyVM::YJIT
was defined, which is true on any build that supports YJIT whether or
not the flag was given -- it would have reported YJIT available on a
build where `--yjit` did nothing, and there is no way to write the ZJIT
equivalent, since RubyVM::ZJIT is defined under `--yjit` too.
bench.rb reports which JIT actually ran rather than a yjit on/off flag,
and a pass whose JIT does not match the one asked for is recorded as an
error instead of being published as a column quietly repeating the
interpreter's number.
RESULTS.md regenerated on Ruby 4.0.6, arm64-darwin25, now including the
4.0.0 row that landed in #2914. ZJIT lands between the interpreter and
YJIT throughout: on master, 145,129 i/s against 121,325 without a JIT
and 225,823 with YJIT. Since that ordering is the first question the
table raises, README.md records where the gap comes from -- ~30
uninlined C method calls per request, each syncing interpreter state
first -- along with what was ruled out (warmup, compilation,
deoptimisation, Grape itself) and the --zjit-stats command that
reproduces the counters.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
4.0.0toDEFAULT_VERSIONSinbenchmark/version_throughput/run.rb, between3.3.5andmaster.RESULTS.mdon the same Ruby 4.0.6 as the previous run.masterrow just after a release.4.0.0 shipped today, so the last released data point in the table was 3.3.5 and every
masterdelta was measured against a release two months and one major behind it. Splitting the row separates what the major delivered from what has landed on master since it was cut.Benchmarks
The last two rows bench the same code:
masterdiffers fromv4.0.0only inlib/grape/version.rb(the 4.1.0 development bump). Their -0.8% / +0.6% spread is therefore a measurement of the benchmark's own noise floor, not drift — a useful calibration of the README's stated ~5-8% figure, and the reason the new README bullet spells the situation out. The pre-existing rows likewise move only by run-to-run noise against the previousRESULTS.md.Test plan
ruby benchmark/version_throughput/run.rbon Ruby 4.0.6 (arm64-darwin25, YJIT available) — output is the committedRESULTS.md.lib/or spec surface touched.🤖 Generated with Claude Code