RUBY-3290 Pin the benchmark server version and submit results to SPS - #3107
Open
comandeo-mongo wants to merge 3 commits into
Open
comandeo-mongo wants to merge 3 commits into
comandeo-mongo wants to merge 3 commits into
Conversation
DRIVERS-2666 requires that performance benchmarks run against a patch-pinned server version, so that a change in benchmark scores reflects a change in the driver rather than a change in the server. The DriverBench variant used the floating "8.0" alias, which resolves to whatever 8.0.x is current at the time of the run. Add an "8.0-perf" value to the mongodb-version axis that sets MONGODB_VERSION=v8.0-perf -- an alias defined by drivers-evergreen-tools (PERF_VERSIONS in mongodl.py) that resolves to exactly 8.0.1 -- and point the DriverBench matrix at it. Note this changes the generated build variant id from DriverBench__mongodb-version~8.0_... to DriverBench__mongodb-version~8.0-perf_... The build variant is part of the performance time series key, so existing benchmark history does not carry over.
Evergreen's perf.send command is deprecated and no longer maintained; the
Signal Processing Service is now fed directly over HTTP. Node, Go and
Python have all already migrated. DRIVERS-2666 requires that benchmark
results reach the analytics backend, so move Ruby over too.
Adds two scripts:
perf-submission-setup.sh writes perf-expansion.yml with is_mainline (so
SPS knows whether the results belong in the change-point time series)
and parsed_order_id. It writes a dedicated file rather than appending to
the shared expansion.yml, which holds the PREPARE_SHELL block scalar.
perf-submission.sh POSTs perf.json to raw_perf_results/cedar_report and
fails the task on a non-200 response. perf.send failed quietly, which
made a dropped submission look exactly like a passing benchmark run.
The raw perf.json is also uploaded to S3 so a run can be inspected from
the task page without going through the analytics backend.
The results payload gains the fields SPS and the other drivers use:
- created_at / completed_at, bracketing each micro-benchmark (and, for
the composites, the whole suite run).
- per-metric metadata with improvement_direction and measurement_unit.
Percentiles are wall-clock times, so they are marked "down"; the score
is throughput, so it is marked "up".
- the primary metric is renamed from "score" to "megabytes_per_second",
matching Node and Go, so the numbers line up across drivers in the
analytics backend.
The rename starts a new time series. That costs nothing here because
pinning the server version already changed the build variant id, which is
itself part of the time series key.
perf-submission-setup.sh was invoked with shell.exec, which expands
${...} in the script text but does not put Evergreen expansions into the
child process environment. The script reads $requester from the
environment, so under set -u it died with:
.evergreen/perf-submission-setup.sh: line 18: requester: unbound variable
Run both scripts with subprocess.exec instead, which is how the Python
driver does it. The expansions are listed explicitly with
include_expansions_in_env rather than using add_expansions_to_env, so
that project secrets stay out of the environment of a script that makes
an outbound network call.
Both scripts now also check up front that the expansions they need are
present, and say what to do about it. This is the boundary that just
failed, and a bare "unbound variable" gives no hint that the problem is
in the command that invoked the script rather than in the script.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two of the four things DRIVERS-2666 asks for.
Pin the benchmark server version
DriverBenchran against the floating8.0alias, which resolves to whatever 8.0.x happens to be current on the day of the run. A server-side performance change was therefore indistinguishable from a driver regression.Adds an
8.0-perfvalue to themongodb-versionaxis, settingMONGODB_VERSION=v8.0-perf— an alias defined by drivers-evergreen-tools (PERF_VERSIONSinmongodl.py) that resolves to exactly 8.0.1.Replace
perf.sendwith the SPS endpointEvergreen's
perf.sendis deprecated and unmaintained; results now go straight to the Signal Processing Service, which owns the time series and change point detection. Node, Go and Python have all already migrated.perf-submission-setup.shcomputesis_mainline(only master-waterfall runs feed the time series) andparsed_order_id.perf-submission.shPOSTsperf.jsonand fails the task on any non-200.perf.sendfailed quietly, which made a dropped submission look exactly like a passing benchmark run.perf.jsonis also uploaded to S3, so a run can be inspected from the task page.The results payload gains what SPS and the other drivers use:
created_at/completed_at, and per-metricmetadatawithimprovement_directionandmeasurement_unit. Percentiles are wall-clock times so they are markeddown; the score is throughput so it is markedup.The primary metric is renamed
score→megabytes_per_second, matching Node and Go.Testing
Evergreen patch 1983 — succeeded on build
DriverBench__mongodb-version~8.0-perf_topology~standalone_ruby~ruby-4.0_os~ubuntu2204. Sinceperf-submission.shexits non-zero on a non-200, a passing task means SPS returned 200.Also checked before that:
mongodlresolvesv8.0-perf→ 8.0.1 with both the archive andcrypt_sharedartifacts available;evergreen validateis clean;rubocopis clean.An earlier patch (1982) failed with
perf-submission-setup.sh: line 18: requester: unbound variable— the setup script was invoked withshell.exec, which expands${...}in the script text but does not put Evergreen expansions into the child process environment. Both scripts now run undersubprocess.exec. The expansions are listed explicitly withinclude_expansions_in_envrather thanadd_expansions_to_env: true, so that project secrets stay out of the environment of a script that makes an outbound network call. Both scripts also now fail early, with an actionable message, if an expansion they need is missing.Notes for the reviewer
This resets the benchmark history, twice over: the build variant id changes (
mongodb-version~8.0→~8.0-perf), and the metric is renamed. Both are part of the time series key. Doing them together means one reset rather than two.Two pieces still have to be done outside this repo before any of this data is visible, and neither is in this PR:
Still open on RUBY-3290 after this: the dedicated
rhel90-dbx-perf-largedistro, change-point alerting, and pinning thespecificationsclone intasks.rake(it clones master unpinned today, so a spec-side dataset change shifts BSON scores on its own).