diff --git a/.evergreen/README.md b/.evergreen/README.md index 96d632c7ad..16a43a2f90 100644 --- a/.evergreen/README.md +++ b/.evergreen/README.md @@ -40,6 +40,53 @@ Because this lives in project settings rather than the repository, it must be recreated if the project is reconfigured. +## Performance benchmarks (RUBY-3290) + +The `DriverBench` build variant runs the spec performance benchmarks +(`rake driver_bench`, see `profile/driver_bench`) on every `master` commit. + +### Pinned server version + +The variant uses the `8.0-perf` value of the `mongodb-version` axis, which +sets `MONGODB_VERSION=v8.0-perf`. That is an alias defined by +drivers-evergreen-tools (`PERF_VERSIONS` in `.evergreen/mongodl.py`) and it +resolves to exactly **8.0.1**, not to "whatever 8.0.x is current". + +This is deliberate, per DRIVERS-2666: benchmark scores are only meaningful if +the server under test is held constant, otherwise a server-side performance +change shows up as a driver regression. + +**Do not bump this version as part of routine server-version maintenance.** +Bumping it invalidates the benchmark baseline: every time series restarts and +the change point detector will flag the bump on every micro-benchmark. If a +bump is genuinely needed, do it deliberately, on its own commit, and say so on +the ticket so the resulting change points are triaged as expected. + +### Result submission + +Results are POSTed to the Signal Processing Service (SPS), which stores the +time series and runs change point detection over it. Two scripts do this: + +- `perf-submission-setup.sh` writes `perf-expansion.yml` with `is_mainline` + (only master-waterfall runs feed the time series) and `parsed_order_id`. +- `perf-submission.sh` POSTs `perf.json` to the + `raw_perf_results/cedar_report` endpoint and **fails the task on a non-200 + response**, because a silently dropped submission is indistinguishable from + a passing benchmark run. + +This replaces Evergreen's `perf.send` command, which is deprecated and no +longer maintained. The raw `perf.json` is also uploaded to S3 and linked from +the task page. + +Two further pieces live outside this repository and must be set up by a +project admin (see RUBY-3290): + +- **Performance Plugins** must be enabled in the Evergreen project settings, + otherwise the task has no Trend Charts tab. +- A **triage context** must exist for the project in the performance + monitoring UI before change points are tagged. + + ## Testing In Docker It is possible to run the test suite in Docker. This executes all of the diff --git a/.evergreen/config.yml b/.evergreen/config.yml index 842c93ff0e..7ac3891ac4 100644 --- a/.evergreen/config.yml +++ b/.evergreen/config.yml @@ -282,9 +282,59 @@ functions: script: | ${PREPARE_SHELL} TEST_CMD="bundle exec rake driver_bench" PERFORMANCE_RESULTS_FILE="$PROJECT_DIRECTORY/perf.json" .evergreen/run-tests.sh - - command: perf.send + # Keep a copy of the raw results attached to the task, so that a run can be + # inspected without going through the analytics backend. + - command: s3.put + params: + aws_key: ${aws_key} + aws_secret: ${aws_secret} + local_file: src/perf.json + optional: true + remote_file: ${UPLOAD_BUCKET}/${version_id}/${build_id}/artifacts/${build_variant}/perf.json + content_type: application/json + permissions: public-read + bucket: mciuploads + display_name: "Performance Results" + # Submit the results to the Signal Processing Service, which owns the time + # series and change point detection. This replaces the deprecated perf.send + # command; see .evergreen/perf-submission.sh. + # Both scripts read the expansions from their environment, so they must be + # run with subprocess.exec: shell.exec only expands ${...} in the script + # text and leaves the child process environment untouched. + # The expansions are listed explicitly rather than using + # add_expansions_to_env, to keep project secrets out of the environment of + # a script that makes an outbound network call. + - command: subprocess.exec + type: test + params: + working_dir: "src" + binary: bash + include_expansions_in_env: + - requester + - revision_order_id + args: + - ".evergreen/perf-submission-setup.sh" + - command: expansions.update + params: + file: src/perf-expansion.yml + - command: subprocess.exec + type: test params: - file: "${PROJECT_DIRECTORY}/perf.json" + working_dir: "src" + binary: bash + include_expansions_in_env: + - project_id + - version_id + - build_variant + - parsed_order_id + - task_name + - task_id + - execution + - is_mainline + env: + PERFORMANCE_RESULTS_FILE: "perf.json" + args: + - ".evergreen/perf-submission.sh" "run tests with orchestration and drivers tools": - command: subprocess.exec @@ -729,6 +779,16 @@ axes: display_name: "8.0" variables: MONGODB_VERSION: "8.0" + # RUBY-3290 / DRIVERS-2666: performance benchmarks must run against a + # patch-pinned server so that score changes reflect driver changes and not + # server changes. "v8.0-perf" is an alias defined by drivers-evergreen-tools + # (.evergreen/mongodl.py, PERF_VERSIONS) that resolves to exactly 8.0.1. + # DO NOT bump this casually: a server bump invalidates the benchmark + # baseline and shows up as a change point. + - id: "8.0-perf" + display_name: "8.0 (perf-pinned)" + variables: + MONGODB_VERSION: "v8.0-perf" - id: "7.0" display_name: "7.0" variables: @@ -1108,7 +1168,7 @@ buildvariants: - matrix_name: DriverBench matrix_spec: ruby: "ruby-4.0" - mongodb-version: "8.0" + mongodb-version: "8.0-perf" topology: standalone os: ubuntu2204 display_name: DriverBench diff --git a/.evergreen/config/axes.yml.erb b/.evergreen/config/axes.yml.erb index a4e0a1cd26..e6d3024dff 100644 --- a/.evergreen/config/axes.yml.erb +++ b/.evergreen/config/axes.yml.erb @@ -24,6 +24,16 @@ axes: display_name: "8.0" variables: MONGODB_VERSION: "8.0" + # RUBY-3290 / DRIVERS-2666: performance benchmarks must run against a + # patch-pinned server so that score changes reflect driver changes and not + # server changes. "v8.0-perf" is an alias defined by drivers-evergreen-tools + # (.evergreen/mongodl.py, PERF_VERSIONS) that resolves to exactly 8.0.1. + # DO NOT bump this casually: a server bump invalidates the benchmark + # baseline and shows up as a change point. + - id: "8.0-perf" + display_name: "8.0 (perf-pinned)" + variables: + MONGODB_VERSION: "v8.0-perf" - id: "7.0" display_name: "7.0" variables: diff --git a/.evergreen/config/common.yml.erb b/.evergreen/config/common.yml.erb index 9d440736d8..f29e39983c 100644 --- a/.evergreen/config/common.yml.erb +++ b/.evergreen/config/common.yml.erb @@ -279,9 +279,59 @@ functions: script: | ${PREPARE_SHELL} TEST_CMD="bundle exec rake driver_bench" PERFORMANCE_RESULTS_FILE="$PROJECT_DIRECTORY/perf.json" .evergreen/run-tests.sh - - command: perf.send + # Keep a copy of the raw results attached to the task, so that a run can be + # inspected without going through the analytics backend. + - command: s3.put params: - file: "${PROJECT_DIRECTORY}/perf.json" + aws_key: ${aws_key} + aws_secret: ${aws_secret} + local_file: src/perf.json + optional: true + remote_file: ${UPLOAD_BUCKET}/${version_id}/${build_id}/artifacts/${build_variant}/perf.json + content_type: application/json + permissions: public-read + bucket: mciuploads + display_name: "Performance Results" + # Submit the results to the Signal Processing Service, which owns the time + # series and change point detection. This replaces the deprecated perf.send + # command; see .evergreen/perf-submission.sh. + # Both scripts read the expansions from their environment, so they must be + # run with subprocess.exec: shell.exec only expands ${...} in the script + # text and leaves the child process environment untouched. + # The expansions are listed explicitly rather than using + # add_expansions_to_env, to keep project secrets out of the environment of + # a script that makes an outbound network call. + - command: subprocess.exec + type: test + params: + working_dir: "src" + binary: bash + include_expansions_in_env: + - requester + - revision_order_id + args: + - ".evergreen/perf-submission-setup.sh" + - command: expansions.update + params: + file: src/perf-expansion.yml + - command: subprocess.exec + type: test + params: + working_dir: "src" + binary: bash + include_expansions_in_env: + - project_id + - version_id + - build_variant + - parsed_order_id + - task_name + - task_id + - execution + - is_mainline + env: + PERFORMANCE_RESULTS_FILE: "perf.json" + args: + - ".evergreen/perf-submission.sh" "run tests with orchestration and drivers tools": - command: subprocess.exec diff --git a/.evergreen/config/standard.yml.erb b/.evergreen/config/standard.yml.erb index 1f4e48ae20..a3a220393b 100644 --- a/.evergreen/config/standard.yml.erb +++ b/.evergreen/config/standard.yml.erb @@ -32,6 +32,10 @@ stable_and_rapid = [ latest_stable_mdb_raw, "rapid" ] + # The patch-pinned MongoDB version used by the performance benchmarks. + # See the "8.0-perf" value of the mongodb-version axis for why this is pinned. + perf_mdb = "8.0-perf".inspect # so it gets quoted as a string + recent_mdb = %w( 9.0 8.2 8.0 7.0 ) %> @@ -54,7 +58,7 @@ buildvariants: - matrix_name: DriverBench matrix_spec: ruby: <%= latest_ruby %> - mongodb-version: <%= latest_stable_mdb %> + mongodb-version: <%= perf_mdb %> topology: standalone os: ubuntu2204 display_name: DriverBench diff --git a/.evergreen/perf-submission-setup.sh b/.evergreen/perf-submission-setup.sh new file mode 100755 index 0000000000..74ec9e75d4 --- /dev/null +++ b/.evergreen/perf-submission-setup.sh @@ -0,0 +1,42 @@ +#!/bin/bash +# Prepares the expansions needed by perf-submission.sh. +# +# The Signal Processing Service needs to know whether a set of results comes +# from the master waterfall (and therefore belongs in the time series used for +# change point detection) or from a patch build. +# +# Writes perf-expansion.yml, which the caller must load with expansions.update. +# A dedicated file is used rather than the shared expansion.yml so that this +# does not have to append to the PREPARE_SHELL block scalar written there. + +set -eu + +# These are Evergreen expansions, which only reach this script if it is run +# with subprocess.exec and include_expansions_in_env. shell.exec does not put +# them in the environment, so check explicitly rather than failing with a bare +# "unbound variable". +for var in requester revision_order_id; do + eval "value=\${$var:-}" + if [ -z "$value" ]; then + echo "Error: expansion '$var' is missing from the environment." >&2 + echo "Run this script with subprocess.exec and include_expansions_in_env." >&2 + exit 1 + fi +done + +out=perf-expansion.yml +: > "$out" + +# shellcheck disable=SC2154 +if [ "${requester}" = "commit" ]; then + echo "is_mainline: true" >> "$out" +else + echo "is_mainline: false" >> "$out" +fi + +# revision_order_id looks like "_" for patches and "" +# for mainline commits. SPS wants just the order number. +# shellcheck disable=SC2154 +echo "parsed_order_id: $(echo "${revision_order_id}" | awk -F'_' '{print $NF}')" >> "$out" + +cat "$out" diff --git a/.evergreen/perf-submission.sh b/.evergreen/perf-submission.sh new file mode 100755 index 0000000000..77186b0925 --- /dev/null +++ b/.evergreen/perf-submission.sh @@ -0,0 +1,52 @@ +#!/bin/bash +# Submits DriverBench results to the Signal Processing Service (SPS), which +# stores the time series and runs change point detection over it. +# +# This replaces Evergreen's perf.send command, which is deprecated and no +# longer maintained. See: +# https://docs.devprod.prod.corp.mongodb.com/performance/getting_started/migrating_from_perfSend_to_sps +# +# Expects the expansions written by perf-submission-setup.sh to be in the +# environment (add_expansions_to_env), plus PERFORMANCE_RESULTS_FILE. + +set -eu + +# See the note in perf-submission-setup.sh: these are Evergreen expansions and +# only reach this script via include_expansions_in_env. +for var in project_id version_id build_variant parsed_order_id task_name task_id execution is_mainline; do + eval "value=\${$var:-}" + if [ -z "$value" ]; then + echo "Error: expansion '$var' is missing from the environment." >&2 + echo "Run this script with subprocess.exec and include_expansions_in_env." >&2 + exit 1 + fi +done + +results_file="${PERFORMANCE_RESULTS_FILE:-perf.json}" + +if [ ! -f "$results_file" ]; then + echo "Error: results file '$results_file' does not exist" >&2 + exit 1 +fi + +endpoint="https://performance-monitoring-api.corp.mongodb.com/raw_perf_results/cedar_report" + +# shellcheck disable=SC2154 +response=$(curl -s -w "\nHTTP_STATUS:%{http_code}" -X 'POST' \ + "${endpoint}?project=${project_id}&version=${version_id}&variant=${build_variant}&order=${parsed_order_id}&task_name=${task_name}&task_id=${task_id}&execution=${execution}&mainline=${is_mainline}" \ + -H 'accept: application/json' \ + -H 'Content-Type: application/json' \ + -d @"$results_file") + +http_status=$(echo "$response" | grep "HTTP_STATUS" | awk -F':' '{print $2}') +response_body=$(echo "$response" | sed '/HTTP_STATUS/d') + +echo "Response Body: $response_body" +echo "HTTP Status: $http_status" + +# Fail the task if the data was not accepted. A silently dropped submission +# looks exactly like a passing benchmark run, so this must be loud. +if [ "$http_status" -ne 200 ]; then + echo "Error: performance data was not submitted (HTTP $http_status)" >&2 + exit 1 +fi diff --git a/profile/driver_bench/base.rb b/profile/driver_bench/base.rb index b65de595ac..ee0e87a8ce 100644 --- a/profile/driver_bench/base.rb +++ b/profile/driver_bench/base.rb @@ -52,16 +52,25 @@ def debug_mode? # Runs the benchmark and returns the score. # - # @return [ Hash ] the score and other - # attributes of the benchmark. + # The started_at and completed_at timestamps bracket the entire + # micro-benchmark (setup through teardown); they are reported to the + # performance analytics backend alongside the metrics. + # + # @return [ Hash ] the + # score and other attributes of the benchmark. def run + started_at = Time.now.utc timings = run_benchmark + completed_at = Time.now.utc + percentiles = Percentiles.new(timings) score = dataset_size / percentiles[50] / 1_000_000.0 { name: self.class.bench_name, score: score, - percentiles: percentiles } + percentiles: percentiles, + started_at: started_at, + completed_at: completed_at } end private diff --git a/profile/driver_bench/suite.rb b/profile/driver_bench/suite.rb index 40259872fb..002cbc004b 100644 --- a/profile/driver_bench/suite.rb +++ b/profile/driver_bench/suite.rb @@ -1,5 +1,7 @@ # frozen_string_literal: true +require 'time' + require_relative 'bson' require_relative 'multi_doc' require_relative 'parallel' @@ -41,6 +43,10 @@ module DriverBench class Suite PERCENTILES = [ 10, 25, 50, 75, 90, 95, 98, 99 ].freeze + # The name of the primary metric reported for every benchmark and + # composite. Matches the name used by the other drivers. + SCORE_METRIC = 'megabytes_per_second' + def self.run! new.run end @@ -49,13 +55,18 @@ def run perf_data = [] benches = Hash.new { |h, k| h[k] = [] } + started_at = Time.now.utc + ALL.each do |klass| result = run_benchmark(klass) perf_data << compile_perf_data(result) append_to_benchmarks(klass, result, benches) end - perf_data += compile_benchmarks(benches) + # The composites are derived from every micro-benchmark, so they are + # timestamped with the span of the whole suite rather than of a single + # benchmark. + perf_data += compile_benchmarks(benches, started_at, Time.now.utc) save_perf_data(perf_data) summarize_perf_data(perf_data) @@ -74,8 +85,10 @@ def run_benchmark(klass) def compile_perf_data(result) percentile_data = PERCENTILES.map do |percentile| - { 'name' => "time-#{percentile}%", - 'value' => result[:percentiles][percentile] } + # Percentiles are wall-clock iteration times, so a smaller number is + # an improvement -- the opposite of the throughput score. + metric("time-#{percentile}%", result[:percentiles][percentile], + direction: 'down', unit: 'seconds') end { @@ -83,21 +96,47 @@ def compile_perf_data(result) 'test_name' => result[:name], 'args' => {}, }, + 'created_at' => iso8601(result[:started_at]), + 'completed_at' => iso8601(result[:completed_at]), 'metrics' => [ - { 'name' => 'score', - 'value' => result[:score] }, + score_metric(result[:score]), *percentile_data ] } end + # The primary metric for every benchmark, named to match the other + # drivers (see the Node and Go implementations) so that the numbers are + # comparable across the performance analytics backend. + def score_metric(score) + metric(SCORE_METRIC, score, direction: 'up', unit: 'megabytes_per_second') + end + + # Builds a single metric entry in the format expected by the Signal + # Processing Service. The metadata drives how the change point detector + # reports a shift: improvement_direction says which way is better. + def metric(name, value, direction:, unit:) + { + 'name' => name, + 'value' => value, + 'metadata' => { + 'improvement_direction' => direction, + 'measurement_unit' => unit + } + } + end + + def iso8601(time) + (time || Time.now.utc).utc.iso8601 + end + def append_to_benchmarks(klass, result, benches) BENCHES.each do |benchmark, list| benches[benchmark] << result[:score] if list.include?(klass) end end - def compile_benchmarks(benches) + def compile_benchmarks(benches, started_at, completed_at) benches.each_key do |key| benches[key] = benches[key].sum / benches[key].length end @@ -110,10 +149,9 @@ def compile_benchmarks(benches) 'test_name' => bench, 'args' => {} }, - 'metrics' => [ - { 'name' => 'score', - 'value' => score } - ] + 'created_at' => iso8601(started_at), + 'completed_at' => iso8601(completed_at), + 'metrics' => [ score_metric(score) ] } end end @@ -125,7 +163,7 @@ def summarize_perf_data(data) next unless item['metrics'].length > 1 item['metrics'].each do |metric| - next if metric['name'] == 'score' + next if metric['name'] == SCORE_METRIC puts format(' %s : %4.4g', metric['name'], metric['value']) end