RUBY-3290 Pin the benchmark server version and submit results to SPS

.evergreen/README.md

@@ -40,6 +40,53 @@ Because this lives in project settings rather than the repository, it must be
 recreated if the project is reconfigured.
 
 
+## Performance benchmarks (RUBY-3290)
+
+The `DriverBench` build variant runs the spec performance benchmarks
+(`rake driver_bench`, see `profile/driver_bench`) on every `master` commit.
+
+### Pinned server version
+
+The variant uses the `8.0-perf` value of the `mongodb-version` axis, which
+sets `MONGODB_VERSION=v8.0-perf`. That is an alias defined by
+drivers-evergreen-tools (`PERF_VERSIONS` in `.evergreen/mongodl.py`) and it
+resolves to exactly **8.0.1**, not to "whatever 8.0.x is current".
+
+This is deliberate, per DRIVERS-2666: benchmark scores are only meaningful if
+the server under test is held constant, otherwise a server-side performance
+change shows up as a driver regression.
+
+**Do not bump this version as part of routine server-version maintenance.**
+Bumping it invalidates the benchmark baseline: every time series restarts and
+the change point detector will flag the bump on every micro-benchmark. If a
+bump is genuinely needed, do it deliberately, on its own commit, and say so on
+the ticket so the resulting change points are triaged as expected.
+
+### Result submission
+
+Results are POSTed to the Signal Processing Service (SPS), which stores the
+time series and runs change point detection over it. Two scripts do this:
+
+- `perf-submission-setup.sh` writes `perf-expansion.yml` with `is_mainline`
+  (only master-waterfall runs feed the time series) and `parsed_order_id`.
+- `perf-submission.sh` POSTs `perf.json` to the
+  `raw_perf_results/cedar_report` endpoint and **fails the task on a non-200
+  response**, because a silently dropped submission is indistinguishable from
+  a passing benchmark run.
+
+This replaces Evergreen's `perf.send` command, which is deprecated and no
+longer maintained. The raw `perf.json` is also uploaded to S3 and linked from
+the task page.
+
+Two further pieces live outside this repository and must be set up by a
+project admin (see RUBY-3290):
+
+- **Performance Plugins** must be enabled in the Evergreen project settings,
+  otherwise the task has no Trend Charts tab.
+- A **triage context** must exist for the project in the performance
+  monitoring UI before change points are tagged.
+
+
 ## Testing In Docker
 
 It is possible to run the test suite in Docker. This executes all of the

.evergreen/config.yml

@@ -282,9 +282,59 @@ functions:
         script: |
           ${PREPARE_SHELL}
           TEST_CMD="bundle exec rake driver_bench" PERFORMANCE_RESULTS_FILE="$PROJECT_DIRECTORY/perf.json" .evergreen/run-tests.sh
-    - command: perf.send
+    # Keep a copy of the raw results attached to the task, so that a run can be
+    # inspected without going through the analytics backend.
+    - command: s3.put
+      params:
+        aws_key: ${aws_key}
+        aws_secret: ${aws_secret}
+        local_file: src/perf.json
+        optional: true
+        remote_file: ${UPLOAD_BUCKET}/${version_id}/${build_id}/artifacts/${build_variant}/perf.json
+        content_type: application/json
+        permissions: public-read
+        bucket: mciuploads
+        display_name: "Performance Results"
+    # Submit the results to the Signal Processing Service, which owns the time
+    # series and change point detection. This replaces the deprecated perf.send
+    # command; see .evergreen/perf-submission.sh.
+    # Both scripts read the expansions from their environment, so they must be
+    # run with subprocess.exec: shell.exec only expands ${...} in the script
+    # text and leaves the child process environment untouched.
+    # The expansions are listed explicitly rather than using
+    # add_expansions_to_env, to keep project secrets out of the environment of
+    # a script that makes an outbound network call.
+    - command: subprocess.exec
+      type: test
+      params:
+        working_dir: "src"
+        binary: bash
+        include_expansions_in_env:
+          - requester
+          - revision_order_id
+        args:
+          - ".evergreen/perf-submission-setup.sh"
+    - command: expansions.update
+      params:
+        file: src/perf-expansion.yml
+    - command: subprocess.exec
+      type: test
       params:
-        file: "${PROJECT_DIRECTORY}/perf.json"
+        working_dir: "src"
+        binary: bash
+        include_expansions_in_env:
+          - project_id
+          - version_id
+          - build_variant
+          - parsed_order_id
+          - task_name
+          - task_id
+          - execution
+          - is_mainline
+        env:
+          PERFORMANCE_RESULTS_FILE: "perf.json"
+        args:
+          - ".evergreen/perf-submission.sh"
 
   "run tests with orchestration and drivers tools":
     - command: subprocess.exec
@@ -729,6 +779,16 @@ axes:
         display_name: "8.0"
         variables:
           MONGODB_VERSION: "8.0"
+      # RUBY-3290 / DRIVERS-2666: performance benchmarks must run against a
+      # patch-pinned server so that score changes reflect driver changes and not
+      # server changes. "v8.0-perf" is an alias defined by drivers-evergreen-tools
+      # (.evergreen/mongodl.py, PERF_VERSIONS) that resolves to exactly 8.0.1.
+      # DO NOT bump this casually: a server bump invalidates the benchmark
+      # baseline and shows up as a change point.
+      - id: "8.0-perf"
+        display_name: "8.0 (perf-pinned)"
+        variables:
+          MONGODB_VERSION: "v8.0-perf"
       - id: "7.0"
         display_name: "7.0"
         variables:
@@ -1108,7 +1168,7 @@ buildvariants:
   - matrix_name: DriverBench
     matrix_spec:
       ruby: "ruby-4.0"
-      mongodb-version: "8.0"
+      mongodb-version: "8.0-perf"
       topology: standalone
       os: ubuntu2204
     display_name: DriverBench

.evergreen/config/axes.yml.erb

@@ -24,6 +24,16 @@ axes:
         display_name: "8.0"
         variables:
           MONGODB_VERSION: "8.0"
+      # RUBY-3290 / DRIVERS-2666: performance benchmarks must run against a
+      # patch-pinned server so that score changes reflect driver changes and not
+      # server changes. "v8.0-perf" is an alias defined by drivers-evergreen-tools
+      # (.evergreen/mongodl.py, PERF_VERSIONS) that resolves to exactly 8.0.1.
+      # DO NOT bump this casually: a server bump invalidates the benchmark
+      # baseline and shows up as a change point.
+      - id: "8.0-perf"
+        display_name: "8.0 (perf-pinned)"
+        variables:
+          MONGODB_VERSION: "v8.0-perf"
       - id: "7.0"
         display_name: "7.0"
         variables:

.evergreen/config/common.yml.erb

@@ -279,9 +279,59 @@ functions:
         script: |
           ${PREPARE_SHELL}
           TEST_CMD="bundle exec rake driver_bench" PERFORMANCE_RESULTS_FILE="$PROJECT_DIRECTORY/perf.json" .evergreen/run-tests.sh
-    - command: perf.send
+    # Keep a copy of the raw results attached to the task, so that a run can be
+    # inspected without going through the analytics backend.
+    - command: s3.put
       params:
-        file: "${PROJECT_DIRECTORY}/perf.json"
+        aws_key: ${aws_key}
+        aws_secret: ${aws_secret}
+        local_file: src/perf.json
+        optional: true
+        remote_file: ${UPLOAD_BUCKET}/${version_id}/${build_id}/artifacts/${build_variant}/perf.json
+        content_type: application/json
+        permissions: public-read
+        bucket: mciuploads
+        display_name: "Performance Results"
+    # Submit the results to the Signal Processing Service, which owns the time
+    # series and change point detection. This replaces the deprecated perf.send
+    # command; see .evergreen/perf-submission.sh.
+    # Both scripts read the expansions from their environment, so they must be
+    # run with subprocess.exec: shell.exec only expands ${...} in the script
+    # text and leaves the child process environment untouched.
+    # The expansions are listed explicitly rather than using
+    # add_expansions_to_env, to keep project secrets out of the environment of
+    # a script that makes an outbound network call.
+    - command: subprocess.exec
+      type: test
+      params:
+        working_dir: "src"
+        binary: bash
+        include_expansions_in_env:
+          - requester
+          - revision_order_id
+        args:
+          - ".evergreen/perf-submission-setup.sh"
+    - command: expansions.update
+      params:
+        file: src/perf-expansion.yml
+    - command: subprocess.exec
+      type: test
+      params:
+        working_dir: "src"
+        binary: bash
+        include_expansions_in_env:
+          - project_id
+          - version_id
+          - build_variant
+          - parsed_order_id
+          - task_name
+          - task_id
+          - execution
+          - is_mainline
+        env:
+          PERFORMANCE_RESULTS_FILE: "perf.json"
+        args:
+          - ".evergreen/perf-submission.sh"
 
   "run tests with orchestration and drivers tools":
     - command: subprocess.exec

.evergreen/config/standard.yml.erb

@@ -32,6 +32,10 @@
 
   stable_and_rapid = [ latest_stable_mdb_raw, "rapid" ]
 
+  # The patch-pinned MongoDB version used by the performance benchmarks.
+  # See the "8.0-perf" value of the mongodb-version axis for why this is pinned.
+  perf_mdb = "8.0-perf".inspect # so it gets quoted as a string
+
   recent_mdb = %w( 9.0 8.2 8.0 7.0 )
 %>
 
@@ -54,7 +58,7 @@ buildvariants:
   - matrix_name: DriverBench
     matrix_spec:
       ruby: <%= latest_ruby %>
-      mongodb-version: <%= latest_stable_mdb %>
+      mongodb-version: <%= perf_mdb %>
       topology: standalone
       os: ubuntu2204
     display_name: DriverBench

.evergreen/perf-submission-setup.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+# Prepares the expansions needed by perf-submission.sh.
+#
+# The Signal Processing Service needs to know whether a set of results comes
+# from the master waterfall (and therefore belongs in the time series used for
+# change point detection) or from a patch build.
+#
+# Writes perf-expansion.yml, which the caller must load with expansions.update.
+# A dedicated file is used rather than the shared expansion.yml so that this
+# does not have to append to the PREPARE_SHELL block scalar written there.
+
+set -eu
+
+# These are Evergreen expansions, which only reach this script if it is run
+# with subprocess.exec and include_expansions_in_env. shell.exec does not put
+# them in the environment, so check explicitly rather than failing with a bare
+# "unbound variable".
+for var in requester revision_order_id; do
+  eval "value=\${$var:-}"
+  if [ -z "$value" ]; then
+    echo "Error: expansion '$var' is missing from the environment." >&2
+    echo "Run this script with subprocess.exec and include_expansions_in_env." >&2
+    exit 1
+  fi
+done
+
+out=perf-expansion.yml
+: > "$out"
+
+# shellcheck disable=SC2154
+if [ "${requester}" = "commit" ]; then
+  echo "is_mainline: true" >> "$out"
+else
+  echo "is_mainline: false" >> "$out"
+fi
+
+# revision_order_id looks like "<username>_<order>" for patches and "<order>"
+# for mainline commits. SPS wants just the order number.
+# shellcheck disable=SC2154
+echo "parsed_order_id: $(echo "${revision_order_id}" | awk -F'_' '{print $NF}')" >> "$out"
+
+cat "$out"

.evergreen/perf-submission.sh

@@ -0,0 +1,52 @@
+#!/bin/bash
+# Submits DriverBench results to the Signal Processing Service (SPS), which
+# stores the time series and runs change point detection over it.
+#
+# This replaces Evergreen's perf.send command, which is deprecated and no
+# longer maintained. See:
+#   https://docs.devprod.prod.corp.mongodb.com/performance/getting_started/migrating_from_perfSend_to_sps
+#
+# Expects the expansions written by perf-submission-setup.sh to be in the
+# environment (add_expansions_to_env), plus PERFORMANCE_RESULTS_FILE.
+
+set -eu
+
+# See the note in perf-submission-setup.sh: these are Evergreen expansions and
+# only reach this script via include_expansions_in_env.
+for var in project_id version_id build_variant parsed_order_id task_name task_id execution is_mainline; do
+  eval "value=\${$var:-}"
+  if [ -z "$value" ]; then
+    echo "Error: expansion '$var' is missing from the environment." >&2
+    echo "Run this script with subprocess.exec and include_expansions_in_env." >&2
+    exit 1
+  fi
+done
+
+results_file="${PERFORMANCE_RESULTS_FILE:-perf.json}"
+
+if [ ! -f "$results_file" ]; then
+  echo "Error: results file '$results_file' does not exist" >&2
+  exit 1
+fi
+
+endpoint="https://performance-monitoring-api.corp.mongodb.com/raw_perf_results/cedar_report"
+
+# shellcheck disable=SC2154
+response=$(curl -s -w "\nHTTP_STATUS:%{http_code}" -X 'POST' \
+  "${endpoint}?project=${project_id}&version=${version_id}&variant=${build_variant}&order=${parsed_order_id}&task_name=${task_name}&task_id=${task_id}&execution=${execution}&mainline=${is_mainline}" \
+  -H 'accept: application/json' \
+  -H 'Content-Type: application/json' \
+  -d @"$results_file")
+
+http_status=$(echo "$response" | grep "HTTP_STATUS" | awk -F':' '{print $2}')
+response_body=$(echo "$response" | sed '/HTTP_STATUS/d')
+
+echo "Response Body: $response_body"
+echo "HTTP Status: $http_status"
+
+# Fail the task if the data was not accepted. A silently dropped submission
+# looks exactly like a passing benchmark run, so this must be loud.
+if [ "$http_status" -ne 200 ]; then
+  echo "Error: performance data was not submitted (HTTP $http_status)" >&2
+  exit 1
+fi

profile/driver_bench/base.rb

@@ -52,16 +52,25 @@ def debug_mode?
 
       # Runs the benchmark and returns the score.
       #
-      # @return [ Hash<name,score,percentiles> ] the score and other
-      #   attributes of the benchmark.
+      # The started_at and completed_at timestamps bracket the entire
+      # micro-benchmark (setup through teardown); they are reported to the
+      # performance analytics backend alongside the metrics.
+      #
+      # @return [ Hash<name,score,percentiles,started_at,completed_at> ] the
+      #   score and other attributes of the benchmark.
       def run
+        started_at = Time.now.utc
         timings = run_benchmark
+        completed_at = Time.now.utc
+
         percentiles = Percentiles.new(timings)
         score = dataset_size / percentiles[50] / 1_000_000.0
 
         { name: self.class.bench_name,
           score: score,
-          percentiles: percentiles }
+          percentiles: percentiles,
+          started_at: started_at,
+          completed_at: completed_at }
       end
 
       private