DRIVERS-3620 Reduce OpenTelemetry tracing overhead, benchmark driver configurations - #3108
Draft
comandeo-mongo wants to merge 11 commits into
Draft
comandeo-mongo wants to merge 11 commits into
comandeo-mongo wants to merge 11 commits into
Conversation
DriverBench only ever measured the default configuration, so no automated benchmark executed the OpenTelemetry code path at all and a large regression in it went unnoticed. Make the driver configuration an axis of the benchmark instead of a property of the environment. A result is now identified by the pair (task, configuration), so every configuration becomes its own time series that can be watched for regressions independently. Comparison runs the tasks under each configuration and records the throughput given up relative to the baseline as a metric of its own, rather than leaving it to be derived by comparing two time series later: recording it directly makes it watchable, and computing it within a run cancels host-to-host variation. Configurations are measured one per subprocess, because OpenTelemetry cannot be reconfigured once its SDK has been installed and the "api-only" configuration requires that the SDK was never loaded. They are run interleaved rather than one to completion and then the next, so that a machine which slows down partway through a run does not put one configuration's samples in the fast half and another's in the slow half and report the difference as overhead. The SDK configurations install no span processor. The cost to attribute to the driver is that of creating and recording spans; a processor would additionally charge the SDK's SpanData conversion, background thread and queue to the driver's account, and add variance that hides small driver-side changes. Sampled spans are still fully recorded without one. Task filters are included because the full suite spends at least a minute per micro-benchmark per configuration, which is too slow for an optimize-and-remeasure loop. Composites are skipped when the task list is filtered, since they average a fixed list of micro-benchmarks.
The comparison is too long to run on a laptop: it runs the tasks once per configuration, several times over, and each micro-benchmark has a 60 second floor. It also has to run undisturbed, since the quantity being measured is a difference between configurations and a machine that sleeps or throttles partway through reports that as overhead. Runs a bounded set of micro-benchmarks rather than the whole suite so that the task fits in a sensible wall clock: the four high-signal single- and multi-doc tasks, which create one operation span and one command span per operation and so show the per-span cost most clearly. Parallel tasks are excluded as disk- and concurrency-bound, and BSON tasks are excluded by Comparison itself since they never reach a server and create no spans. All three of the task list, the configuration list and the repetition count are expansions, so a patch can widen them without a config change. The results are uploaded as a task artifact rather than sent to the performance store. What is wanted here is a baseline to compare later runs against by hand, not a trend series. The task needs its own exec_timeout_secs; the global 5400 is not enough.
…to OperationTracer
4 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DRIVERS-3620
Enabling OpenTelemetry cost up to 26% of throughput on small-document tasks, almost all of it in driver code that ran before the sampling decision was known. This PR removes that cost and adds benchmarking to catch this class of regression.
Tracing hot path:
Benchmarking:
off,api-only,sdk-never,sdk-parent-1pct,sdk-always).rake driver_bench:compareruns them interleaved, one configuration per process, and records each configuration's overhead against the untraced baseline as its own metric.driver-bench-otelruns the comparison on a bounded set of high-signal tasks.Measured on Evergreen (overhead vs
off, before -> after this PR):Spec change documenting the guidelines and benchmark configurations: mongodb/specifications#1986
Testing: tracer unit specs, OTel unified spec tests against a local replica set (
OTEL_SPEC_TESTS=1), a local allocation mini-bench (traced command allocations fell from ~+115/op to ~+10/op in the non-recording configurations), and two Evergreen comparison runs.