Skip to content

[BENCHMARK] sync instrument recording benchmarks - #4470

Open
dbarker wants to merge 14 commits into
open-telemetry:mainfrom
dbarker:benchmark_sync_instruments
Open

[BENCHMARK] sync instrument recording benchmarks#4470
dbarker wants to merge 14 commits into
open-telemetry:mainfrom
dbarker:benchmark_sync_instruments

Conversation

@dbarker

@dbarker dbarker commented Aug 21, 2026

Copy link
Copy Markdown
Member

Contributes to #4317 and #4416

Adds sync instruments recording benchmarks covering all instrument types and aggregation types. Includes Exemplar filters and bound instruments when enabled.

Changes

  • Disabled instrument benchmarks (Counter, Histogram, Gauge, and bound variants)
  • Counter benchmarks
    • Drop aggregation
    • Sum aggregation (by attribute count, by cardinality, by thread count)
  • Histogram benchmarks
    • Drop aggregation
    • Explicit bucket and Base2 aggregation (by attribute count, by cardinality, by thread count)
  • Gauge benchmarks
    • Drop aggregation
    • Last value aggregation (by attribute count, by cardinality, by thread count)
  • ExemplarFilter
    • Counter with AlwaysOff, AlwaysOn, and TraceBased filters
    • Histogram (Default and Base2Expo) with AlwaysOff and AlwaysOn filters
  • Bound instruments (counter and histogram)
    • Drop aggregation
    • Sum aggregation (by thread count)
    • Histogram (default and base2) aggregation (by thread count)

For significant contributions please make sure you have completed the following items:

  • CHANGELOG.md updated for non-trivial changes
  • Unit tests have been added
  • Changes in public API reviewed

@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.09%. Comparing base (cd252b4) to head (cb8bd08).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #4470   +/-   ##
=======================================
  Coverage   83.09%   83.09%           
=======================================
  Files         519      519           
  Lines       20253    20253           
=======================================
  Hits        16827    16827           
  Misses       3426     3426           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dbarker
dbarker marked this pull request as ready for review August 22, 2026 20:13
@dbarker
dbarker requested a review from a team as a code owner August 22, 2026 20:13

@dbarker dbarker left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some interesting findings from this benchmark that warrant a followup.

  • Need to optimize the drop aggregation path to behave like a disabled metrics stream
  • Need to investigate the performance regression for unbound instruments when building with ABIv2 and the bound instrument preview.

// BM_Record_Counter_Disabled_ByThreads/threads:4 0.335 ns 0.335 ns 1860780336
// BM_Record_Counter_Drop_ByThreads/threads:1 181 ns 181 ns 3883111
// BM_Record_Counter_Drop_ByThreads/threads:2 336 ns 336 ns 3412070
// BM_Record_Counter_Drop_ByThreads/threads:4 548 ns 519 ns 1265888

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is an opportunity to reduce recording impact with drop aggregation to behave as if the stream was disabled. Currently the sync storage computes the attribute hash and gains locks when drop aggregation is set.

ABIv1: Disabled vs. Drop (by threads recording with one instrument)

Instrument Disabled Drop Overhead vs. Disabled
Counter (threads:1) 0.223 ns 181 ns 812x
Counter (threads:2) 0.334 ns 336 ns 1006x
Counter (threads:4) 0.335 ns 548 ns 1636x
Histogram Explicit (threads:1) 0.215 ns 180 ns 837x
Histogram Explicit (threads:2) 0.215 ns 198 ns 921x
Histogram Explicit (threads:4) 0.227 ns 556 ns 2450x

// BM_Record_Counter_Disabled_ByThreads/threads:1 0.225 ns 0.225 ns 3106874136
// BM_Record_Counter_Disabled_ByThreads/threads:2 0.235 ns 0.235 ns 2399463178
// BM_Record_Counter_Disabled_ByThreads/threads:4 0.235 ns 0.235 ns 3078807728
// BM_Record_Counter_Drop_ByThreads/threads:1 291 ns 291 ns 2434016

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When building with ABIv2 and OPENTELEMETRY_HAVE_METRICS_BOUND_INSTRUMENTS_PREVIEW = ON there is a significant performance regression with unbound sync instrument recording compared to ABIv1.

Benchmark (threads:1) ABIv1 no bound preview ABIv2 bound preview ON Delta
Counter Drop 181 ns 291 ns +61%
Counter Sum 183 ns 293 ns +60%
Histogram Drop 180 ns 288 ns +60%
Histogram Explicit 192 ns 297 ns +55%
Histogram Base2Expo 209 ns 300 ns +44%

// BM_Record_Histogram_Explicit_Exemplar_AlwaysOn 453 ns 453 ns 1573692
// BM_Record_Histogram_Base2Expo_Exemplar_AlwaysOff 303 ns 303 ns 2274437
// BM_Record_Histogram_Base2Expo_Exemplar_AlwaysOn 453 ns 453 ns 1550092
// BM_Record_BoundCounter_Disabled_ByThreads/threads:1 0.221 ns 0.221 ns 3142560276

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The speed up for bound instruments compared to unbound is impressive at ~16-50x.

ABIv2: Unbound vs. Bound recording overhead (single thread) with nominal attributes (3)

Instrument Unbound Bound Speedup
Counter Drop 291 ns 5.81 ns 50x
Counter Sum 293 ns 10.7 ns 27x
Histogram Drop 288 ns 6.19 ns 47x
Histogram Explicit Buckets 297 ns 11.8 ns 25x
Histogram Base2 Exponential 300 ns 19.3 ns 16x

// BM_Record_Counter_Sum_ByThreads/threads:1 183 ns 183 ns 3725731
// BM_Record_Counter_Sum_ByThreads/threads:2 313 ns 313 ns 1731700
// BM_Record_Counter_Sum_ByThreads/threads:4 658 ns 605 ns 1074776
// BM_Record_Counter_Sum_ByAttributes/0 21.7 ns 21.7 ns 32259034

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The attribute count has the highest impact during recording and cardinality is not a major factor for latency.

ABIv1 Counter Sum: Attribute count vs. Cardinality

Attribute count (single time series, varying key count per call)

Attribute count 0 1 10 128
Counter Sum 21.7 ns 63.7 ns 665 ns 11,495 ns
Histogram Explicit Buckets 25.4 ns 73.7 ns 662 ns 10,666 ns
Histogram Base2 Exponential 27.4 ns 72.1 ns 662 ns 10,619 ns

Cardinality (3 attributes, N distinct time series, cycling through all)

Active time series 10 500 2000 4000 (overflow)
Counter Sum 189 ns 187 ns 185 ns 195 ns
Histogram Explicit Buckets 202 ns 202 ns 205 ns
Histogram Base2 Exponential 207 ns 203 ns 206 ns

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant