[BENCHMARK] sync instrument recording benchmarks - #4470
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4470 +/- ##
=======================================
Coverage 83.09% 83.09%
=======================================
Files 519 519
Lines 20253 20253
=======================================
Hits 16827 16827
Misses 3426 3426 🚀 New features to boost your workflow:
|
…ad. General cleanup and ordering. Fix thread safety for random generator use.
…and bound instruments
dbarker
left a comment
There was a problem hiding this comment.
Some interesting findings from this benchmark that warrant a followup.
- Need to optimize the drop aggregation path to behave like a disabled metrics stream
- Need to investigate the performance regression for unbound instruments when building with ABIv2 and the bound instrument preview.
| // BM_Record_Counter_Disabled_ByThreads/threads:4 0.335 ns 0.335 ns 1860780336 | ||
| // BM_Record_Counter_Drop_ByThreads/threads:1 181 ns 181 ns 3883111 | ||
| // BM_Record_Counter_Drop_ByThreads/threads:2 336 ns 336 ns 3412070 | ||
| // BM_Record_Counter_Drop_ByThreads/threads:4 548 ns 519 ns 1265888 |
There was a problem hiding this comment.
There is an opportunity to reduce recording impact with drop aggregation to behave as if the stream was disabled. Currently the sync storage computes the attribute hash and gains locks when drop aggregation is set.
ABIv1: Disabled vs. Drop (by threads recording with one instrument)
| Instrument | Disabled | Drop | Overhead vs. Disabled |
|---|---|---|---|
| Counter (threads:1) | 0.223 ns | 181 ns | 812x |
| Counter (threads:2) | 0.334 ns | 336 ns | 1006x |
| Counter (threads:4) | 0.335 ns | 548 ns | 1636x |
| Histogram Explicit (threads:1) | 0.215 ns | 180 ns | 837x |
| Histogram Explicit (threads:2) | 0.215 ns | 198 ns | 921x |
| Histogram Explicit (threads:4) | 0.227 ns | 556 ns | 2450x |
| // BM_Record_Counter_Disabled_ByThreads/threads:1 0.225 ns 0.225 ns 3106874136 | ||
| // BM_Record_Counter_Disabled_ByThreads/threads:2 0.235 ns 0.235 ns 2399463178 | ||
| // BM_Record_Counter_Disabled_ByThreads/threads:4 0.235 ns 0.235 ns 3078807728 | ||
| // BM_Record_Counter_Drop_ByThreads/threads:1 291 ns 291 ns 2434016 |
There was a problem hiding this comment.
When building with ABIv2 and OPENTELEMETRY_HAVE_METRICS_BOUND_INSTRUMENTS_PREVIEW = ON there is a significant performance regression with unbound sync instrument recording compared to ABIv1.
| Benchmark (threads:1) | ABIv1 no bound preview | ABIv2 bound preview ON | Delta |
|---|---|---|---|
| Counter Drop | 181 ns | 291 ns | +61% |
| Counter Sum | 183 ns | 293 ns | +60% |
| Histogram Drop | 180 ns | 288 ns | +60% |
| Histogram Explicit | 192 ns | 297 ns | +55% |
| Histogram Base2Expo | 209 ns | 300 ns | +44% |
| // BM_Record_Histogram_Explicit_Exemplar_AlwaysOn 453 ns 453 ns 1573692 | ||
| // BM_Record_Histogram_Base2Expo_Exemplar_AlwaysOff 303 ns 303 ns 2274437 | ||
| // BM_Record_Histogram_Base2Expo_Exemplar_AlwaysOn 453 ns 453 ns 1550092 | ||
| // BM_Record_BoundCounter_Disabled_ByThreads/threads:1 0.221 ns 0.221 ns 3142560276 |
There was a problem hiding this comment.
The speed up for bound instruments compared to unbound is impressive at ~16-50x.
ABIv2: Unbound vs. Bound recording overhead (single thread) with nominal attributes (3)
| Instrument | Unbound | Bound | Speedup |
|---|---|---|---|
| Counter Drop | 291 ns | 5.81 ns | 50x |
| Counter Sum | 293 ns | 10.7 ns | 27x |
| Histogram Drop | 288 ns | 6.19 ns | 47x |
| Histogram Explicit Buckets | 297 ns | 11.8 ns | 25x |
| Histogram Base2 Exponential | 300 ns | 19.3 ns | 16x |
| // BM_Record_Counter_Sum_ByThreads/threads:1 183 ns 183 ns 3725731 | ||
| // BM_Record_Counter_Sum_ByThreads/threads:2 313 ns 313 ns 1731700 | ||
| // BM_Record_Counter_Sum_ByThreads/threads:4 658 ns 605 ns 1074776 | ||
| // BM_Record_Counter_Sum_ByAttributes/0 21.7 ns 21.7 ns 32259034 |
There was a problem hiding this comment.
The attribute count has the highest impact during recording and cardinality is not a major factor for latency.
ABIv1 Counter Sum: Attribute count vs. Cardinality
Attribute count (single time series, varying key count per call)
| Attribute count | 0 | 1 | 10 | 128 |
|---|---|---|---|---|
| Counter Sum | 21.7 ns | 63.7 ns | 665 ns | 11,495 ns |
| Histogram Explicit Buckets | 25.4 ns | 73.7 ns | 662 ns | 10,666 ns |
| Histogram Base2 Exponential | 27.4 ns | 72.1 ns | 662 ns | 10,619 ns |
Cardinality (3 attributes, N distinct time series, cycling through all)
| Active time series | 10 | 500 | 2000 | 4000 (overflow) |
|---|---|---|---|---|
| Counter Sum | 189 ns | 187 ns | 185 ns | 195 ns |
| Histogram Explicit Buckets | 202 ns | 202 ns | 205 ns | — |
| Histogram Base2 Exponential | 207 ns | 203 ns | 206 ns | — |
Contributes to #4317 and #4416
Adds sync instruments recording benchmarks covering all instrument types and aggregation types. Includes Exemplar filters and bound instruments when enabled.
Changes
For significant contributions please make sure you have completed the following items:
CHANGELOG.mdupdated for non-trivial changes