Skip to content

Add telemetry metrics for dropped/skipped events - #12319

Open
jpbempel wants to merge 3 commits into
masterfrom
jpbempel/add-debugger-telemetry
Open

Add telemetry metrics for dropped/skipped events#12319
jpbempel wants to merge 3 commits into
masterfrom
jpbempel/add-debugger-telemetry

Conversation

@jpbempel

@jpbempel jpbempel commented Aug 27, 2026

Copy link
Copy Markdown
Member

What Does This Do

when snapshots are dropped (Queue full) or skipped (rate limit or eval timeout) we are sending telemetry metrics with te number of events (snapshots) involved. metrics are tagged by reason

we are removing the old DebuggerMetrics and the skip cause to replace by this new telemetry metrics.
More telemetry metrics will be added later

Motivation

Additional Notes

Contributor Checklist

Jira ticket: DEBUG-5839

when snapshots are dropped (Queue full) or skipped (rate limit or
eval timeout) we are sending telemetry metrics with te number of
events (snapshots) involved. metrics are tagged by reason

we are removing the old DebuggerMetrics and the skip cause to replace
by this new telemetry metrics.
More telemetry metrics will be added later
@jpbempel
jpbempel requested review from a team as code owners August 27, 2026 15:05
@jpbempel
jpbempel requested review from dudikeleti and sarahchen6 and removed request for a team August 27, 2026 15:05
@jpbempel jpbempel added the comp: debugger Dynamic Instrumentation label Aug 27, 2026
@dd-octo-sts

dd-octo-sts Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@jpbempel jpbempel added type: refactoring type: feature Enhancements and improvements labels Aug 27, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 560d92d527

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@datadog-prod-us1-4 datadog-prod-us1-4 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Datadog Autotest: FAIL

The metric collector can lose event counts during a concurrent reset. Some timeout and disabled-session paths also omit or use an incorrect skip reason.

Open Bits AI session

🤖 Datadog Autotest · Commit 560d92d · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest

@datadog-prod-us1-4

datadog-prod-us1-4 Bot commented Aug 27, 2026

Copy link
Copy Markdown

🎯 Code Coverage (details)
Patch Coverage: 92.86%
Overall Coverage: 58.80% (+0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 288b633 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Aug 27, 2026

Copy link
Copy Markdown

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1787862580 1787862907
end_time 2026-08-27T20:31:11 2026-08-27T20:36:38
git_branch master jpbempel/add-debugger-telemetry
git_commit_sha 34aaf8a 288b633
start_time 2026-08-27T20:29:41 2026-08-27T20:35:08
See matching parameters
Baseline Candidate
ci_job_id 1991996609 1991996609
ci_pipeline_id 133773588 133773588
cpu_model Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
git_commit_date 1787861896 1787861896

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 9 metrics, 6 unstable metrics.

See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-41766.838ns; +43214.024ns] or [-12.164%; +12.585%]
unstable
[-79.009µs; +34.893µs] or [-19.713%; +8.706%]
unstable
[-89.699µs; +42.169µs] or [-21.458%; +10.088%]
unstable
[-288.543µs; +444.155µs] or [-22.743%; +35.009%]
same
scenario:basic same same same unstable
[-66.484µs; +310.548µs] or [-6.433%; +30.051%]
unstable
[-120.311op/s; +216.929op/s] or [-5.534%; +9.979%]
scenario:loop same same same same same
Request duration reports for reports
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (400.79 µs) : 345, 456
.   : milestone, 401,
basic (347.473 µs) : 342, 353
.   : milestone, 347,
loop (8.121 ms) : 8057, 8185
.   : milestone, 8121,
section candidate
noprobe (378.732 µs) : 326, 431
.   : milestone, 379,
basic (341.821 µs) : 333, 351
.   : milestone, 342,
loop (8.115 ms) : 8052, 8179
.   : milestone, 8115,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 400.79 µs [345.161 µs, 456.419 µs]
basic 347.473 µs [342.094 µs, 352.852 µs]
loop 8.121 ms [8.057 ms, 8.185 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 378.732 µs [326.269 µs, 431.196 µs]
basic 341.821 µs [333.045 µs, 350.597 µs]
loop 8.115 ms [8.052 ms, 8.179 ms]

@dd-octo-sts

dd-octo-sts Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.80 s 14.57 s [+0.7%; +2.4%] (maybe worse)
startup:insecure-bank:tracing:Agent 13.62 s 13.64 s [-1.0%; +0.6%] (no difference)
startup:petclinic:appsec:Agent 17.37 s 17.29 s [-0.5%; +1.4%] (no difference)
startup:petclinic:iast:Agent 17.48 s 17.46 s [-0.6%; +0.8%] (no difference)
startup:petclinic:profiling:Agent 17.35 s 17.20 s [-0.2%; +1.9%] (no difference)
startup:petclinic:sca:Agent 17.48 s 17.42 s [-0.5%; +1.2%] (no difference)
startup:petclinic:tracing:Agent 16.53 s 16.67 s [-2.0%; +0.4%] (no difference)

Commit: 288b633a · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@sarahchen6 sarahchen6 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like Jacoco is complaining about test coverage, but otherwise LGTM

make INSTANCE private
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: debugger Dynamic Instrumentation type: feature Enhancements and improvements type: refactoring

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants