feat(evals): add deploy-regression scenario (measure organic metric tool adoption)

[karl-power]

Why

metric-saturation measures whether an agent can use metric tools when the scenario forces it to. deploy-regression measures the complement: organic metric-tool adoption — the incident is fully solvable from traces + logs, the planted metrics only corroborate, and the transcript (adoption) rubric records whether the agent reaches for metric tools it doesn't strictly need, without letting that inflate the outcome score.

Summary

• Scenario: checkout-api staged rollout of 2.0.0-rc1 pauses at 3/6 pods; the new build throws a TypeError on fixed-amount promo codes (FREESHIP, FLAT5, GIFT10) → ~7-8% of checkouts 500, only on updated pods, only for fixed-amount codes.
• Nothing pre-labeled: no service.version stamping — the smoking gun is a k8s.pod.name cross-tab on failed checkouts joined against the deployment.rollout event logs; promo type has no label and the TypeError stack is error-sampled (~1-in-7).
• Calibration traps: two co-timed ones (an innocent completed payment-service rollout minutes before the onset, and a deploy-introduced benign DeprecationWarning flood stepping at the same boundaries as the errors) plus background distractors (healthy payment child spans, constant caught-exception noise, pre-deploy 502 burst, noisy-but-benign queue-depth gauge).
• Metrics corroborate only (keyed by service, not pod/version): they confirm the onset but cannot reveal the pod split or the defect — ignoring metric tools entirely is a valid solve path, which is what makes measured adoption organic.
• Also included: eval-viewer "copy batch name" button next to the batch picker.

Eval results (claude-opus-4-6)

Metric	hyperdx	clickhouse	Δ (clickhouse)
Combined score	64%	81%	+16pp
Programmatic	69%	75%	+6pp
Judge mean (weighted)	64%	94%	+30pp
Adoption (organic metric-tool use)	0%	0%	—
Tool calls (mean)	19.7	25.0	+5.3
Wall clock (s, mean)	232.5	244.0	+11.5
Termination	final_answer 3/3	final_answer 3/3	—
N	3	3	—

• The organic-adoption readout: neither arm touched a metric tool (0% adoption) — with traces/logs sufficient, agents skip metrics entirely; this is the baseline future MCP ergonomics changes can move.
• Judge mean drives the gap: hyperdx answers lost points on correctness/evidence grounding (e.g. blames_the_deploy 33% vs 100%), not on trap calibration (all false-blame negatives clean on both arms).

References

• Linear Issue: Closes HDX-4787
• Related PRs: feat(evals): harden the metric-saturation scenario #2717

[changeset-bot]

🦋 Changeset detected

Latest commit: 10d6f38

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@hyperdx/hdx-eval	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ignored	Preview	Jul 24, 2026 3:16pm
hyperdx-storybook	Ignored	Preview	Jul 24, 2026 3:16pm

[github-actions]

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

• Large diff: 1279 production lines changed (threshold: 1000)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 6
• Production lines changed: 1279 (+ 758 in test files, excluded from tier calculation)
• Branch: karl/hdx-eval-deploy-regression-scenario
• Author: karl-power

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[greptile-apps]

Greptile Summary

Adds a deploy-regression evaluation scenario for measuring organic metric-tool adoption.

• Generates deterministic traces, logs, and corroborating metrics for a staged checkout-api rollout regression.
• Adds grading rubrics and comprehensive scenario tests, including regression coverage for false-blame phrasing.
• Registers and documents the scenario.
• Adds a batch-name copy control to the evaluation viewer and prevents overlapping feedback timers.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failures remain.

Important Files Changed

Filename	Overview
packages/hdx-eval/src/scenarios/deploy-regression/generate.ts	Defines the deterministic staged-rollout regression telemetry, distractors, prompt customization, and scenario configuration.
packages/hdx-eval/src/scenarios/deploy-regression/ground-truth.json	Defines outcome, false-blame, transcript-adoption, and judge criteria for the new scenario.
packages/hdx-eval/src/tests/deploy-regression.test.ts	Verifies telemetry invariants, scenario difficulty, rubric behavior, and prompt budget configuration.
packages/hdx-eval/scripts/viewer/public/app.js	Implements batch-name clipboard feedback while cancelling the previous reset timer on repeated clicks.
packages/hdx-eval/src/scenarios/index.ts	Registers deploy-regression with the evaluation scenario catalog.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  G[deploy-regression generator] --> T[Deterministic traces and logs]
  G --> M[Corroborating metrics]
  T --> A[Agent investigation]
  M --> A
  A --> P[Programmatic outcome checks]
  A --> J[LLM judge]
  A --> R[Transcript adoption checks]
  P --> S[Combined outcome score]
  J --> S
  R --> O[Separate adoption readout]

Loading

_{Reviews (3): Last reviewed commit: "feat(evals): add deploy-regression scena..." | Re-trigger Greptile}

[github-actions]

E2E Test Results

✅ All tests passed • 242 passed • 1 skipped • 963s

Status	Count
✅ Passed	242
❌ Failed	0
⚠️ Flaky	1
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

Deep Review

✅ No critical issues found.

This PR adds a self-contained eval scenario (deploy-regression) — synthetic trace/log/metric generation plus its ground-truth/rubric — and one additive "copy batch name" button in the eval-viewer. No production runtime code paths, auth, data mutations, or migrations are touched. The generator's RNG usage, time/rollout arithmetic, defect-sampling, and streaming flush logic are internally consistent and correct; the viewer button handles clipboard failure and timer resets cleanly.

🟡 P2 -- recommended

• packages/hdx-eval/src/scenarios/deploy-regression/generate.ts:530 -- the new scenario's generate() has nontrivial structural invariants (staggered onset, exactly 3/6 failing pods, ~1-in-7 stack sampling, fixed vs. percent promo split) but no visible dedicated coverage; the volumeFactor/collectScenario test hooks suggest a shared parametrized suite exists.
  • Fix: Confirm a shared scenario test exercises deploy-regression at low volumeFactor and asserts its key invariants, or add one alongside the sibling scenarios.
  • _{testing-reviewer}

🔵 P3 nitpicks (2)

• .changeset/eval-deploy-regression-scenario.md:2 -- the summary is labeled feat(...) but the bump is patch; a new scenario is a feature-level addition, so minor is the conventional level (negligible impact for this internal package).

  • Fix: Change the bump to minor if the package follows conventional-commit semver, or leave as-is if patch bumps are intentional for internal tooling.

• packages/hdx-eval/src/scenarios/deploy-regression/generate.ts:358 -- the corroborating 5xx metric uses Math.max(baselineFail, round(reqs * failFrac)), so once the defect is active the baseline error contribution is dropped rather than summed, making the metric marginally under-count total 5xx versus the trace floor.

  • Fix: Use addition (baselineFail + round(reqs * failFrac)) if exact aggregate-vs-trace parity matters; otherwise leave it, since metrics are corroborating-only by design.

---

Reviewers (7): correctness, testing, maintainability, project-standards, agent-native, kieran-typescript, julik-frontend-races.

Testing gaps: Confirm the shared scenario suite (or a new test) asserts deploy-regression structural invariants — staggered error onset, 3-of-6 failing-pod split, and the deterministic ~1-in-7 stack-trace sampling — at reduced volumeFactor.

_{Environment note: git/bash and network tools were unavailable this run, so the review covered the working-tree state of the six changed files (generate.ts, ground-truth.json, scenarios/index.ts, viewer app.js/index.html, changeset) rather than a computed base-SHA diff. The change is additive, so this is substantially complete, but line-level deltas for the two modified viewer files were inferred from current content.}

[brandon-pereira]

Love the idea of this PR, a couple feedback items:

• I don't think the way we measure metric tool adoption is fair on ClickHouse, because it checks for calls to the ClickStack MCP but ClickHouse runs cant call those tools, maybe we check if the tool args have the specific metrics in them instead?
• Not for this PR - but can you explore what changes need to be made to improve the ClickStack MCP? Is it system prompting changes to promote our tools OR new tools that make metrics easier to work with?