feat(mcp): add clickstack_emerging_signals tool (+7% on service-health-check)

[brandon-pereira]

What

Adds a ClickStack MCP tool, clickstack_emerging_signals — a two-window Drain pattern novelty detector. It mines log/event patterns in a baseline window and a current window and set-differences them:

• emerging — present now, absent from baseline ("new") or >= minShareRatio× more frequent (default 3×)
• disappeared — the reverse ("gone" / "shifted")

It answers "what is new or gone?" — which the existing tools structurally can't.

Why

Tool	Question
clickstack_event_patterns	common patterns in one window?
clickstack_event_deltas	which attribute value distribution differs between two row groups?
clickstack_emerging_signals (new)	what patterns appeared / vanished over time?

event_deltas compares value distributions within a shared population, so it can't surface a brand-new log template (a feature-flag line that started an hour ago has no baseline distribution to shift — it just appears). The two are complementary: "what's different about these rows" vs "what's new since then".

How

Runs the event_patterns sample-and-mine pipeline twice (once per window) and set-differences the result:

1. Two non-overlapping windows in (current + earlier baseline; overlaps rejected).
2. Sample each window (default 10k rows) and Drain-mine into generalized templates, collapsing high-cardinality noise so template-level diffing works.
3. Match templates across windows by a normalized string (<*> unified, whitespace collapsed, lower-cased), since Drain assigns fresh cluster ids each run.
4. Express each pattern's frequency as a share of its window (comparable across volumes), then set-difference.

Calibration: minShareRatio (default 3×) ignores routine volume wobble; an empty emerging list is a valid "nothing novel" answer, and the tool is told not to fabricate findings.

Eval results

Dual-slot A/B vs main, identical seeded data (seed 42, same anchor), Opus judge (averaged over n=3 and n=5 runs).

Win — service-health-check, the scenario that exercises novelty detection:

branch	main	Δ
72%	62%	+10%

Driven by notes_new_log_pattern: 67% branch vs 0% main (main missed the planted new log template every run). Matches an earlier controlled measure: agent surfaced the signal in 0/10 runs without the tool, 8/10 with it.

No effect elsewhere. The other scenarios don't exercise novelty detection, and their deltas average to noise — no regressions:

Scenario	Δ
error-root-cause	~0 (both at ceiling)
latency-spike	+2%
noisy-signals	~0
segmented-regression	+2%

(These scenarios swing 20–40 pts per run, so small deltas are variance, not signal.)

Testing

tsc / eslint / knip clean on the feature files. Changeset included (@hyperdx/api minor).

[changeset-bot]

🦋 Changeset detected

Latest commit: 1362bb5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ready	Preview, Comment	Jul 22, 2026 4:33pm
hyperdx-storybook	Ready	Preview, Comment	Jul 22, 2026 4:33pm

[github-actions]

🔴 Tier 4 — Critical

Touches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD.

Why this tier:

• Large diff: 599 production lines changed (threshold: 400)

Additional context: agent branch (claude/emerging-signals-mcp-tool)

Review process: Deep review from a domain expert. Synchronous walkthrough may be required.
SLA: Schedule synchronous review within 2 business days.

Stats

• Production files changed: 3
• Production lines changed: 599 (+ 381 in test files, excluded from tier calculation)
• Branch: claude/emerging-signals-mcp-tool
• Author: brandon-pereira

To override this classification, remove the review/tier-4 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

[greptile-apps]

Greptile Summary

This PR adds a new MCP tool for comparing event patterns across two time windows. The main changes are:

• New clickstack_emerging_signals tool registration and documentation.
• Shared window-mining helper extracted from event-pattern queries.
• Pattern shift classification for emerging and disappeared templates.
• Unit and integration tests for threshold behavior and tool output.

Confidence Score: 5/5

This looks safe to merge.

• No blocking issues found in the changed code.
• The threshold fixes cover exact-boundary shifts and stable patterns with minShareRatio: 1.
• The extracted mining helper keeps the existing event-pattern response path intact.

Important Files Changed

Filename	Overview
packages/api/src/mcp/tools/query/emergingSignals.ts	Adds the emerging-signals tool, input schema, two-window mining flow, and shift classification.
packages/api/src/mcp/tools/query/runEventPatterns.ts	Extracts shared window mining while preserving the existing event-pattern formatting path.
packages/api/src/mcp/tools/query/index.ts	Registers the new emerging-signals query tool.
packages/api/src/mcp/tools/query/tests/emergingSignalsClassify.test.ts	Adds coverage for new, shifted, disappeared, boundary, and stable pattern classification.
packages/api/src/mcp/tests/emergingSignalsTool.int.test.ts	Adds integration coverage for tool schema exposure, window validation, and new-pattern detection.

_{Reviews (7): Last reviewed commit: "fix(mcp): require directional change in ..." | Re-trigger Greptile}

[github-actions]

E2E Test Results

✅ All tests passed • 243 passed • 1 skipped • 1058s

Status	Count
✅ Passed	243
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

[github-actions]

Deep Review

✅ No critical issues found. The new clickstack_emerging_signals tool is well-structured, the classifyShift core is thoroughly unit-tested (including float-boundary and directional-guard cases), bodyExpression is allowlist-sanitized, and both ClickHouse clients are closed in finally. No P0/P1 ship-blockers. The items below are recommended hardening and coverage gaps.

🟡 P2 -- recommended

• packages/api/src/mcp/tools/query/emergingSignals.ts:54 -- The disappeared branch returns on curShare === 0 with no minimum-baseline floor, while the mirror emerging path (baseShare === 0) requires curShare >= newPatternShareFloor; a one-off baseline template (single sampled row) that stops recurring is reported as gone and inflates disappearedCount, the exact one-off noise the emerging side filters.
  • Fix: Gate the curShare === 0 case on a baseline floor symmetric to newPatternShareFloor, e.g. baseShare >= 2 / baseRes.sampledCount.
• packages/api/src/mcp/tools/query/runEventPatterns.ts:60 -- getSource and getConnectionById are awaited outside the try/catch that converts failures into the { error } union, so a Mongo blip rejects mineWindowPatterns, rejects the handler's Promise.all, and surfaces as a generic error instead of the clean MCP error the file's own query/mining guards produce.
  • Fix: Wrap the source/connection lookups in the same try/catch returning mcpServerError(...).
  • _{correctness, adversarial}
• packages/api/src/mcp/tools/query/runEventPatterns.ts:252 -- New handler-level behavior is untested: normalizeTemplate (the cross-window join key that all diffing depends on), the additive ingest merge, and the window-overlap rejection have no coverage, so a keying or clobber regression would silently corrupt every emerging/disappeared verdict.
  • Fix: Add unit tests asserting normalizeTemplate keeps distinct placeholder placements distinct while collapsing whitespace/case, that same-key clusters accumulate rather than overwrite, and that touching windows pass while overlapping windows return mcpUserError.
  • _{testing, correctness}

🔵 P3 nitpicks (4)

• packages/api/src/mcp/tools/query/emergingSignals.ts:187 -- The overlap guard rejects only true overlap, not a baseline placed entirely after the current window, so transposed windows silently invert emerging/disappeared labels.
  • Fix: Also reject when base.startDate >= cur.endDate so the baseline must precede the current window.
  • _{correctness, adversarial}
• packages/api/src/mcp/tools/query/emergingSignals.ts:277 -- When the current window samples exactly one row, newPatternShareFloor becomes 2 while the max achievable curShare is 1, so every brand-new pattern is rejected and emerging is silently empty, yet the low-data warning only fires at sampledCount === 0.
  • Fix: Cap the floor below the max share, or extend the "NO DATA" warning to fire below a small sample threshold, not only at zero.
• packages/api/src/mcp/tools/query/runEventPatterns.ts:177 -- The mining queries pass only { max_execution_time: 30 }, diverging from the shared MCP_CLICKHOUSE_SETTINGS (readonly: 2 plus a client requestTimeout) that every other MCP query path uses, dropping a defense-in-depth guardrail on an agent-controlled where in SQL mode.
  • Fix: Reuse MCP_CLICKHOUSE_SETTINGS and set requestTimeout on the client to match the sibling query tools.
• packages/api/src/mcp/tools/query/emergingSignals.ts:204 -- Each emerging_signals call resolves the same source and connection twice (once per window) because mineWindowPatterns re-fetches them, doubling the Mongo lookups per invocation.
  • Fix: Resolve source/connection once and pass them into the per-window mining, or memoize within the call.

---

Reviewers (3): correctness, adversarial, testing.

Testing gaps: normalizeTemplate, ingest accumulation, window-overlap rejection, newPatternShareFloor derivation, ranking/topN slicing, and empty-window warning are all unexercised — only the pure classifyShift function is tested. (Note: the review ran with a degraded shell — git/Bash were unavailable — so the diff was reconstructed from file reads; the always-on maintainability/project-standards/agent-native personas could not be dispatched.)