feat: Claude Managed Agents for ClickStack alert investigations (+ EE extensibility seams)#2646
Open
jordan-simonovski wants to merge 14 commits into
Open
feat: Claude Managed Agents for ClickStack alert investigations (+ EE extensibility seams)#2646jordan-simonovski wants to merge 14 commits into
jordan-simonovski wants to merge 14 commits into
Conversation
When a Claude-typed alert fires, start an Anthropic managed-agent session in-process instead of POSTing an outbound webhook, inject a structured alert payload, and persist an AgentRun. A poll sweep on the check-alerts task detects idle sessions, fetches the agent's root-cause summary, and delivers it to the webhook's configured Slack URL. - AgentRun model: windowed dedupe key (per alert per cooldown window) + TTL so recurring alerts re-investigate; status lifecycle with a transient delivering claim and attempt cap. - pollAndDeliverAgentSessions: atomic claim so concurrent sweeps can't double-post, bounded delivery retries, GET-independent age-out, multi-block summary extraction with empty-summary retry, Slack section chunking. - startAgentSession: firing-edge-only, HTTPS Slack delivery guard, idempotent re-fire handling. - Provisioning: MCP reachability preflight before creating Anthropic resources; auto-allow the read-only ClickStack MCP toolset; idempotent agent delete (404). - MCP server: allowlist the configured public host so managed agents (and the tunnel) can reach it without disabling DNS-rebinding protection. - WebhookForm: Claude webhook URL is the Slack result destination (HTTPS/Slack validated); drop the dead body prefill.
Add a typed, fail-open extension registry (agentRunExtensions.ts) with three lifecycle hooks — onProvisionAgent (swap the standing system prompt), onSessionStart (swap/append the kickoff prompt, contribute run metadata) and onBeforeDelivery (contribute Slack footer links) — plus a downstream-owned registration point (extensions/index.ts) that upstream never edits, and an additive AgentRun.metadata (Mixed) field. Hook runners isolate per-extension failures and emit a span per invocation plus a low-cardinality outcome counter (hyperdx.agent_run.extension_events), so a broken extension never blocks starting a session or delivering to Slack. With no extensions registered and the default webhook body, behaviour is unchanged. Claude webhooks now honour their user-editable body template as the agent kickoff prompt (previously documented but ignored), falling back to the built-in enriched payload for empty, blank-rendering, or broken templates. The seam result types are consumed only by hyperdx-ee, so they are tagged @public and knip is configured with tags:[-public] to recognise intentional public API. Pre-existing model exports with no downstream consumer (IAgentRun, AgentRunStatus, IManagedAgent, IAnthropicIntegration, AnthropicIntegrationDocument, ManagedAgentData) are unexported/removed so the unused-export check passes honestly rather than by loosening it. Also maps the newly-added AlertState.PENDING in the enriched webhook status mapping, which upstream left unmapped and which broke the branch type-check.
Add a "Designing for Downstream (EE) Extensibility" section to AGENTS.md capturing the seam patterns (extension registries, downstream-owned registration files, additive metadata, swappable defaults) and the @public / knip convention for seam exports that only downstream consumes, plus a Key Principles bullet. Include the changeset for the managed-agent extension seams.
Replace the bare 'anthropic'/'openai' provider string literals (in config, the getAIModel switch, and the managed-agent key resolver) with an AIProvider enum, so the accepted provider values have one source of truth. Behaviour is unchanged — the enum members are the same env strings.
…ropicKey seam The managed-agent Anthropic key is now resolved from the environment (AI_API_KEY when AI_PROVIDER is Anthropic, else the legacy ANTHROPIC_API_KEY) via a new resolveAnthropicKey extension seam (fail-open, last-registered wins). A downstream distribution can register a resolver to supply a per-team key, which takes precedence over the env key. Remove the per-team, UI-managed key from open source: the AnthropicIntegration model and the /anthropic-key GET/PUT/DELETE endpoints are gone. Per-team key storage (the security-sensitive, encrypted-at-rest surface) is now a downstream concern injected through the seam. A non-Anthropic AI_API_KEY is never sent to api.anthropic.com. Tests inject the key through the resolver seam.
The managed-agent Anthropic key is resolved from the server environment in open source, so the key-entry card, the useAnthropicKey/useSaveAnthropicKey/ useDeleteAnthropicKey hooks, and the hasKey gating are removed. The Agents section shows the create-agent form when the feature is enabled and notes the env vars; provisioning surfaces a clear error if no key is configured. Per-team, UI-managed keys move downstream (hyperdx-ee).
These two dashboard-template JSONs were dropped in by an earlier work-in-progress commit but never wired into dashboardTemplates/index.ts and are unreferenced anywhere, so they never shipped as templates. Remove them.
🦋 Changeset detectedLatest commit: 3459c1d The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
github-actions
Bot
added
the
review/tier-4
Contributor
🔴 Tier 4 — CriticalTouches auth, data models, config, tasks, OTel pipeline, ClickHouse, or CI/CD. Why this tier:
Review process: Deep review from a domain expert. Synchronous walkthrough may be required. Stats
|
Contributor
Greptile SummaryThis PR adds Claude Managed Agents for automated ClickStack alert investigations: when a "Claude" webhook fires, an in-process Anthropic agent session is started, it investigates via the ClickStack MCP server, and delivers an evidence-linked summary to Slack. An extension seam layer (
Confidence Score: 5/5Safe to merge — the feature flag defaults to off and all previously flagged issues are resolved. The new code is well-structured and feature-gated — the entire surface no-ops when HDX_MANAGED_AGENTS_ENABLED is off, so existing alert delivery is completely unaffected. All four issues raised in the previous review round are addressed. The only remaining findings are style-level inconsistencies (numeric fields quoted as strings in the default template, and the MCP allowlist not gated on the feature flag), neither of which affects correctness or security. No files require special attention before merging. WebhookForm.tsx and mcp/app.ts have minor inconsistencies worth a follow-up but are not blocking. Important Files Changed
Sequence Diagram%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant CA as checkAlerts cron
participant T as template.ts
participant AA as anthropicAgents.ts
participant Anthropic as Anthropic API
participant Mongo as MongoDB
participant Slack as Slack
CA->>T: renderAlertTemplate state ALERT
T->>T: notifyChannel webhook service Claude
T->>AA: handleStartAgentSession(webhook, message)
AA->>AA: check IS_MANAGED_AGENTS_ENABLED
AA->>AA: compileClaudeWebhookBody or buildAgentPrompt
AA->>AA: startAgentSession(teamId, prompt, deliverToUrl)
AA->>Mongo: AgentRun.findOne(dedupeKey)
AA->>Anthropic: POST /v1/sessions
AA->>Anthropic: POST /v1/sessions/:id/events kickoff
AA->>Mongo: AgentRun.create(dedupeKey, sessionId)
Note over CA,Slack: next check-alerts tick
CA->>AA: pollAndDeliverAgentSessions()
AA->>Mongo: find status running
AA->>Anthropic: GET /v1/sessions/:id poll status
alt session idle
AA->>Mongo: findOneAndUpdate status delivering
AA->>Anthropic: GET /v1/sessions/:id/events fetch summary
AA->>Slack: postMessageToWebhook summary
AA->>Mongo: status delivered
else still running or timed out
AA->>Mongo: failRun or skip
end
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant CA as checkAlerts cron
participant T as template.ts
participant AA as anthropicAgents.ts
participant Anthropic as Anthropic API
participant Mongo as MongoDB
participant Slack as Slack
CA->>T: renderAlertTemplate state ALERT
T->>T: notifyChannel webhook service Claude
T->>AA: handleStartAgentSession(webhook, message)
AA->>AA: check IS_MANAGED_AGENTS_ENABLED
AA->>AA: compileClaudeWebhookBody or buildAgentPrompt
AA->>AA: startAgentSession(teamId, prompt, deliverToUrl)
AA->>Mongo: AgentRun.findOne(dedupeKey)
AA->>Anthropic: POST /v1/sessions
AA->>Anthropic: POST /v1/sessions/:id/events kickoff
AA->>Mongo: AgentRun.create(dedupeKey, sessionId)
Note over CA,Slack: next check-alerts tick
CA->>AA: pollAndDeliverAgentSessions()
AA->>Mongo: find status running
AA->>Anthropic: GET /v1/sessions/:id poll status
alt session idle
AA->>Mongo: findOneAndUpdate status delivering
AA->>Anthropic: GET /v1/sessions/:id/events fetch summary
AA->>Slack: postMessageToWebhook summary
AA->>Mongo: status delivered
else still running or timed out
AA->>Mongo: failRun or skip
end
Reviews (3): Last reviewed commit: "fix(app): persist the Claude default bod..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
Deep ReviewScope: ✅ No critical issues found. The new surface is feature-gated, team-scoped on every endpoint, and the delivery/MCP/provisioning URLs are HTTPS/SSRF-checked. The recommendations below are reliability and maintainability polish, not merge blockers. 🟡 P2 — recommended
🔵 P3 nitpicks (2)
Reviewers (12): correctness, security, adversarial, reliability, api-contract, performance, kieran-typescript, testing, maintainability, project-standards, agent-native, learnings-researcher. Testing gaps: The poll/deliver state machine has several concurrency edges worth explicit coverage — the |
Contributor
E2E Test Results✅ All tests passed • 238 passed • 3 skipped • 1575s
Tests ran across 4 shards in parallel. |
Reliability + gating fixes from the Greptile and Deep reviews:
- Bound every outbound Anthropic call with an AbortController timeout and the
one Slack delivery per run with a local timeout, so a hung endpoint can no
longer stall the shared check-alerts sweep (the poll loop runs inline on that
cron). Kept the loop sequential — running it concurrently would cross-
contaminate per-run setBusinessContext trace attribution.
- Roll back partially-provisioned Anthropic resources (best-effort delete the
environment and vault) when a later provisioning step or the local persist
fails, so a partial failure doesn't leave orphans the user can't remove.
- On losing the dedup race (duplicate-key), best-effort delete the now-untracked
Anthropic session instead of leaving it running and consuming quota.
- Feature-gate the runtime paths on IS_MANAGED_AGENTS_ENABLED: the poll sweep
and handleStartAgentSession no-op when off, and creating a service=claude
webhook is rejected. Enable the flag in .env.test so the suites exercise it.
- Use ISO-8601 timestamps (startTimeISO/endTimeISO) in webhook templates so the
default Claude body matches buildAgentPrompt instead of emitting Unix ms.
- Add a compound {status, createdAt} index backing the poll query.
- Document encryption.ts / HDX_ENCRYPTION_KEY as downstream (EE) scaffolding
(OSS reads the key from env), fix the stale IS_MANAGED_AGENTS_ENABLED comment,
and replace the committed personal ngrok subdomain with a placeholder.
The default Claude webhook body's time_range used {{startTime}}/{{endTime}}
(raw Unix ms), so the agent received stringified integers. Switch to the new
{{startTimeISO}}/{{endTimeISO}} variables (added to the shared template
variable set and the picker) so the default matches the built-in agent prompt's
ISO times.
Address the follow-up Greptile/Deep review findings: - Add WebhookService.Claude to the external-API v2 externalWebhookSchema union so Claude webhooks serialise instead of 500ing (and are no longer omitted from list responses while still counted). - Delete the vault (holds the team's ClickStack bearer credential) and the environment when an agent is deleted, not just the agent object, so removing an agent doesn't orphan the access key on Anthropic. - Cap the poll sweep's wall-clock (MAX_SWEEP_DURATION_MS) so it can't block the next check-alerts tick during an incident storm; leftover runs defer to the next sweep. - Clean up the Anthropic session on ANY post-creation failure in startAgentSession (not just the dedup race), and reuse isDuplicateKeyError. - Roll back the created agent too (not just env/vault) on provisioning failure. - Add an AbortController timeout to verifyMcpReachable, matching the other calls. - Type the runMetadata merge (drop the Object.assign any), match the DELETE response shape to sibling endpoints, and refresh the stale WebhookService comment + managed-agents changeset (env key, not encrypted-at-rest).
onSubmit's default-body block had Generic and IncidentIO but not Claude, so an unedited Claude webhook was tested with CLAUDE_WEBHOOK_BODY but saved with an empty body. Add the Claude case so save matches test.
| # Anthropic requires a PUBLIC HTTPS URL for the agent's MCP server (its cloud | ||
| # sandbox reaches it directly), so localhost won't work. For local testing, | ||
| # expose your instance's /api/mcp via a tunnel and set it here, e.g.: | ||
| HDX_MANAGED_AGENTS_MCP_URL=https://YOUR-SUBDOMAIN.ngrok-free.app/api/mcp |
Member
There was a problem hiding this comment.
chore: We should document these values in a markdown file or doc specific to how to setup these agents - there isn't really any user facing documentation for this feature atm.
Additionally, thoughts on putting this behind a feature flag while we test it internally? NVM: I see that we don't - its behind a FF that is true by default. We may want to set it to false? Thoughts?
For the docs, recommend suggesting .env.local at the root for storing values
| them). Use the shared helpers in | ||
| `packages/api/src/utils/instrumentation.ts`. See | ||
| [`agent_docs/observability.md`](agent_docs/observability.md). | ||
| 8. **EE extensibility**: this repo is upstream of an enterprise fork — build |
Member
There was a problem hiding this comment.
issue: We should not reference EE from OSS, same for the PR description.
Member
polish(non-blocking): Didn't realize I created an agent (wasn't scroll down enough) - we should disable the create agent button and/or change to "Created!" |
Member
|
idea: currently the claude agent implementation is very coupled to slack webook type - it would be cool if instead we just let someone choose a webhook type from the agent UI (ex SRE Responder [ Slack Webhook |
| {service === WebhookService.Generic && [ | ||
| {(service === WebhookService.Generic || | ||
| service === WebhookService.Claude) && [ | ||
| <label className=".mantine-TextInput-label" key="1"> |
Member
There was a problem hiding this comment.
issue: Why do we show webhook headers, body, etc on the Claude agent but you can't specify a webhook URL (only Slack URL)? The Slack alert type does not support these headers
Member
|
issue: I was able to get this working - feedback - If I wanted to get alerted immediately when something happens, I currently need to duplicate the alert so it fires an alert (ie on call notification). it would be nice if it fired webhooks like this:
Currently I get a slack message with a solution to a problem I wasn't aware of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds Claude Managed Agents that automatically investigate firing ClickStack
alerts and deliver an evidence-linked root-cause summary to Slack, provisioned
and managed in-product — plus an extensibility layer so the enterprise fork
(hyperdx-ee) can extend the flow without editing OSS files.
Why
When an alert fires, an on-call engineer usually starts the same investigation
by hand: re-run the alert query, pull related logs/traces/metrics, check recent
deploys, follow the runbook. This wires up a managed Claude agent (against the
team's ClickStack MCP server) to do that first pass automatically and post the
summary to Slack, so responders start from findings instead of a blank page.
The feature is upstream of an enterprise fork, so it's built with extension
seams from day one rather than inline hooks EE would have to fork — see the new
"Designing for Downstream (EE) Extensibility" section in
AGENTS.md.What's in it
vault + MCP credential) from Team Settings, with an MCP-reachability preflight.
Claudewebhook service starts an in-process agentsession on the firing edge, dedupes per alert/window, and a poll loop delivers
the summary to Slack when the session idles (with retries + attempt caps).
agentRunExtensions.ts, fail-open + instrumented):onProvisionAgent(swap system prompt),onSessionStart(swap/append kickoffprompt, add run metadata),
onBeforeDelivery(Slack footer links), andresolveAnthropicKey(per-team key). A downstream-ownedextensions/index.tsis the registration point OSS never edits;
AgentRun.metadatais the additiveextension-data field.
bodytemplate as the agent prompt (previously documented but ignored), falling back
to the built-in enriched payload for empty/blank/broken templates.
AI_API_KEYwithAI_PROVIDER=anthropic, or legacyANTHROPIC_API_KEY; via a newAIProviderenum). The per-team, UI-managed key (encrypted in Mongo) is removed from OSS
and becomes a downstream concern injected through
resolveAnthropicKey— thesecurity-sensitive surface stays out of open source.
How OSS → EE extensibility works
The whole point of the seam is that hyperdx-ee extends this flow without
editing any OSS file — so upstream merges into the fork never conflict.
The seam pattern (why merges stay conflict-free):
flowchart TB subgraph OSS["OSS core (upstream — EE never edits)"] flow["anthropicAgents.ts<br/>managed-agent alert flow"] runner["agentRunExtensions.ts<br/>hook runners (fail-open + instrumented)"] reg["extensions/index.ts<br/>registration point — empty in OSS"] flow -->|"calls run*Extensions()"| runner runner -->|"iterates registered extensions"| list[("registered<br/>extensions[]")] end subgraph EE["hyperdx-ee (downstream fork)"] eereg["extensions/index.ts<br/>(body REPLACED wholesale)"] ext["EE extension<br/>implements AgentRunExtension"] eereg -->|"registerAgentRunExtension()"| ext end ext -.->|"pushes into"| list reg -.->|"same file, replaced not edited<br/>→ no merge conflict"| eereg classDef oss fill:#e8f0fe,stroke:#4285f4; classDef ee fill:#fef7e0,stroke:#f9ab00; class flow,runner,reg,list oss; class eereg,ext ee;EE only ever replaces the body of
extensions/index.ts(a file upstream commitsto never editing) and implements the
AgentRunExtensioncontract. The core flowand the hook runners are untouched, so upstream merges never conflict on them.
Where the four hooks fire along the alert lifecycle:
sequenceDiagram participant Alert as Firing alert participant Flow as anthropicAgents.ts participant Hooks as run*Extensions() participant Ext as EE extension (0..n) participant Anthropic participant Slack Note over Flow,Ext: Provision (once, per agent) Flow->>Hooks: runProvisionExtensions(defaultSystemPrompt) Hooks->>Ext: onProvisionAgent() Ext-->>Hooks: systemPrompt? (override) Hooks-->>Flow: resolved system prompt Note over Flow,Anthropic: Key resolution Flow->>Hooks: runAnthropicKeyExtensions(teamId) Hooks->>Ext: resolveAnthropicKey() Ext-->>Hooks: apiKey? (per-team) Hooks-->>Flow: key — else fall back to env key Alert->>Flow: alert fires → start session Note over Flow,Ext: Session start Flow->>Hooks: runSessionStartExtensions(prompt) Hooks->>Ext: onSessionStart() Ext-->>Hooks: promptOverride? / promptSuffix? / runMetadata? Hooks-->>Flow: kickoff prompt (+ persisted AgentRun.metadata) Flow->>Anthropic: send kickoff Note over Flow,Slack: Delivery (every attempt — idempotent) Flow->>Hooks: runDeliveryExtensions(summary) Hooks->>Ext: onBeforeDelivery() Ext-->>Hooks: footerLinks? Hooks-->>Flow: footer links Flow->>Slack: post summary + footerEach hook is fail-open — a throwing extension is recorded (span + counter + log)
and contributes nothing; the core flow never fails because of an extension. With
zero extensions registered (the OSS default) every runner returns a neutral
value and behaviour is exactly as before the seams.
Notes
HDX_MANAGED_AGENTS_ENABLED=true. Safe to merge disabled.resolveAnthropicKeyscaffold are stashed in the hyperdx-ee repo for thedownstream re-add (after the CHC security review of per-team key storage).
@hyperdx/api,@hyperdx/app).webhook body), behaviour is unchanged.
Testing
agentRunExtensions(9),anthropicAgents(42),renderAlertTemplate(82),ai(17) — all green.make ci-lintandknipgreen across all packages.resolves HDX-4344