Add native AWS CloudWatch and Azure Monitor alert ingress - #2
Open
sarora-eightfold wants to merge 3 commits into
Open
Add native AWS CloudWatch and Azure Monitor alert ingress#2sarora-eightfold wants to merge 3 commits into
sarora-eightfold wants to merge 3 commits into
Conversation
CloudWatch alarms reach us through an SNS topic, but GoAlert could not be subscribed to one directly for three independent reasons: SNS requires the endpoint to fetch a SubscribeURL to confirm the subscription and nothing did that, so zero messages were ever delivered; the SNS envelope has no `summary` field and carries the alarm as a JSON string inside `Message`; and SNS always sends `text/plain`, which genericapi.ServeCreateAlert ignores because it only unmarshals `application/json`. That last point is upstream target#4463. Add a `cloudwatch` integration key type and a handler at POST /api/v2/cloudwatch/incoming that performs the subscription handshake and verifies the SNS message signature, so a topic can be subscribed directly with no Lambda or forwarder in between. The key type is a full integration rather than a reuse of TypeGeneric because the webhook URL shown in the UI is generated server-side from the key type alone (IntegrationKey.Href). Reusing TypeGeneric would hand the user a /api/v2/generic/incoming URL that silently fails to confirm as an SNS subscription -- reproducing the exact bug this change fixes. Because the UI is driven entirely by IntegrationKeyTypes and Href, no frontend changes are needed. Notable implementation details: - The body is parsed regardless of Content-Type, and bounded with MaxBytesReader rather than io.LimitReader so an oversized body yields a 413 instead of silently truncating into a misleading 400. - Signature verification is split into pure functions (canonical string, verify, cert parse) so they are testable with no I/O. `Subject` is a *string because AWS omits the field from the string-to-sign entirely when absent, which a plain string cannot distinguish from an empty value. - Both outbound fetches are host-allowlisted with an anchored pattern, and the client blocks redirects: the allowlist only covers the first hop, so a single 302 from an allowlisted host would otherwise reach link-local addresses. - The signing-cert cache is bounded with FIFO eviction because the cert URL path is attacker-supplied and would otherwise grow without limit. - A freshness window on the signed Timestamp bounds replay of a captured envelope. Tradeoff: retries arriving over an hour late are rejected. - Dedup is hex sha256(AlarmName), matching the CloudWatch alarm Lambdas that post to PagerDuty, so one alarm cannot produce two alerts across ingress paths. It is never nil, since a nil dedup silently falls back to a content hash that changes on every state transition and would break both idempotency and the OK close. - NewStateReason is capped before assembling details so a verbose reason cannot push AlarmDescription, which carries the runbook URL, past the length limit. - INSUFFICIENT_DATA and a stray OK with no open alert both create nothing and return 2xx; non-2xx is reserved for infrastructure failure so SNS retries only when a retry could help. Tests: table-driven unit tests for the canonical string, allowlist (including the unanchored-suffix bypass), signature and freshness, and the alarm mapping; plus a smoke test that generates an RSA key, serves a self-signed cert from an httptest server, and drives the real crypto and allowlist paths end to end. These are the first unit tests in any GoAlert ingress package. Verified against live AWS SNS: subscription confirms, an alarm creates one alert with the runbook URL preserved, and re-delivery is suppressed as a duplicate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Azure Monitor delivers alerts by having an action group POST a webhook. Those
webhooks point at PagerDuty's Events API today; this adds a native GoAlert
destination, as a sibling to the existing cloudwatch integration.
The generic endpoint cannot serve this: Azure sends application/json so it
clears the content-type gate, but ServeCreateAlert expects a flat
{summary, details, action, dedup, meta} body while Azure nests everything under
data.essentials / data.alertContext with no top-level summary -- so every
delivery would create an alert with a blank summary rather than an error. There
is also nothing mapping Azure's alertId onto the dedup field.
Unlike SNS there is no subscription handshake and no signature, so the handler
makes no outbound requests at all -- there is no analogue of cloudwatch's host
allowlist or certificate cache. The integration key in the URL is the only
credential, which makes the webhook URL credential-grade; the docs say so
explicitly for this key type.
Parsing:
- Only the common alert schema is accepted. A legacy-schema payload is rejected
with a message naming the fix (enable the common alert schema on the receiver)
rather than being degraded to the fallback, which would produce content-free
alerts with no indication why. 400 rather than 5xx, since Azure does not retry
4xx and a misconfigured receiver is a permanent condition.
- Dispatch is on the presence of condition.allOf, not on a conditionType
allowlist. Every metric and log criteria shape shares that envelope, so this
covers SingleResource, MultipleResource (used by multi-resource and
resource-group-scoped rules), DynamicThreshold and WebtestLocationAvailability
with one code path. conditionType still selects the two behaviours that
genuinely differ: suppressing the dynamic threshold, which is a sensitivity
artifact rather than a limit, and the log query/link lines.
- Prometheus rule groups get their own branch: the shape carries no
conditionType and no condition, only expression/labels/annotations.
- Service Health and activity-log payloads render from properties, with HTML
stripped and string-containing-JSON fields left opaque.
- Anything unrecognised builds a best-effort alert from essentials, which is
present on every payload regardless of type, and logs the
signalType/monitorService/conditionType triple so a newly-routed alert type
announces itself instead of silently producing thin alerts.
signalType is deliberately not the discriminator: Platform and Prometheus metric
alerts share signalType "Metric", and Log Alerts V2, Azure Backup and
ActivityLog Administrative all share "Log". Branching on it would feed unrelated
payloads to the wrong renderer.
Dedup is sha256(essentials.alertId). Azure alerts are stateful, so one alert
object carries the whole lifecycle and the Fired and Resolved deliveries share an
alertId -- which is what lets the Resolved delivery close the alert its Fired
delivery opened. originAlertId is deliberately not used: it is per-rule for
metric alerts, so a single missed close would hold the dedup key and mute that
rule permanently. Status comes from essentials.monitorCondition, never
alertContext.status, which can disagree with it because the underlying incident
resolved while the alert fired.
A json.UnmarshalTypeError on an individual field is tolerated rather than fatal.
Azure documents threshold and dimension values as strings but is not consistent
across shapes, and a hard failure means a 400, which Azure does not retry -- so
one oddly-typed field would lose the page instead of one value.
Verified against the target tenant's inventory, measured path-agnostically
across all resource types rather than by assuming where the action-group
reference lives: 495 metric rules, 8 log (V2) rules and 3 Prometheus rule groups
route to a PagerDuty action group. All three shapes are natively parsed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds two native alert-ingress integrations so CloudWatch and Azure Monitor can reach GoAlert directly, with no Lambda, forwarder or middleware. They are deliberate siblings: same handler shape, same payload-mapper structure, same dedup discipline, same smoke-test layout.
Neither can use the existing
genericendpoint, for different reasons — SNS is blocked on transport and handshake, Azure on payload shape.ticket: https://eightfoldai.atlassian.net/browse/ENG-206319
AWS CloudWatch (via SNS)
CloudWatch alarms reach us through an SNS topic, but GoAlert could not be subscribed to one directly for three independent reasons: SNS requires the endpoint to fetch a SubscribeURL to confirm the subscription and nothing did that, so zero messages were ever delivered; the SNS envelope has no
summaryfield and carries the alarm as a JSON string insideMessage; and SNS always sendstext/plain, which genericapi.ServeCreateAlert ignores because it only unmarshalsapplication/json. That last point is upstream target#4463.Add a
cloudwatchintegration key type and a handler at POST /api/v2/cloudwatch/incoming that performs the subscription handshake and verifies the SNS message signature, so a topic can be subscribed directly with no Lambda or forwarder in between.The key type is a full integration rather than a reuse of TypeGeneric because the webhook URL shown in the UI is generated server-side from the key type alone (IntegrationKey.Href). Reusing TypeGeneric would hand the user a /api/v2/generic/incoming URL that silently fails to confirm as an SNS subscription -- reproducing the exact bug this change fixes. Because the UI is driven entirely by IntegrationKeyTypes and Href, no frontend changes are needed.
Notable implementation details:
Subjectis a *string because AWS omits the field from the string-to-sign entirely when absent, which a plain string cannot distinguish from an empty value.Azure Monitor
Azure Monitor delivers alerts by having an action group POST a webhook. Azure sends
application/jsonso it clears the content-type gate that blocks SNS, butServeCreateAlertexpects a flat body while Azure nests everything underdata.essentials/data.alertContextwith no top-level summary — so every delivery would create an alert with a blank summary rather than an error. Nothing maps Azure'salertIdonto the dedup field either.Adds an
azureMonitorkey type and a handler at POST /api/v2/azuremonitor/incoming.Unlike SNS there is no subscription handshake and no signature, so this handler makes no outbound requests at all — there is no analogue of cloudwatch's host allowlist or certificate cache. The integration key in the URL is therefore the only credential, which makes the webhook URL credential-grade; the docs say so explicitly for this key type.
Parsing:
condition.allOf, not aconditionTypeallowlist. Every metric and log criteria shape shares that envelope, so one code path coversSingleResourceMultipleMetricCriteria,MultipleResourceMultipleMetricCriteria(used by multi-resource and resource-group-scoped rules),DynamicThresholdCriteriaandWebtestLocationAvailabilityCriteria.conditionTypestill selects the two behaviours that genuinely differ: suppressing the dynamic threshold, which is a sensitivity artifact rather than a limit, and the log query/link lines.conditionTypeand nocondition, onlyexpression/labels/annotations.properties, with HTML stripped and string-containing-JSON fields left opaque.essentials, which is present on every payload regardless of type, and logs thesignalType/monitorService/conditionTypetriple so a newly-routed alert type announces itself instead of silently producing thin alerts.signalTypeis deliberately not the discriminator. Platform and Prometheus metric alerts both reportsignalType: "Metric", and Log Alerts V2, Azure Backup and ActivityLog Administrative all report"Log". Branching on it would feed unrelated payloads to the wrong renderer.Dedup is
sha256(essentials.alertId). Azure alerts are stateful, so one alert object carries the whole lifecycle and the Fired and Resolved deliveries share analertId— which is what lets the Resolved delivery close the alert its Fired delivery opened.originAlertIdis deliberately not used: it is per-rule for metric alerts, so a single missed close would hold the dedup key and mute that rule permanently. Status comes fromessentials.monitorCondition, neveralertContext.status, which can disagree with it because the underlying incident resolved while the alert fired.A
json.UnmarshalTypeErroron an individual field is tolerated rather than fatal. Azure documentsthresholdand dimension values as strings but is not consistent across shapes, and a hard failure means a 400 — which Azure does not retry — so one oddly-typed field would lose the page instead of one value.Verification
CloudWatch — verified end-to-end against live AWS SNS: the subscription confirms, a real alarm creates one alert with the runbook URL preserved in details, and re-delivery is suppressed as a duplicate.
Azure Monitor — the target tenant's inventory was measured path-agnostically across all resource types, rather than by assuming where the action-group reference lives (it differs per rule type, which is how Prometheus was initially missed): 495 metric rules, 8 log (V2) rules and 3 Prometheus rule groups route to a PagerDuty action group. All three shapes are natively parsed.
Both packages have table-driven unit tests — the first unit tests in any GoAlert ingress package — plus a smoke test each.
go build ./...,go vetandgofmtare clean and the full suite passes.New CW option in integrations: