Skip to content

Add native AWS CloudWatch and Azure Monitor alert ingress - #2

Open
sarora-eightfold wants to merge 3 commits into
masterfrom
add-aws-support
Open

Add native AWS CloudWatch and Azure Monitor alert ingress#2
sarora-eightfold wants to merge 3 commits into
masterfrom
add-aws-support

Conversation

@sarora-eightfold

@sarora-eightfold sarora-eightfold commented Jul 31, 2026

Copy link
Copy Markdown

Adds two native alert-ingress integrations so CloudWatch and Azure Monitor can reach GoAlert directly, with no Lambda, forwarder or middleware. They are deliberate siblings: same handler shape, same payload-mapper structure, same dedup discipline, same smoke-test layout.

Neither can use the existing generic endpoint, for different reasons — SNS is blocked on transport and handshake, Azure on payload shape.

ticket: https://eightfoldai.atlassian.net/browse/ENG-206319

AWS CloudWatch (via SNS)

CloudWatch alarms reach us through an SNS topic, but GoAlert could not be subscribed to one directly for three independent reasons: SNS requires the endpoint to fetch a SubscribeURL to confirm the subscription and nothing did that, so zero messages were ever delivered; the SNS envelope has no summary field and carries the alarm as a JSON string inside Message; and SNS always sends text/plain, which genericapi.ServeCreateAlert ignores because it only unmarshals application/json. That last point is upstream target#4463.

Add a cloudwatch integration key type and a handler at POST /api/v2/cloudwatch/incoming that performs the subscription handshake and verifies the SNS message signature, so a topic can be subscribed directly with no Lambda or forwarder in between.

The key type is a full integration rather than a reuse of TypeGeneric because the webhook URL shown in the UI is generated server-side from the key type alone (IntegrationKey.Href). Reusing TypeGeneric would hand the user a /api/v2/generic/incoming URL that silently fails to confirm as an SNS subscription -- reproducing the exact bug this change fixes. Because the UI is driven entirely by IntegrationKeyTypes and Href, no frontend changes are needed.

Notable implementation details:

  • The body is parsed regardless of Content-Type, and bounded with MaxBytesReader rather than io.LimitReader so an oversized body yields a 413 instead of silently truncating into a misleading 400.
  • Signature verification is split into pure functions (canonical string, verify, cert parse) so they are testable with no I/O. Subject is a *string because AWS omits the field from the string-to-sign entirely when absent, which a plain string cannot distinguish from an empty value.
  • Both outbound fetches are host-allowlisted with an anchored pattern, and the client blocks redirects: the allowlist only covers the first hop, so a single 302 from an allowlisted host would otherwise reach link-local addresses.
  • The signing-cert cache is bounded with FIFO eviction because the cert URL path is attacker-supplied and would otherwise grow without limit.
  • A freshness window on the signed Timestamp bounds replay of a captured envelope. Tradeoff: retries arriving over an hour late are rejected.
  • Dedup is hex sha256(AlarmName), matching the CloudWatch alarm Lambdas that post to PagerDuty, so one alarm cannot produce two alerts across ingress paths. It is never nil, since a nil dedup silently falls back to a content hash that changes on every state transition and would break both idempotency and the OK close.
  • NewStateReason is capped before assembling details so a verbose reason cannot push AlarmDescription, which carries the runbook URL, past the length limit.
  • INSUFFICIENT_DATA and a stray OK with no open alert both create nothing and return 2xx; non-2xx is reserved for infrastructure failure so SNS retries only when a retry could help.

Azure Monitor

Azure Monitor delivers alerts by having an action group POST a webhook. Azure sends application/json so it clears the content-type gate that blocks SNS, but ServeCreateAlert expects a flat body while Azure nests everything under data.essentials / data.alertContext with no top-level summary — so every delivery would create an alert with a blank summary rather than an error. Nothing maps Azure's alertId onto the dedup field either.

Adds an azureMonitor key type and a handler at POST /api/v2/azuremonitor/incoming.

Unlike SNS there is no subscription handshake and no signature, so this handler makes no outbound requests at all — there is no analogue of cloudwatch's host allowlist or certificate cache. The integration key in the URL is therefore the only credential, which makes the webhook URL credential-grade; the docs say so explicitly for this key type.

Parsing:

  • Only the common alert schema is accepted. A legacy-schema payload is rejected with a message naming the fix (enable the common alert schema on the receiver) rather than degraded to the fallback, which would produce content-free alerts with no indication why. 400 rather than 5xx, since Azure does not retry 4xx and a misconfigured receiver is a permanent condition.
  • Dispatch is on the presence of condition.allOf, not a conditionType allowlist. Every metric and log criteria shape shares that envelope, so one code path covers SingleResourceMultipleMetricCriteria, MultipleResourceMultipleMetricCriteria (used by multi-resource and resource-group-scoped rules), DynamicThresholdCriteria and WebtestLocationAvailabilityCriteria. conditionType still selects the two behaviours that genuinely differ: suppressing the dynamic threshold, which is a sensitivity artifact rather than a limit, and the log query/link lines.
  • Prometheus rule groups get their own branch — that shape carries no conditionType and no condition, only expression/labels/annotations.
  • Service Health and activity-log payloads render from properties, with HTML stripped and string-containing-JSON fields left opaque.
  • Anything unrecognised builds a best-effort alert from essentials, which is present on every payload regardless of type, and logs the signalType/monitorService/conditionType triple so a newly-routed alert type announces itself instead of silently producing thin alerts.

signalType is deliberately not the discriminator. Platform and Prometheus metric alerts both report signalType: "Metric", and Log Alerts V2, Azure Backup and ActivityLog Administrative all report "Log". Branching on it would feed unrelated payloads to the wrong renderer.

Dedup is sha256(essentials.alertId). Azure alerts are stateful, so one alert object carries the whole lifecycle and the Fired and Resolved deliveries share an alertId — which is what lets the Resolved delivery close the alert its Fired delivery opened. originAlertId is deliberately not used: it is per-rule for metric alerts, so a single missed close would hold the dedup key and mute that rule permanently. Status comes from essentials.monitorCondition, never alertContext.status, which can disagree with it because the underlying incident resolved while the alert fired.

A json.UnmarshalTypeError on an individual field is tolerated rather than fatal. Azure documents threshold and dimension values as strings but is not consistent across shapes, and a hard failure means a 400 — which Azure does not retry — so one oddly-typed field would lose the page instead of one value.


Verification

CloudWatch — verified end-to-end against live AWS SNS: the subscription confirms, a real alarm creates one alert with the runbook URL preserved in details, and re-delivery is suppressed as a duplicate.

Azure Monitor — the target tenant's inventory was measured path-agnostically across all resource types, rather than by assuming where the action-group reference lives (it differs per rule type, which is how Prometheus was initially missed): 495 metric rules, 8 log (V2) rules and 3 Prometheus rule groups route to a PagerDuty action group. All three shapes are natively parsed.

Both packages have table-driven unit tests — the first unit tests in any GoAlert ingress package — plus a smoke test each. go build ./..., go vet and gofmt are clean and the full suite passes.

New CW option in integrations:

Screenshot 2026-08-01 at 7 36 21 PM Screenshot 2026-07-31 at 4 31 40 PM Screenshot 2026-08-01 at 7 36 04 PM

Sarthak Arora and others added 2 commits July 31, 2026 16:54
CloudWatch alarms reach us through an SNS topic, but GoAlert could not be
subscribed to one directly for three independent reasons: SNS requires the
endpoint to fetch a SubscribeURL to confirm the subscription and nothing did
that, so zero messages were ever delivered; the SNS envelope has no `summary`
field and carries the alarm as a JSON string inside `Message`; and SNS always
sends `text/plain`, which genericapi.ServeCreateAlert ignores because it only
unmarshals `application/json`. That last point is upstream target#4463.

Add a `cloudwatch` integration key type and a handler at
POST /api/v2/cloudwatch/incoming that performs the subscription handshake and
verifies the SNS message signature, so a topic can be subscribed directly with
no Lambda or forwarder in between.

The key type is a full integration rather than a reuse of TypeGeneric because
the webhook URL shown in the UI is generated server-side from the key type
alone (IntegrationKey.Href). Reusing TypeGeneric would hand the user a
/api/v2/generic/incoming URL that silently fails to confirm as an SNS
subscription -- reproducing the exact bug this change fixes. Because the UI is
driven entirely by IntegrationKeyTypes and Href, no frontend changes are
needed.

Notable implementation details:

- The body is parsed regardless of Content-Type, and bounded with
  MaxBytesReader rather than io.LimitReader so an oversized body yields a 413
  instead of silently truncating into a misleading 400.
- Signature verification is split into pure functions (canonical string,
  verify, cert parse) so they are testable with no I/O. `Subject` is a *string
  because AWS omits the field from the string-to-sign entirely when absent,
  which a plain string cannot distinguish from an empty value.
- Both outbound fetches are host-allowlisted with an anchored pattern, and the
  client blocks redirects: the allowlist only covers the first hop, so a single
  302 from an allowlisted host would otherwise reach link-local addresses.
- The signing-cert cache is bounded with FIFO eviction because the cert URL
  path is attacker-supplied and would otherwise grow without limit.
- A freshness window on the signed Timestamp bounds replay of a captured
  envelope. Tradeoff: retries arriving over an hour late are rejected.
- Dedup is hex sha256(AlarmName), matching the CloudWatch alarm Lambdas that
  post to PagerDuty, so one alarm cannot produce two alerts across ingress
  paths. It is never nil, since a nil dedup silently falls back to a content
  hash that changes on every state transition and would break both idempotency
  and the OK close.
- NewStateReason is capped before assembling details so a verbose reason cannot
  push AlarmDescription, which carries the runbook URL, past the length limit.
- INSUFFICIENT_DATA and a stray OK with no open alert both create nothing and
  return 2xx; non-2xx is reserved for infrastructure failure so SNS retries
  only when a retry could help.

Tests: table-driven unit tests for the canonical string, allowlist (including
the unanchored-suffix bypass), signature and freshness, and the alarm mapping;
plus a smoke test that generates an RSA key, serves a self-signed cert from an
httptest server, and drives the real crypto and allowlist paths end to end.
These are the first unit tests in any GoAlert ingress package.

Verified against live AWS SNS: subscription confirms, an alarm creates one
alert with the runbook URL preserved, and re-delivery is suppressed as a
duplicate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Azure Monitor delivers alerts by having an action group POST a webhook. Those
webhooks point at PagerDuty's Events API today; this adds a native GoAlert
destination, as a sibling to the existing cloudwatch integration.

The generic endpoint cannot serve this: Azure sends application/json so it
clears the content-type gate, but ServeCreateAlert expects a flat
{summary, details, action, dedup, meta} body while Azure nests everything under
data.essentials / data.alertContext with no top-level summary -- so every
delivery would create an alert with a blank summary rather than an error. There
is also nothing mapping Azure's alertId onto the dedup field.

Unlike SNS there is no subscription handshake and no signature, so the handler
makes no outbound requests at all -- there is no analogue of cloudwatch's host
allowlist or certificate cache. The integration key in the URL is the only
credential, which makes the webhook URL credential-grade; the docs say so
explicitly for this key type.

Parsing:

- Only the common alert schema is accepted. A legacy-schema payload is rejected
  with a message naming the fix (enable the common alert schema on the receiver)
  rather than being degraded to the fallback, which would produce content-free
  alerts with no indication why. 400 rather than 5xx, since Azure does not retry
  4xx and a misconfigured receiver is a permanent condition.
- Dispatch is on the presence of condition.allOf, not on a conditionType
  allowlist. Every metric and log criteria shape shares that envelope, so this
  covers SingleResource, MultipleResource (used by multi-resource and
  resource-group-scoped rules), DynamicThreshold and WebtestLocationAvailability
  with one code path. conditionType still selects the two behaviours that
  genuinely differ: suppressing the dynamic threshold, which is a sensitivity
  artifact rather than a limit, and the log query/link lines.
- Prometheus rule groups get their own branch: the shape carries no
  conditionType and no condition, only expression/labels/annotations.
- Service Health and activity-log payloads render from properties, with HTML
  stripped and string-containing-JSON fields left opaque.
- Anything unrecognised builds a best-effort alert from essentials, which is
  present on every payload regardless of type, and logs the
  signalType/monitorService/conditionType triple so a newly-routed alert type
  announces itself instead of silently producing thin alerts.

signalType is deliberately not the discriminator: Platform and Prometheus metric
alerts share signalType "Metric", and Log Alerts V2, Azure Backup and
ActivityLog Administrative all share "Log". Branching on it would feed unrelated
payloads to the wrong renderer.

Dedup is sha256(essentials.alertId). Azure alerts are stateful, so one alert
object carries the whole lifecycle and the Fired and Resolved deliveries share an
alertId -- which is what lets the Resolved delivery close the alert its Fired
delivery opened. originAlertId is deliberately not used: it is per-rule for
metric alerts, so a single missed close would hold the dedup key and mute that
rule permanently. Status comes from essentials.monitorCondition, never
alertContext.status, which can disagree with it because the underlying incident
resolved while the alert fired.

A json.UnmarshalTypeError on an individual field is tolerated rather than fatal.
Azure documents threshold and dimension values as strings but is not consistent
across shapes, and a hard failure means a 400, which Azure does not retry -- so
one oddly-typed field would lose the page instead of one value.

Verified against the target tenant's inventory, measured path-agnostically
across all resource types rather than by assuming where the action-group
reference lives: 495 metric rules, 8 log (V2) rules and 3 Prometheus rule groups
route to a PagerDuty action group. All three shapes are natively parsed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@sarora-eightfold sarora-eightfold changed the title Add native AWS CloudWatch (via SNS) alert ingress Add native AWS CloudWatch and Azure Monitor alert ingress Aug 1, 2026
Comment thread cloudwatch/cloudwatch.go Dismissed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants