Skip to content

audit_export_status heartbeat: emit on state change + slow keepalive (currently 1/min/agent, ~1440 events/agent/day of noise) #280

Description

@initializ-mk

Problem

Every agent emits an audit_export_status heartbeat once per minute:

{"ts":"2026-07-11T16:36:14Z","event":"audit_export_status","schema_version":"1.0","org_id":"org_…","workspace_id":"ws_…","entity_id":"agt-…","entity_type":"agent","fields":{"sinks":[{"connected":0,"drops_dial":0,"drops_timeout":0,"name":"stderr","writes_ok":2179},{"connected":1,"drops_dial":1,"drops_timeout":0,"name":"unix-socket","writes_ok":2178}]}}

That's ~1,440 events per agent per day in steady state:

  • At fleet scale, heartbeats become the dominant row count in the audit-events collection (storage + index pressure in security-next for a signal that's overwhelmingly "still fine").
  • Any bounded read window fills with them — the console's per-agent Activity view (last 50 events) was ~50 minutes of heartbeats crowding out actual behavior events. (The console now excludes audit_export_status from its builder-facing fetches as a mitigation, but the volume problem is at the source.)

Why not just "emit on change" or just "every 15 min"

  • Pure emit-on-change is the wrong shape for an audit-integrity signal. This event exists to prove audit events aren't being silently dropped. Silent-when-healthy makes "no events for an hour" ambiguous: healthy, or heartbeat/pipeline dead? Silence-as-success is the anti-pattern this event is supposed to prevent.
  • "Change" must mean state transitions anywaywrites_ok is a cumulative counter that increments every interval, so naive payload-diffing changes nothing. Change = connected flips or drops_dial/drops_timeout increments.
  • Pure 15-min interval fixes volume (→ 96/day) but delays detection of a disconnected sink or a drop burst by up to 15 minutes — the one moment the event earns its keep.

Proposal: hybrid

  1. Emit immediately on state change: any sink's connected transitions, or any drops_* counter increments.
  2. Otherwise emit a keepalive on a slow interval — 15 min (or hourly), configurable (e.g. AUDIT_STATUS_KEEPALIVE_INTERVAL).
  3. Optionally tag the reason so consumers can distinguish: fields.reason: "keepalive" | "state_change".

Healthy steady state: 1,440 → 24–96 events/agent/day. Failures surface within seconds. Liveness stays provable (a missing keepalive past the interval is itself a signal consumers can alert on).

Consumers

  • security-next / console: no changes required; the console already treats audit_export_status as operational telemetry (visible in Security · Events, excluded from builder-facing Activity windows).
  • audit-forwarder: pass-through, unaffected.
  • If the heartbeat participates in the per-entity prev_hash chain, reducing frequency just makes chain links sparser — no consumer depends on 1-minute cadence, but worth confirming nothing assumes it.

Related contract-family issues: #198 (seq scope), #278 (auth event correlation ids).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions