Problem
Every agent emits an audit_export_status heartbeat once per minute:
{"ts":"2026-07-11T16:36:14Z","event":"audit_export_status","schema_version":"1.0","org_id":"org_…","workspace_id":"ws_…","entity_id":"agt-…","entity_type":"agent","fields":{"sinks":[{"connected":0,"drops_dial":0,"drops_timeout":0,"name":"stderr","writes_ok":2179},{"connected":1,"drops_dial":1,"drops_timeout":0,"name":"unix-socket","writes_ok":2178}]}}
That's ~1,440 events per agent per day in steady state:
- At fleet scale, heartbeats become the dominant row count in the audit-events collection (storage + index pressure in security-next for a signal that's overwhelmingly "still fine").
- Any bounded read window fills with them — the console's per-agent Activity view (last 50 events) was ~50 minutes of heartbeats crowding out actual behavior events. (The console now excludes
audit_export_status from its builder-facing fetches as a mitigation, but the volume problem is at the source.)
Why not just "emit on change" or just "every 15 min"
- Pure emit-on-change is the wrong shape for an audit-integrity signal. This event exists to prove audit events aren't being silently dropped. Silent-when-healthy makes "no events for an hour" ambiguous: healthy, or heartbeat/pipeline dead? Silence-as-success is the anti-pattern this event is supposed to prevent.
- "Change" must mean state transitions anyway —
writes_ok is a cumulative counter that increments every interval, so naive payload-diffing changes nothing. Change = connected flips or drops_dial/drops_timeout increments.
- Pure 15-min interval fixes volume (→ 96/day) but delays detection of a disconnected sink or a drop burst by up to 15 minutes — the one moment the event earns its keep.
Proposal: hybrid
- Emit immediately on state change: any sink's
connected transitions, or any drops_* counter increments.
- Otherwise emit a keepalive on a slow interval — 15 min (or hourly), configurable (e.g.
AUDIT_STATUS_KEEPALIVE_INTERVAL).
- Optionally tag the reason so consumers can distinguish:
fields.reason: "keepalive" | "state_change".
Healthy steady state: 1,440 → 24–96 events/agent/day. Failures surface within seconds. Liveness stays provable (a missing keepalive past the interval is itself a signal consumers can alert on).
Consumers
- security-next / console: no changes required; the console already treats
audit_export_status as operational telemetry (visible in Security · Events, excluded from builder-facing Activity windows).
- audit-forwarder: pass-through, unaffected.
- If the heartbeat participates in the per-entity
prev_hash chain, reducing frequency just makes chain links sparser — no consumer depends on 1-minute cadence, but worth confirming nothing assumes it.
Related contract-family issues: #198 (seq scope), #278 (auth event correlation ids).
Problem
Every agent emits an
audit_export_statusheartbeat once per minute:{"ts":"2026-07-11T16:36:14Z","event":"audit_export_status","schema_version":"1.0","org_id":"org_…","workspace_id":"ws_…","entity_id":"agt-…","entity_type":"agent","fields":{"sinks":[{"connected":0,"drops_dial":0,"drops_timeout":0,"name":"stderr","writes_ok":2179},{"connected":1,"drops_dial":1,"drops_timeout":0,"name":"unix-socket","writes_ok":2178}]}}That's ~1,440 events per agent per day in steady state:
audit_export_statusfrom its builder-facing fetches as a mitigation, but the volume problem is at the source.)Why not just "emit on change" or just "every 15 min"
writes_okis a cumulative counter that increments every interval, so naive payload-diffing changes nothing. Change =connectedflips ordrops_dial/drops_timeoutincrements.Proposal: hybrid
connectedtransitions, or anydrops_*counter increments.AUDIT_STATUS_KEEPALIVE_INTERVAL).fields.reason: "keepalive" | "state_change".Healthy steady state: 1,440 → 24–96 events/agent/day. Failures surface within seconds. Liveness stays provable (a missing keepalive past the interval is itself a signal consumers can alert on).
Consumers
audit_export_statusas operational telemetry (visible in Security · Events, excluded from builder-facing Activity windows).prev_hashchain, reducing frequency just makes chain links sparser — no consumer depends on 1-minute cadence, but worth confirming nothing assumes it.Related contract-family issues: #198 (seq scope), #278 (auth event correlation ids).