docs(ax): entry 47 — the cap built to bound stuck events produces them

[lilyshen0722]

What

@sam asked for the 71-count line verbatim in the AX audit. I re-derived the count instead of quoting it, and the re-derivation changed the entry.

169 events retired at attempts >= 3, not 71 — and #1166 did not slow the rate.

The measurement, and the trap in it

day (UTC)	retired	window
08-22	36	04:40 → 23:30
08-23	35	02:10 → 04:06
08-24	76	11:00 → 23:50
08-25	22	00:10 → 05:50

#1166 reached every seat 08-24 (file-sync 09:40:01Z, all ten pollers restarted by 10:26Z). A before/after split on that cutover gives 71 before / 98 after and reads as "the fix made it worse".

It didn't. The 31-hour gap from 08-23 04:06 to 08-24 11:00 is the fleet being down, and it spans the restart. The daily shape says something narrower and worse: the rate is unchanged, so over-claiming was never the dominant cause.

What it actually is

attempts increments when the poller claims an event. The requeue pass returns any delivered row older than AGENT_EVENT_REQUEUE_DELIVERED_MINUTES (default 10) to pending. So an event whose turn merely runs long is reclaimed from the agent currently processing it, three times, then retired — while the turn is still running.

The threshold's own comment states its premise:

The 10-min default accommodates legitimately-long-running tool calls — codex exec for multi-slide LLM generation can take 3-5 min — without re-firing while the agent is still processing.

#1166 measured this seat over 11.5h: median 128s, p90 669s (11.1 min), 13 turns over 600s, max 1153s. The p90 turn exceeds the threshold.

Sampled post-fix retirement: created 08-22T22:57Z, last delivered 08-25T05:32Z, retired 08-25T05:50Z, attempts: 3. Nothing crashed.

This is also the origin of the ~50-hour stale redeliveries driving the current session — the requeue loop, seen from inside the agent.

Why it's an AX entry

Nothing reports it to the party who loses work. The agent acks an event that is already failed; the sender sees a message that was delivered; the cap logs a count with no owner. Two of the retired events were @sprint-review's requests to gate a PR — asked twice, in the pod, neither reaching anyone.

The failure mode of a safety valve is that it looks like the absence of a problem.

Not fixed here

Deliberately. Raising the default moves the boundary without making a long turn distinguishable from a dead poller — attempts counts claims, and slow work and crashed work are the same document. The shape that distinguishes them is a lease the poller extends, bound to the turn. That's a design call, not a one-line default flip, so this entry states the constraint and leaves the choice open.

🤖 Generated with Claude Code