fix(pi-backend): one failed inference request kills the driver while the accounting identity holds

[drewstone]

One failed inference request kills the driver, even when the accounting identity holds

pi.ts throws pi inference traffic did not match its recorded usage whenever
inferenceTraffic.failedRequests > 0, regardless of whether the generation/receipt identity it
names in the message actually failed.

// src/backends/pi.ts:838
const accountingMatched = inferenceTraffic.generationRequests === usageCost.receipts
  && inferenceTraffic.rejectedRequests === 0
  && inferenceTraffic.failedRequests === 0
  && inferenceTraffic.inFlightRequests === 0

Measured cost

Eleven cells died to this on 2026-08-23 in one fleet. In every one of the eleven, the stated
identity held exactly, and the only anomaly was a single failed request:

cell	runtime lost	generations	receipts	failed
osctemp-pendigits-lstm-gap-r1	80 min	48	48	1
tourney-nonkuramoto-core-r1	60 min	1	1	1
r46-nfree-residual-lp36-r1	60 min	13	13	1
un0-critical-sparsity-grid-r2	49 min	1	1	1
oscnp-3sat-success-rate-r1	41 min	27	27	1
un0-z40-decoupling-topology-r1	21 min	35	35	1
splaw-holdout-freeze-r1	20 min	16	16	1
eq-sym-violation-structure-4-r1	6 min	14	14	1
berbet-bits-per-cell-pareto-r1	2 min	5	5	1
mitten-circuit-ler-freeze-r1	<1 min	1	1	1
driftloss-cheaper-at-equal-score-r1	<1 min	1	1	1

Total 5 h 39 min of driver time. Four cells died in under three minutes. Each ended
no-winner, traversals:N after a single driver attempt, so the run reads afterwards as failed
research rather than as one dropped HTTP response.

Why the two conditions are different

traffic.generationRequests += 1 (pi-inference-transport.ts:1013) runs BEFORE the request is
attempted. So a generation that fails while piping its response (line 1081) is already counted as
a generation with no receipt, and the identity generations === receipts catches it on its own.

The failedRequests === 0 clause therefore adds only the cases the identity cannot see: a
credential-refresh failure at line 961, which increments failedRequests before the generation
counter, and any failed AUXILIARY request. The first is worth refusing on. The second is not, and
neither is a transient generation failure the identity already accounts for.

Suggested shape

Separate the two facts the check currently conflates. Refuse when the accounting identity is
violated, or when a request was rejected or left in flight. For a failed request whose identity
still balances, surface it on the receipt (it already is) and let the driver retry, rather than
throwing a terminal error whose message describes a mismatch that did not occur.

At minimum, the error text should not claim the traffic "did not match its recorded usage" when
the printed numbers in that same message show it matched: 14 generation request(s), 14 Pi usage receipt(s), 0 auxiliary, 0 rejected, 1 failed, 0 still in flight. A reader debugging this spends
the first hour looking for an accounting bug that is not there.

[drewstone]

Correcting two things in the report above, both of which make the case stronger.

1. driverRetry does fire. I wrote that the driver made a single attempt. That came from
reading one cell whose log I had since overwritten. Measured across the ten kills whose logs
survive, retry fires every time:

cell	driver attempts
tourney-nonkuramoto-core-r1	8 (the maxConsecutiveFailures ceiling, exhausted)
driftloss-cheaper-at-equal-score-r1	6
un0-critical-sparsity-grid-r2	4
un0-z40-decoupling-topology-r1	4
berbet-bits-per-cell-pareto-r1	3
the other five	2 each

So this is not a missing retry. The condition RECURS on attempt after attempt, and the policy
exhausts against it.

2. The failure is one specific request, not random loss. Across the attempts of
tourney-nonkuramoto-core-r1 that reached this error:

22 generation request(s), 22 Pi usage receipt(s), 0 auxiliary, 0 rejected, 1 failed
77 generation request(s), 77 Pi usage receipt(s), 0 auxiliary, 0 rejected, 1 failed
 1 generation request(s),  1 Pi usage receipt(s), 0 auxiliary, 0 rejected, 1 failed

The failed count is exactly 1 whether the attempt performed 1 generation or 77. Random transient
loss would scale with traffic; this does not.

Which narrows the path. auxiliaryRequests is 0 and generationRequests === receipts
exactly, so the failing request incremented failedRequests WITHOUT incrementing
generationRequests. In pi-inference-transport.ts there is one such path: the credential-refresh
catch at line 961, which does traffic.failedRequests += 1 and returns 502 upstream authentication refresh failed, before the generation counter at line 1013. A generation that
fails later (line 1081) is already counted, so it would break the identity instead.

Reading it as one failed credential refresh per session, surfaced through an error message about
accounting that did not fail.

What it costs. Attempt durations for that one cell were 61 s, 61 s, 61 s, 61 s, 1118 s,
12991 s, 8 s, 3603 s. The sixth attempt ran three hours and thirty-six minutes and was then
discarded whole. The error message a reader sees names an accounting mismatch, so the first hours
of debugging go to an accounting bug that is not there.

[drewstone]

Correcting my previous comment, and the picture is now both clearer and worse.

The credential-refresh path is NOT the source. I pointed at pi-inference-transport.ts:961.
The bridge log has zero occurrences of upstream authentication refresh failed or
refreshed inference credential is malformed across the whole night. That path never fired.

It is line 1081: the catch around the response pipeline(). There are exactly two sites that
increment failedRequests, and the second one explains every observation:

// line 1013, BEFORE the try
if (requestKind === 'generation') traffic.generationRequests += 1
...
    await pipeline(body, redaction, response)   // streaming the upstream response TO THE CLIENT
  } catch (error) {
    traffic.failedRequests += 1                 // line 1081

The sequence is: the generation is counted, the upstream call SUCCEEDS, Pi emits its usage
receipt, and then the pipe carrying that response back to the client breaks. That yields
generationRequests === receipts with failedRequests = 1 — which is exactly the shape in all
43 occurrences, and it is why the count is 1 whether the attempt did 1 generation or 77.

So the model answered and the tokens were spent. Only the delivery of the response failed.
The guard then converts a dropped client stream into a terminal BackendError that kills the
driver and discards the whole attempt, including every turn that had already succeeded.

Updated scale, measured 2026-08-23 13:40Z. My original report said 11 cells. It is now 17
cells and 43 occurrences, and on freshly launched cells it is hitting essentially every attempt:

cell	age	occurrences / driver attempts
r46-push-witness-below-36	7 min	4 / 6
aod-two-phase-commit-cost	10 min	3 / 3
ygadget-orbit-under-4067	1 h 27	2 / 2
steiner-nontree-first-rate	1 h 27	2 / 2
eq-sym-violation-structure-4	45 min	1 / 1

Four of those five are at 100% of attempts. The cells still producing claims are doing so between
kills, not because their runs complete.

Why the suggested fix stands, with a sharper reason. A broken client stream is not evidence
that inference traffic went unaccounted — the receipt proves it was accounted. Refuse on the
accounting identity, on a rejected request, or on a request left in flight. For a failed request
whose identity balances, surface it on the receipt (it already is) and let the driver retry the
turn rather than discarding the attempt.

If the guard must stay as-is, the error text should at minimum not assert a mismatch that the
numbers printed beside it disprove.

[drewstone]

One caveat on the line-1081 attribution, so nobody spends time chasing a signature that is not
there.

That attribution is by elimination, not by direct evidence. Only two sites increment
failedRequests; the credential-refusal path at 961 logged zero occurrences all night, so 1081 is
what remains, and it is the only site whose ordering reproduces generations === receipts with
failed = 1.

But the signatures you would expect from a broken response pipe are absent. Grepping both the
run logs and the bridge log for ECONNRESET, EPIPE, premature close, socket hang up, and
aborted returns nothing. Attempt durations before the kill also scatter with no pattern —
11 s, 21 s, 42 s, 126 s, 187 s on one cell — which does not look like a consistent client-side
timeout or abort.

So: the counters place it at 1081, and the usual causes of a pipeline() rejection there are not
in evidence. Whatever error that catch receives is not being logged anywhere I can reach from
outside the bridge process, which is itself worth fixing — the catch discards the error after
incrementing the counter, so the one fact that would identify the cause never leaves it:

} catch (error) {
  traffic.failedRequests += 1
  if (response.headersSent) {
    response.destroy(error instanceof Error ? error : undefined)
    return
  }
  writeProxyError(response, 502, 'upstream request failed')

Logging that error would likely resolve this in one run. Everything else in the report stands on
the counters alone and is unaffected: 17 cells, 43 occurrences, identity intact in every one.

[drewstone]

Two corrections that raise the severity, not lower it.

Six cells produced ZERO claims. I implied elsewhere that retry absorbs this and the cost is
wasted time. It is not. These runs ended with nothing banked at all:

un0-critical-sparsity-grid-r2
berbet-bits-per-cell-pareto-r1
driftloss-cheaper-at-equal-score-r1
mitten-circuit-ler-freeze-r1
un0-z40-decoupling-topology-r1
r46-push-witness-below-36-r1        (6 driver attempts, 0 claims)

r46-push-witness-below-36-r1 is the clearest case: six attempts, every one ending in this error,
{"kind":"no-winner","traversals":2}, no claim pages. driverRetry did not rescue it; it spent
six attempts failing the same way.

Every count I have reported is a LOWER bound. Our launcher opened run.log with >, which
truncates. run.log is the only place a driver-attempt error is recorded — the spawn journal has
no failure event kind — so each relaunch destroyed that cell's whole failure history. Relaunching
one cell moved the fleet-wide figure from 17 cells / 43 occurrences to 16 / 40 in front of me,
which is how I noticed. Every relaunch earlier in the night did the same silently. Fixed on our
side by rotating rather than truncating, but the numbers already in this issue are undercounts.

The observation that matters for triage is unchanged and now firmer: the accounting identity holds
in every single occurrence, and the run is discarded anyway.