Preserve the audit envelope when two found blocks land in one confirm-publish window

[kiwidream]

Fixes #133.

The interleaving

Two distinct found blocks can land inside one confirm→publish window, because
production landings run on two tails that never exclude each other across hashes:
_submit_synchronous_block_candidate on a miner connection thread, and the accounting
actor. The disposition lease is per-hash, so two different block hashes hold two
different leases and serialise against nothing.

1. Block A lands on the synchronous client-thread tail. Confirm allocates publication
   ordinal N under the payout-balance lock and the publication order guard — and
   releases both when the confirm scope returns, before the publish scope is entered.
2. Inside A's confirm→publish gap, block B lands on the accounting-actor tail, confirms
   with ordinal N+1, and publishes. The durable floor is now N+1.
3. A reaches publish_success with sequence N while the current identity is N+1. The
   superseded branch returned published=False without writing anything — no error,
   no log.

The durable Postgres audit record, canonical bundle, body, block accounting, share credit
and settlement all survived. What was lost is the published evidence pointer for block
A: a permanent, silent gap in the public audit trail for a block that was mined and paid
correctly. Because the repair path only ever repairs the current identity and the
finalized outbox row leaves no replay source, the loss was irrecoverable.

The second shape converged to the same place: if B was confirmed but not yet published, A
raised "audit publication is behind the durable ledger floor", was retained as retryable,
and then hit the silent skip once B published.

Neither ingredient is wrong alone. The publication-ordinal floor gating is correct for
same-hash replay, which is what it was designed for. The dual-tail topology predates it.

Which fix shape, and why

Shape 1 — write the envelope for a ledger-proven superseded ordinal.

The alternative was to serialise synchronous-route landings through the accounting actor.
That was rejected. It would put the miner connection thread behind another block's full
landing — which contains a synchronous full-ledger accepted_share_stats() aggregate on a
cold cache — reintroducing exactly the latency and lock-convoy problem on the found-block
path that this repository has a documented history of. It also changes the topology of the
most consequential code here to fix a store-level bug, and would leave the store silently
lossy for any other concurrent publisher, so the evidence trail would stay one topology
change away from the same permanent gap.

The fix separates two invariants the old code conflated behind a single floor check:

• the evidence path is one mutable pointer that must advance monotonically in
  sequence — floor gating is correct and stays exactly as it was;
• the live envelope is a per-block immutable artifact on its own height+hash path,
  and a superseded ordinal's envelope competes with nothing.

Being behind the floor means you may not become current. It does not mean your own
envelope is not yours to write.

Why the write is opt-in

Sequence-versus-floor alone is not sufficient authority to write. A process whose
writer lease has been taken over is also behind the floor, and it must stay fenced out of
the shared audit root entirely — its in-memory report may be stale, and publishing a stale
report as a block's public evidence is worse than leaving the envelope missing.

Only the caller knows which situation it is in. So the write is opt-in through a new
restore_superseded_envelope keyword defaulting to False, and only
_publish_finalization_evidence — the live writer finalizing a block it confirmed itself
in this landing — passes True, and only after proving its lease is still live. See the
review round below: an earlier revision granted that authority unconditionally at this call
site, which was not sufficient.

This was not the first design. The first version keyed the write on sequence-versus-floor
alone, and the A1 process gate's C3 "B wins and late A" scenario caught it: C3 models
this same interleaving across two real processes where A's writer lease has been revoked,
and asserts stale A never repairs N plus full immobility of the shared audit root. That
gate is right.

It also has a useful consequence: because the default behaviour is unchanged, no
existing assertion had to be flipped. The test diff is purely additive.

What is now guaranteed

• Every block finalized by the live writer writes its live audit envelope, regardless of
  how landings interleave — provided the writer can still prove its lease is live.
• A process that has lost its writer lease cannot touch the shared audit root, including a
  writer deposed inside its own confirm→publish gap. The A1 process gate's C3 immobility
  invariant stays literally true.
• A lease proof that fails never fails the landing. It degrades to the previously shipped
  behaviour of skipping the restore, and says so in the log.
• The current evidence reference still names the highest published sequence and never
  moves backwards. The cached identity, evidence state and retention pruning are untouched,
  and the call still reports published=False.
• The write is rolled back if any post-write identity assertion fails.
• A present envelope is held to the same immutable-content standard as the main publish
  path and the parent directory is fsynced, so a corrupt or truncated envelope is surfaced
  rather than silently retained forever.
• The event is no longer silent — it logs, on stderr because worker stdout is a strict
  JSON-lines protocol, and emitted by the caller after every lock scope has been released
  so an undrained stderr cannot block the found-block path.

The landing topology is deliberately unchanged; block_candidates.py is untouched and
block_finalization.py only opts in at the publish call.

Review round: lease-gating the restore

An adversarial review returned merge with changes with four findings. All four are
addressed in the second commit.

F1 — the opt-in claim was falsified. "A deposed writer never reaches the opt-in call
site" is false for a writer deposed inside its own confirm→publish gap: it confirms with
a valid lease, loses the lease during the gap, and everything from there to the publish
call is lease-unfenced. That is exactly the A1 process gate's C3-"late" topology. It went
undetected because the gate harness publishes with the default False, so it never
exercised the production opt-in.

Content divergence was independently impossible — report and persistence digests bind to
the durably confirmed ordinal — so this was a coherence defect against C3's immobility
invariant, not a corruption path. That invariant is cheap to keep literally true, so it
is kept.

_publish_finalization_evidence now asks the coordinator's bounded, non-blocking
exact-session fence (require_fresh_lease_for_external_side_effect) for restore authority
before entering the publication order guard, holding no store lock across the
verification. The asymmetry is deliberate:

• a clean proof grants restore_superseded_envelope=True;
• a lease failure (WriterLeaseRenewalDeferred, ShutdownInProgress — verified to be the
  complete set the fence can raise) withholds the restore, logs it, and lets the
  publication proceed unchanged.

The restore is a repair of an evidence pointer, not the publication itself. Degrading to
False is exactly the previously shipped behaviour, whereas failing the found-block path
on a lease hiccup would be a live regression — the same latency concern that ruled out the
alternative fix shape. The fence stays a no-op for ledgers without the writer lease guard,
never waits on the lease row, and its internal hard-exit arming for a genuinely dead lease
is left intact.

F2 — the production opt-in was untested. Flipping the call site to False passed all
254 tests. A new coordinator-level regression test now drives the real two-tail topology:
the synchronous landing is paused inside its confirm→publish gap while a second
distinct-hash candidate rides the submitter→accounting-actor handoff, lands, and publishes.
It asserts both live envelopes exist. Verified to fail with the call site reverted:

FAIL: test_interleaved_distinct_hash_landings_write_both_live_envelopes
    self.assertTrue(result.envelope_a_exists)
AssertionError: False is not true

A sibling test pins the degrade path — a fence that raises withholds the restore without
failing either landing.

F3 — the present-file early return was too permissive. It accepted any bytes and
fsynced nothing, so a truncated or corrupt envelope was silently retained and could never
be repaired. It now holds present bytes to the main publish path's immutable-content
standard (the comparison is extracted into a helper shared with that path), raises the same
invalid/conflicts errors, and fsyncs the parent directory on a match.

F4 — the diagnostic printed under locks. It ran while holding both the store lock and
the publication-order flock, so an undrained stderr could block the found-block path with
locks held. The store now records the write on the returned publication
(AuditPublication.superseded_envelope_written, defaulted so every existing construction
site is unchanged) and the caller emits the message after both lock scopes have exited.

Tests

The regression test is the point of this change — no existing test interleaved two
distinct-hash landings, which is why the defect survived two review passes.

Added, both at the store level against a real SingleWriterShareLedger with deterministic
threading.Event handoffs (no sleeps):

• test_interleaved_distinct_landings_preserve_the_superseded_envelope — the synchronous
  tail confirms A and releases the balance lock and publication guard before publishing;
  the actor tail lands and publishes B inside that gap; A then publishes superseded.
  Holding either lock across the gap would make the interleaving unreachable, so the test
  deliberately does not.
• test_superseded_landing_converges_after_the_peer_publishes — the second shape: A
  raises behind the floor, B publishes, A's retry lands its envelope.

Confirmed failing before the fix. Verified with the new keyword present but the restore
path neutered, so the failure is behavioural rather than a missing-argument TypeError:

FAIL: test_interleaved_distinct_landings_preserve_the_superseded_envelope
    self.assertTrue(a.envelope_path.exists())
AssertionError: False is not true

FAIL: test_superseded_landing_converges_after_the_peer_publishes
    self.assertTrue(envelope_a.exists())
AssertionError: False is not true

Ran 2 tests in 0.005s
FAILED (failures=2)

Suites (python -m unittest, matching CI):

Suite	Result
tests.test_prism_audit_artifacts	86 tests, OK (84 before, +2 new)
tests.test_prism_block_candidates	165 tests, OK (163 before, +2 new)
tests.test_prism_block_finalization	5 tests, OK

256 total.

Postgres gates: all pass on the final tree, run locally against a Postgres 16 container
(test/test-prism-postgres-ledger.sh, verified real exit 0, no Traceback or
GateFailure in the output):

prism postgres ledger PASS shares=14 lease=replay startup-retry persist-fence sql-window maturity=reorg carry-replay integrity multipage-window=9000
prism postgres A1 gate PASS exact-transition-parity empty-fresh empty-legacy ordinary-decoy-binding temporary-decoy-binding durable-floor-gaps
prism postgres A1 migration gate PASS M0-M11 bigint-bounds invalid-definitions migration-advisory-waiter
prism postgres A1 revert gate PASS round-trip legacy-confirm loud-failure noop-idempotent
prism postgres A1 process gate PASS C1-confirmation-order C2-A-wins-confirmation-publication C3-B-wins-late-A

Full discovery: 1967 tests. On Linux this is clean. On the macOS reviewing host, 2 failures
and 3 errors appear in test_prism_job_builder (the splice spool paths); these were
verified to reproduce identically on the pristine base commit with an empty diff, so they
are pre-existing and platform-specific, not from this change.

Lint in Docker: ruff 0.12.5 check --select E4,E7,E9,F on all changed files — all checks
passed; compileall clean.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.