Cap the escalated landing budget below the watchdog tolerance

[kiwidream]

Fixes #125.

A landing-class ledger step could stay heartbeat-silent for roughly twice the escalated landing budget, and that escalated budget was allowed to reach the watchdog tolerance itself. At the default tolerance the watchdog hard-exited the coordinator mid-landing; because the escalation counter lives only in memory, the restart dropped back to the base budget and replayed the same doomed attempt, deferring found-block settlement while the pool restart-looped. Accounting was never at risk — the outbox row stays pending and replays idempotently — but settlement was.

The arithmetic

One landing step costs a ledger admission wait (winning the writer lock or read semaphore) plus the SQL statement. Under statement_timeout, _remaining_operation_timeout returns the constant statement budget, so the admission wait got the full landing budget — and then the statement got a fresh one.

	before (120s watchdog)	after (120s watchdog)
landing base budget	30s	30s
escalated cap granted	120s	60s (derived, watchdog x 0.5)
admission wait	up to 120s, heartbeat-silent	sliced, stamps a heartbeat every slice
statement	120s, heartbeat-silent	60s, heartbeat-silent
worst-case silent span per step	~240s	60s
watchdog tolerance	120s	120s
outcome	240s > 120s — watchdog hard-exits mid-landing	60s < 120s — a full statement of margin

Escalation now runs 30s -> 60s -> 60s at the default tolerance instead of 30s -> 60s -> 120s.

Production override. Production runs PRISM_WATCHDOG_TIMEOUT_SECONDS=300, which widened the window but did not close it: the admission wait was unbounded by any heartbeat regardless of tolerance. At 300s the derived ceiling is 150s, so the reviewed 120s cap is still granted in full — deployments using that override see no change to the escalation ladder, only the newly non-silent admission wait.

The change

Both halves called for in the issue:

Half A — the admission wait is no longer heartbeat-silent. PsqlShareLedger._operation_gate waited in a single blocking acquire(). That wait is local admission, not database work: no statement has been sent, so no server-side deadline bounds it and the ledger reports nothing until the gate opens. A new operation_progress thread-local scope — shaped exactly like its operation_timeout / statement_timeout siblings — installs a callback plus a slice; when one is installed the acquire is served in slices and the callback fires between them.

With no hook installed the path is byte-for-byte what it was: one blocking acquire, same LedgerOperationTimeout message. So non-block-work callers pay nothing. The caller's deadline stays authoritative, the loop always attempts the acquire at least once (an expired budget still fails exactly where it did), the callback never runs while the thread holds the gate, and an exception from it propagates rather than being swallowed inside a lock wait.

Both statement-timeout scopes in BlockCandidateService run on the watchdog-monitored block-work owner thread, so they install that hook — duck-typed, like the statement_timeout / operation_timeout probes beside them — using _record_block_submitter_wait as the callback and the already-watchdog-clamped _block_work_wait_slice as the slice. Ledgers predating the hook keep working unchanged.

Half B — the cap is derived from the configured watchdog, not hardcoded. DEFAULT_BLOCK_LANDING_DB_TIMEOUT_MAX_SECONDS (120s) and the watchdog default (120s) were independent literals that could drift. _block_landing_db_timeout now clamps both the base and the escalation cap to watchdog_timeout_seconds * BLOCK_LANDING_DB_TIMEOUT_WATCHDOG_FRACTION (0.5), read from the configured coordinator. The reviewed cap becomes an upper bound that the derived ceiling can only lower. DEFAULT_PRISM_WATCHDOG_TIMEOUT_SECONDS also replaces the bare literal in the env default so there is one source for the tolerance.

The issue offered persisting per-hash escalation state in the outbox row as an alternative to capping. Capping is taken here: once the budget cannot outrun the tolerance, the restart-loses-escalation path stops being reachable, so no new persisted state is needed.

Tests

New coverage, all confirmed failing on 2.x.x before the change:

• test_escalated_landing_budget_stays_inside_watchdog_tolerance — fails on base with AssertionError: 120.0 not less than or equal to 60.0, the exact arithmetic above. Also pins the 300s override still granting the full 120s cap.
• test_landing_scope_stamps_progress_during_ledger_admission — fails on base with AssertionError: 0 != 2 (no hook installed).
• Four share_ledger tests covering the sliced wait stamping progress while blocked and still raising at the caller's deadline, the unhooked path staying a single acquire, and hook validation / nesting / exception propagation — all fail on base with AttributeError: 'PsqlShareLedger' object has no attribute 'operation_progress'.
• test_landing_scope_accepts_ledgers_without_a_progress_hook — pins the duck-typed fallback.

Two existing tests assert budgets above half the default tolerance; both now set watchdog_timeout_seconds explicitly rather than weakening their assertions, so the escalation ladder and the per-hash reset are still asserted.

Verification

suite	result
tests.test_prism_block_candidates	166 tests, OK
tests.test_prism_share_ledger	173 tests, OK
tests.test_prism_background_services	26 tests, OK
full unittest discover -s tests	1972 tests, only the 5 known macOS-only os.splice failures in test_prism_job_builder (pass on Linux)
python -m compileall docker lab tests examples scripts	clean

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.