fix(pruning): make prune candidacy independent of repair-hint history

[mickvandijke]

Production symptom

Nodes accumulate out-of-range chunks indefinitely. The widths tell the story:

• client PUT / fresh-replication storage admission can reach width 20 (SELF_CLOSENESS_GATE_WIDTH = K_BUCKET_SIZE);
• stored-record retention/pruning responsibility uses width 9 (close_group_size + STORAGE_ADMISSION_MARGIN);
• storage commitments include only keys where self is in the strict closest 7;
• fleet telemetry shows ~35% of stored chunks in current commitments — closely matching 7/20 — while very few records ever reach a prune audit or deletion.

Root cause

Two defects in src/replication/pruning.rs:

1. Repair-proof-gated candidacy (introduced in 4cce554, retuned in d04c5aa). After the 3-day hysteresis, evaluate_record_prune_key filtered the strict close group through peers_with_mature_repair_proofs and, when fewer than 6 of 7 peers had a mature repair-hint proof, silently dropped the key from consideration (a debug! line only). Repair proofs are created exclusively when this node sends replica hints during neighbor sync — and neighbor sync contacts the NEIGHBOR_SYNC_SCOPE = 20 peers closest to self, while prune confirmation concerns the 7 peers closest to the key. For a record outside width 9 there is no overlap guarantee between those sets, so six mature proofs could never accumulate: the record never became a candidate, was never audited, and was never deleted. Prune liveness depended on unrelated, self-centric neighbor-sync history.

2. Retention veto blocked the hysteresis clock. The commitment_state.is_held check returned before set_record_out_of_range, so commitment retention postponed the start of the 3-day hysteresis instead of only vetoing deletion — serializing every prune behind the gossip retention window.

A third, contributing problem: every intermediate state (hysteresis pending, proof-deferred, budget/bootstrap-deferred, held) was debug-level or absent, so the stall was invisible in production logs.

What changed

Candidacy is now unconditional. A record continuously outside the storage-retention width (9) for PRUNE_HYSTERESIS_DURATION is a prune candidate — full stop. It never depends on repair-hint proofs, bootstrap state, audit budget, or prior neighbor-sync contact.

The timer starts immediately. The out-of-range first-seen timestamp is recorded the moment self is observed outside width 9, even while a retained commitment still contains the key. The commitment veto now applies to deletion (and skips the pointless audit), never to the clock. The timestamp remains process-local (restart resets it, per the conservative spec).

Audit targets come straight from the routing table. An auditable candidate challenges the CURRENT strict closest 7 to the key with the nonce-bound possession audit — peers_with_mature_repair_proofs is deleted, and PrunePassContext loses its current_sync_epoch / repair_proof_now fields. The repair-proof maturity gate is unchanged for the responsible-chunk anti-outsourcing audit (audit.rs), where it belongs.

Deletion rules are unchanged where they matter. Deletion still requires 6 of 7 valid positive possession proofs (prune_proofs_needed untouched); missing, timed-out, bootstrapping, malformed, absent, and digest-mismatching responses never count positively. Immediately before deletion, a revalidation (revalidate_record_prune_candidate, now a pure function) re-checks width-9 responsibility, the hysteresis, commitment retention (TOCTOU), and that the positive reports still satisfy the current close group — membership churn invalidates stale reports. Below-threshold candidates keep their record, timestamp, and candidacy for later passes.

Scheduling gates are deferrals, not resets. Bootstrap state defers the audit; the per-pass challenge budget (64, unchanged — bounded per-pass network work) defers the audit; a retained answerable commitment vetoes deletion. None of them removes candidate status or restarts hysteresis.

Explicit state machine + telemetry. The lifecycle is now explicit in code — RecordPruneKeyState::{InRange, HysteresisPending, Candidate}, PruneCandidateDisposition::{HeldByCommitment, BootstrapDeferred, BudgetDeferred, Unauditable, Auditable}, PruneRevalidationOutcome::{…, AuditFailed, Delete} — and each pass emits one aggregate INFO line: total, in_range, newly_marked, cleared, hysteresis_pending, candidates, held_by_commitment, bootstrap_deferred, budget_deferred, audits_attempted, audit_below_threshold, pruned (plus paid-list counts). All counters are also new public PruneResult fields. No per-chunk INFO lines.

Paid-list entry pruning, payment verification, commitment pricing, client PUT admission, and the 6/7 threshold are untouched.

Why this restores prune liveness

Every out-of-range record now deterministically reaches an audit of the peers actually responsible for it right now, instead of waiting on neighbor-sync hint history that may never exist for key-close peers outside the node's own neighborhood. The conservative failure mode is preserved throughout: uncertainty (timeouts, absent peers, churned groups, held commitments, budget exhaustion) always retains data.

Tests

No prune test manufactures repair proofs anymore (the record_repair_proofs_for_peers E2E helper is deleted).

Unit (15 new in pruning.rs, 34 total): immediate marking even when held by a commitment; not-a-candidate inside the hysteresis; candidacy at hysteresis with an empty RepairProofs table targeting the full strict close group; held/bootstrap/budget deferrals preserving first-seen and budget; back-in-range clearing; revalidation 6/7-delete vs 5/7-retain; stale-close-group proof rejection; re-commit TOCTOU veto; and response grading (absent digest, digest mismatch, wrong digest count, challenge-id mismatch, Rejected, unexpected body, Bootstrapping never counts).

E2E (5, all against live testnets):

• test_prune_pass_requires_remote_confirmation_before_delete — bootstrap gate defers but preserves candidacy and the original first-seen across passes; below-threshold audits surface in stats; retained until every target proves possession.
• test_prune_veto_for_committed_out_of_range_key — the veto blocks deletion while the timer still starts; prunes once no longer committed.
• prune_deletes_at_proof_threshold_and_retains_below_it — the production-width scenario: close group 7, retention width 9, pruner outside both, empty RepairProofs; 5/7 retains across repeated passes (audits attempted every pass), 6/7 deletes.
• prune_marks_immediately_and_candidacy_waits_for_hysteresis — real non-zero hysteresis: marked on first pass, pending with stable first-seen, then matures, audits, and prunes.
• paid_prune_requires_paid_close_group_confirmations — unchanged semantics, updated to the new context.

Docs

docs/REPLICATION_DESIGN.md: rule 20 now scopes the repair-proof prerequisite to the responsible-chunk audit ONLY (with the self-nearest vs key-nearest rationale); rule 22, Section 11, and scenarios 36/50 rewritten around the new lifecycle; Section 11 gains an explicit taxonomy separating storage admission, retained storage responsibility, commitment inclusion, prune candidacy, prune-audit scheduling, prune deletion confirmation (a direct current-close-group possession check requiring 6/7 positive proofs), and general anti-outsourcing audits. Also fixes the stale claim that paid-list pruning is local-state-only (it has required 15-of-20 paid confirmations since d04c5aa).

Validation

cargo fmt --all -- --check                          → clean
cargo clippy --all-targets --all-features -- -D warnings -D clippy::panic \
  -D clippy::unwrap_used -D clippy::expect_used     → 0 warnings
cargo test --lib                                    → 700 passed; 0 failed
cargo test --lib replication::pruning               → 34 passed
cargo test --lib replication::{types,audit,neighbor_sync} → 35 + 32 + 24 passed
cargo test --test e2e --features test-utils prune   → 5 passed; 0 failed (310s)
cargo doc --no-deps --features test-utils           → no warnings

Rollout notes / remaining risks

• Backlog surge: fleets have accumulated out-of-range records; hysteresis starts at first post-upgrade observation, so a wave of candidates lands ~3 days after deploy. The 64-challenge/pass budget and rotating cursor bound per-pass work; watch the new budget_deferred counter to see drain progress.
• First-seen state is process-local (per spec): a frequently-restarting node restarts its hysteresis. Persisting the timestamp is a possible follow-up.
• HeldByCommitment skips the audit as well as the deletion (no budget spent on unde­letable candidates); if pre-warming audit evidence during the hold window is preferred, it is a one-line change in schedule_prune_candidate.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR fixes stored-record pruning liveness by making prune candidacy and audit targeting independent of neighbor-sync repair-hint history, and by ensuring the out-of-range hysteresis clock starts immediately (even when deletion is temporarily vetoed by retained commitments). It also adds an explicit prune lifecycle/state model and richer per-pass telemetry to make stalls observable in production.

Changes:

• Refactors stored-record pruning to make candidacy unconditional after hysteresis and to audit the current strict close group directly from the routing table (removing prune-path repair-proof gating).
• Ensures commitment retention only vetoes deletion (and skips audit) rather than blocking the hysteresis timer from starting.
• Updates E2E/unit tests and replication design docs to reflect the new lifecycle, targeting, and telemetry.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
tests/e2e/replication.rs	Removes test-only repair-proof manufacturing and asserts new prune lifecycle/telemetry behaviors against live nodes.
src/replication/pruning.rs	Implements the new prune lifecycle state machine, unconditional candidacy, direct close-group audit targeting, and expanded per-pass metrics/logging.
src/replication/mod.rs	Removes pruning’s dependency on neighbor-sync epoch/proof-time fields and simplifies constant usage/imports.
docs/REPLICATION_DESIGN.md	Updates rules and scenarios to scope repair-proof prerequisites to responsible-chunk audits only, and documents the new prune lifecycle and widths taxonomy.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.