Persist Slurm run state, retry shards, and collect winners

[nabinchha]

Priority Level

High

Task Summary

Implement durable run, shard, attempt, readiness, winner, retry, and collection behavior so every public control operation works from persisted records and normalized scheduler observations.

Technical Details & Implementation Plan

• Persist immutable run intent and resolved-plan references plus stable shard identities and record ranges.
• Create monotonically numbered attempts with scheduler array/job identities and atomic lifecycle updates.
• Give resumable shards one locked shard-owned dataset workspace; use attempt-local datasets when resume is disabled.
• Atomically replace attempt readiness snapshots with restrictive permissions and validated monotonic revisions.
• Validate semantic client results and candidate output manifests before winner publication.
• Publish exactly one immutable winner record per successful shard without overwrite.
• Reconcile persisted state with active and accounting scheduler observations in a fresh process.
• Classify retryable shards without winners and create new attempts from the pinned plan and artifact digests.
• Normalize and authorize collection destinations through the selected workspace and writable mount mapping.
• Submit collection as a zero-GPU CPU Slurm job using the run's pinned client image.
• Enumerate planned shards and validated winners only, process Parquet with bounded memory, stage beside the destination, verify row counts, and publish atomically without overwrite.

Acceptance criteria

• Failed, partial, stale, or scheduler-inconsistent attempts never publish winners.
• A winning shard is sealed against further retry.
• Concurrent resume writers, finalizers, retry requests, and collection requests fail or converge safely.
• Status reconciles attempt, readiness, generation, retry, and collection evidence without an in-memory controller.
• Accounting lag remains explicit and does not become guessed success or failure.
• Collection rejects missing, malformed, mismatched, or duplicate winners before reading bulk data.
• Collection never recursively globs attempt directories or selects a latest attempt.
• Large Parquet inputs are not processed by the login-host CLI.
• Interruption, validation failure, destination collision, timeout, and out-of-memory failure expose no partial final dataset.

Out of scope

• Generic object-store output.
• Cross-run resume or concurrent writers to one shard.
• Resident monitoring.
• Arbitrary unmanaged input trees for collection.

Investigation / Context

This is the persistent-state and collection lane for #850. Scheduler IDs are observations; stable run, shard, and attempt IDs remain the application identities.

Agent Plan / Findings

Implement persistence and locking first, then winner validation/publication, fresh-process reconciliation, retry, and winner-driven CPU collection.

Dependencies

Blocked by the shared-contract and fake-infrastructure foundation work tracked by #850.