Persist per-session vardiff difficulty across miner reconnects

[kiwidream]

Refs #132 (item 1). Targets 2.x.x.

The feedback loop this breaks

Vardiff targets one accepted share per 15 s per connection. Both deployed Stratum lanes share the same 2^32 ceiling and differ only in where a session starts:

	standard lane	high-diff lane (port 4334)
start difficulty	16,384	500,000
minimum difficulty	1,024	500,000
maximum difficulty	4,294,967,296	4,294,967,296
target share interval	15 s	15 s
retarget interval	300 s	300 s
max step up/down	4x	4x

Choosing a lane therefore does not change what a miner can reach — it changes how long that miner floods before it converges.

Until now, a reconnect reset every session to its lane's start difficulty. At 4x per 300 s a large miner needs three to five retarget cycles — 15 to 25 minutes — to climb back to a difficulty it had already reached, submitting far above target the entire time, alongside everyone else who reconnected at the same moment. That is the loop the 2026-07-16 operator note names as "miner reconnect storms that reset vardiff and re-flood the share path":

coordinator slows → watchdog exits → every miner reconnects → every session resets to lane start → share rate spikes → coordinator slows further

Reconnect is the amplifier, not the ceiling. This PR removes it.

What changed

A bounded, TTL'd retention store (SessionDifficultyStore in lab/prism/vardiff_service.py) holds each worker's last converged share difficulty, keyed by (listener name, exact Stratum username). It is an OrderedDict behind a leaf lock with LRU eviction at the entry cap. A hit refreshes recency but deliberately not recorded_monotonic: the TTL measures the age of the value, not of the last access.

Lane scoping keeps the two lanes' policies independent. A reconnect to the same port — the reconnect-storm case — always hits; a lane switch is simply a miss that behaves exactly as today.

Recorded at two points: at the commit point of every successful retarget (so a session that dies without a clean disconnect still retains its value), and on disconnect_client once retirement is claimed, outside the coordinator lock, behind a guard that can never break disconnect cleanup.

Adopted on the first mining.authorize of a connection only — a live session re-authorizing already holds a converged value and is left alone. Adoption lands before password parsing, which preserves the existing precedence exactly: an explicit d= computes its target from requested_difficulty and still outranks the resume, while an md=-only password clamps the just-resumed value through the share_difficulty fallback.

Clamped so a stale or absurd entry can never be adopted as-is. The ceiling is min(lane max difficulty, lane start × resume factor) — at the default 1024x that is 16,777,216 on the standard lane and 512,000,000 on the high-diff lane, i.e. five 4x retarget steps, exactly the climb reconnects used to repeat.

A retained value below the lane start is adopted unchanged, on purpose. Vardiff targets a share rate, so a session sitting at its converged difficulty is on target by definition; cold-starting it higher would give the pool fewer shares, not more.

New tunables

env var	default	meaning
PRISM_STRATUM_VARDIFF_RESUME	1	Master switch. 0 restores lane-start reconnect behaviour exactly.
PRISM_STRATUM_VARDIFF_RESUME_TTL_SECONDS	900	How long a converged difficulty stays adoptable.
PRISM_STRATUM_VARDIFF_RESUME_MAX_ENTRIES	8192	LRU bound on retained state (2x the deployed 4096 connection cap).
PRISM_STRATUM_VARDIFF_RESUME_MAX_START_FACTOR	1024	Plausibility ceiling as a multiple of the lane's own start difficulty. Validated >= 1 at startup.

All four are documented in .env.example.

On the TTL default. 900 s is deliberately short. With PRISM_STRATUM_VARDIFF_IDLE_SWEEP_SECONDS=0 as deployed there is no idle step-down sweep, so retargeting only samples on an accepted share — a session resumed above its true hashrate produces few shares and cannot readily step itself down. A short TTL keeps retained values reflective of recent hashrate and is the shipped mitigation; re-enabling the idle sweep would be the structural one. That coupling is called out in .env.example next to the knob.

For that mitigation to mean anything the TTL must age from real convergence, not from the last time the value was touched. It does: re-recording an unchanged difficulty refreshes the entry's timestamp only when the connection produced at least one accepted share. A session that resumes a retained value, submits nothing and disconnects leaves the clock running, so a reconnect loop shorter than the TTL can no longer keep a stale value alive indefinitely. See the review-fix commit below.

Observability

This area had none: no metric for a session pinned at max_difficulty, and no per-lane share rate. Added to the renderer:

• qbit_prism_vardiff_sessions_at_max_difficulty (gauge) — sessions sitting at their effective vardiff ceiling.
• qbit_prism_vardiff_lane_accepted_shares_total{lane} (counter) and qbit_prism_vardiff_lane_accepted_shares_per_second{lane} (gauge). The per-second gauge uses the same elapsed formula as qbit_prism_shares_per_second, so the two are directly comparable. The lane counter increments before vardiff's enabled check, so lane load stays observable where vardiff is off.
• qbit_prism_vardiff_resume_total{outcome} over a bounded label set (resumed, clamped, overridden, expired, miss, rejected, disabled) and qbit_prism_vardiff_resume_retained_sessions (gauge). resumed and clamped count attempts; overridden counts an adopted value that an explicit d=/md= superseded in the same authorize, so resumed + clamped - overridden is the number that actually stuck.

The lane label set is deterministic: configured listener profiles in order, then any additional observed lane names, sorted.

The at-ceiling census respects the codebase's lock order (client vardiff lock → coordinator lock, never the reverse): membership is snapshotted under the coordinator lock alone, released, and only then is each client's vardiff lock taken. The two locks are never held together.

Metrics output is pinned byte-for-byte by the frozen-reference parity test, so reference_render_metrics_payload in tests/test_prism_metrics.py gains a mirrored reference_vardiff_convergence_metrics_lines at the matching position, plus needles so an empty fixture cannot pass silently.

Operational effect on the next coordinator restart

Worth stating precisely, because the boundary matters: the store is in-process. It survives miner-side reconnects, network flaps, load-balancer churn and slow-coordinator timeouts — every reconnect storm that does not kill the coordinator. It does not survive a coordinator process restart.

So on the next restart the store starts empty and that first reconnect wave still climbs from lane start, exactly as today. What changes is everything after: the store refills as sessions converge, and each subsequent reconnect — including the repeat reconnects that follow a watchdog exit while the coordinator is still settling — resumes at the converged value instead of re-climbing. The loop loses its amplifier from the second wave onward. Durable cross-restart retention would need a store this PR deliberately does not add; see follow-ups.

Rollback is an env flip, not a revert: PRISM_STRATUM_VARDIFF_RESUME=0.

Review fixes (second commit)

An independent adversarial review of the first commit found one major defect, four minor findings and two nits. The second commit applies the accepted subset.

Major, confirmed by execution — the TTL could be laundered. SessionDifficultyStore.record unconditionally overwrote recorded_monotonic, and the disconnect seam recorded on every disconnect with no share evidence. A session that resumed a retained difficulty, submitted nothing and disconnected therefore re-stamped that value with a fresh clock: with a 900 s TTL and 800 s reconnect cycles the value survived past 4000 s and in principle forever — defeating the mitigation the short TTL exists to provide, in exactly the zero-share case that the disabled idle sweep makes dangerous. The store now refreshes the timestamp only on a first record, a value that actually moved, or an unchanged value backed by an accepted share on that connection (tracked by a new per-connection ClientState.vardiff_accepted_any). Regression tests cover both directions: a silent reconnect loop expires on schedule, a share-backed one does not.

Nit — undelivered values are no longer retained. The disconnect path read pending_share_difficulty or share_difficulty; pending_share_difficulty is advertised with a future job, so a disconnect racing an in-flight retarget retained a difficulty the miner was never given. Disconnect now retains the delivered value only. Nothing is lost — a retarget whose paired send completed already records its own value at that commit point.

Minor — resume counters made unambiguous. Added the overridden outcome described above.

Three review findings are accepted as documented limitations rather than code changes, and the docs now say so plainly:

• Retention is last-writer-wins per (listener, exact username), so rigs deliberately sharing one username share one retained value. Per-rig worker names are the real fix; PRISM_STRATUM_MAX_CONNECTIONS_PER_USERNAME bounds concurrency but not sequential reuse, and the clamp remains the backstop. Documented in .env.example.
• The per-lane _per_second gauge is a since-start average and structurally cannot show a transient storm. Its HELP text now says so and points at rate() over the _total counter.
• The at-ceiling census is O(connections) locks per scrape; the cost, the 4096-connection cap and why it is acceptable are recorded in a comment at convergence_snapshot.

The review separately cleared, with reasoning: lock discipline throughout (the convergence lock is a strict leaf; no client vardiff lock is ever held across a socket write, so a slow miner cannot stall the scrape), the difficulty_generation bump ordering against job delivery, the high-diff minimum_advertised_difficulty floor on resumed sessions, authorize failure paths, prune/lookup double-counting, and that the parity test is a genuine transcription rather than a tautological delegation.

Verification

• python -m compileall -q docker lab tests examples scripts — clean.
• tests.test_prism_vardiff_resume, tests.test_prism_vardiff_service, tests.test_prism_vardiff — 66 tests, OK.
• tests.test_prism_stratum_session, tests.test_prism_stratum_restart_bind, tests.test_prism_metrics, tests.test_prism_coordinator_config, tests.test_auxpow_vardiff_coordinator — 164 tests, OK.
• Full unittest discover — 1996 tests; the only failures are the five known os.splice cases in tests/test_prism_job_builder.py, which fail on macOS, pass on Linux, and live in a file this PR does not touch. The review independently ran the full suite on Linux: 1996 green, zero failures.

tests/test_prism_vardiff_resume.py covers the three acceptance cases — resume after reconnect (converged 262,144 resumed rather than the 16,384 lane start), TTL expiry falling back to lane start, and an implausible retained 1e15 clamped to the 16,777,216 lane ceiling and asserted not adopted — plus LRU eviction, per-lane scoping, d= precedence, md=-only clamping, no re-resume on re-authorize, the kill switch, and the new metric series. The second commit adds the TTL-laundering regression in both directions, the delivered-value retention case, the retarget_locked commit-point recording path, the end-to-end env → StratumConfig → coordinator → store sizing chain (which getattr defaults would otherwise mask on a rename), and an authorize path where reserve_client_username fails.

Follow-ups (deliberately not in this PR)

Kept out to stay reviewable; all are #132 items.

• Lane steering (2.x.x: bound share-ingest rate across the standard and high-diff lanes #132 item 2). Nothing here reads across lanes. The retention key already holds per-lane converged values per username, and the new per-lane rate series is the load signal a steering policy needs; a future pass could consult the other lane's retained entry as a hashrate estimate.
• Asymmetric convergence (2.x.x: bound share-ingest rate across the standard and high-diff lanes #132 item 3). Resume removes the reconnect case, but a genuinely new large miner still climbs from lane start over three to five cycles. calculate_next_difficulty already receives an EWMA difficulty_estimate, so a larger first-window step could gate on client.vardiff_difficulty_estimate is None without touching the steady-state bound.
• The 300 s retarget interval (2.x.x: bound share-ingest rate across the standard and high-diff lanes #132 item 4). Unchanged. Note the interaction found here: with the idle sweep disabled, retargeting samples only on accepted shares, which is what makes an over-high resume sticky and what pins the short TTL default.
• Shedding paths (2.x.x: bound share-ingest rate across the standard and high-diff lanes #132 item 6). The lane counter increments inside vardiff's accept accounting, so a future shedding design has to decide whether shed shares count toward lane rate, or the per-lane series will undercount true lane load in degraded mode.
• Cross-restart durability. Retention is in-process by design; a durable store is the natural next increment if the first post-restart wave proves costly in practice.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.