perf(utils): opt-in chunked demean_map dispatch + corrected solver-phase memory attribution

[igerber]

Summary

• Opt-in memory lever for the Rust FE-absorption kernel: _demean_map_rust can dispatch variables in balanced near-equal column blocks via the internal DIFF_DIFF_DEMEAN_CHUNK_COLS env knob (read per call; invalid/non-positive values fall back silently, mirroring the DIFF_DIFF_BACKEND convention). Default is OFF - a single full-width dispatch, byte-identical to the perf(rust): compiled MAP demeaning kernel - rayon across columns, 1.4-2.8x on FE-absorption workloads #603 behavior, so no baseline moves and no default-path performance change. Balanced bounds matter: a naive fixed-stride split leaves a small remainder block that runs nearly serial across the rayon pool and measured ~+11% wall-clock on the firm-panel workload.
• Chunking is exact partitioning: each column's MAP loop is fully independent (no cross-column arithmetic), so partitioning changes neither demeaned values nor iteration counts. Locked by exact-equality tests (chunked-rust vs single-call-rust, assert_array_equal) plus the house cross-engine parity contract vs _demean_map_numpy (iteration-count equality + assert_allclose atol=1e-12).
• Corrects a wrong memory attribution from perf(rust): compiled MAP demeaning kernel - rayon across columns, 1.4-2.8x on FE-absorption workloads #603: the rust-backend firm-panel fit's ~21GB peak RSS (vs ~13.4GB numpy-backend) is dominated by the downstream solver phase (the faer path holds an owned copy of the ~2.5GB stacked design plus SVD workspace), not by the demean kernel's dispatch marshalling as the performance-plan narrative claimed. Measured (probes recorded in docs/performance-plan.md): width-16 chunked dispatch cuts the kernel's transients to near-numpy footprint in isolation, yet the end-to-end fit still peaks ~19-21GB (only ~5-12% below unchunked) at +2-7% wall-clock. That failed the pre-agreed default-on gate (>=20% RSS reduction AND <=5% slowdown), hence opt-in. A measurement-methodology note is recorded alongside: single-run macOS ru_maxrss is unreliable under ambient memory pressure (the compressor deflates resident peaks; one probe read 15.1GB for a configuration that reproducibly peaks ~19-20GB).
• TODO hygiene: new Actionable/Performance row for rust solve_ols peak memory on wide designs (the actual dominator; DIFF_DIFF_BACKEND=python is the interim OOM escape hatch); the solver-bound SunAbraham speed row moves Actionable -> Parked per the correctness-first decision (QR-reuse would remove the SVD's independent near-collinearity detector - deliberate defense-in-depth for the no-silent-failures contract - and perturb all coefficients at the last digits; re-open on practitioner demand, gated on fixest/R end-to-end parity).

Methodology references (required if estimator / math changes)

• Method name(s): Method of alternating projections (MAP) for N-way fixed-effects absorption - dispatch-level change only; no estimand, weighting, variance, or identification change. Per-column results and iteration counts are invariant to partitioning by construction.
• Paper / source link(s): docs/methodology/REGISTRY.md "Absorbed Fixed Effects with Survey Weights" (Rust-kernel Note extended with the opt-in chunking sentence).
• Any intentional deviations from the source (and why): None - numpy remains canonical; the kernel contract is unchanged (no Rust source modified in this PR).

Validation

• Tests added/updated: tests/test_rust_backend.py::TestDemeanMapKernel +5 cases (chunked-vs-single-call EXACT equality, weighted + unweighted; balanced-boundary splits with kernel call-count assertions proving the multi-chunk path ran; non-converging variable in a later chunk still named in the warning; estimator-level DiD(absorb=) ATT/SE parity at chunk=2; default-is-single-dispatch proof) and a standalone TestDemeanChunkResolver (env parsing incl. invalid-value fallback; runs without the Rust build). All chunk tests delenv-guard against ambient env exports. Targeted sweeps: 461 passed (rust build) / 371 passed (DIFF_DIFF_BACKEND=python) across test_rust_backend, test_utils, test_within_transform, test_sun_abraham, test_estimators.
• Backtest / simulation / notebook evidence (if applicable): measured probes quoted in docs/performance-plan.md (frozen-code, 3-repeat arms + single-run probes; identity gate vs committed after-baselines at 1e-13-1e-16 incl. the exempt stress scenario, both arms). No committed baseline changes - default behavior is identical to perf(rust): compiled MAP demeaning kernel - rayon across columns, 1.4-2.8x on FE-absorption workloads #603.

Security / privacy

• Confirm no secrets/PII in this PR: Yes

---

Generated with Claude Code

[github-actions]

Overall Assessment

✅ Looks good — no unmitigated P0/P1 findings.

Executive Summary

• The change is dispatch-level only for Rust MAP demeaning; no estimand, weighting, variance/SE, inference, or default estimator behavior changes were introduced.
• The methodology registry documents the opt-in chunking note, and the implementation preserves the documented numpy-canonical per-column MAP contract.
• New tests cover chunked-vs-single Rust equality, weighted/unweighted parity, boundary splits, warning-name preservation, estimator-level ATT/SE parity, and resolver behavior.
• Remaining wide-design Rust solver memory pressure is properly tracked in TODO.md, so it is informational rather than a blocker.
• One minor documentation rendering issue: an inline env-var code span is split across lines.

Methodology

• Severity: P3 informational
  Impact: The affected method is MAP fixed-effect absorption via the optional Rust demean_map dispatch. The registry explicitly documents chunked balanced column blocks as an opt-in internal knob with unchanged per-column outputs/iteration counts, and the Rust kernel’s source contract is per-column independent. This is an implementation choice, not a methodology deviation. See docs/methodology/REGISTRY.md:L4342-L4358, diff_diff/utils.py:L2745-L2807, rust/src/demean.rs:L1-L13.
  Concrete fix: None required.

Code Quality

No findings.

Performance

• Severity: P3 informational
  Impact: Rust wide-design peak RSS is still dominated by the solver phase, not the demean kernel. This limitation is explicitly tracked in TODO.md:L49 and described in docs/performance-plan.md:L79-L94, so it is properly deferred.
  Concrete fix: No PR action required; future work can pursue the listed borrow-view/workspace-reuse/numpy-routing options.

Maintainability

No findings.

Tech Debt

No untracked tech-debt findings. The new solver-memory item is tracked under the actionable performance backlog in TODO.md:L49.

Security

No findings. The new env knob is read-only configuration and does not introduce secret handling or external input execution.

Documentation/Tests

• Severity: P3
  Impact: docs/performance-plan.md:L89-L90 splits the inline code span for DIFF_DIFF_BACKEND=python across two physical lines. Some Markdown renderers may display this as DIFF_DIFF_BACKEND= python, which is a mildly misleading env-var example.
  Concrete fix: Keep the code span on one line: `DIFF_DIFF_BACKEND=python`.

I did not run the full pytest suite; review was based on the provided diff plus targeted source inspection and a small sanity check of the balanced-boundary arithmetic.