fix(reports): exact Student-t p-values and a paired comparison section

[uipreliga]

Why

welch_t_test treated the Welch t-statistic as a z-score (2.0 * NormalDist().cdf(-t)). The normal distribution has lighter tails than Student's t, so this understated p and manufactured significance exactly in the small-replicate regime coder_eval runs in. The in-code comment claimed the approximation "overestimates p slightly" — backwards. Measured: t=2.5 at df=4 is p=0.0668 (not significant); the old code reported 0.0124 (falsely significant). The 2-variant report fixture printed p=0.005 where the exact value is 0.106 — understated ~20x.

What changed

Exact t-test. Two-tailed Student-t p-value via the regularized incomplete beta (Lentz continued fraction, Numerical Recipes 6.4) plus proper Welch–Satterthwaite degrees of freedom. Pure stdlib — math.lgamma/log1p — so the module stays zero-dependency; no scipy. Cross-validated against scipy.stats.ttest_ind(equal_var=False) over 200 randomized group pairs, worst absolute difference 1.4e-13. Also fixes the docstring's claim of an exact fallback that never existed, and makes zero-variance-with-different-means return p=0.0 (a deterministic difference) instead of 1.0.

Paired comparison. Both variants run the same tasks, so an unpaired Welch test carries between-task difficulty variance in its denominator and is systematically underpowered. Adds paired_t_test and a ## Paired Comparison section for 2-variant experiments. The existing paired block moves out of ## Replicate Statistics and loses its replicate_count > 1 gate, so single-replicate A/B — the common case — gets it too.

Pairing is over per-task mean scores. Replicate slots within a task share the task effect and are not independent; pairing them individually would understate the standard error. The task is the unit of analysis.

One source of truth. _collect_variant_series existed twice and had drifted: the markdown copy divides duration_seconds by replicate_count (correct — VariantResult sums it across replicates) while the HTML copy used it raw, so the HTML report showed replicate-inflated durations. Both now call one collect_variant_series. Likewise both reporters render one shared paired_comparison(), and the HTML report gains the paired section it never had.

Coherent interval. The paired section printed a percentile-bootstrap CI beside an exact-t p-value — two inference models on the same quantity, free to disagree, and degenerate over the 2–3 task means it resamples. paired_t_ci now derives the interval from the same distribution as the p-value, so the two always agree about whether 0 is excluded.

Behavioural deltas (all intended)

• p-values in 2-variant reports get larger (honest) at small n; identical groups still return exactly 1.0.
• The 2-variant snapshot CI widens from [-0.450, +0.200] to [-4.255, +4.005] beside p = 0.766. The bootstrap was resampling two task means and reporting a falsely narrow interval; with 2 tasks you know almost nothing, and the interval now says so.
• A 2-variant experiment with only one common task explains why it has no paired result instead of rendering nothing.
• HTML durations are no longer replicate-inflated.

Notes for review

• The ## Paired Comparison note is deliberately descriptive and makes no power claim. Pairing trades degrees of freedom for correlation and is not unconditionally sharper — the committed 2-variant snapshot is itself a counter-example (Welch p=0.658, paired p=0.766).
• Tasks whose two variants ran unequal replicate counts are no longer excluded. That constraint only existed to zip replicate slots; under mean-pairing it discarded real data and biased toward fully-completed tasks.
• Non-finite guards: student_t_two_tailed_p fails closed to 1.0 (garbage must never read as significant), the list helpers return None (rendered —), and regularized_incomplete_beta raises rather than returning NaN.
• tests/test_reports_stats_nonfinite.py enumerates the module instead of listing helpers by hand, so a statistic added later is covered automatically. It caught the regularized_incomplete_beta NaN case while being written.
• Multiple-comparison correction is deliberately out of scope: both reporters gate p-values on exactly 2 variants, so there is no all-pairs family to correct.

Testing

make verify — 3462 passed, 2 skipped, coverage 90.88%. Reference values pinned against scipy without taking a scipy dependency; both markdown snapshots regenerated and reviewed rather than loosened.

🤖 Generated with Claude Code

[github-actions]

Claude finished @uipreliga's task in 44s —— View job

---

I'll analyze this and get back to you.