Time-averaged consistency gate uses the wrong degrees of freedom

[gabrielfrasantos]

Severity: high
Domain: math / estimators
Status: VERIFIED — read + hand-traced against a210d34 on 2026-08-10
Suggested labels: bug, numerical-correctness, estimators

Summary

ConsistencyMetrics::IsTimeAveragedConsistent() clamps the χ² quantile to the 10-degree-of-freedom
entry whenever numSamples · Dim > 10, then divides by numSamples. The resulting bound shrinks
towards zero as the averaging window grows, so a perfectly consistent filter is always rejected.
The gate inverts its own meaning.

Location

numerical/math/ConsistencyMetrics.hpp

Evidence

Tables hold only 10 entries:

static constexpr std::size_t kMaxChiSquareDim = 10;

static constexpr std::array<float, kMaxChiSquareDim> kChi2Hi95 = {
    5.023886f, 7.377759f, 9.348404f, 11.143480f, 12.832502f,
    14.449376f, 16.012764f, 17.534546f, 19.022768f, 20.483177f
};

const std::size_t dof = numSamples * Dim;
const float hi = (dof <= detail::kMaxChiSquareDim)
                     ? detail::kChi2Hi95[dof - 1] / static_cast<float>(numSamples)
                     : detail::kChi2Hi95[detail::kMaxChiSquareDim - 1] / static_cast<float>(numSamples);

Trace

Dim = 1, numSamples = 100 → dof = 100 > 10, so the 10-dof entry is used:

hi = 20.483177 / 100 = 0.2048
lo =  3.246973 / 100 = 0.0325

expected average NEES for a consistent filter ≈ 1.0     => REJECTED
correct bound: χ²₀.₉₇₅(100)/100 ≈ 129.56/100 ≈ 1.296

The implemented upper bound is more than 6× too small, and the error grows with numSamples.

Suggested fix

Replace the table lookup for dof > kMaxChiSquareDim with a closed-form approximation. The
Wilson–Hilferty transform is accurate, cheap and allocation-free:

χ²_p(k) ≈ k · ( 1 − 2/(9k) + z_p · sqrt(2/(9k)) )³

with z_{0.975} = 1.959964, z_{0.025} = −1.959964. Add tests at dof = 1, 10, 100, 1000
against reference quantiles.

Notes

This was reported as EF-04 in the 2026-08-03 audit and was not fixed by the #227 refactor
(which did fix the neighbouring EF-05 singular-precheck issue). It remains open.