Summary
_summary() in the benchmark script computes "p95" as max(samples), identical to its own "maximum" field.
Details
scripts/benchmark_runtime.py:105-110 (_summary) returns "p95": max(samples) and "maximum": max(samples) — the same formula, with no statistics.quantiles or similar. docs/performance.md explicitly documents "the benchmark reports the median, p95, and maximum for seven samples" as two distinct figures, and .github/workflows/tests.yml:65 gates CI on --check against this mislabeled "p95" budget.
Impact
The CI gate is actually checking against the true max (stricter/noisier than a real p95 would be), and the printed "p95"/"maximum" columns are always equal — misleading when debugging a budget failure.
Suggested fix
Compute an actual percentile (e.g. statistics.quantiles(samples, n=20)[18] or nearest-rank over more iterations), or rename the field to avoid implying a distinct statistic.
Summary
_summary()in the benchmark script computes"p95"asmax(samples), identical to its own"maximum"field.Details
scripts/benchmark_runtime.py:105-110(_summary) returns"p95": max(samples)and"maximum": max(samples)— the same formula, with nostatistics.quantilesor similar.docs/performance.mdexplicitly documents "the benchmark reports the median, p95, and maximum for seven samples" as two distinct figures, and.github/workflows/tests.yml:65gates CI on--checkagainst this mislabeled "p95" budget.Impact
The CI gate is actually checking against the true max (stricter/noisier than a real p95 would be), and the printed "p95"/"maximum" columns are always equal — misleading when debugging a budget failure.
Suggested fix
Compute an actual percentile (e.g.
statistics.quantiles(samples, n=20)[18]or nearest-rank over more iterations), or rename the field to avoid implying a distinct statistic.