Problem
The macOS performance budgets added in #503 were calibrated while the deterministic /usr/bin/python3 probe tail was still present. They intentionally allow very large tail changes:
- server startup P95: 10,000ms absolute / 100% relative
- full refresh P95: 5,000ms absolute / 25% relative
- time to first environment P95: 10,000ms absolute / 100% relative
PR #506 eliminates that tail. Its two final validations report:
| Metric |
Run 1 |
Run 2 |
Existing baseline |
| Full refresh P95 |
1,435ms |
1,256ms |
30,320ms |
| Server startup P95 |
691ms |
- |
7,356ms |
| Time to first environment P95 |
745ms |
- |
7,433ms |
With a post-fix baseline near 1-1.5 seconds, the current absolute budgets could allow a multi-second regression before blocking. That would normalize away much of #506's improvement.
Scope
After #506 merges and successful exact-base baseline artifacts exist:
- inspect repeated post-fix macOS baselines and hosted-runner variance;
- tighten macOS startup, refresh, and time-to-first P95 budgets with measured headroom;
- retain the dual absolute/relative rule so small noisy changes do not fail;
- update comparator tests and
docs/QUALITY_SNAPSHOTS.md;
- leave Linux and Windows budgets unchanged unless new evidence independently justifies a change.
Acceptance criteria
- Missing or malformed metrics continue to fail closed.
- Synthetic regressions beyond each new macOS limit fail tested comparator cases.
- Repeated unchanged-head macOS runs remain green.
- A recurrence of a 15-second interpreter probe tail is unambiguously blocked.
- Full lint, CI, coverage, performance, and CodeQL checks pass.
Depends on #506.
Problem
The macOS performance budgets added in #503 were calibrated while the deterministic
/usr/bin/python3probe tail was still present. They intentionally allow very large tail changes:PR #506 eliminates that tail. Its two final validations report:
With a post-fix baseline near 1-1.5 seconds, the current absolute budgets could allow a multi-second regression before blocking. That would normalize away much of #506's improvement.
Scope
After #506 merges and successful exact-base baseline artifacts exist:
docs/QUALITY_SNAPSHOTS.md;Acceptance criteria
Depends on #506.