Skip to content

Tighten macOS performance budgets after eliminating the 30-second tail #507

Description

Problem

The macOS performance budgets added in #503 were calibrated while the deterministic /usr/bin/python3 probe tail was still present. They intentionally allow very large tail changes:

  • server startup P95: 10,000ms absolute / 100% relative
  • full refresh P95: 5,000ms absolute / 25% relative
  • time to first environment P95: 10,000ms absolute / 100% relative

PR #506 eliminates that tail. Its two final validations report:

Metric Run 1 Run 2 Existing baseline
Full refresh P95 1,435ms 1,256ms 30,320ms
Server startup P95 691ms - 7,356ms
Time to first environment P95 745ms - 7,433ms

With a post-fix baseline near 1-1.5 seconds, the current absolute budgets could allow a multi-second regression before blocking. That would normalize away much of #506's improvement.

Scope

After #506 merges and successful exact-base baseline artifacts exist:

  • inspect repeated post-fix macOS baselines and hosted-runner variance;
  • tighten macOS startup, refresh, and time-to-first P95 budgets with measured headroom;
  • retain the dual absolute/relative rule so small noisy changes do not fail;
  • update comparator tests and docs/QUALITY_SNAPSHOTS.md;
  • leave Linux and Windows budgets unchanged unless new evidence independently justifies a change.

Acceptance criteria

  • Missing or malformed metrics continue to fail closed.
  • Synthetic regressions beyond each new macOS limit fail tested comparator cases.
  • Repeated unchanged-head macOS runs remain green.
  • A recurrence of a 15-second interpreter probe tail is unambiguously blocked.
  • Full lint, CI, coverage, performance, and CodeQL checks pass.

Depends on #506.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestimportantIssue identified as high-priority

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions