Skip to content

Build without PEP 517 isolation in nox sessions to keep cargo caches warm#15146

Closed
reaperhulk wants to merge 4 commits into
mainfrom
claude/gha-test-performance-hwd684
Closed

Build without PEP 517 isolation in nox sessions to keep cargo caches warm#15146
reaperhulk wants to merge 4 commits into
mainfrom
claude/gha-test-performance-hwd684

Conversation

@reaperhulk

Copy link
Copy Markdown
Member

PEP 517 build isolation creates a temporary virtualenv with a randomized path for every build, so PYO3_PYTHON changes on each install. cryptography-cffi's build script registers rerun-if-env-changed=PYO3_PYTHON, so cryptography-cffi and cryptography-rust recompile on every CI job even with a fully warm rust-cache — and on free-threaded (non-abi3) builds the pyo3 crates recompile too, since the maturin-generated interpreter config also embeds the ephemeral path. See PyO3/pyo3#6113 (the upstream PYO3_BASE_PYTHON fix is deferred to pyo3 0.30).

This installs the [build-system].requires into the session venv (constrained by ci-constraints-requirements.txt, same as the flake session already does) and passes --no-build-isolation, so the build backend runs from the stable .nox/<session> interpreter and cargo fingerprints stay valid across runs.

Verified locally: with isolation, every reinstall recompiles the two crates (~30s of cargo on 4 cores); without it, a warm-cache reinstall with identical env compiles nothing (cargo finishes in 0.05s).

Note: this PR also bumps the rust-cache key. PR runs restore main's caches (whose fingerprints contain the old ephemeral paths) and rust-cache skips saving on an exact key hit, so without a key roll the stable-fingerprint cache would never get saved and the win would only appear once caches roll naturally. CI timing comparisons will be posted on this PR.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Lj3BsMN8xmPQQ89d6nzDcZ


Generated by Claude Code

claude added 4 commits July 3, 2026 18:51
…warm

PEP 517 build isolation creates a temporary virtualenv with a randomized
path for every build, so PYO3_PYTHON changes on each install. Our
cryptography-cffi build script registers
rerun-if-env-changed=PYO3_PYTHON, so cryptography-cffi and
cryptography-rust recompiled on every CI job even with a fully warm
Swatinem/rust-cache (and on free-threaded builds the pyo3 crates
recompiled too, since the non-abi3 interpreter config also embeds the
ephemeral path). See PyO3/pyo3#6113

Install the build requirements into the session venv and pass
--no-build-isolation instead: the build backend then runs from the
stable .nox/<session> interpreter, making the cargo fingerprints stable
across CI runs. Verified locally: with isolation every reinstall
recompiles the crates (~30s); without it a warm-cache reinstall
compiles nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lj3BsMN8xmPQQ89d6nzDcZ
rust-cache skips saving on an exact key hit, so without a roll the
caches would keep the fingerprints recorded from isolated builds and
every job would keep rebuilding once per run.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lj3BsMN8xmPQQ89d6nzDcZ
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Lj3BsMN8xmPQQ89d6nzDcZ

Copy link
Copy Markdown
Member Author

CI timing results (4 measured runs on this PR)

Run Condition Wall clock
main 28675190534 / 28659707361 baseline (isolation, warm cache) 5m47s / 7m16s
A 28678670499 fix, stale -5 caches from main 5m03s
B 28679077977 fix, cold caches (-6 key roll) 5m50s
C 28679399945 fix, warm stable caches 4m58s
D 28679807220 fix, warm stable caches 6m07s

What the logs prove: the fix works as intended. In the warm runs (C/D), the pip-install build (nox --install-only) compiles only the 8 workspace crates — the pyo3 dependency chain is fully cache-stable on every platform, including the free-threaded jobs where it previously rebuilt every run. Example: windows 3.14t nox --install-only dropped from ~112s (baseline, compiles pyo3 + workspace) to ~80s (workspace only). A pleasant side-effect measured in run B: a full cache roll now only costs ~1 minute of wall clock, since the whole dependency tree builds in ~50s on the Linux runners.

What it doesn't show: total wall clock and per-job composites are within run-to-run noise (windows-tests composite: baseline 197–261s vs warm runs 168–249s; linux Create nox environment median ~40s in all runs). Two reasons, both now understood:

  1. rust-cache doesn't cache workspace crates (default cache-workspace-crates: false), and those are the dominant remaining compile cost (~40–50s/job). This PR makes their fingerprints stable, which is the prerequisite — but enabling workspace-crate caching also requires solving cargo's mtime-based freshness on fresh checkouts (naively pinning mtimes to a fixed date would make cargo miss real source changes; it needs git restore-mtime-style per-file commit times, or sccache). Deliberately left as follow-up.
  2. The rust session's cargo test build has its own pyo3/pyo3-ffi rebuild each run (no PYO3_PYTHON involved — pre-existing and unrelated to build isolation). Also follow-up.

So: correct fix, verified mechanism, ~30s of real redundant compile removed per job, but the headline wall-clock win is gated on the workspace-cache follow-up this enables.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Lj3BsMN8xmPQQ89d6nzDcZ


Generated by Claude Code

@reaperhulk reaperhulk closed this Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants