experimental/air: warm snapshot cache for the plain_tar path

[ben-hansen-db]

Summary

Adds a local warm cache for the air run plain_tar snapshot path (dirty working tree / no git ref), so a repeated submission re-reads and re-tars only the files that changed.

• Keyed by (repo path, config path, include_paths) → $TMPDIR/databricks/.air/<sha256>/, holding a warm tar + a manifest.json of each file's size, mtime, and byte-range in the tar.
• On the next run: stat the current file set; unchanged members are copied verbatim from the warm tar (no disk re-read), only changed/new files are re-read, deletions drop out. Change detection is size+mtime — the same fingerprint DABs file-sync uses.
• --no-cache bypasses it and re-packs from scratch.
• Engages only above 64 MiB (below that a plain re-pack is cheap). The cache rebuild compresses with klauspost/pgzip (parallel gzip) at DefaultCompression.

Relationship to #6571

Builds on #6571 (parallel gzip + Go-native tar packer), now merged into main. The two are orthogonal: #6571 speeds up the from-scratch pack; this PR avoids re-reading unchanged files. The --no-cache fall-back path calls the merged Go/pgzip packer, so both paths are parallel.

When does the cache help?

• Cold page cache (first run after checkout/reboot, CI, or working set > RAM): scattered small-file reads were 6,464 ms for a large folder (~84 MB) vs 14 ms to read the warm tar sequentially.
• Very large folders (multi-GB, hundreds of thousands of files) that can't stay in page cache: a warm hit packaged in ~6.6 s vs ~20 s cold.

On a warm page cache the win is smaller — parallel gzip (#6571) is the dominant factor there — but the cache still trims the pack by reusing unchanged bytes:

pack phase (large folder, ~476 MB, warm page cache)	time
no cache, parallel gz6	752 ms
cache hit, parallel gz6	413 ms

Correctness

Two MAJOR review findings (from an earlier review pass) are addressed:

• Concurrent same-key rebuilds no longer race a shared snapshot.tar.tmp + non-atomic manifest write.
• A crash mid-rebuild can no longer leave a manifest whose byte offsets describe a different tar layout (which would have silently corrupted a later verbatim-reuse rebuild).

Each build writes a uniquely named snapshot.<id>.tar that is never overwritten, and installs the manifest atomically (unique temp + rename) only after its tar is durable. A manifest and the tar it indexes are always a consistent pair — concurrent rebuilds are last-writer-wins on the manifest rather than interleaving, and no lock is needed. Superseded/orphaned tars are cleaned up best-effort. Covered by a rotation test.

Known follow-ups (not in this PR)

• No eviction of stale cache directories across keys — each distinct repo/config/include-set leaves a dir under $TMPDIR (within a key, superseded tars are already cleaned up).
• The list phase (git ls-files + lstat of all index entries) is the next bottleneck for large folders.
• plain_tar upload isn't content-addressed, so an unchanged tree still re-uploads.

---

This pull request and its description were written by Isaac.

[eng-dev-ecosystem-bot]

Integration test report

Commit: fc3da1b

Run: 34533009808

	Env	💚​RECOVERED	✅​pass	🙈​skip	Time
💚​	aws linux	1	275	16	5:51
💚​	aws windows	1	277	14	4:05
💚​	azure linux	1	274	16	4:54
💚​	azure windows	1	276	14	3:13
💚​	gcp linux	1	275	16	5:45
💚​	gcp windows	1	277	14	3:21

	Test Name	aws linux	aws windows	azure linux	azure windows	gcp linux	gcp windows
💚​	TestAccept	💚​R	💚​R	💚​R	💚​R	💚​R	💚​R

Top 1 slowest tests (at least 2 minutes):

duration	env	testname
3:23	aws windows	TestAccept

[vinchenzo-db]

Claude says:

No-change hit can hard-fail under a concurrent same-key rebuild (minor, correctness)
snapshot_cache.go, the hit path:
if old != nil && old.DirName == dirName && fileExists(oldTarPath) && !snapshotChanged(files, old) {
    return gzipFile(oldTarPath, outputTarball)
}
There's a TOCTOU between fileExists(oldTarPath) and gzipFile's os.Open. If a concurrent same-key rebuild finishes and its cleanupOldTars removes oldTarPath in that window (on Linux, where os.Remove of an unopened file succeeds), gzipFile returns "failed to open warm tar" and the whole air run submit errors out instead of just re-packing. The rebuild path already degrades gracefully when the old tar is gone (old = nil); the hit path doesn't. Cheap fix: if gzipFile fails to open the warm tar, fall through to rebuildWarmSnapshot rather than propagating. Rare (needs two simultaneous submits of the same repo+config+includes), so minor — but the failure mode is a user-visible error on a path that's supposed to be a pure optimization.

everything else lgtm