experimental/air: warm snapshot cache + parallel gzip for plain_tar

[ben-hansen-db]

Summary

The air run plain_tar snapshot path — used when the working tree is dirty or has no git ref, i.e. the normal iterate-and-resubmit dev loop — re-walked, re-tarred, and re-gzipped the entire source tree on every submission. For a large tree (e.g. universe with include_paths) that is seconds to minutes of pure packaging on every run.

This adds a local warm cache for that path plus parallel gzip:

• Warm cache keyed by (repo path, config path, include_paths) under $TMPDIR/databricks/.air/<sha256>/: an uncompressed snapshot.tar + a manifest.json recording each file's size, mtime (ns), and byte-range in the tar. On the next run we stat the current file set; unchanged members are copied verbatim from the warm tar (no disk re-read), only changed/new files are re-read, and deletions drop out. Change detection is size+mtime — the same fingerprint DABs file-sync uses. Keyed so different repos / configs / include-sets never collide.
• Parallel gzip via klauspost/pgzip (MIT), a drop-in for compress/gzip. gzip over the whole tar is the dominant packaging cost and is paid on every run (even a no-change hit re-gzips); pgzip spreads it across cores — ~18× faster on a 470 MiB tar. Level is BestSpeed since upload size doesn't matter here, only latency.
• --no-cache bypasses the cache and re-packs from scratch (the old shell-tar path).
• The cache only engages above 64 MiB (below that a plain re-pack is cheap enough not to bother).

Results — local packaging only

These are packaging latency (git file walk + tar + gzip), measured by calling the packaging functions directly. They do NOT include the workspace upload or API / run-creation round trips. For plain_tar the upload is unchanged by this PR — it still uploads the full compressed tarball each run (the upload name is timestamped, not content-addressed).

Measured on universe (32-core host, warm page cache):

Target	files / size	BEFORE (no cache, every run)	AFTER cold	AFTER warm hit	AFTER +1 file
research	7.4k	2,456 ms	726 ms	557 ms	650 ms
research + js + spark	27.8k / 476 MB	8,525 ms	1,662 ms	934 ms	1,415 ms
whole universe	534k / ~4 GB	~2–3 min (est., serial gz6)	19.9 s	6.6 s	15.2 s

Isolated gzip step (470 MiB tar): compress/gzip 2,123 ms → pgzip 119 ms.

Known limitations / follow-ups (not in this PR)

• The list phase is now the bottleneck. On the whole-universe warm hit, ~6 s of the 6.6 s is git ls-files + ~534k os.Lstat calls (36k of them index entries not present in this sparse checkout). Parallelizing the stat pass and/or skipping skip-worktree entries would cut this.
• gzip is O(total) on every rebuild. A 1-file change still rewrites and re-gzips the whole warm tar (see WARM+1chg vs HIT). Storing per-member gzip streams (concatenated into one valid .tar.gz) would make a rebuild O(changed bytes).
• Concurrent runs on the same cache key race on the fixed snapshot.tar.tmp path — needs a unique temp name + lock.
• No cache eviction — the warm tar (uncompressed tree size) lives in $TMPDIR per key indefinitely.
• plain_tar upload isn't content-addressed, so even a no-change run re-uploads.
• No .nextchanges fragment yet (experimental command) — add if we want a changelog entry.

Validation

• Correctness verified end-to-end: the produced tarball passes gzip -t, its entry count matches the manifest exactly, extracted content matches source, symlinks are preserved, and index-but-not-checked-out files are correctly skipped (sparse checkout).
• Unit tests in snapshot_cache_test.go; the internal/build license test passes for the new pgzip dep (annotated // MIT in go.mod + NOTICE).

---

This pull request and its description were written by Isaac.

[ben-hansen-db]

Superseded by the split requested in review: #6571 (parallel gzip, standalone) and #6572 (warm snapshot cache, stacked on #6571). Closing this combined draft.