experimental/air: warm snapshot cache for the plain_tar path - #6572
Open
ben-hansen-db wants to merge 4 commits into
Open
experimental/air: warm snapshot cache for the plain_tar path#6572ben-hansen-db wants to merge 4 commits into
ben-hansen-db wants to merge 4 commits into
Conversation
Collaborator
Integration test reportCommit: ed64774
Top 22 slowest tests (at least 2 minutes):
|
Contributor
Approval status: pending
|
ben-hansen-db
force-pushed
the
air-plain-tar-pgzip
branch
from
September 8, 2026 21:52
b732496 to
8155ccf
Compare
Stacked on the parallel-gzip change. Add a local warm cache for the plain_tar snapshot path, keyed by (repo, config, include_paths) under $TMPDIR/databricks/.air/<key>: an uncompressed snapshot.tar plus a manifest of each file's size+mtime and byte range. Later runs stat the file set, copy unchanged members verbatim from the warm tar, and re-read only changed files before gzipping the upload. --no-cache bypasses it and re-packs from scratch (now also parallel-gzip, from the parent PR). The cache engages only above 64 MiB. The cache's payoff is largest when the working set does not fit the OS page cache: cold, scattered small-file reads cost seconds (6.5 s for research, 84 MB) versus ~ms to read the warm tar sequentially. When the tree is already warm in RAM, parallel gzip accounts for most of the gain and the cache adds little. Co-authored-by: Isaac <no-reply@databricks.com>
Match the parent PR: DefaultCompression rather than BestSpeed in newGzFile, so the cached tarball is re-gzipped at the same level as the --no-cache path and the upload stays small. Parallel compression makes the higher level nearly free. Co-authored-by: Isaac <no-reply@databricks.com>
The new --no-cache flag on `air run` adds a line to its --help output, which the experimental/air/config-help acceptance test pins. Regenerate the golden. Co-authored-by: Isaac <no-reply@databricks.com>
Isaac Review flagged two MAJOR correctness bugs in the warm cache: - Concurrent `air run` on the same cache key wrote the same snapshot.tar.tmp and raced the rename plus a non-atomic manifest write, interleaving into a corrupt tar/manifest pair. - rebuildWarmSnapshot renamed the new tar into place before saving the manifest, so a crash between the two left a manifest whose byte offsets described a different tar layout -- silently corrupting a later verbatim-reuse rebuild. Fix both by binding the manifest to a per-build, uniquely named tar (snapshot.<id>.tar) that is never overwritten, and installing the manifest atomically (unique temp + rename) only after its tar is durable. A manifest and the tar it indexes are therefore always a consistent pair: there is no window where offsets describe a mismatched tar, and concurrent rebuilds are last-writer-wins on the manifest rather than interleaving, so no lock is needed. Superseded and orphaned tars are cleaned up best-effort. Adds a rotation test. Co-authored-by: Isaac <no-reply@databricks.com>
ben-hansen-db
force-pushed
the
air-warm-snapshot-cache-v2
branch
from
September 9, 2026 15:44
710e750 to
ed64774
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #6571 (parallel gzip). Adds a local warm cache for the
air runplain_tar snapshot path (dirty working tree / no git ref), so a repeated submission re-reads and re-tars only the files that changed.(repo path, config path, include_paths)→$TMPDIR/databricks/.air/<sha256>/, holding a warm tar + amanifest.jsonof each file's size, mtime, and byte-range in the tar.--no-cachebypasses it and re-packs from scratch (also parallel-gzip, from the parent PR).When does the cache actually help?
Honestly: parallel gzip (#6571) is the dominant win in the common warm-page-cache case. The cache's own contribution shows up in two situations:
Pack-phase decomposition at the shipped level (research+js+spark, 476 MB, warm page cache; enumerate ~634 ms is common to every path):
Parallel gzip removes ~90% of the pack cost by itself; the warm-tar reuse trims a further ~340 ms warm, and far more when the page cache is cold.
Correctness (addressed after Isaac Review)
Two MAJOR review findings are fixed in this PR:
snapshot.tar.tmp+ non-atomic manifest write.Each build writes a uniquely named
snapshot.<id>.tarthat is never overwritten, and installs the manifest atomically (unique temp + rename) only after its tar is durable. So a manifest and the tar it indexes are always a consistent pair — concurrent rebuilds are last-writer-wins on the manifest rather than interleaving, and no lock is needed. Superseded/orphaned tars are cleaned up best-effort. Covered by a rotation test.Known follow-ups (not in this PR)
$TMPDIR(within a key, superseded tars are already cleaned up).This pull request and its description were written by Isaac.