experimental/air: content-address the plain_tar upload - #6578
Draft
ben-hansen-db wants to merge 2 commits into
Draft
experimental/air: content-address the plain_tar upload#6578ben-hansen-db wants to merge 2 commits into
ben-hansen-db wants to merge 2 commits into
Conversation
Collaborator
Integration test reportCommit: 5362b47
Top 14 slowest tests (at least 2 minutes):
|
git_archive snapshots are already content-addressed: a repeat submission at the same commit reuses the uploaded tarball and skips packaging + upload. plain_tar (dirty working tree) used a timestamped name, so it re-packaged and re-uploaded the full tarball on every submission, even when nothing changed. Name the plain_tar tarball by a working-tree fingerprint (sha256 over each file's path, size and mtime) and run the same snapshotExists skip for both modes. An unchanged resubmit now reuses the remote object and moves no bytes. The listing is captured once and threaded into packaging, so the tree is walked only once. The fingerprint is size+mtime, not content, matching DABs file-sync. Verified on df1: a second submission of an unchanged tree logs "snapshot upload skipped; reusing ..." and returns the identical remote path. Co-authored-by: Isaac <no-reply@databricks.com>
ben-hansen-db
force-pushed
the
air-plain-tar-content-addressed
branch
from
September 9, 2026 16:03
9f7d8be to
b9aa788
Compare
- Strengthen the dedup test: count import-file calls and assert the second
(unchanged) submit adds zero, instead of asserting a path-keyed set has one
entry. The set couldn't distinguish a skipped submit from a re-upload to the
same content-addressed name; the counter can (verified it fails when the skip
is disabled).
- Fold snapshotPackagingVersion into computePlainTarKey so a packaging-logic
bump invalidates plain_tar keys too, not just plainTarKeyVersion.
- Fix stale comments now that plain_tar is content-addressed: modePlainTar
("not cacheable") and snapshotExists ("git_archive" only).
Co-authored-by: Isaac <no-reply@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the
plain_tarupload content-addressed so an unchanged working tree skips packaging and upload — the mechanismgit_archivealready used, now generalized to both modes. Standalone; independent of the parallel-gzip change (#6571).Today
git_archivenames its tarball by(commit, include_paths)and, if that object is already uploaded, reuses it and moves no bytes.plain_tar(dirty working tree / no ref — the normal iterate-and-resubmit loop) used a timestamped name, so it re-packaged and re-uploaded the full tarball on every submission, even when nothing changed.This change:
snapshotFilesreturns each file'ssize+mtime(captured free from theLstatit already did) instead of just paths.computePlainTarKeyhashes the sortedpath+size+mtimeset into a working-tree fingerprint (namespaced with its own version tag).snapshotTarNamenamesplain_tar<dir>_<fingerprint>.tar.gz(was<dir>_<timestamp>.tar.gz) and returns the file listing so packaging reuses it — the tree is walked once, not twice.uploadSnapshotViaDABsruns the existingsnapshotExistsskip for both modes: if the object already exists, it reuses the remote path and uploads nothing.Validation
experimental/air/cmdpackage tests pass;vetandgofmtclean. The old "unique timestamped name" test is now a "content-addressed + dedup on unchanged" test (mirrors the git_archive caching test); addedcomputePlainTarKeyproperty tests.End-to-end on df1, with this branch's binary. Submitting an unchanged universe tree twice: the second run logs
snapshot upload skipped; reusing …<fingerprint>.tar.gz, returns the identical remotecode_source_path, and skips packaging + upload. Both runs launched (1×A10):research(~27 MB)research+js+spark(~97 MB)The win is proportional to what would otherwise be re-uploaded: negligible for a tiny tree (the submit is dominated by fixed API/MLflow/auth overhead), ~8× once the tarball is non-trivial. The slow first submit reflects this branch's single-threaded shell
tar -czf— parallel gzip is the separate experimental/air: parallel gzip for the plain_tar snapshot packer #6571; the two are complementary (this PR speeds the unchanged resubmit, experimental/air: parallel gzip for the plain_tar snapshot packer #6571 speeds the first/changed one).Notes / trade-offs
git_archive— no eviction here.This pull request and its description were written by Isaac.