compact: authenticate a gap object before dropping its bytes, #10093

[mr-raj12]

Fixes #10093. Rebased on current master, now that #10094 is merged; what is left here is the #10093 change itself, in one commit.

superseded_gap_ranges took the object headers in a pack's gaps at face value. magic, chunk id, meta_size and data_size all came out of the bytes being examined, and (offset, obj_size) was then handed to compact as a range to delete. A wrong data_size in a header whose chunk id is indexed elsewhere stretched that range past the object and into the gap bytes behind it, which were then dropped without ever being looked at. Gaps are where the only copy of a chunk can sit (a backup that crashed before writing its index, or a stale index), so that is not free.

A gap object's bytes now get dropped only if two independent things agree:

• validate accepts the header and metadata slot — the same check the repair walk applies, via PackReader._validation_problem, so the MAX_VALIDATED_META_SIZE guard against an oversized slot read applies here too.
• the object's total size matches the chunk index entry's obj_size. That size decides how far the dropped range reaches, so the index entry serves as a second source for it, one that does not come from the bytes being examined.

The header goes through PackReader._parse_header first, so one that does not parse, or that overruns its gap, ends the walk over that gap. The metadata slot is read together with the header (GAP_META_READAHEAD), so this costs no extra round trip in the common case.

Fail either check and the object keeps its bytes; the walk just continues past it. A corrupt length field can still push the walk to a wrong offset, but at a wrong offset nothing passes both checks, so nothing gets dropped.

validate is threaded in from the callers that hold a key: compact, repo-compress, check --repair. Without one, nothing in a gap is dropped at all. Worth calling out: borg debug delete-obj opens the repository without a key, so it still deletes its target object but no longer reclaims superseded gap bytes from the pack it rewrites.

Changed by the rebase

• object_validator (the old resync_validator) already moved to repoobj.py with check --repair: resync past corrupt object headers when rebuilding the chunks index #10094, so that part is gone from this branch.
• master's object_validator also ties data_size to the csize recorded in the authenticated metadata, so data_size is no longer outside what validate covers. The obj_size cross-check is kept: it is a second source that does not come from the object's own bytes, and it is what rules out a forged size in the modes whose tag is unkeyed. The docs, the docstrings and test_superseded_gap_ranges_rejects_a_size_the_index_contradicts say so now; that test pins the index check on its own with an accept_all validator, and asserts the real validator rejects it too.
• the gap walk reuses master's PackReader._validation_problem instead of reading the metadata slot inline, so both walks share one implementation and the oversized-slot guard. _parse_header returns (hdr, problem) on master, and its MAX_DATA_SIZE check made the branch's own size check redundant.

New docs section "Gap bytes" in docs/internals/packs.rst.

Full suite: 2992 passed, 1107 skipped. ruff and black clean.

[codecov]

Codecov Report

❌ Patch coverage is 91.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.76%. Comparing base (607ab62) to head (c128990).
⚠️ Report is 7 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/repository.py	89.47%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #10167   +/-   ##
=======================================
  Coverage   87.75%   87.76%           
=======================================
  Files         103      103           
  Lines       18821    18836   +15     
  Branches     2906     2907    +1     
=======================================
+ Hits        16517    16532   +15     
  Misses       1600     1600           
  Partials      704      704           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

@mr-raj12 please rebase onto current master. and then ask claude for feedback.

[ThomasWaldmann]

@mr-raj12 ping

[ThomasWaldmann]

review by claude fable 5.1 max

Verdict. The authentication part is right and I found no correctness bug in it. I would still ask for one change before merging: drop the obj_size equality check. It turns a class of superseded duplicates into bytes nothing can ever reclaim, which master reclaims today.

1. Blocking: the size-equality check regresses reclaim. Equal chunk ids mean equal plaintext, not equal stored size. I ran superseded_gap_ranges from this branch with a real validator (object_validator over a chacha20-poly1305 key) on the same chunk stored twice, the gap copy in one pack and the indexed copy at another location:

gap copy	indexed copy	dropped
lz4, 310 B	lz4, 310 B	yes
lz4, 310 B	zstd,3, 297 B	no
none, 6450 B	lz4, 310 B	no
obfuscate,110,lz4, 847 B	obfuscate,110,lz4, 632 B	no
obfuscate,250,lz4, 329 B	obfuscate,250,lz4, 329 B	yes

Sizes differ whenever two clients use different compression, when borg repo-compress ran between the crash and the compact, and on every single store with obfuscate levels 1 to 6 and 110 to 123, which pad randomly. Only Padmé, level 250, is deterministic. Master drops all of these. Worse, they get stuck: the index rebuild in cache.py keeps one copy per id, so after borg check --repair the others are superseded gaps again, and every later rewrite copies them verbatim. The docs' "for borg check --repair to re-index" does not hold for them.

The check buys nothing that the validator does not already provide. The PR text itself notes that data_size is pinned to the authenticated csize, and meta_size is covered by the tag. In the none-* modes the tag is an unkeyed checksum, so a deliberate forgery passes, but whoever can plant a forged object in a pack can also rewrite the index fragments or delete packs, so the index is no independent source there either. Against accidental corruption the checksum suffices. Suggest requiring only the validator. The "second source" bullet in packs.rst and the docstrings go with it, test_superseded_gap_ranges_rejects_a_size_the_index_contradicts reduces to its object_validator assertion, and a new test should show that a differently sized authoritative copy is still reported.

2. Minor: the new read-ahead constant duplicates an existing one. META_READ_SIZE = 1024 in repository.py already serves the same purpose for the repair walk in iter_headers. The comment on GAP_META_READAHEAD claims a metadata slot is at most 112 bytes over all key modes. I measured 113 for the AEAD modes with obfuscate,250, and it is not a bound by construction. Reuse the existing constant and drop the claim.

3. Minor: docs wording. "payloads are user content stored as it is" is only true with --compression none. "stored unencrypted" is what the argument needs.

4. Test nit. DATA_SIZE_OFFSET and META_SIZE_OFFSET are already imported into repository_test.py from repoobj_test.py. The new tests recompute them by hand (hdr_size - 4, len(OBJ_MAGIC) + 1).

Verified fine:

• validate is threaded from the three callers that hold a key. borg debug delete-obj opens with manifest=False and loses gap reclaim, as documented.
• borg compact always has a manifest, since with_repository defaults to manifest=True. Every manifest=None test site was converted.
• ChunkIndex has get() through MutableMapping.
• _parse_header is bounded by the gap end. _validation_problem does its own extra read for a slot longer than the read-ahead, under the existing MAX_VALIDATED_META_SIZE guard. A store-backed pack still costs one round trip per gap object, as before.
• Continuing past a rejected object by its own size can desync the rest of the gap, but nothing validates at a wrong offset, so that is conservative.

Checks run at c128990: ruff, black and git diff --check are clean.

tests	result
repository_test.py, repo_compress_cmd_test.py, compact_cmd_test.py	186 passed, 16 skipped
check_cmd_test.py, debug_cmds_test.py	44 passed, 47 skipped
repoobj_test.py	55 passed

[ThomasWaldmann]

@mr-raj12 ping?!