Auto-recover corrupt packfiles in packfile maintenance - #2052
Open
tyrielv wants to merge 1 commit into
Open
Conversation
tyrielv
force-pushed
the
tyrielv/packfile-corruption-recovery
branch
from
July 9, 2026 21:22
cacf77d to
284fdb5
Compare
tyrielv
marked this pull request as ready for review
July 9, 2026 22:00
tyrielv
force-pushed
the
tyrielv/packfile-corruption-recovery
branch
from
July 14, 2026 22:43
284fdb5 to
bcc2b74
Compare
Derrick Stolee (derrickstolee)
approved these changes
Jul 22, 2026
When a packfile in the shared object cache is corrupt or truncated (e.g. from a
past disk-full event), 'git multi-pack-index write/verify' fails with "could not
load pack N". The existing self-heal only deletes and rewrites the
multi-pack-index (MIDX), which does not fix the underlying pack: the rewrite
re-scans the same bad pack and keeps failing. The corruption then recurs
indefinitely.
PackfileMaintenanceStep now routes write/verify failures through a recovery path
that, when git reports a pack-load failure:
- Detection (always runs, even with recovery disabled): verifies each pack in
the object cache with 'git verify-pack' and reports every unreadable pack via
telemetry (Operation=FoundCorruptPack). The "could not load pack N" ordinal is
an internal MIDX position, not a filename, so per-pack verification is how we
find the actual bad file.
- Removal (gated, see kill switch below): deletes each corrupt pack's files
(.pack/.idx/.keep/.rev; Operation=DeletedCorruptPack), then deletes and
regenerates the MIDX from the packs that remain (fast path, no full repack).
Missing objects are re-fetched on demand.
- Corrupt prefetch pack (special case): prefetch packs are incremental and
ordered by timestamp, so a corrupt one invalidates every later prefetch pack
too - leaving a hole would let the newest surviving timestamp advance past it
so a later prefetch never backfills the gap. Recovery removes the corrupt
prefetch pack and every later prefetch pack
(Operation=DeletedHealthyPrefetchPack for the healthy ones removed purely due
to ordering), then requests a prefetch (via a callback GitMaintenanceScheduler
wires to a PrefetchStep, only when using a cache server) to re-download them
and rebuild the commit-graph.
Kill switch: the destructive pack removal is gated by a new git config,
gvfs.enable-packfile-recovery (default true). When false, GVFS still detects and
reports corrupt packs (Operation=FoundCorruptPack, then
CorruptPackRecoverySkipped) but deletes nothing and does not request a prefetch;
the non-destructive MIDX rewrite still runs, so behavior degrades to today's.
This gives a field kill switch without a redeploy if the destructive path ever
misbehaves.
This is stacked on the git-output bounding change: recovery runs additional git
commands (verify-pack, MIDX rewrites) against the corrupt repo, so it relies on
that change to keep a noisy stderr from OOM-ing the mount mid-recovery.
Review follow-ups:
- prefetchRestoreNeeded is now set only after a corrupt prefetch pack is
actually removed (RemovePackFileSet returns whether the .pack file was
deleted), instead of as soon as one is detected. If deletion is blocked,
the restore no longer runs while the corrupt pack is still present.
- DetectAndRemoveCorruptPacks now remembers, for the lifetime of a single
maintenance run, that it already reported corrupt packs with recovery
disabled, and skips the redundant per-pack verify-pack rescan on later
MIDX failures in that same run.
- DetectAndRemoveCorruptPacks now parses the corrupt pack's filename directly
out of the write/verify failure's stderr when git includes it (e.g.
"packfile pack-1234.pack does not match index" / "wrong index v2 file size
in pack-1234.idx"), and verifies only that candidate instead of every pack
in the object cache. This only helps when git actually names the file,
which it does for the verify-triggered failures this code mostly handles
(not for the rarer write-path "could not load pack N", which is genuinely
an unresolvable internal ordinal - confirmed by reading git's midx-write.c).
Falls back to verifying every pack whenever no candidate can be parsed, or
the parsed candidate turns out to be healthy, so detection is never less
thorough than before.
Tests:
- A verify failure reporting a pack-load error removes the corrupt pack and
rewrites the MIDX from the remaining good packs (recovery enabled).
- With recovery disabled, the same failure still verifies each pack and reports
the corrupt one but deletes nothing.
- A corrupt prefetch pack removes it and every later prefetch pack, keeps the
earlier healthy one, and requests a prefetch.
- A verify failure that names the corrupt pack directly verifies only that
pack (fast path).
- A verify failure that names a pack which turns out to be healthy falls back
to verifying every pack (fallback path).
Assisted-by: Claude Sonnet 5
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
tyrielv
force-pushed
the
tyrielv/packfile-corruption-recovery
branch
from
August 7, 2026 22:44
490f99e to
9ec7abf
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When a packfile in the shared object cache is corrupt/truncated (e.g. from a past disk-full event),
git multi-pack-index write/verifyfails withcould not load pack N. The existing self-heal only deletes+rewrites the multi-pack-index (MIDX), which does not fix the underlying pack — the rewrite re-scans the same bad pack and keeps failing, so the corruption recurs indefinitely.Change
PackfileMaintenanceSteproutes write/verify failures through a recovery path. When git reports a pack-load failure:git verify-packand reports every unreadable pack via telemetry (Operation=FoundCorruptPack). TheNincould not load pack Nis an internal MIDX position, not a filename, so per-pack verification is how we find the actual bad file..pack/.idx/.keep/.rev;Operation=DeletedCorruptPack), then deletes+regenerates the MIDX from the packs that remain (fast path, no full repack). Missing objects are re-fetched on demand.Operation=DeletedHealthyPrefetchPackfor the healthy ones removed purely due to ordering), then requests a prefetch (via a callbackGitMaintenanceSchedulerwires to aPrefetchStep, only when using a cache server) to re-download them and rebuild the commit-graph.Kill switch
The destructive pack removal is gated by a new git config
gvfs.enable-packfile-recovery(defaulttrue). Whenfalse, GVFS still detects and reports corrupt packs (FoundCorruptPack→CorruptPackRecoverySkipped) but deletes nothing and requests no prefetch; the non-destructive MIDX rewrite still runs, so behavior degrades to today's. This is a field kill switch — no redeploy needed if the destructive path ever misbehaves.Telemetry
Distinct, queryable
Operationvalues so rollout can be watched:FoundCorruptPack(even when disabled),CorruptPackRecoverySkipped,DeletedCorruptPack,DeletedHealthyPrefetchPack,PrefetchRestoreUnavailable.Tests
Full unit suite: 885 passed, 0 failed (11 pre-existing native-hook skips). The pre-existing
PackfileMaintenanceRewriteOnBadVerify(empty-stderr verify failure → MIDX rewrite only, no pack detection) is unchanged.Follow-up
A functional test that corrupts a real pack and asserts the mount self-heals (rather than looping) is tracked for the streaming follow-up PR.