Fix duplicate fetches and 502s on concurrent cache misses - #329
Open
montehurd wants to merge 2 commits into
Open
Fix duplicate fetches and 502s on concurrent cache misses#329montehurd wants to merge 2 commits into
montehurd wants to merge 2 commits into
Conversation
storeArtifact returned a CacheResult holding an open file handle. A handle has one read position, so it can only ever serve a single caller, which is what blocks sharing one fetch between concurrent requests. Return the artifact and its storage path instead, and let each caller open its own reader through openStoredArtifact. Threading that type through fetchAndCache, fetchAndCacheFromURL and their error paths is mechanical; behaviour is unchanged.
A cache miss went from checkCache straight to an upstream fetch with nothing tracking in-flight work, so N concurrent requests for one uncached artifact produced N upstream fetches and N stores to the same key. That is the CI shape: parallel jobs installing overlapping dependencies against a cold cache. The duplicate stores also fail requests, racing fileblob's per-key ".attrs" sidecar into a partial read served as a 502. Over 12 runs of 8 simultaneous requests for one uncached tarball, against bb2205a: before, 8 fetches per run and 12 of 96 responses were 502; after, 1 fetch per run and none failed. Route both miss paths through a shared in-flight map keyed on the artifact, including the download URL and upstream-declared hash so callers expecting different bytes never share a fetch. singleflight does not fit: Do gives waiters no way to leave, while DoChan lets the caller running the fetch abandon it, breaking storeArtifact's scan-on-disconnect contract. Deciding roles under a mutex gives both behaviours. The fetch runs on the first caller's context and is seen through; waiters leave when their own clients do. This removes the sidecar trigger on this path. The race is in fileblob and three writers bypass this path entirely, so it is fixed separately. Fewer failures now reach the circuit breaker, so it trips later. Sixteen concurrent callers against real file:// storage fail 10 of 10 runs on main and pass 10 of 10 here. Other tests pin key discrimination, failure propagation, resolver-path coalescing, per-caller readers, waiter cancellation, key release and panic safety. allocs/op is unchanged. mockStorage gains a mutex so concurrent tests can use it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
A cache miss goes from
checkCachestraight to an upstream fetch with nothing tracking in-flight work, so N concurrent requests for one uncached artifact produce N upstream fetches and N stores to the same storage key. That is the CI shape: parallel jobs installing overlapping dependencies against a cold cache.The duplicate stores also fail requests, racing fileblob's per-key
.attrssidecar into a partial read served as a 502.Over 12 runs of 8 simultaneous requests for one uncached tarball, against
bb2205a:The fix
Both miss paths route through a shared in-flight map keyed on the artifact. The key includes the download URL and upstream-declared hash, so callers expecting different bytes never share a fetch.
x/sync/singleflightwould be the obvious tool and is already used for ECR tokens, but neither mode fits:Dogives waiters no way to leave, whileDoChanlets the caller running the fetch abandon it on its own cancellation, which breaks the scan-on-disconnect contract instoreArtifact. Deciding the roles under a mutex makes both behaviours available.Two commits. The first returns the stored artifact from
storeArtifactinstead of an open reader, a mechanical refactor with no behaviour change. The second adds the coalescing. The reproduction tests fail at bothmainand the refactor commit, so a bisect lands on the right one.Scope
cacheMetadataBlob,storeContainerMetadataand the Gradle build cachePUTreach storage without coming through here.fileblob's and is addressed separately in Stop writing fileblob's .attrs sidecar #328.Green on ubuntu, macOS and Windows.