You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Pulling from the big conda channels through the proxy doesn't work. mamba/conda request /conda/conda-forge/linux-64/repodata.json and either get a 502 (with cache_metadata: true) or a truncated body (default config, caching off).
Cause
repodata.json is plain JSON and huge. conda-forge linux-64 is roughly 440 MB uncompressed and a few tens of MB gzipped. Two things bite:
Since fix(handler): preserve upstream content-encoding for cached metadata #304 the ProxyCached path sends Accept-Encoding: identity so cached metadata bytes stay verbatim. That's the right call for signed indexes (Release, repomd.xml, APKINDEX.tar.gz), but conda's repodata goes through the same path, so every fetch now pulls the full uncompressed file over the upstream hop instead of the gzip'd one Go used to negotiate. With the default http_timeout of 30s the download can get cut off mid-stream; the 200 has already been written by then, so the client just receives a short body and fails to parse it.
As far as I can tell conda is the only ProxyCached ecosystem where this matters. The others serve indexes that are already compressed or tiny, and their upstreams return the same bytes whether you ask for gzip or not.
Proposed Fix
Request gzip explicitly for repodata.json and current_repodata.json instead of identity. Setting Accept-Encoding ourselves already disables Go's transparent decompression, so the compressed bytes plus the Content-Encoding: gzip header get cached and replayed as-is — the same mechanism #304 relies on for identity. Both hops stay compressed, the cache stores the small blob, and metadata_max_size then applies to the compressed size, which fits easily.
conda, mamba and pixi all send Accept-Encoding: gzip for .json URLs and decode the header themselves, so serving it compressed is fine. repodata.json.bz2 is already compressed and should stay on identity.
This needs the ProxyCached path to accept a per-call Accept-Encoding (identity remains the default, so apk/apt/rpm/go/hex are untouched).
Pulling from the big conda channels through the proxy doesn't work.
mamba/condarequest/conda/conda-forge/linux-64/repodata.jsonand either get a 502 (withcache_metadata: true) or a truncated body (default config, caching off).Cause
repodata.jsonis plain JSON and huge. conda-forge linux-64 is roughly 440 MB uncompressed and a few tens of MB gzipped. Two things bite:ProxyCachedpath sendsAccept-Encoding: identityso cached metadata bytes stay verbatim. That's the right call for signed indexes (Release,repomd.xml,APKINDEX.tar.gz), but conda's repodata goes through the same path, so every fetch now pulls the full uncompressed file over the upstream hop instead of the gzip'd one Go used to negotiate. With the defaulthttp_timeoutof 30s the download can get cut off mid-stream; the 200 has already been written by then, so the client just receives a short body and fails to parse it.ReadMetadata, which enforcesmetadata_max_size(100 MiB by default). 440 MB doesn't fit, so we returnErrMetadataTooLarge→ 502. This half actually predates fix(handler): preserve upstream content-encoding for cached metadata #304: the limit always counted the decoded bytes, so linux-64 never fit in the metadata cache. fix(handler): preserve upstream content-encoding for cached metadata #304 just broke the uncached path as well.As far as I can tell conda is the only
ProxyCachedecosystem where this matters. The others serve indexes that are already compressed or tiny, and their upstreams return the same bytes whether you ask for gzip or not.Proposed Fix
Request
gzipexplicitly forrepodata.jsonandcurrent_repodata.jsoninstead ofidentity. SettingAccept-Encodingourselves already disables Go's transparent decompression, so the compressed bytes plus theContent-Encoding: gzipheader get cached and replayed as-is — the same mechanism #304 relies on for identity. Both hops stay compressed, the cache stores the small blob, andmetadata_max_sizethen applies to the compressed size, which fits easily.conda, mamba and pixi all send
Accept-Encoding: gzipfor.jsonURLs and decode the header themselves, so serving it compressed is fine.repodata.json.bz2is already compressed and should stay onidentity.This needs the
ProxyCachedpath to accept a per-callAccept-Encoding(identity remains the default, so apk/apt/rpm/go/hex are untouched).Related: #304, #305