Skip to content

conda: large repodata.json fails through the proxy #321

Description

@pinguinfuss

Pulling from the big conda channels through the proxy doesn't work. mamba/conda request /conda/conda-forge/linux-64/repodata.json and either get a 502 (with cache_metadata: true) or a truncated body (default config, caching off).

Cause

repodata.json is plain JSON and huge. conda-forge linux-64 is roughly 440 MB uncompressed and a few tens of MB gzipped. Two things bite:

As far as I can tell conda is the only ProxyCached ecosystem where this matters. The others serve indexes that are already compressed or tiny, and their upstreams return the same bytes whether you ask for gzip or not.

Proposed Fix

Request gzip explicitly for repodata.json and current_repodata.json instead of identity. Setting Accept-Encoding ourselves already disables Go's transparent decompression, so the compressed bytes plus the Content-Encoding: gzip header get cached and replayed as-is — the same mechanism #304 relies on for identity. Both hops stay compressed, the cache stores the small blob, and metadata_max_size then applies to the compressed size, which fits easily.

conda, mamba and pixi all send Accept-Encoding: gzip for .json URLs and decode the header themselves, so serving it compressed is fine. repodata.json.bz2 is already compressed and should stay on identity.

This needs the ProxyCached path to accept a per-call Accept-Encoding (identity remains the default, so apk/apt/rpm/go/hex are untouched).

Related: #304, #305

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions