Add per-ecosystem artifact retention (age-based eviction)

[pinguinfuss]

Summary

Cached artifacts currently stay in storage forever unless storage.max_size is set, and that limit is a single global LRU across all ecosystems. There is no way to say "keep npm tarballs for 30 days, OCI blobs for 14 days, and Maven artifacts indefinitely". This issue proposes a storage.retention block that evicts artifacts per ecosystem (and optionally per package) after they have not been accessed for a configurable duration.

Current behaviour

• storage.max_size triggers size-based LRU eviction, global for all ecosystems (internal/server/eviction.go). It is DB-driven via artifacts.last_accessed_at.
• gradle.build_cache.max_age / max_size evict Gradle build-cache entries only, storage-listing-driven (internal/server/gradle_cache_eviction.go).
• Nothing else is ever evicted. Without max_size, the cache grows without bound.

Proposed configuration

Follows the existing cooldown pattern (default + ecosystems map + packages map keyed by PURL):

storage:
  max_size: "10GB"          # unchanged, size-based LRU still applies
  retention:
    # Evict artifacts that have not been accessed for longer than this.
    # "0" or empty disables age-based eviction (current behaviour).
    default: "0"
    ecosystems:
      npm: "30d"
      oci: "14d"
      alpine: "7d"
      maven: "0"            # explicitly unlimited
    packages:               # optional, keys as in cooldown.packages
      "pkg:npm/lodash": "0"
    sweep_interval: "10m"

Environment overrides: PROXY_STORAGE_RETENTION_DEFAULT, PROXY_STORAGE_RETENTION_SWEEP_INTERVAL (maps have no env override, same as cooldown).

Semantics

• Retention is measured from the last access, not from the fetch time. Artifacts are immutable, so a fixed max age would only force re-downloads of actively used packages. For artifacts that were never served (for example mirrored ones), fetched_at is used as fallback: COALESCE(last_accessed_at, fetched_at) < cutoff.
• Resolution order: package override → ecosystem override → default. A value of "0" at any level means "never evict by age" and stops the lookup.
• Durations accept a d suffix, using the same parser as cooldown.
• Valid ecosystem keys are the literals the handlers pass to the artifact fetch (also the first segment of the storage path): npm, pypi, cargo, gem, golang, hex, pub, maven, nuget, composer, conan, conda, cran, julia, swift, deb, rpm, alpine, helm, oci. Unknown keys fail validation so typos are not silently ignored.
• The retention sweep runs before the size-based LRU pass, so max_size operates on the already-pruned set.

Implementation outline

• internal/config/config.go: add RetentionConfig under StorageConfig, validation of durations and ecosystem keys, parse helpers.
• internal/database/queries.go: query for expired artifacts per ecosystem (join artifacts → versions → packages.ecosystem, batch limit as in GetLeastRecentlyUsedArtifacts) plus SELECT DISTINCT ecosystem FROM packages.
• internal/server/retention.go (new): sweeper modelled on evictLRU, reusing store.Delete and ClearArtifactCache. Start it from Server.Start next to startEvictionLoop.
• Tests: internal/server/retention_test.go using the harness from eviction_test.go; config parsing and validation cases in config_test.go.
• Docs: config.example.yaml, storage section in docs/configuration.md, README example config.
• Optional in the same PR: a proxy_artifacts_evicted_total{reason, ecosystem} counter covering both LRU and retention eviction (neither has a metric today).

Notes and edge cases

• Mirrored artifacts. Packages preloaded with proxy mirror for offline use are evicted once they age out, even if never served. Document that the clock starts at mirror time. A pinned flag would need a schema migration and is out of scope here.
• Multiple replicas on PostgreSQL. Concurrent sweeps are safe: Delete and ClearArtifactCache are idempotent, matching the existing LRU behaviour.
• Key naming. Artifact ecosystem keys differ from metadata-cache keys in a few places (deb vs debian, oci vs oci-tags/oci-manifest). Retention uses the artifact keys.
• Existing bug worth fixing alongside. The comment on gradle.build_cache.max_age promises "7d", but validation uses time.ParseDuration and rejects it. Switching to the day-aware parser fixes both.

Out of scope

• Retention for the metadata cache (_metadata/ is never evicted today; separate issue).
• Pinning mirrored artifacts.
• Retention based on fetch age instead of last access.

[pinguinfuss]

@andrew is that something worth implementing?

[andrew]

Yeah sounds good!