Conversation
When CarrierWave storage moved from the SFTP asset host (assets.codebar.io) to S3 in July 2025, existing sponsor logo files were not migrated. Around 500 logos referenced by the sponsors table return 403 because the objects do not exist in the prod-sponsor-logos bucket. Add a SponsorLogoRestore service plus a sponsor_logos:restore rake task (make target restore_sponsor_logos) that: - reads the live /sponsors page and HEAD-checks each logo URL against the bucket, treating anything but 200 as missing - looks the old asset path up in the Wayback Machine CDX index (all 500 missing logos are archived there) - downloads the archived image and re-uploads it to S3 under the same key the sponsors table references, with a public-read ACL - verifies the upload via the public URL The task is idempotent: logos already present are skipped, failed restores are reported and can be retried by re-running. It needs no database access and reads the same AWS_* environment variables the CarrierWave initializer uses, so it can run from any machine.
Apply the 11 confirmed findings from the code review of feature/restore-sponsor-logos: - Send limit=100000 (no Ruby underscore) to the Wayback CDX API (#8) - Reject 200 HTML error pages as image downloads; '<' bodies are only accepted as SVG markup (#7) - Treat a CDX 200 HTML response as an unavailable index instead of an empty one (#4); retry the CDX fetch once (#1) - Rescue network exceptions in the availability-check loop and record them as per-logo failures instead of crashing the run (#1) - Set explicit Net::HTTP open/read timeouts (5s/30s) (#3) - Distinguish permanent 404 misses from transient failures; honour Retry-After on 429 responses (#5) - Signal restore outcomes with explicit [:ok]/[:failed, reason] tuples instead of return-type sniffing (#2) - Read bucket/region from a shared AWS_ASSETS constant so the defaults live in one place (config/initializers/aws_assets.rb) (#9) - Make the retry delay injectable so tests do not sleep (#10) - Stub SPONSORS_URL in the spec instead of hardcoding the URL (#12) - Add tests for the HTML-download rejection, availability-check failures, CDX unavailability, 404 short-circuit, and 429 handling (#6) Specs: 11 examples green in ~1.4s; rubocop clean; zeitwerk:check passes.
Cover the gaps the code review flagged in the sponsor logo restore work: - Rake task contract: summary output, service invocation, failure listing and non-zero abort (spec/lib/tasks/sponsor_logos_rake_spec.rb) - HTTP plumbing: get! raise path, relative Location redirects, redirect limit exhaustion (spec/services/sponsor_logo_restore/http_spec.rb) - Politeness pacing: delay before each download, retry delay between attempts, Retry-After honoured on 429s - content_type extension mapping for all branches - Idempotent re-run asserts the CDX lookup is skipped entirely - SVG-markup bodies accepted by the image sniffer (HTML still rejected)
… runs Enable learning-oriented runs of the restore task before committing to the full ~500-logo batch: - RESTORE_LIMIT=N restores only the first N missing logos; the rest are reported as deferred (a new summary line) and left for later runs. - DRY_RUN=1 rehearses the full pipeline -- page scan, HEAD checks, CDX matching, archive downloads with validation -- without uploading anything to S3, so it also works before AWS credentials are set. Both are idempotent and covered by specs; the rake task announces dry runs and prints deferred counts.
The first real dry run surfaced two problems: - The Wayback CDX index query for the full asset prefix can exceed the 30s read timeout used for ordinary probes, so both attempts failed with Net::ReadTimeout. The CDX fetch now uses its own 180s read timeout, and reports its retry. - The task was silent for minutes (page scan, 721 availability checks, CDX fetch). It now reports phase progress to stderr: found count, availability ticks every 50 probes, CDX fetch start/retry, and restore progress every 10 logos. The reporter is injectable; the default writes to stderr, keeping the stdout summary clean.
Practice runs kept re-paying the two slow, stable costs: ~720 availability HEAD probes and the multi-minute Wayback CDX index query. Both are now cached on disk (tmp/cache/sponsor_logos, default TTL 6h) so a second run skips them and finishes in seconds. - Availability classifications are cached per URL; post-upload verification HEADs always stay live. - The raw CDX response is cached and re-parsed each run. - REFRESH_CACHE=1 forces a refetch; CACHE_TTL_MINUTES bounds staleness. - Caching is disabled in the test environment (SPONSOR_LOGO_CACHE=1 opts in) so the suite stays hermetic; cache behaviour is covered by spec/services/sponsor_logo_restore_caching_spec.rb, which proves the second run makes no classification HEADs and no CDX request while verification checks stay live.
The first practice run hit HTTP 503 from web.archive.org on the CDX query. The fetch now allows 3 attempts and honours the Retry-After header on throttled responses, falling back to a 15s backoff (CDX_RETRY_DELAY, injectable as cdx_retry_delay) when the header is absent. Failure reasons now carry the CDX-specific status instead of a generic message.
… variants The first real batch revealed why 10 of 25 downloads failed: the old asset host was already dying before the migration, and Wayback's final crawl (2025-06-17) stored Cloudflare 522 error pages instead of logos for some files. The download failures were the magic-byte check correctly rejecting those stubs. - The CDX index now ranks captures: complete image captures (200 + image/* mimetype) beat error stubs, latest capture wins within a rank, so a 2025 error page no longer displaces a 2023 logo. - restore falls back to the CarrierWave thumb variant (thumb_<filename>, max 178x65) when no intact capture of the original exists -- the thumbs were crawled months earlier and fit the page's small-image styling. - Batch analysis against the real CDX data: 489 of 500 missing logos recover from the exact filename, 7 more via thumbs; ~4 have no intact capture anywhere (origin was 522 at every capture).
The full recovery run surfaced sponsor 873, whose avatar is a favicon.ico. Its Wayback capture is intact (200, image/x-icon) but the magic-byte check only knew JPEG/PNG/GIF/SVG. Recognise the ICO header (00 00 01 00) and upload .ico files as image/x-icon. Last remaining wayback-recoverable logo; only the three 522-casualty sponsors remain manual.
…red logos A verified restore left the availability cache saying '403' until the TTL expired, so every run in between re-classified the restored logos as missing and re-downloaded all of them from Wayback (the full run after the batch restored 496 logos then re-attempted all 500 on the next run). The post-upload verification result (200) is now written to the availability cache, so the next run classifies those logos as present instantly. With this, a warm-cache run on the finished recovery state does one page fetch, one archive download (the one still-missing logo), one upload, and one verification -- seconds instead of ~30 minutes.
The review P1 covered discovery-phase exceptions, but a download-time Net::OpenTimeout still propagated out of restore_one and aborted the whole batch (hit for real at logo 150/500). Failures mid-restore are now recorded per logo and the batch carries on.
Collaborator
Author
|
This PR is not actually meant to be merged, just a place to keep the git commits, ugly history and all and to collect evidence for future analysis. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Around 500 of the 721 sponsor logos on codebar.io/sponsors return HTTP 403. When CarrierWave storage moved from the SFTP asset host (assets.codebar.io) to S3 in July 2025 (8739cf7), the existing logo files were never migrated, but the sponsors table still references the old filenames.
This adds a rake task that restores the missing files from the Wayback Machine, which archived the old asset host. All 500 missing logos were confirmed present in the archive (matched by sponsor ID + filename, case-insensitive), and sampled downloads verified as valid PNG/JPEG.
Before
After
How it works
sponsor_logos:restore(make targetrestore_sponsor_logos) runs the newSponsorLogoRestoreservice:assets.codebar.io/b/uploads/sponsor/avatar*once and matches missing logos by sponsor ID + filename.The service is split into
SponsorLogoRestore(orchestration + S3),SponsorLogoRestore::Http(Net::HTTP plumbing) andSponsorLogoRestore::Wayback(CDX index + archive downloads).Idempotency and credentials
AWS_ACCESS_KEY,AWS_SECRET_ACCESS_KEY,AWS_REGIONdefaulting toeu-north-1,S3_BUCKET_NAMEdefaulting toprod-sponsor-logos), plus an optionalSPONSORS_URLoverride for staging.mise.local.toml) and runmake restore_sponsor_logos. ~500 archive downloads with a 1s politeness delay take a few minutes.Testing
spec/services/sponsor_logo_restore_spec.rbcovers: restore of a missing logo found in the archive, skip-when-present idempotency, download retry, unmatched logos reported as failed, failed upload verification, and S3 errors surfaced as failures. All HTTP is stubbed with WebMock and the S3 client is an instance double, so no network is touched.bundle exec rails zeitwerk:checkpasses; rubocop clean on all new files.Limitations: discovery is driven by the /sponsors page, so logos of hidden-level sponsors (shown on event pages but not /sponsors) are out of scope for now; the page-level check can be pointed at any URL via
SPONSORS_URL.Small-batch and dry-run usage
Learn from small runs before the full batch:
DRY_RUN=1 make restore_sponsor_logosrehearses the whole pipeline (page scan, HEAD checks, CDX matching, archive downloads with magic-byte validation) without uploading anything to S3 — it also works before AWS credentials are set.RESTORE_LIMIT=5 make restore_sponsor_logosrestores only the first 5 missing logos; the rest are reported as deferred and left for later runs.DRY_RUN=1 RESTORE_LIMIT=5) they are a complete dress rehearsal of 5 logos with zero S3 writes.tmp/cache/sponsor_logos, TTL 6h) so repeat runs finish in seconds;REFRESH_CACHE=1forces a refetch andCACHE_TTL_MINUTESbounds staleness. Post-upload verification checks are always live.