Skip to content

Generate sitemap.xml - #2882

Draft
mroderick wants to merge 1 commit into
masterfrom
feature/issue-2347-sitemap
Draft

mroderick wants to merge 1 commit into
masterfrom
feature/issue-2347-sitemap

Conversation

@mroderick

Copy link
Copy Markdown
Collaborator

Closes #2347.

Crawlers requesting /sitemap.xml fell through to the chapter catch-all route and got a 404, producing log noise. The previous attempt (#2349) shipped sitemap_generator with a static sitemap.xml.gz containing only the root URL, which went stale and did not fix the 404s for the bare /sitemap.xml path.

This PR replaces the static-file approach with a dynamic endpoint:

  • SitemapsController renders sitemap.xml with entries for static pages, chapters, workshops, events and meetings, each with a lastmod value from updated_at.
  • Each section is fragment-cached, keyed by its records' latest updated_at and bounded by expires_in. A new workshop invalidates only the workshops section; chapters, events and meetings stay cached until their own data changes, and stale sections (for example after a record is deleted) self-heal after a day. Cache keys use .to_f because raw Time values in cache keys are stringified with second precision.
  • New updated_at indexes on workshops, events and meetings keep the per-request cache-key computation off sequential scans; the response sets public HTTP caching (1 hour) so the CDN absorbs repeat crawler fetches.
  • Gzip is enabled in the Heroku nginx config (config/nginx.conf.erb) for proxied responses - the Heroku router does not compress, and Cloudflare only re-compresses when the origin has not already.
  • sitemap_generator, config/sitemap.rb and the committed public/sitemap.xml.gz are removed; robots.txt now points at /sitemap.xml.
  • The migration builds the three indexes concurrently (disable_ddl_transaction! + algorithm: :concurrently), following the repo's existing convention.
Notes for reviewers
  • The endpoint is unauthenticated by design: it lists URLs of pages that are already public (chapter pages, workshops, events, meetings, static guides).
  • Fragment-cached sections pin absolute URLs built from the request host. Production is single-host (codebar.io), so this is fine; it only matters if the app is ever served under a second hostname.
  • Deploy check: no leftover Heroku Scheduler entry should run rake sitemap:refresh - the gem is gone, so such an entry would fail post-deploy.
  • External consumers that pinned https://codebar.io/sitemap.xml.gz (e.g. an old Search Console submission) will now get a 404; robots.txt points crawlers at the new path.

Testing

  • New request specs (spec/requests/sitemap_spec.rb) cover the XML response and content type, URL coverage for each record type, lastmod, public Cache-Control headers, and cache invalidation: new records appear once their section key changes, and deleted records disappear once the section expires (via travel).
  • Full suite: bundle exec parallel_rspec spec/ -n 3 - 1448 examples, 0 failures.
  • RuboCop clean on all changed Ruby files.

Crawlers requesting /sitemap.xml hit the chapter catch-all route and got
a 404. The previous attempt (PR #2349) shipped sitemap_generator with a
static sitemap.xml.gz containing only the root URL, which went stale and
did not fix the 404s for the bare /sitemap.xml path.

Serve the sitemap from a dynamic endpoint instead:

- Add SitemapsController and an xml.builder view rendering static pages,
  active chapters, workshops, events and meetings, each with a lastmod
  value from updated_at.
- Fragment-cache each section, keyed by its records' latest updated_at
  and bounded by expires_in, so unchanged sections are not re-rendered
  and stale sections (for example after record deletions) self-heal.
  Timestamps use to_f because raw Time values in cache keys are
  stringified with second precision.
- Add updated_at indexes on workshops, events and meetings so the cache
  key computation does not do a sequential scan per request, and set
  public HTTP caching (1 hour) on the response so the CDN absorbs
  repeat crawler fetches.
- Enable gzip in the Heroku nginx config for proxied responses, so the
  uncompressed XML is compressed (the Heroku router does not compress).
- Remove sitemap_generator and the committed public/sitemap.xml.gz;
  robots.txt now points at /sitemap.xml.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate sitemap.xml

1 participant