Generate sitemap.xml - #2882
Draft
mroderick wants to merge 1 commit into
Draft
Generate sitemap.xml#2882mroderick wants to merge 1 commit into
mroderick wants to merge 1 commit into
Conversation
Crawlers requesting /sitemap.xml hit the chapter catch-all route and got a 404. The previous attempt (PR #2349) shipped sitemap_generator with a static sitemap.xml.gz containing only the root URL, which went stale and did not fix the 404s for the bare /sitemap.xml path. Serve the sitemap from a dynamic endpoint instead: - Add SitemapsController and an xml.builder view rendering static pages, active chapters, workshops, events and meetings, each with a lastmod value from updated_at. - Fragment-cache each section, keyed by its records' latest updated_at and bounded by expires_in, so unchanged sections are not re-rendered and stale sections (for example after record deletions) self-heal. Timestamps use to_f because raw Time values in cache keys are stringified with second precision. - Add updated_at indexes on workshops, events and meetings so the cache key computation does not do a sequential scan per request, and set public HTTP caching (1 hour) on the response so the CDN absorbs repeat crawler fetches. - Enable gzip in the Heroku nginx config for proxied responses, so the uncompressed XML is compressed (the Heroku router does not compress). - Remove sitemap_generator and the committed public/sitemap.xml.gz; robots.txt now points at /sitemap.xml.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2347.
Crawlers requesting
/sitemap.xmlfell through to the chapter catch-all route and got a 404, producing log noise. The previous attempt (#2349) shippedsitemap_generatorwith a staticsitemap.xml.gzcontaining only the root URL, which went stale and did not fix the 404s for the bare/sitemap.xmlpath.This PR replaces the static-file approach with a dynamic endpoint:
SitemapsControllerrenderssitemap.xmlwith entries for static pages, chapters, workshops, events and meetings, each with alastmodvalue fromupdated_at.updated_atand bounded byexpires_in. A new workshop invalidates only the workshops section; chapters, events and meetings stay cached until their own data changes, and stale sections (for example after a record is deleted) self-heal after a day. Cache keys use.to_fbecause rawTimevalues in cache keys are stringified with second precision.updated_atindexes on workshops, events and meetings keep the per-request cache-key computation off sequential scans; the response sets public HTTP caching (1 hour) so the CDN absorbs repeat crawler fetches.config/nginx.conf.erb) for proxied responses - the Heroku router does not compress, and Cloudflare only re-compresses when the origin has not already.sitemap_generator,config/sitemap.rband the committedpublic/sitemap.xml.gzare removed;robots.txtnow points at/sitemap.xml.disable_ddl_transaction!+algorithm: :concurrently), following the repo's existing convention.Notes for reviewers
rake sitemap:refresh- the gem is gone, so such an entry would fail post-deploy.https://codebar.io/sitemap.xml.gz(e.g. an old Search Console submission) will now get a 404;robots.txtpoints crawlers at the new path.Testing
spec/requests/sitemap_spec.rb) cover the XML response and content type, URL coverage for each record type,lastmod, publicCache-Controlheaders, and cache invalidation: new records appear once their section key changes, and deleted records disappear once the section expires (viatravel).bundle exec parallel_rspec spec/ -n 3- 1448 examples, 0 failures.