Cache scraper responses in tmp/cache

README.md

@@ -134,7 +134,7 @@ thor docs:manifest  # Create the manifest file used by the app
 thor docs:generate  # Generate/scrape a documentation
 thor docs:page      # Generate/scrape a documentation page
 thor docs:package   # Package a documentation for use with docs:download
-thor docs:clean     # Delete documentation packages
+thor docs:clean     # Delete documentation packages and cached responses
 
 # Console
 thor console        # Start a REPL

docs/adding-docs.md

@@ -12,7 +12,7 @@ Adding a documentation may look like a daunting task but once you get the hang o
 5. Using the `thor docs:page [my_doc] [path]` command, check that the scraper works properly. Files will appear in the `public/docs/[my_doc]/` directory (but not inside the app as the command doesn't touch the index). `path` in this case refers to either the remote path (if using `UrlScraper`) or the local path (if using `FileScraper`).
 6. Generate the full documentation using the `thor docs:generate [my_doc] --force` command. Additionally, you can use the `--verbose` option to see which files are being created/updated/deleted (useful to see what changed since the last run), and the `--debug` option to see which URLs are being requested and added to the queue (useful to pin down which page adds unwanted URLs to the queue).
 7. Start the server, open the app, enable the documentation, and see how everything plays out.
-8. Tweak the scraper/filters and repeat 5) and 6) until the pages and metadata are ok.
+8. Tweak the scraper/filters and repeat 5) and 6) until the pages and metadata are ok. Only the first run downloads the pages; the ones after that are served from the [response cache](./scraper-reference.md#response-cache), until you run `thor docs:clean`.
 9. To customize the pages' styling, create an SCSS file in the `assets/stylesheets/pages/` directory and import it in `application.css.scss`. Both the file and CSS class should be named `_[type]` where [type] is equal to the scraper's `type` attribute (documentations with the same type share the same custom CSS and JS). Setting the type to `simple` will apply the general styling rules in `assets/stylesheets/pages/_simple.scss`, which can be used for documentations where little to no CSS changes are needed.
 10. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works).
 11. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. The icon spritesheet is automatically generated when you (re)start your local DevDocs instance.

docs/maintainers.md

@@ -76,7 +76,7 @@ In addition to the [publicly-documented commands](https://github.com/freeCodeCam
 
 - `thor docs:clean`
 
-  Shortcut command to delete all package files (once uploaded via `thor docs:upload`, they are not needed anymore).
+  Shortcut command to delete all package files (once uploaded via `thor docs:upload`, they are not needed anymore), as well as the responses cached by the scrapers in `tmp/cache` (see the [Scraper Reference](./scraper-reference.md#response-cache)).
 
 ## Shell completion for fish
 

docs/scraper-reference.md

@@ -211,6 +211,34 @@ It is useful to preserve whitespaces of code segments within non-pre blocks, bec
 
 
 
+## Response cache
+
+`UrlScraper` stores every response it fetches in `tmp/cache/[slug]` and serves subsequent runs from there. Tweaking filters and running `thor docs:generate` again is therefore fast and doesn't put any load on the source site. (`FileScraper` doesn't need a cache, as it already reads from the local filesystem.)
+
+The cache never expires. Run `thor docs:clean` to empty it, which is required to pick up changes made to the source site. Only successful responses are stored, so timeouts, 404s and server errors are requested anew on the next run.
+
+Each response is stored in its own file, named after a hash of the request — changing a scraper's `params` or `headers` invalidates its cache. The files are JSON, in the entry schema of the [HTTP Archive (HAR) format](http://www.softwareishard.com/blog/har-12-spec/), so that it's easy to see what a scraper got back:
+
+```json
+{
+  "startedDateTime": "2026-08-15T11:05:10.430Z",
+  "time": 412,
+  "request": {
+    "method": "GET",
+    "url": "https://vite.dev/guide/",
+    "headers": [{ "name": "User-Agent", "value": "DevDocs" }]
+  },
+  "response": {
+    "status": 200,
+    "headers": [{ "name": "Content-Type", "value": "text/html; charset=utf-8" }],
+    "content": { "size": 57302, "mimeType": "text/html; charset=utf-8", "text": "<!doctype html>…" }
+  },
+  "_effectiveUrl": "https://vite.dev/guide/"
+}
+```
+
+(Fields irrelevant here are elided. Redirections are followed transparently, so an entry only holds the last response of a chain, and `_effectiveUrl` is the URL it ended up at.)
+
 ## Keeping scrapers up-to-date
 
 In order to keep scrapers up-to-date the `get_latest_version(opts)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations.

fish/completions/devdocs.fish

@@ -27,7 +27,7 @@ end
 complete -c devdocs -f
 
 # Commands (the `docs:` namespace is implied)
-complete -c devdocs -n __fish_is_first_arg -a clean -d 'Delete documentation packages'
+complete -c devdocs -n __fish_is_first_arg -a clean -d 'Delete documentation packages and cached responses'
 complete -c devdocs -n __fish_is_first_arg -a commit -d 'Commit the generated documentations'
 complete -c devdocs -n __fish_is_first_arg -a download -d 'Download documentation packages'
 complete -c devdocs -n __fish_is_first_arg -a generate -d 'Generate a documentation'

lib/docs.rb

@@ -25,6 +25,9 @@ module Docs
   mattr_accessor :store_path
   self.store_path = File.expand_path '../public/docs', @@root_path
 
+  mattr_accessor :cache_path
+  self.cache_path = File.expand_path '../tmp/cache', @@root_path
+
   mattr_accessor :rescue_errors
   self.rescue_errors = false
 

lib/docs/core/request.rb

@@ -13,18 +13,31 @@ def self.run(*args, &block)
       request.run
     end
 
+    # The ResponseCache the request reads from and writes to, if any.
+    attr_reader :cache
+
     def initialize(url, options = {})
-      super url.to_s, DEFAULT_OPTIONS.merge(options)
+      options = DEFAULT_OPTIONS.merge(options)
+      @cache = options.delete(:cache)
+      super url.to_s, options
+    end
+
+    def cached_response
+      cache.get(self) if cache
     end
 
     def response=(value)
-      value.extend Response if value
+      if value
+        value.extend Response
+        cache.set(self, value) if cache
+      end
       super
     end
 
     def run
       instrument 'response.request', url: base_url do |payload|
-        response = super
+        cached = cached_response
+        response = cached ? finish(cached) : super
         payload[:response] = response
         response
       end

lib/docs/core/requester.rb

@@ -38,6 +38,31 @@ def queue(request)
       super
     end
 
+    # Cached responses are set aside instead of being handed over right away,
+    # because delivering them from here would nest one request's callbacks
+    # inside the previous one's and blow the stack on large documentations.
+    def add(request)
+      if response = request.cached_response
+        cached_responses << [request, response]
+      else
+        super
+      end
+    end
+
+    def run
+      loop do
+        while pair = cached_responses.shift
+          pair[0].finish(pair[1])
+        end
+        break if queued_requests.empty? && multi.easy_handles.empty?
+        super
+      end
+    end
+
+    def cached_responses
+      @cached_responses ||= []
+    end
+
     def on_response(&block)
       @on_response ||= []
       @on_response << block if block

lib/docs/core/response_cache.rb

@@ -0,0 +1,165 @@
+require 'base64'
+require 'fileutils'
+require 'json'
+require 'time'
+
+module Docs
+  # Stores the responses fetched by a scraper on the filesystem so that
+  # subsequent runs are served from disk instead of hitting the network.
+  #
+  # Each scraper gets its own directory (tmp/cache/<slug>) in which every
+  # response is stored as an HTTP Archive (HAR) entry:
+  # http://www.softwareishard.com/blog/har-12-spec/
+  #
+  # An archive is a log of many entries; keeping one entry per file instead
+  # means the cache stays incremental (a run that's interrupted keeps whatever
+  # it fetched, and reading one page doesn't parse the whole archive), at the
+  # cost of the files not being valid archives on their own.
+  #
+  # Run `thor docs:clean` to throw the cached responses away.
+  class ResponseCache
+    # Written to every cache directory, so that .clean can tell the scraper
+    # caches apart from the other things living in tmp/cache (e.g. the assets
+    # cache of the web app).
+    MARKER_FILENAME = '.scraper_cache'
+
+    EXTENSION = '.json'
+
+    # Deletes the cache directory of every scraper.
+    def self.clean
+      Dir[File.join(Docs.cache_path, '*', MARKER_FILENAME)].each do |marker|
+        FileUtils.rm_rf File.dirname(marker)
+      end
+    end
+
+    attr_reader :path
+
+    def initialize(path)
+      @path = path
+    end
+
+    # Returns the response stored for the given request, or nil.
+    def get(request)
+      entry = JSON.parse File.read(path_for(request), encoding: Encoding::UTF_8)
+      response = Typhoeus::Response.new(deserialize(entry))
+      response.cached = true
+      response
+    rescue Errno::ENOENT
+      nil
+    rescue StandardError
+      # Ignore (and overwrite) entries we can't make sense of.
+      nil
+    end
+
+    # Stores the response of the given request.
+    def set(request, response)
+      return unless cache_response?(response)
+
+      prepare
+      file = path_for(request)
+      temp = "#{file}.#{Process.pid}.tmp"
+      File.binwrite temp, JSON.pretty_generate(serialize(request, response))
+      File.rename temp, file
+    end
+
+    def path_for(request)
+      File.join path, "#{request.cache_key}#{EXTENSION}"
+    end
+
+    private
+
+    # Responses that aren't plain successes are left out, so that transient
+    # failures don't stick around forever.
+    def cache_response?(response)
+      !response.mock && !response.cached? && response.code == 200
+    end
+
+    # Redirections are followed transparently, so an entry only ever holds the
+    # last response of a chain. The url it ended up at doesn't fit anywhere in
+    # the spec, hence the custom field; HAR reserves the underscore prefix for
+    # those. Fields we don't collect are left at -1 or empty, as prescribed.
+    def serialize(request, response)
+      wait = (response.starttransfer_time.to_f * 1000).round
+      receive = (response.total_time.to_f * 1000).round - wait
+
+      { 'startedDateTime' => Time.now.utc.iso8601(3),
+        'time' => wait + receive,
+        'request' => {
+          'method' => request.options.fetch(:method, :get).to_s.upcase,
+          'url' => request.base_url.to_s,
+          'httpVersion' => '',
+          'cookies' => [],
+          'headers' => name_value_pairs(request.options[:headers]),
+          'queryString' => name_value_pairs(request.options[:params]),
+          'headersSize' => -1,
+          'bodySize' => 0 },
+        'response' => {
+          'status' => response.code,
+          'statusText' => response.status_message.to_s,
+          'httpVersion' => response.http_version ? "HTTP/#{response.http_version}" : '',
+          'cookies' => [],
+          'headers' => name_value_pairs(response.headers),
+          'content' => content(response),
+          'redirectURL' => '',
+          'headersSize' => -1,
+          'bodySize' => response.body.to_s.bytesize },
+        'cache' => {},
+        'timings' => { 'send' => -1, 'wait' => wait, 'receive' => receive },
+        '_effectiveUrl' => response.effective_url.to_s }
+    end
+
+    def deserialize(entry)
+      response = entry['response']
+      { code: response['status'],
+        headers: header_hash(response['headers']),
+        body: body(response['content']),
+        effective_url: entry['_effectiveUrl'],
+        return_code: :ok,
+        total_time: entry['time'].to_f / 1000 }
+    end
+
+    def content(response)
+      body = response.body.to_s
+      text = body.dup.force_encoding(Encoding::UTF_8)
+      result = { 'size' => body.bytesize,
+                 'mimeType' => (response.headers || {})['Content-Type'].to_s }
+
+      if text.valid_encoding?
+        result['text'] = text
+      else
+        # The spec's way out for bodies that aren't valid JSON strings.
+        result['text'] = Base64.strict_encode64(body)
+        result['encoding'] = 'base64'
+      end
+
+      result
+    end
+
+    # Responses come off the wire as binary, and are handed back as such, so
+    # that scrapers see the same thing whether or not the cache was used.
+    def body(content)
+      text = content['text'].to_s
+      content['encoding'] == 'base64' ? Base64.strict_decode64(text) : text.b
+    end
+
+    def name_value_pairs(hash)
+      (hash || {}).flat_map do |name, value|
+        Array(value).map { |value| { 'name' => name.to_s, 'value' => value.to_s } }
+      end
+    end
+
+    def header_hash(pairs)
+      (pairs || []).each_with_object({}) do |pair, hash|
+        name, value = pair['name'], pair['value']
+        hash[name] = hash.key?(name) ? Array(hash[name]) << value : value
+      end
+    end
+
+    def prepare
+      return if @prepared
+      FileUtils.mkdir_p path
+      FileUtils.touch File.join(path, MARKER_FILENAME)
+      @prepared = true
+    end
+  end
+end

lib/docs/core/scrapers/url_scraper.rb

@@ -39,11 +39,15 @@ def request_all(urls, &block)
     end
 
     def request_options
-      options = { params: self.class.params, headers: self.class.headers }
+      options = { params: self.class.params, headers: self.class.headers, cache: response_cache }
       options[:accept_encoding] = 'gzip' if self.class.force_gzip
       options
     end
 
+    def response_cache
+      @response_cache ||= ResponseCache.new(File.join(Docs.cache_path, self.class.slug))
+    end
+
     def process_response?(response)
       if response.error?
         raise <<~ERROR

lib/tasks/docs.thor

@@ -159,9 +159,10 @@ class DocsCLI < Thor
     handle_doc_not_found_error(error)
   end
 
-  desc 'clean', 'Delete documentation packages'
+  desc 'clean', 'Delete documentation packages and cached responses'
   def clean
     File.delete(*Dir[File.join Docs.store_path, '*.tar.gz'])
+    Docs::ResponseCache.clean
     puts 'Done'
   end
 

test/lib/docs/core/request_test.rb

@@ -66,4 +66,48 @@ def request(url = self.url, **options)
       assert_includes response.singleton_class.ancestors, Docs::Response
     end
   end
+
+  describe "with a :cache" do
+    let :cache_path do
+      File.join tmp_path, 'request_cache'
+    end
+
+    let :cache do
+      Docs::ResponseCache.new cache_path
+    end
+
+    let :fetched_response do
+      Typhoeus::Response.new(
+        code: 200, headers: {}, body: 'body', effective_url: url, return_code: :ok)
+    end
+
+    def request(**options)
+      super(url, cache: cache, **options)
+    end
+
+    after do
+      FileUtils.rm_rf cache_path
+    end
+
+    it "isn't passed on as a request option" do
+      assert_equal cache, request.cache
+      refute request.options.key?(:cache)
+    end
+
+    it "stores the response" do
+      request.response = fetched_response
+      assert cache.get(request)
+    end
+
+    it "returns the cached response instead of making a request" do
+      request.response = fetched_response
+      response = request.run
+      assert response.cached?
+      assert_equal 'body', response.body
+    end
+
+    it "makes a request when the response isn't cached" do
+      assert_equal self.response, request.run
+    end
+  end
 end