diff --git a/README.md b/README.md index 17b9ec4dbe..ca8128c1f6 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,7 @@ thor docs:manifest # Create the manifest file used by the app thor docs:generate # Generate/scrape a documentation thor docs:page # Generate/scrape a documentation page thor docs:package # Package a documentation for use with docs:download -thor docs:clean # Delete documentation packages +thor docs:clean # Delete documentation packages and cached responses # Console thor console # Start a REPL diff --git a/docs/adding-docs.md b/docs/adding-docs.md index 5051d5ee31..8dcd7c088d 100644 --- a/docs/adding-docs.md +++ b/docs/adding-docs.md @@ -12,7 +12,7 @@ Adding a documentation may look like a daunting task but once you get the hang o 5. Using the `thor docs:page [my_doc] [path]` command, check that the scraper works properly. Files will appear in the `public/docs/[my_doc]/` directory (but not inside the app as the command doesn't touch the index). `path` in this case refers to either the remote path (if using `UrlScraper`) or the local path (if using `FileScraper`). 6. Generate the full documentation using the `thor docs:generate [my_doc] --force` command. Additionally, you can use the `--verbose` option to see which files are being created/updated/deleted (useful to see what changed since the last run), and the `--debug` option to see which URLs are being requested and added to the queue (useful to pin down which page adds unwanted URLs to the queue). 7. Start the server, open the app, enable the documentation, and see how everything plays out. -8. Tweak the scraper/filters and repeat 5) and 6) until the pages and metadata are ok. +8. Tweak the scraper/filters and repeat 5) and 6) until the pages and metadata are ok. Only the first run downloads the pages; the ones after that are served from the [response cache](./scraper-reference.md#response-cache), until you run `thor docs:clean`. 9. To customize the pages' styling, create an SCSS file in the `assets/stylesheets/pages/` directory and import it in `application.css.scss`. Both the file and CSS class should be named `_[type]` where [type] is equal to the scraper's `type` attribute (documentations with the same type share the same custom CSS and JS). Setting the type to `simple` will apply the general styling rules in `assets/stylesheets/pages/_simple.scss`, which can be used for documentations where little to no CSS changes are needed. 10. To add syntax highlighting or execute custom JavaScript on the pages, create a file in the `assets/javascripts/views/pages/` directory (take a look at the other files to see how it works). 11. Add the documentation's icon in the `public/icons/docs/[my_doc]/` directory, in both 16x16 and 32x32-pixels formats. The icon spritesheet is automatically generated when you (re)start your local DevDocs instance. diff --git a/docs/maintainers.md b/docs/maintainers.md index 5a13b09142..81140f279d 100644 --- a/docs/maintainers.md +++ b/docs/maintainers.md @@ -76,7 +76,7 @@ In addition to the [publicly-documented commands](https://github.com/freeCodeCam - `thor docs:clean` - Shortcut command to delete all package files (once uploaded via `thor docs:upload`, they are not needed anymore). + Shortcut command to delete all package files (once uploaded via `thor docs:upload`, they are not needed anymore), as well as the responses cached by the scrapers in `tmp/cache` (see the [Scraper Reference](./scraper-reference.md#response-cache)). ## Shell completion for fish diff --git a/docs/scraper-reference.md b/docs/scraper-reference.md index d5ed5074fa..9bf407bc4d 100644 --- a/docs/scraper-reference.md +++ b/docs/scraper-reference.md @@ -211,6 +211,34 @@ It is useful to preserve whitespaces of code segments within non-pre blocks, bec +## Response cache + +`UrlScraper` stores every response it fetches in `tmp/cache/[slug]` and serves subsequent runs from there. Tweaking filters and running `thor docs:generate` again is therefore fast and doesn't put any load on the source site. (`FileScraper` doesn't need a cache, as it already reads from the local filesystem.) + +The cache never expires. Run `thor docs:clean` to empty it, which is required to pick up changes made to the source site. Only successful responses are stored, so timeouts, 404s and server errors are requested anew on the next run. + +Each response is stored in its own file, named after a hash of the request — changing a scraper's `params` or `headers` invalidates its cache. The files are JSON, in the entry schema of the [HTTP Archive (HAR) format](http://www.softwareishard.com/blog/har-12-spec/), so that it's easy to see what a scraper got back: + +```json +{ + "startedDateTime": "2026-08-15T11:05:10.430Z", + "time": 412, + "request": { + "method": "GET", + "url": "https://vite.dev/guide/", + "headers": [{ "name": "User-Agent", "value": "DevDocs" }] + }, + "response": { + "status": 200, + "headers": [{ "name": "Content-Type", "value": "text/html; charset=utf-8" }], + "content": { "size": 57302, "mimeType": "text/html; charset=utf-8", "text": "…" } + }, + "_effectiveUrl": "https://vite.dev/guide/" +} +``` + +(Fields irrelevant here are elided. Redirections are followed transparently, so an entry only holds the last response of a chain, and `_effectiveUrl` is the URL it ended up at.) + ## Keeping scrapers up-to-date In order to keep scrapers up-to-date the `get_latest_version(opts)` method should be overridden. If `self.release` is defined, this should return the latest version of the documentation. If `self.release` is not defined, it should return the Epoch time when the documentation was last modified. If the documentation will never change, simply return `1.0.0`. The result of this method is periodically reported in a "Documentation versions report" issue which helps maintainers keep track of outdated documentations. diff --git a/fish/completions/devdocs.fish b/fish/completions/devdocs.fish index d7db77ace2..b6fdf087ec 100644 --- a/fish/completions/devdocs.fish +++ b/fish/completions/devdocs.fish @@ -27,7 +27,7 @@ end complete -c devdocs -f # Commands (the `docs:` namespace is implied) -complete -c devdocs -n __fish_is_first_arg -a clean -d 'Delete documentation packages' +complete -c devdocs -n __fish_is_first_arg -a clean -d 'Delete documentation packages and cached responses' complete -c devdocs -n __fish_is_first_arg -a commit -d 'Commit the generated documentations' complete -c devdocs -n __fish_is_first_arg -a download -d 'Download documentation packages' complete -c devdocs -n __fish_is_first_arg -a generate -d 'Generate a documentation' diff --git a/lib/docs.rb b/lib/docs.rb index a09aae76df..89cf7a20e7 100644 --- a/lib/docs.rb +++ b/lib/docs.rb @@ -25,6 +25,9 @@ module Docs mattr_accessor :store_path self.store_path = File.expand_path '../public/docs', @@root_path + mattr_accessor :cache_path + self.cache_path = File.expand_path '../tmp/cache', @@root_path + mattr_accessor :rescue_errors self.rescue_errors = false diff --git a/lib/docs/core/request.rb b/lib/docs/core/request.rb index b1e26e68d6..6d18cc1609 100644 --- a/lib/docs/core/request.rb +++ b/lib/docs/core/request.rb @@ -13,18 +13,31 @@ def self.run(*args, &block) request.run end + # The ResponseCache the request reads from and writes to, if any. + attr_reader :cache + def initialize(url, options = {}) - super url.to_s, DEFAULT_OPTIONS.merge(options) + options = DEFAULT_OPTIONS.merge(options) + @cache = options.delete(:cache) + super url.to_s, options + end + + def cached_response + cache.get(self) if cache end def response=(value) - value.extend Response if value + if value + value.extend Response + cache.set(self, value) if cache + end super end def run instrument 'response.request', url: base_url do |payload| - response = super + cached = cached_response + response = cached ? finish(cached) : super payload[:response] = response response end diff --git a/lib/docs/core/requester.rb b/lib/docs/core/requester.rb index 7413a506db..7646c65021 100644 --- a/lib/docs/core/requester.rb +++ b/lib/docs/core/requester.rb @@ -38,6 +38,31 @@ def queue(request) super end + # Cached responses are set aside instead of being handed over right away, + # because delivering them from here would nest one request's callbacks + # inside the previous one's and blow the stack on large documentations. + def add(request) + if response = request.cached_response + cached_responses << [request, response] + else + super + end + end + + def run + loop do + while pair = cached_responses.shift + pair[0].finish(pair[1]) + end + break if queued_requests.empty? && multi.easy_handles.empty? + super + end + end + + def cached_responses + @cached_responses ||= [] + end + def on_response(&block) @on_response ||= [] @on_response << block if block diff --git a/lib/docs/core/response_cache.rb b/lib/docs/core/response_cache.rb new file mode 100644 index 0000000000..b88f82012b --- /dev/null +++ b/lib/docs/core/response_cache.rb @@ -0,0 +1,165 @@ +require 'base64' +require 'fileutils' +require 'json' +require 'time' + +module Docs + # Stores the responses fetched by a scraper on the filesystem so that + # subsequent runs are served from disk instead of hitting the network. + # + # Each scraper gets its own directory (tmp/cache/) in which every + # response is stored as an HTTP Archive (HAR) entry: + # http://www.softwareishard.com/blog/har-12-spec/ + # + # An archive is a log of many entries; keeping one entry per file instead + # means the cache stays incremental (a run that's interrupted keeps whatever + # it fetched, and reading one page doesn't parse the whole archive), at the + # cost of the files not being valid archives on their own. + # + # Run `thor docs:clean` to throw the cached responses away. + class ResponseCache + # Written to every cache directory, so that .clean can tell the scraper + # caches apart from the other things living in tmp/cache (e.g. the assets + # cache of the web app). + MARKER_FILENAME = '.scraper_cache' + + EXTENSION = '.json' + + # Deletes the cache directory of every scraper. + def self.clean + Dir[File.join(Docs.cache_path, '*', MARKER_FILENAME)].each do |marker| + FileUtils.rm_rf File.dirname(marker) + end + end + + attr_reader :path + + def initialize(path) + @path = path + end + + # Returns the response stored for the given request, or nil. + def get(request) + entry = JSON.parse File.read(path_for(request), encoding: Encoding::UTF_8) + response = Typhoeus::Response.new(deserialize(entry)) + response.cached = true + response + rescue Errno::ENOENT + nil + rescue StandardError + # Ignore (and overwrite) entries we can't make sense of. + nil + end + + # Stores the response of the given request. + def set(request, response) + return unless cache_response?(response) + + prepare + file = path_for(request) + temp = "#{file}.#{Process.pid}.tmp" + File.binwrite temp, JSON.pretty_generate(serialize(request, response)) + File.rename temp, file + end + + def path_for(request) + File.join path, "#{request.cache_key}#{EXTENSION}" + end + + private + + # Responses that aren't plain successes are left out, so that transient + # failures don't stick around forever. + def cache_response?(response) + !response.mock && !response.cached? && response.code == 200 + end + + # Redirections are followed transparently, so an entry only ever holds the + # last response of a chain. The url it ended up at doesn't fit anywhere in + # the spec, hence the custom field; HAR reserves the underscore prefix for + # those. Fields we don't collect are left at -1 or empty, as prescribed. + def serialize(request, response) + wait = (response.starttransfer_time.to_f * 1000).round + receive = (response.total_time.to_f * 1000).round - wait + + { 'startedDateTime' => Time.now.utc.iso8601(3), + 'time' => wait + receive, + 'request' => { + 'method' => request.options.fetch(:method, :get).to_s.upcase, + 'url' => request.base_url.to_s, + 'httpVersion' => '', + 'cookies' => [], + 'headers' => name_value_pairs(request.options[:headers]), + 'queryString' => name_value_pairs(request.options[:params]), + 'headersSize' => -1, + 'bodySize' => 0 }, + 'response' => { + 'status' => response.code, + 'statusText' => response.status_message.to_s, + 'httpVersion' => response.http_version ? "HTTP/#{response.http_version}" : '', + 'cookies' => [], + 'headers' => name_value_pairs(response.headers), + 'content' => content(response), + 'redirectURL' => '', + 'headersSize' => -1, + 'bodySize' => response.body.to_s.bytesize }, + 'cache' => {}, + 'timings' => { 'send' => -1, 'wait' => wait, 'receive' => receive }, + '_effectiveUrl' => response.effective_url.to_s } + end + + def deserialize(entry) + response = entry['response'] + { code: response['status'], + headers: header_hash(response['headers']), + body: body(response['content']), + effective_url: entry['_effectiveUrl'], + return_code: :ok, + total_time: entry['time'].to_f / 1000 } + end + + def content(response) + body = response.body.to_s + text = body.dup.force_encoding(Encoding::UTF_8) + result = { 'size' => body.bytesize, + 'mimeType' => (response.headers || {})['Content-Type'].to_s } + + if text.valid_encoding? + result['text'] = text + else + # The spec's way out for bodies that aren't valid JSON strings. + result['text'] = Base64.strict_encode64(body) + result['encoding'] = 'base64' + end + + result + end + + # Responses come off the wire as binary, and are handed back as such, so + # that scrapers see the same thing whether or not the cache was used. + def body(content) + text = content['text'].to_s + content['encoding'] == 'base64' ? Base64.strict_decode64(text) : text.b + end + + def name_value_pairs(hash) + (hash || {}).flat_map do |name, value| + Array(value).map { |value| { 'name' => name.to_s, 'value' => value.to_s } } + end + end + + def header_hash(pairs) + (pairs || []).each_with_object({}) do |pair, hash| + name, value = pair['name'], pair['value'] + hash[name] = hash.key?(name) ? Array(hash[name]) << value : value + end + end + + def prepare + return if @prepared + FileUtils.mkdir_p path + FileUtils.touch File.join(path, MARKER_FILENAME) + @prepared = true + end + end +end diff --git a/lib/docs/core/scrapers/url_scraper.rb b/lib/docs/core/scrapers/url_scraper.rb index 1daa1c0526..08d5e153e1 100644 --- a/lib/docs/core/scrapers/url_scraper.rb +++ b/lib/docs/core/scrapers/url_scraper.rb @@ -39,11 +39,15 @@ def request_all(urls, &block) end def request_options - options = { params: self.class.params, headers: self.class.headers } + options = { params: self.class.params, headers: self.class.headers, cache: response_cache } options[:accept_encoding] = 'gzip' if self.class.force_gzip options end + def response_cache + @response_cache ||= ResponseCache.new(File.join(Docs.cache_path, self.class.slug)) + end + def process_response?(response) if response.error? raise <<~ERROR diff --git a/lib/tasks/docs.thor b/lib/tasks/docs.thor index 601ff0bbb3..f500ee89ca 100644 --- a/lib/tasks/docs.thor +++ b/lib/tasks/docs.thor @@ -159,9 +159,10 @@ class DocsCLI < Thor handle_doc_not_found_error(error) end - desc 'clean', 'Delete documentation packages' + desc 'clean', 'Delete documentation packages and cached responses' def clean File.delete(*Dir[File.join Docs.store_path, '*.tar.gz']) + Docs::ResponseCache.clean puts 'Done' end diff --git a/test/lib/docs/core/request_test.rb b/test/lib/docs/core/request_test.rb index 4e7353465d..b242dd58c3 100644 --- a/test/lib/docs/core/request_test.rb +++ b/test/lib/docs/core/request_test.rb @@ -66,4 +66,48 @@ def request(url = self.url, **options) assert_includes response.singleton_class.ancestors, Docs::Response end end + + describe "with a :cache" do + let :cache_path do + File.join tmp_path, 'request_cache' + end + + let :cache do + Docs::ResponseCache.new cache_path + end + + let :fetched_response do + Typhoeus::Response.new( + code: 200, headers: {}, body: 'body', effective_url: url, return_code: :ok) + end + + def request(**options) + super(url, cache: cache, **options) + end + + after do + FileUtils.rm_rf cache_path + end + + it "isn't passed on as a request option" do + assert_equal cache, request.cache + refute request.options.key?(:cache) + end + + it "stores the response" do + request.response = fetched_response + assert cache.get(request) + end + + it "returns the cached response instead of making a request" do + request.response = fetched_response + response = request.run + assert response.cached? + assert_equal 'body', response.body + end + + it "makes a request when the response isn't cached" do + assert_equal self.response, request.run + end + end end diff --git a/test/lib/docs/core/requester_test.rb b/test/lib/docs/core/requester_test.rb index 18a17ec3fc..2e07acd73d 100644 --- a/test/lib/docs/core/requester_test.rb +++ b/test/lib/docs/core/requester_test.rb @@ -148,5 +148,67 @@ def stub_request(url) requester.run end end + + context "with a cache" do + let :cache_path do + File.join tmp_path, 'requester_cache' + end + + let :cache do + Docs::ResponseCache.new cache_path + end + + let :options do + { request_options: { cache: cache } } + end + + def cache_response(url, body) + Docs::Request.new(url, cache: cache).response = Typhoeus::Response.new( + code: 200, headers: {}, body: body, effective_url: url, return_code: :ok) + end + + after do + FileUtils.rm_rf cache_path + end + + it "serves the requests from the cache" do + cache_response url, 'body' + requester.on_response { |response| @response = response } + requester.request(url) + requester.run + assert @response.cached? + assert_equal 'body', @response.body + end + + it "serves the urls returned by the callbacks from the cache" do + cache_response url, 'body' + cache_response 'http://example.com/one', 'one' + cache_response 'http://example.com/two', 'two' + + bodies = [] + requester.on_response do |response| + bodies << response.body + ['http://example.com/one', 'http://example.com/two'] if response.body == 'body' + end + + requester.request(url) + requester.run + assert_equal %w(body one two), bodies + end + + it "requests the urls that aren't cached" do + cache_response url, 'body' + stub_request 'http://example.com/one' + + requester.on_response do |response| + @count = @count.to_i + 1 + ['http://example.com/one'] if response.cached? + end + + requester.request(url) + requester.run + assert_equal 2, @count + end + end end end diff --git a/test/lib/docs/core/response_cache_test.rb b/test/lib/docs/core/response_cache_test.rb new file mode 100644 index 0000000000..eadbb3c5fe --- /dev/null +++ b/test/lib/docs/core/response_cache_test.rb @@ -0,0 +1,134 @@ +require_relative '../../../test_helper' +require_relative '../../../../lib/docs' + +class DocsResponseCacheTest < Minitest::Spec + let :path do + File.join tmp_path, 'response_cache' + end + + let :cache do + Docs::ResponseCache.new path + end + + let :request do + Docs::Request.new 'http://example.com/page', cache: cache + end + + let :response do + Typhoeus::Response.new( + code: 200, + headers: { 'Content-Type' => 'text/html' }, + body: '', + effective_url: 'http://example.com/page', + return_code: :ok + ).tap { |response| response.extend Docs::Response } + end + + after do + FileUtils.rm_rf path + end + + describe "#set" do + it "stores the response" do + cache.set request, response + assert File.exist?(cache.path_for(request)) + end + + it "marks the directory as a scraper cache" do + cache.set request, response + assert File.exist?(File.join(path, Docs::ResponseCache::MARKER_FILENAME)) + end + + it "ignores mocked responses" do + response.mock = true + cache.set request, response + refute File.exist?(cache.path_for(request)) + end + + it "ignores unsuccessful responses" do + response.options[:code] = 404 + cache.set request, response + refute File.exist?(cache.path_for(request)) + end + + it "ignores responses that came from the cache" do + response.cached = true + cache.set request, response + refute File.exist?(cache.path_for(request)) + end + + it "stores the response as a HAR entry" do + cache.set request, response + entry = JSON.parse File.read(cache.path_for(request)) + assert_equal 'GET', entry['request']['method'] + assert_equal 'http://example.com/page', entry['request']['url'] + assert_equal 200, entry['response']['status'] + assert_equal '', entry['response']['content']['text'] + assert_equal 'text/html', entry['response']['content']['mimeType'] + assert_equal 'http://example.com/page', entry['_effectiveUrl'] + assert_includes entry['response']['headers'], + { 'name' => 'Content-Type', 'value' => 'text/html' } + end + + it "base64-encodes bodies that aren't valid UTF-8" do + response.options[:body] = "\xC3".b + cache.set request, response + entry = JSON.parse File.read(cache.path_for(request)) + assert_equal 'base64', entry['response']['content']['encoding'] + assert_equal "\xC3".b, cache.get(request).body + end + end + + describe "#get" do + it "returns nil when nothing is stored for the request" do + assert_nil cache.get(request) + end + + it "returns the stored response" do + cache.set request, response + result = cache.get(request) + assert_equal response.code, result.code + assert_equal response.body, result.body + assert_equal response.headers.to_h, result.headers.to_h + assert_equal response.effective_url.to_s, result.effective_url + assert result.cached? + end + + it "returns the body as binary, like a response off the wire" do + cache.set request, response + assert_equal Encoding::BINARY, cache.get(request).body.encoding + end + + it "returns nil when the stored response can't be read" do + cache.set request, response + File.binwrite cache.path_for(request), 'garbage' + assert_nil cache.get(request) + end + end + + describe ".clean" do + before do + @cache_path = Docs.cache_path + Docs.cache_path = File.join(tmp_path, 'cache') + end + + after do + FileUtils.rm_rf Docs.cache_path + Docs.cache_path = @cache_path + end + + it "deletes the scraper caches" do + cache = Docs::ResponseCache.new(File.join(Docs.cache_path, 'scraper')) + cache.set request, response + Docs::ResponseCache.clean + refute File.exist?(cache.path) + end + + it "leaves other directories alone" do + other = File.join(Docs.cache_path, 'assets') + FileUtils.mkdir_p other + Docs::ResponseCache.clean + assert File.exist?(other) + end + end +end diff --git a/test/lib/docs/core/scrapers/url_scraper_test.rb b/test/lib/docs/core/scrapers/url_scraper_test.rb index 7469ce4070..2c3dd136d6 100644 --- a/test/lib/docs/core/scrapers/url_scraper_test.rb +++ b/test/lib/docs/core/scrapers/url_scraper_test.rb @@ -53,25 +53,29 @@ class Scraper < Docs::UrlScraper scraper.send :request_all, 'urls', &block end + let :cache do + scraper.send :response_cache + end + it "runs a Requester with the given urls" do - mock(Docs::Requester).run('urls', request_options: {params: {}, headers: {"User-Agent" => "DevDocs"}}) + mock(Docs::Requester).run('urls', request_options: {params: {}, headers: {"User-Agent" => "DevDocs"}, cache: cache}) result end it "runs a Requester with .headers as :request_options" do stub(Scraper).headers { { testheader: true } } - mock(Docs::Requester).run('urls', request_options: {params: {}, headers: {testheader: true}}) + mock(Docs::Requester).run('urls', request_options: {params: {}, headers: {testheader: true}, cache: cache}) result end it "runs a Requester with default .headers as :request_options" do - mock(Docs::Requester).run('urls', request_options: {params: {}, headers: {"User-Agent" => "DevDocs"}}) + mock(Docs::Requester).run('urls', request_options: {params: {}, headers: {"User-Agent" => "DevDocs"}, cache: cache}) result end it "runs a Requester with .params as :request_options" do stub(Scraper).params { { test: true } } - mock(Docs::Requester).run('urls', request_options: {params: {test: true}, headers: {"User-Agent" => "DevDocs"}}) + mock(Docs::Requester).run('urls', request_options: {params: {test: true}, headers: {"User-Agent" => "DevDocs"}, cache: cache}) result end