From 72dc1d08d253ae42ab3df6282023c920f972f5ea Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 6 Sep 2026 02:49:59 +0200 Subject: [PATCH 1/6] docs(concepts): update server architecture and default http strategy --- src/content/docs/creating-custom-feeds.mdx | 2 +- src/content/docs/web-application/concepts.mdx | 8 ++++++-- 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/src/content/docs/creating-custom-feeds.mdx b/src/content/docs/creating-custom-feeds.mdx index 31b098d2..18fac1a4 100644 --- a/src/content/docs/creating-custom-feeds.mdx +++ b/src/content/docs/creating-custom-feeds.mdx @@ -22,7 +22,7 @@ Prefer [Getting Started](/getting-started/) (URL paste) and the [Feed Directory] 3. Validate: `html2rss validate your-config.yml` 4. Live-check: `html2rss test your-config.yml`, then ship with `html2rss apply your-config.yml` 5. Mount into `html2rss-web` or contribute to html2rss-configs -6. Escalate to `strategy: botasaurus` (or `auto` with `BOTASAURUS_SCRAPER_URL`) only when Faraday is not enough +6. Escalate to `strategy: botasaurus` (or `auto` with `BOTASAURUS_SCRAPER_URL`) only when default HTTP fetching is not enough `html2rss feed` is a Thor alias for `apply`. `html2rss auto` aliases `scrape` (one-shot, no YAML). diff --git a/src/content/docs/web-application/concepts.mdx b/src/content/docs/web-application/concepts.mdx index a0e36e4f..beefa085 100644 --- a/src/content/docs/web-application/concepts.mdx +++ b/src/content/docs/web-application/concepts.mdx @@ -13,7 +13,7 @@ Production Docker Compose defaults `AUTO_SOURCE_ENABLED` to `true`. Non-compose ## Botasaurus companion -`html2rss-web` talks to `botasaurus-scrape-api` at `BOTASAURUS_SCRAPER_URL` (Compose sets `http://botasaurus:4010`). That service renders JavaScript-heavy pages when Faraday alone is not enough. Auto-source uses Faraday first, then Botasaurus when configured. +`html2rss-web` talks to `botasaurus-scrape-api` at `BOTASAURUS_SCRAPER_URL` (Compose sets `http://botasaurus:4010`). That service renders JavaScript-heavy pages when default HTTP fetching alone is not enough. Auto-source uses the default HTTP strategy (HTTPX) first, then Botasaurus when configured. ## Three tokens (plain English) @@ -27,9 +27,13 @@ Generate secrets with `openssl rand -hex 32`. Demo quickstart hard-codes `CHANGE Optional: `SENTRY_DSN` (web) and `BOTASAURUS_SENTRY_DSN` (scraper) for error reporting. They are separate projects; leave both unset until you want monitoring. +## Server & async reactor + +`html2rss-web` runs on the Falcon application server, an asynchronous fiber-based server supporting HTTP/1.1 and HTTP/2. Falcon cooperates with HTTPX for non-blocking upstream I/O and coalesces concurrent feed requests in memory to prevent cache stampedes. + ## HTTP vs HTTPS -Plain HTTP works on localhost and LAN. The app sets HSTS and CSP `upgrade-insecure-requests` only when the request is HTTPS (including when a reverse proxy sends `X-Forwarded-Proto: https`). Put Caddy, Nginx, or Traefik in front for public TLS. +Plain HTTP works on localhost and LAN. The app sets HSTS and CSP `upgrade-insecure-requests` only when the request is HTTPS (including when a reverse proxy sends `X-Forwarded-Proto: https`). For public TLS, put Caddy, Nginx, or Traefik in front, or configure direct in-process TLS with Falcon. ## Ecosystem map From 55583d8540712c9ff9b650f31df310994797bb6c Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 6 Sep 2026 02:50:05 +0200 Subject: [PATCH 2/6] docs(guides): update strategy migration notes and direct tls deployment --- .../docs/ruby-gem/guides/ai-agent-workflows.mdx | 2 +- .../docs/ruby-gem/guides/backward-compatibility.mdx | 6 ++++++ .../docs/ruby-gem/guides/capturing-feed-configs.mdx | 2 +- .../ruby-gem/guides/handling-dynamic-content.mdx | 4 ++-- src/content/docs/web-application/deployment.mdx | 12 ++++++++++++ 5 files changed, 22 insertions(+), 4 deletions(-) diff --git a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx index 7ead82f0..be9c85d2 100644 --- a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx +++ b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx @@ -50,7 +50,7 @@ Put `BOTASAURUS_SCRAPER_URL` in the MCP `env` block — a shell export does not ## Golden loops -**Articles now:** `scrape` (or `batch_scrape`). `strategy: "auto"` already falls back to Botasaurus when configured — do not retry with explicit `faraday` after `auto`. Empty items can still be success; follow `next_step`. +**Articles now:** `scrape` (or `batch_scrape`). `strategy: "auto"` already falls back to Botasaurus when configured — do not retry with explicit `default` after `auto`. Empty items can still be success; follow `next_step`. **Durable YAML:** optional `inspect` → `recon` → `capture` → `test` → `apply`. diff --git a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx index 8de5059b..13bdeda0 100644 --- a/src/content/docs/ruby-gem/guides/backward-compatibility.mdx +++ b/src/content/docs/ruby-gem/guides/backward-compatibility.mdx @@ -7,6 +7,12 @@ import { Code } from "@astrojs/starlight/components"; This page outlines breaking changes, purged legacy aliases, and migration steps when upgrading the gem. +## 0.30.0 (Unreleased) + +1. **Default HTTP strategy (`:default`)** — Migrated from Faraday to HTTPX. The canonical strategy name across configs, CLI, Ruby API, and MCP is `:default`. +2. **Deprecated strategy aliases** — `:faraday` and `:httpx` remain accepted as aliases for `:default` with deprecation warnings. Update configurations and scripts to `strategy: default` (or `strategy: auto`). +3. **HTTP/2 & SSRF protection** — The default transport provides native HTTP/2 multiplexing via ALPN and enforced private-network IP filtering. + ## 0.27.0 Upgrade notes for configs and integrators targeting gem **0.27.0** (current `master` after `0.26.0`): diff --git a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx index cad7f99c..bcd64832 100644 --- a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx @@ -46,7 +46,7 @@ Print YAML to stdout: Common options: -- `--strategy` — `auto`, `faraday`, `botasaurus`, or `local_file` (default `auto`) +- `--strategy` — `auto`, `default`, `botasaurus`, or `local_file` (default `auto`) - `--items_selector` — CSS selector hint for item cards - `--limit` — maximum articles kept while deriving selectors (default `25`) - `--max-redirects` / `--max-requests` — request budget overrides diff --git a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx index 503c9d14..44c6ec3a 100644 --- a/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx +++ b/src/content/docs/ruby-gem/guides/handling-dynamic-content.mdx @@ -9,7 +9,7 @@ Some websites load their content dynamically using JavaScript. Static fetch path ## Solution -Default `strategy: auto` automatically tries `faraday` first, then falls back to `botasaurus` when `BOTASAURUS_SCRAPER_URL` is configured. This handles many JS-rendered listing pages without needing custom configuration. +Default `strategy: auto` automatically tries `default` first, then falls back to `botasaurus` when `BOTASAURUS_SCRAPER_URL` is configured. This handles many JS-rendered listing pages without needing custom configuration. When a site requires browser rendering or anti-bot bypass by default, you can explicitly set `strategy: botasaurus` and configure request controls under `request.botasaurus`: @@ -102,7 +102,7 @@ Browser-based extraction uses more resources than static HTTP fetching because i - Executes JavaScript and handles DOM events - Manages browser pools and network emulation -Use static HTTP fetching (`faraday`) for static content, and lean on `auto` or explicit `botasaurus` strategies when browser rendering is required. See the [Strategy Reference](/ruby-gem/reference/strategy/) for details. +Use static HTTP fetching (`default`) for static content, and lean on `auto` or explicit `botasaurus` strategies when browser rendering is required. See the [Strategy Reference](/ruby-gem/reference/strategy/) for details. ## Related Topics diff --git a/src/content/docs/web-application/deployment.mdx b/src/content/docs/web-application/deployment.mdx index d32230dd..9b64a116 100644 --- a/src/content/docs/web-application/deployment.mdx +++ b/src/content/docs/web-application/deployment.mdx @@ -79,6 +79,18 @@ Example `.env` beside Compose: After `docker compose up -d`, check `docker compose logs caddy --tail 20` for certificate issuance. +### Direct TLS termination (Falcon) + +Falcon also supports in-process TLS termination and HTTP/2 negotiation by pointing to your certificate and private key: + + + ## Custom feeds mount Auto-source is the default. To add curated YAML feeds, bind-mount `feeds.yml` (see [Advanced Feeds](/creating-custom-feeds/)): From d3d6945604417ec77850cc9258558bfd33c3a4f1 Mon Sep 17 00:00:00 2001 From: Gil Desmarais Date: Sun, 6 Sep 2026 02:50:15 +0200 Subject: [PATCH 3/6] docs(adapters): document default httpx strategy, cli flags, mcp schemas, and error handling --- .../docs/ruby-gem/reference/auto-source.mdx | 2 +- .../docs/ruby-gem/reference/cli-reference.mdx | 6 ++-- .../docs/ruby-gem/reference/configuration.mdx | 4 +-- .../docs/ruby-gem/reference/mcp-server.mdx | 8 ++--- .../docs/ruby-gem/reference/strategy.mdx | 30 ++++++++++++------- .../docs/troubleshooting/troubleshooting.mdx | 10 +++---- 6 files changed, 34 insertions(+), 26 deletions(-) diff --git a/src/content/docs/ruby-gem/reference/auto-source.mdx b/src/content/docs/ruby-gem/reference/auto-source.mdx index 1ec09d9d..d8be1f06 100644 --- a/src/content/docs/ruby-gem/reference/auto-source.mdx +++ b/src/content/docs/ruby-gem/reference/auto-source.mdx @@ -35,7 +35,7 @@ What each scraper does: - **`microdata`:** Extracts HTML Microdata annotations (`itemscope itemtype`). - **`microformats2`:** Parses Microformats2 `h-entry` markup, extracting `p-name`, `e-content`, `u-url`, `dt-published`, `p-author`, `p-category`, and `u-photo` / `u-featured` media. - **`json_state`:** Walks in-page JSON (`