diff --git a/src/content/docs/creating-custom-feeds.mdx b/src/content/docs/creating-custom-feeds.mdx index 31b098d2..6a0ffcea 100644 --- a/src/content/docs/creating-custom-feeds.mdx +++ b/src/content/docs/creating-custom-feeds.mdx @@ -22,7 +22,7 @@ Prefer [Getting Started](/getting-started/) (URL paste) and the [Feed Directory] 3. Validate: `html2rss validate your-config.yml` 4. Live-check: `html2rss test your-config.yml`, then ship with `html2rss apply your-config.yml` 5. Mount into `html2rss-web` or contribute to html2rss-configs -6. Escalate to `strategy: botasaurus` (or `auto` with `BOTASAURUS_SCRAPER_URL`) only when Faraday is not enough +6. Keep `strategy: auto`; if results are incomplete, confirm the companion scraper is configured before adding site-specific controls `html2rss feed` is a Thor alias for `apply`. `html2rss auto` aliases `scrape` (one-shot, no YAML). diff --git a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx index 7ead82f0..d1e4cf7f 100644 --- a/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx +++ b/src/content/docs/ruby-gem/guides/ai-agent-workflows.mdx @@ -50,7 +50,7 @@ Put `BOTASAURUS_SCRAPER_URL` in the MCP `env` block — a shell export does not ## Golden loops -**Articles now:** `scrape` (or `batch_scrape`). `strategy: "auto"` already falls back to Botasaurus when configured — do not retry with explicit `faraday` after `auto`. Empty items can still be success; follow `next_step`. +**Articles now:** `scrape` (or `batch_scrape`). Keep `strategy: "auto"` so html2rss can choose the available fetch path. Empty items can still be success; follow `next_step` instead of retrying with another strategy. **Durable YAML:** optional `inspect` → `recon` → `capture` → `test` → `apply`. diff --git a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx index cad7f99c..3242e10d 100644 --- a/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx +++ b/src/content/docs/ruby-gem/guides/capturing-feed-configs.mdx @@ -29,9 +29,6 @@ Print YAML to stdout: `, `window.__NEXT_DATA__`, `window.__NUXT__`, `window.STATE`) for arrays with `title`/`url` pairs. -- **`xhr_articles`:** Reuses JSON XHR/fetch bodies captured during a Botasaurus **browser** scrape (no extra HTTP). Empty for Faraday and Botasaurus HTTP-request tiers. +- **`xhr_articles`:** Reuses JSON XHR/fetch bodies captured during browser execution (no extra request). It is empty when the selected fetch path does not capture browser responses. - **`wordpress_api`:** Detects `` and pulls posts from the REST API. See [WordPress API](/ruby-gem/reference/wordpress-api/). - **`sitemap`:** Locates XML sitemaps (``, `/sitemap.xml`, or `/robots.txt`), filtering by priority and recency, with Google News tags (``). - **`meta_oembed`:** OpenGraph/Twitter meta tags plus JSON oEmbed (``). diff --git a/src/content/docs/ruby-gem/reference/cli-reference.mdx b/src/content/docs/ruby-gem/reference/cli-reference.mdx index 3554048f..9115faaa 100644 --- a/src/content/docs/ruby-gem/reference/cli-reference.mdx +++ b/src/content/docs/ruby-gem/reference/cli-reference.mdx @@ -37,7 +37,7 @@ Command: `html2rss scrape [URL]` Available options: -- `--strategy`: Optional request strategy (`auto`, `faraday`, `botasaurus`, `local_file`). Defaults to `auto`, which tries `faraday` -> `botasaurus`. +- `--strategy`: Optional request strategy (`auto`, `default`, `botasaurus`, `local_file`). Defaults to `auto`, which chooses an available fetch path based on scrape results. Use a concrete value only for diagnosis or strategy-specific controls. - `--format`: Output format for the auto-sourced feed (`rss` or `jsonfeed`). Defaults to `rss`. - `--limit`: Maximum number of articles to extract during discovery (defaults to `25`). - `--items_selector`: Optional CSS selector hint for item extraction. @@ -76,11 +76,11 @@ When no extractable items are found, `scrape` classifies likely causes instead o Known anti-bot interstitial responses (for example Cloudflare challenge pages) are surfaced explicitly as blocked-surface errors. -If all fallback tiers run but still extract zero items, html2rss raises: +If `auto` exhausts the available fetch paths without items, html2rss raises: - `No RSS feed items extracted after auto fallback ...` -If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so the `auto` Botasaurus tier can run. +If failures continue after URL/surface fixes, ensure `BOTASAURUS_SCRAPER_URL` is set so `auto` can use the companion scraper. Start by changing the input URL to a direct listing/update page, then move to explicit selectors if needed. @@ -123,7 +123,7 @@ Command: `html2rss apply YAML_FILE [feed_name]` Available options: -- `--strategy`: Request strategy override (`auto`, `faraday`, `botasaurus`, `local_file`). +- `--strategy`: Request strategy override (`auto`, `default`, `botasaurus`, `local_file`). - `--params`: Dynamic parameters passed as key-value pairs (e.g. `--params id:42 section:news`). - `--max-redirects`: Maximum redirects to follow per request. - `--max-requests`: Total request budget allowed for this feed build. @@ -150,7 +150,7 @@ Command: `html2rss capture [URL]` Available options: -- `--strategy`: Optional request strategy (`auto`, `faraday`, `botasaurus`, `local_file`). Defaults to `auto`. +- `--strategy`: Optional request strategy (`auto`, `default`, `botasaurus`, `local_file`). Defaults to `auto`. - `--items_selector`: Optional CSS selector hint for item extraction. - `--limit`: Maximum number of articles to keep (defaults to `25`). - `--max-redirects`: Maximum redirects to follow per request. @@ -158,7 +158,7 @@ Available options: - `--input`: Local HTML file path to read input from without making network requests. - `--explain`: Print capture quality JSON to stderr (`articles_count`, `channel_title`, `has_selectors`, `segment_strategy`, `selected_strategy`, `admission_drops`). Stdout stays YAML. -When AutoFallback (or a pinned strategy) selects a concrete transport, the printed YAML includes a top-level `strategy:` so later `html2rss apply` uses the same hop. +Capture records the concrete strategy that produced the draft as a top-level `strategy:` so later `html2rss apply` can reproduce that fetch path. ### MCP diff --git a/src/content/docs/ruby-gem/reference/configuration.mdx b/src/content/docs/ruby-gem/reference/configuration.mdx index 9dad7c29..2ed14a38 100644 --- a/src/content/docs/ruby-gem/reference/configuration.mdx +++ b/src/content/docs/ruby-gem/reference/configuration.mdx @@ -16,7 +16,6 @@ Global configuration is thread-safe and implements an RCU (Read-Copy-Update) mec Html2rss.configure do |config| config.log_level = :info config.min_ttl = 60 -config.default_strategy = :faraday config.headers = { 'User-Agent' => 'MyCustomUserAgent/1.0' } end `} @@ -76,15 +75,15 @@ Defines HTTP headers that are globally appended/prepended to all requests. You c ### `default_strategy` -Sets the default scraper strategy name used when a feed configuration doesn't specify a `strategy`. The strategy name must correspond to a registered strategy. +Overrides the strategy used when a feed configuration does not specify one. Keep the gem default (`auto`) unless every feed in the process requires a fixed fetch mode. - **Type**: `Symbol`, `String`, or `nil` -- **Default**: `nil` (falls back to the gem default strategy, usually `auto`) +- **Default**: `nil` (uses `auto`) `botasaurus`. -- **`faraday`**: Makes a direct HTTP request. It is fast but does not execute JavaScript. -- **`botasaurus`**: Delegates fetching to a Botasaurus scrape API. Included in the `auto` chain; requires `BOTASAURUS_SCRAPER_URL` when that tier runs (or when you pin `strategy: botasaurus`). +- **`auto`** (default): Chooses from the available fetch paths based on scrape results. Use this for normal scraping. +- **`default`**: Makes a direct request without executing JavaScript. Use it only when you need a fixed request mode. +- **`botasaurus`**: Uses the companion scraper for browser rendering and browser-specific controls. Requires `BOTASAURUS_SCRAPER_URL`. - **`local_file`**: Reads HTML content directly from a local file on disk without making network requests. `strategy` is a top-level config key. Request-specific controls live under `request`. -`auto` falls back to `botasaurus` when `faraday` errors or extracts zero items. Pin a concrete strategy when you need a specific transport (for example, forcing `botasaurus` directly). +`auto` is the recommended interface: html2rss evaluates the scrape result and can use the configured companion scraper when needed. Pin a concrete strategy only for diagnosis, reproducibility, or strategy-specific controls. ## `auto` (default) -The default strategy chain is: - -`faraday` -> `botasaurus` - When the config uses `auto_source` (no custom `selectors`), each strategy attempt: 1. Runs AutoSource scrapers, starting with `native_feed` for same-origin RSS/Atom (direct feed URLs parse without HTML AutoSource). @@ -31,24 +27,24 @@ Gates, defaults, and Status fields: [Auto Source — Entry URL resolution](/ruby Auto fallback shares one request budget across all strategy attempts. The baseline reserves a NativeFeed follow-up slot and, when entry resolution is enabled, `max_probes` probe slots plus one retry GET. Raise `request.max_requests` (or `--max-requests`) when builds exhaust the budget. -Under `auto`, `Html2rss.feed_result(...).status` exposes scrape telemetry: `selected_strategy`, `attempt_count`, `strategy_attempts`, and when resolution ran `entry_url`, `scrape_url`, and `entry_resolution` (see [Managing Feed Configs](/ruby-gem/guides/managing-feed-configs/#ruby-api-feedresult)). Auto fallback decisions are also visible at `LOG_LEVEL=info` (hidden at the default `LOG_LEVEL=warn`). +Under `auto`, `Html2rss.feed_result(...).status` exposes scrape telemetry: `selected_strategy`, `attempt_count`, `strategy_attempts`, and when resolution ran `entry_url`, `scrape_url`, and `entry_resolution` (see [Managing Feed Configs](/ruby-gem/guides/managing-feed-configs/#ruby-api-feedresult)). Strategy selection is also visible at `LOG_LEVEL=info` (hidden at the default `LOG_LEVEL=warn`). -## `faraday` +## `default` -`faraday` is the standard static HTTP transport. When no custom `User-Agent` header is supplied in `headers`, `html2rss` sends `html2rss/` by default. +`default` is the fixed direct-request mode. It does not execute JavaScript. Keep `auto` unless you are diagnosing fetch behavior or must prevent browser rendering. -HTML is accepted when `Content-Type` includes `text/html`, or when the body sniffs as HTML (``. The shared request policy enforces redirect, request, timeout, and private-network safeguards and accepts supported compressed HTML responses. ## `botasaurus` -`botasaurus` delegates page fetching to a Botasaurus scrape API endpoint. It runs as the second tier of `auto`, or when you pin `strategy: botasaurus`. +`botasaurus` delegates page fetching to the companion scrape API. `auto` can use it when configured; pin `strategy: botasaurus` only when you need browser-specific behavior. Requirements: - `BOTASAURUS_SCRAPER_URL` set to your Botasaurus scrape API base URL (for example `http://localhost:4010`) -- pin `strategy: botasaurus` when you want to skip Faraday and force this transport directly +- pin `strategy: botasaurus` only when you need to force browser rendering or configure the controls below -html2rss enforces local request policy preflight and a feed-build timeout budget. The Botasaurus scrape API splits **total** scrape time (queue, boot, navigate, wait) from **work** time (navigate, selector wait, and scroll after the browser is ready). Feed YAML `wait_timeout_seconds` is validated against the work cap (`1..30` by default); the gem forwards `request.botasaurus` options to the scrape API without shrinking retries or waits to fit a remaining feed budget. Faraday transport timeout for POST `/scrape` is the lesser of the remaining feed-build budget and `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` plus a small buffer (default **45s** scrape total, **47s** transport cap). +html2rss enforces local request policy preflight and a feed-build timeout budget. The Botasaurus scrape API splits **total** scrape time (queue, boot, navigate, wait) from **work** time (navigate, selector wait, and scroll after the browser is ready). Feed YAML `wait_timeout_seconds` is validated against the work cap (`1..30` by default); the gem forwards `request.botasaurus` options to the scrape API without shrinking retries or waits to fit a remaining feed budget. Transport timeout for POST `/scrape` is the lesser of the remaining feed-build budget and `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` plus a small buffer (default **45s** scrape total, **47s** transport cap). When the scrape API reports a timed-out stage, `Html2rss::RequestService::RequestTimedOut` may include `timeout_phase` (`queue`, `boot`, or `work`) and the same value in the exception message (`timeout_phase=`). Transport-hop timeouts leave `timeout_phase` unset. @@ -152,7 +148,7 @@ Because no remote network connection is established, `local_file` skips remote n Use this split consistently: -- `strategy`: selects `auto`, `faraday`, `botasaurus`, or `local_file` +- `strategy`: selects `auto`, `default`, `botasaurus`, or `local_file` - `headers`: top-level headers shared by remote strategies (defaults to `html2rss/` User-Agent) - `request.max_redirects`: redirect limit for the request session (default: `5`) - `request.max_requests`: total request budget for the whole feed build (default: `10`) diff --git a/src/content/docs/troubleshooting/troubleshooting.mdx b/src/content/docs/troubleshooting/troubleshooting.mdx index 6f30a58c..f2cc4b23 100644 --- a/src/content/docs/troubleshooting/troubleshooting.mdx +++ b/src/content/docs/troubleshooting/troubleshooting.mdx @@ -56,9 +56,9 @@ If your feed is empty, check the following: Known anti-bot interstitial patterns (for example Cloudflare or DataDome challenge pages) are surfaced as blocked-surface errors instead of silent empty extraction results. -When all auto fallback tiers complete but still extract zero items, html2rss raises `No RSS feed items extracted after auto fallback ...`. +When `auto` exhausts the available fetch paths without items, html2rss raises `No RSS feed items extracted after auto fallback ...`. -If failures continue after URL/surface fixes, configure `BOTASAURUS_SCRAPER_URL` or use `--strategy botasaurus` for browser rendering. +If failures continue after URL/surface fixes, configure `BOTASAURUS_SCRAPER_URL` so `auto` can use the companion scraper. Pin `--strategy botasaurus` only for browser-specific diagnosis or controls. ### DataDome And Similar Challenges @@ -68,7 +68,7 @@ DataDome (and similar) challenge pages are classified as a blocked surface. Pref 2. `strategy: botasaurus` with `BOTASAURUS_SCRAPER_URL` so the browser tier can complete navigation 3. A session that already has resolved challenge cookies when the site still blocks automated browsers -Do not expect Faraday-only fetches to succeed against active DataDome interstitials. +Do not expect static HTTP fetches to succeed against active DataDome interstitials. ### Botasaurus Setup & Diagnostics @@ -95,8 +95,8 @@ Then run with: Common configuration-related errors: -- **`UnsupportedResponseContentType`:** The website returned content that html2rss can't parse (not HTML or JSON). Faraday still treats bodies that sniff as HTML (`= 1`; default `300`; Retry-After header backoff duration returned on timeouts | | `PORT` | app listen port; compose uses `4000` | | `RACK_ENV` | Rack environment; quickstart uses `development`, production examples use `production` | -| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; Faraday client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api | +| `REQUEST_TIMEOUT_SECONDS` | optional; total web-request timeout in seconds; default `55` | +| `WEB_CONCURRENCY` | optional integer `>= 1`; default `2` in production (`1` in development); number of web worker processes | +| `BOTASAURUS_SCRAPE_TIMEOUT_SECONDS` | optional; HTTP client cap for POST `/scrape` in seconds; default `45`; mirrors `SCRAPE_TIMEOUT_SECONDS` on botasaurus-scrape-api | | `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` | optional; maximum `wait_timeout_seconds` in feed YAML; default `30`; mirrors `SCRAPE_WORK_TIMEOUT_SECONDS` on botasaurus-scrape-api | | `SCRAPE_WORK_TIMEOUT_SECONDS` | optional on botasaurus-scrape-api; post-boot navigate, selector wait, and scroll budget in seconds; default `30`; set on the scraper service in compose | -| `RACK_TIMEOUT_SERVICE_TIMEOUT` | optional; total web-request timeout in seconds; default `55` | | `HTML2RSS_TOTAL_TIMEOUT_SECONDS` | optional; internal gem feed-build timeout in seconds; default `50` | ## Timeout Chain When fetching feeds from complex websites, multiple components work together. Configure timeouts so outer layers wait longer than inner layers and the scraper exhausts its budget before the web tier aborts the request. -Default compose aligns **scrape total (45) ≤ feed build (50) ≤ Rack (55)**. The **work** budget (**30s**) applies on botasaurus-scrape-api only after the browser is ready; browser boot and queue time count against the **total** scrape wall (**45s**), not against `wait_timeout_seconds`. +Runtime defaults align **scrape total (45) ≤ feed build (50) ≤ web request timeout (55)**. The **work** budget (**30s**) applies on botasaurus-scrape-api only after the browser is ready; browser boot and queue time count against the **total** scrape wall (**45s**), not against `wait_timeout_seconds`. 1. **botasaurus-scrape-api (`SCRAPE_TIMEOUT_SECONDS`)**: **45s** handler wall (queue, boot, navigate, wait). 2. **botasaurus-scrape-api (`SCRAPE_WORK_TIMEOUT_SECONDS`)**: **30s** post-boot navigate, `wait_for_selector`, and scroll. -3. **html2rss client (`BOTASAURUS_SCRAPE_TIMEOUT_SECONDS`)**: **45s** Faraday cap for POST `/scrape` (mirrors scrape total). +3. **html2rss client (`BOTASAURUS_SCRAPE_TIMEOUT_SECONDS`)**: **45s** client cap for POST `/scrape` (mirrors scrape total). 4. **Feed YAML (`wait_timeout_seconds`)**: **1..30**, capped by `BOTASAURUS_SCRAPE_WORK_TIMEOUT_SECONDS` (default **30**). 5. **html2rss-web gem budget (`HTML2RSS_TOTAL_TIMEOUT_SECONDS`)**: **50s** for scrape plus extraction. -6. **html2rss-web (`RACK_TIMEOUT_SERVICE_TIMEOUT`)**: **55s** Rack outer wall. +6. **html2rss-web (`REQUEST_TIMEOUT_SECONDS`)**: **55s** outer web-request wall. -Set matching `SCRAPE_*` on botasaurus-scrape-api and `BOTASAURUS_SCRAPE_*` on html2rss-web when you override defaults. See the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md) for compose examples. +Set matching `SCRAPE_*` on botasaurus-scrape-api and `BOTASAURUS_SCRAPE_*` on html2rss-web when you override defaults. Set `REQUEST_TIMEOUT_SECONDS` explicitly when changing the outer web-request wall. See the `Environment & Runtime Flags` table in [`docs/README.md`](https://github.com/html2rss/html2rss-web/blob/main/docs/README.md) for compose examples.