Release v0.9.3

CHANGELOG.md

@@ -5,6 +5,66 @@ All notable changes to Crawl4AI will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.3] - 2026-08-31
+
+0.9.3 is a security release. It closes five coordinated-disclosure advisories in the PDF processing path and the Docker Playground UI, and ships the 33 bug fixes that accumulated on `develop` since 0.9.2, most of them in the Docker server. There are no new features and no breaking changes. Users who accept untrusted URLs on the Docker server, or who open PDFs from sources they do not control, should upgrade.
+
+### Security
+
+The PDF path was the common thread. `PDFContentScrapingStrategy` is selectable from an untrusted Docker API request body, and it fetches with `requests` outside the browser, so none of the Chromium-side egress or resource controls applied to it.
+
+- **Arbitrary file write via PDF image-write fields (CWE-22, high)**: `PDFContentScrapingStrategy` had no field allowlist, so an untrusted request body could set `save_images_locally` and `image_save_dir` and make the server write extracted images to a path of the caller's choosing. Those fields are now filtered at the trust boundary and `extract_images` is forced off for untrusted bodies. Credit: Zhixi "Jace" Sun. (GHSA-xpp7-j28w-2gvx)
+- **SSRF via PDF download redirects (CWE-918, high)**: the PDF download followed redirects without consulting any destination policy, so a public URL that redirected to an internal address reached it. Redirects are now resolved by hand with a per-hop destination check, bounded at five hops, and the peer IP of the response actually read back is validated to close DNS rebinding. The Docker server installs its egress policy into this path at boot. Credit: Nguyen Tran Thanh Lam ([c240030](https://github.com/c240030)). (GHSA-q5rj-45vw-vp2g)
+- **Denial of service via unbounded PDF size and page count (CWE-400, medium)**: a remote PDF was streamed to disk and parsed with no cap on bytes or pages. Downloads now stop at `max_pdf_bytes` (100 MiB default), enforced on the running total rather than the caller-supplied `content-length`, and parsing stops at `max_pdf_pages` (2000 default). Untrusted bodies cannot raise their own caps. The Docker config now ships a non-zero `limits.wall_clock_s` of 300 seconds. Credit: Nguyen Tran Thanh Lam ([c240030](https://github.com/c240030)). (GHSA-v2rm-hvrj-2x9q)
+- **XSS via unescaped PDF text in `cleaned_html` (CWE-79, medium)**: paragraph text taken verbatim from a PDF was written into `cleaned_html` without escaping, so markup embedded in a PDF survived into the result and executed when rendered. Paragraph text is now escaped like every other sink in that function. Credit: Nguyen Tran Thanh Lam ([c240030](https://github.com/c240030)). (GHSA-7g3g-vhm6-79f3)
+- **DOM-based XSS in the Docker Playground leading to API token theft (CWE-79, high)**: the result viewer reset syntax highlighting with `element.innerHTML = element.textContent`, which re-parsed attacker-controlled crawled content as live HTML in the operator's session. The round trip is removed; highlight.js renders safely from `textContent`. Credit: [e1codes](https://github.com/e1codes). (GHSA-m446-hp3q-qfxp)
+
+All reporters are credited in `SECURITY-CREDITS.md`. GitHub Security Advisories accompany this release.
+
+### Fixed
+
+This release also carries the bug fixes that accumulated on `develop` since 0.9.2.
+
+**Docker server**
+
+- PDF scraping is supported by default, and requests selecting `PDFContentScrapingStrategy` are routed to `PDFCrawlerStrategy` so the pairing works without extra configuration. (#2094, #2150)
+- The egress proxy chains through an upstream `HTTP_PROXY` / `HTTPS_PROXY` instead of ignoring it. (#2142)
+- Junk proxy environment values fall through to the next candidate, and non-http proxy schemes are refused. (#2094)
+- Compose v5 compatibility, clearer warnings on legacy fields, and better playground error handling. (#2094)
+- `CRAWL4AI_API_TOKEN` is forwarded through compose, and `.llm.env` is optional rather than required. (#2094)
+- The disabled-hooks 403 distinguishes the removed `hooks.code` field from other rejections. (#2094)
+- `output_path` is declared a deprecated no-op rather than silently ignored. (#2094)
+- `GET /monitor` redirects to the dashboard UI. (#2157, issue #2091)
+- Failed crawl results are preserved instead of dropped, for both batch and single-URL requests. (#2094, #2134, issue #2133)
+- An unavailable IPv6 loopback no longer breaks startup. (#2081, issue #2078)
+- `mcp` is capped below 2 so the v1 low-level API used by `mcp_bridge` keeps working. (#2148, thanks @weike-zhang)
+- Commented environment variable lines in compose are aligned with the environment list. (#2156)
+
+**Crawler and core**
+
+- `ManagedBrowser` no longer leaks a Playwright driver process when the browser fails to launch inside `__aenter__`. (#2160)
+- `PDFCrawlerStrategy` placeholder responses are no longer vetoed as anti-bot blocks, which previously failed every PDF crawl and burned the retry budget. (#2138, issue #2135)
+- Cookies are carried across manual PDF redirect hops, so gated and CDN-signed PDFs download correctly. (#2159)
+- Unconditional `setTimeout` waits are removed from the overlay and consent removal scripts, which could hang a crawl under a restrictive CSP. (#2139)
+- `remove_overlay_elements` no longer removes `<body>` when the body carries a global popup class. (#2163, thanks @Nalhin)
+- The body-visibility timeout is configurable and validated, and the timeout warning is emitted even with `verbose=False`. (#2117, #2131, #2145, issues #2116, #2129, #2144)
+
+**Documentation**
+
+- Self-hosting and migration guides updated for 0.9.x. (#2093)
+- The `PDFCrawlerStrategy` plus `PDFContentScrapingStrategy` pairing requirement is documented.
+
+### Tests
+
+- `tests/unit/test_pdf_download_limits.py`: 22 tests covering per-hop destination validation, DNS rebinding, redirect bounds, byte and page caps, and untrusted-body clamping.
+- `tests/unit/test_pdf_html_escaping.py`: escaping of PDF paragraph text in `cleaned_html`.
+- `deploy/docker/tests/test_security_pdf_image_write.py`: rejection of image-write fields from untrusted bodies.
+- Docker endpoint coverage for crawl failures and for the per-URL `crawler_configs` PDF guard.
+
+### Breaking Changes
+
+None.
+
 ## [0.9.0] - 2026-06-18
 
 0.9.0 is a major, secure-by-default release of the Crawl4AI Docker API server. The out-of-the-box deployment is now hardened with defense in depth: authentication is on by default, the server binds loopback unless you give it a token, and the network request body is treated as an untrusted trust boundary. This release contains breaking changes for the self-hosted HTTP server only. The core pip library (SDK / in-process use) is unchanged.

Dockerfile

@@ -1,7 +1,7 @@
 FROM python:3.12-slim-bookworm AS build
 
 # C4ai version
-ARG C4AI_VER=0.9.2
+ARG C4AI_VER=0.9.3
 ENV C4AI_VERSION=$C4AI_VER
 LABEL c4ai.version=$C4AI_VER
 

README.md

@@ -37,9 +37,11 @@ Limited slots._
 
 Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community.
 
-[✨ Check out latest update v0.9.2](#-recent-updates)
+[✨ Check out latest update v0.9.3](#-recent-updates)
 
-✨ **New in v0.9.2**: Maintenance patch release. Fixes a `MemoryAdaptiveDispatcher` task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU (`ENABLE_GPU=true`) Docker builds. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.2.md)
+✨ **New in v0.9.3**: Security release. Closes five coordinated-disclosure advisories: arbitrary file write, SSRF, and denial of service in the PDF processing path, plus two XSS issues in the Docker Playground. Also ships 33 bug fixes across the Docker server, crawler, and PDF handling. No new features, no breaking changes. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.3.md)
+
+✨ Recent v0.9.2: Maintenance patch release. Fixes a `MemoryAdaptiveDispatcher` task/page leak when a streaming crawl is closed, Docker Playground "Advanced Config" and Monitor WebSocket auth, Playwright headless-shell packaging, and GPU (`ENABLE_GPU=true`) Docker builds. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.2.md)
 
 ✨ Recent v0.9.0: Major secure-by-default release of the Docker API server. Auth is on by default, the server binds loopback unless given a token, and the request body is now an untrusted trust boundary. [Release notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.0.md)
 
@@ -565,6 +567,23 @@ async def test_news_crawl():
 ## ✨ Recent Updates
 
 <details open>
+<summary><strong>Version 0.9.3 Release Highlights - Security Release</strong></summary>
+
+A security release closing five coordinated-disclosure advisories. Four are in the PDF processing path: an arbitrary file write through `PDFContentScrapingStrategy` image-write fields, an SSRF where the PDF download followed redirects into internal addresses, a denial of service from unbounded PDF size and page count, and an XSS from unescaped PDF text in `cleaned_html`. The fifth is a DOM-based XSS in the Docker Playground that could expose the operator's API token.
+
+It also carries 33 bug fixes that accumulated since 0.9.2: PDF scraping now works out of the box on the Docker server, the egress proxy chains through an upstream proxy, failed crawl results are reported instead of dropped, a Playwright driver leak on failed browser launch is fixed, and PDF crawls are no longer wrongly flagged as anti-bot blocks.
+
+No new features, no breaking changes. Two defaults changed: PDF downloads now cap at 100 MiB and 2000 pages, and the Docker `limits.wall_clock_s` is now 300 seconds instead of 0.
+
+```bash
+pip install -U crawl4ai
+```
+
+[Full v0.9.3 Release Notes →](https://github.com/unclecode/crawl4ai/blob/main/docs/blog/release-v0.9.3.md)
+
+</details>
+
+<details>
 <summary><strong>Version 0.9.2 Release Highlights - Maintenance Bug Fixes</strong></summary>
 
 A maintenance patch release with bug fixes across the dispatcher, Docker, and GPU builds. `MemoryAdaptiveDispatcher` no longer leaks crawl tasks and browser pages when a streaming crawl is closed. Docker fixes cover the Playground "Advanced Config" 400, the Monitor WebSocket 500 under JWT auth, and Playwright headless-shell packaging. `ENABLE_GPU=true` Docker builds no longer fail on the CUDA toolkit.

SECURITY-CREDITS.md

@@ -18,3 +18,6 @@ We thank the following security researchers for their responsible disclosure:
 | UDU_RisePho | GitHub: [hoanggxyuuki](https://github.com/hoanggxyuuki) | Chromium launch-flag RCE class via extra_args (0.9.0) | 2026-06-18 |
 | Y4tacker | GitHub: [Y4tacker](https://github.com/Y4tacker) | Hook system exec() sandbox escape (MRO chain RCE), Chromium launch-arg injection (--utility-cmd-prefix RCE), HTTP crawler path traversal arbitrary file write | 2026-07-09 |
 | Rafael | GitHub: [rafaelfiguereod-stack](https://github.com/rafaelfiguereod-stack) | Reported SSRF, LLM key exfiltration, and auth gaps (already fixed / not exploitable in current code) | 2026-07-09 |
+| Zhixi "Jace" Sun | Independent security researcher | Arbitrary file write via unconfined PDFContentScrapingStrategy image-write fields in untrusted config bodies (0.9.3) | 2026-08-24 |
+| Nguyen Tran Thanh Lam | GitHub: [c240030](https://github.com/c240030) | SSRF via PDF download redirects, DoS via unbounded PDF size and page count, XSS via unescaped PDF text in cleaned_html (0.9.3) | 2026-07-27 |
+| e1codes | GitHub: [e1codes](https://github.com/e1codes) | DOM-based XSS in the Docker Playground leading to operator API-token theft (0.9.3) | 2026-07-24 |

crawl4ai/__version__.py

@@ -1,7 +1,7 @@
 # crawl4ai/__version__.py
 
 # This is the version that will be used for stable releases
-__version__ = "0.9.2"
+__version__ = "0.9.3"
 
 # For nightly builds, this gets set during build process
 __nightly_version__ = None

crawl4ai/async_configs.py

@@ -214,6 +214,29 @@ class UntrustedConfigError(ValueError):
         "override_navigator", "magic", "process_in_browser", "shared_data",
         "session_id",
     },
+    # PDFContentScrapingStrategy has no scalar allowlist (it is kept for its
+    # value fields), so its filesystem-write knobs must be blocked explicitly:
+    # image_save_dir is an unconfined write destination and save_images_locally
+    # turns the write on. extract_images stays allowed - it only returns the
+    # image bytes base64-inline in the response, no disk write.
+    "PDFContentScrapingStrategy": {
+        "image_save_dir", "save_images_locally",
+    },
+}
+
+# Field names that must NEVER be set from an untrusted body on ANY allowed
+# type, checked regardless of per-type allowlist. This is the fail-closed
+# backstop: the per-type maps above only cover BrowserConfig/CrawlerRunConfig,
+# so a strategy that (now or in future) exposes a filesystem-write, code, or
+# routing constructor arg is caught here even with no explicit allowlist.
+# Presence => 400 (loud), matching js_code/extra_args behavior.
+UNTRUSTED_GLOBAL_FORBIDDEN_FIELDS = {
+    # filesystem write / read sinks
+    "image_save_dir", "save_images_locally", "downloads_path", "user_data_dir",
+    "output_path", "save_path", "file_path", "local_path", "storage_state",
+    # code / command execution
+    "js_code", "js_code_before_wait", "c4a_script", "init_scripts",
+    "code", "command", "hook", "hooks",
 }
 
 # Scalar knobs an untrusted body MAY set, per class. A field not listed here is
@@ -244,6 +267,7 @@ class UntrustedConfigError(ValueError):
         "fetch_ssl_certificate",
         # timing / waiting
         "wait_until", "page_timeout", "wait_for", "wait_for_timeout",
+        "body_visibility_timeout",
         "wait_for_images", "delay_before_return_html", "mean_delay", "max_range",
         # scrolling / rendering
         "ignore_body_visibility", "scan_full_page", "scroll_delay",
@@ -272,6 +296,8 @@ class UntrustedConfigError(ValueError):
 _MAX_TIMEOUT_MS = 60_000
 _MAX_SCROLL_STEPS = 1000
 _MAX_VIEWPORT = 4000
+_MAX_PDF_BYTES = 100 * 1024 * 1024
+_MAX_PDF_PAGES = 2000
 
 
 def _filter_untrusted_fields(type_name: str, params: dict) -> dict:
@@ -280,7 +306,7 @@ def _filter_untrusted_fields(type_name: str, params: dict) -> dict:
     allowlist = UNTRUSTED_FIELD_ALLOWLIST.get(type_name)  # None => keep all non-forbidden
     out = {}
     for key, value in params.items():
-        if key in forbidden:
+        if key in UNTRUSTED_GLOBAL_FORBIDDEN_FIELDS or key in forbidden:
             raise UntrustedConfigError(
                 f"field '{key}' is not permitted on {type_name} from an untrusted request"
             )
@@ -299,7 +325,7 @@ def _cap_timeout(v):
         return min(int(v), _MAX_TIMEOUT_MS)
 
     if type_name == "CrawlerRunConfig":
-        for f in ("page_timeout", "wait_for_timeout"):
+        for f in ("page_timeout", "wait_for_timeout", "body_visibility_timeout"):
             if f in params:
                 params[f] = _cap_timeout(params[f])
         if isinstance(params.get("max_scroll_steps"), int):
@@ -308,6 +334,17 @@ def _cap_timeout(v):
         for f in ("viewport_width", "viewport_height"):
             if isinstance(params.get(f), int):
                 params[f] = max(1, min(params[f], _MAX_VIEWPORT))
+    elif type_name == "PDFContentScrapingStrategy":
+        # This class has no field allowlist, so without clamping a body could
+        # simply raise its own caps back to unbounded and re-open the DoS.
+        for f, cap in (("max_pdf_bytes", _MAX_PDF_BYTES), ("max_pdf_pages", _MAX_PDF_PAGES)):
+            if f in params:
+                v = params[f]
+                # <=0 or non-int would read as "no limit"; pin to the cap.
+                params[f] = cap if not isinstance(v, int) or v <= 0 else min(v, cap)
+        # Rasterizing every page is the most expensive thing this strategy can
+        # do, and nothing about untrusted crawling needs it.
+        params["extract_images"] = False
     return params
 
 
@@ -1463,6 +1500,8 @@ class CrawlerRunConfig():
                         Default: False.
         ignore_body_visibility (bool): If True, ignore whether the body is visible before proceeding.
                                        Default: True.
+        body_visibility_timeout (int): Maximum time in ms to wait for the body to become visible.
+                                       Default: 30000.
         scan_full_page (bool): If True, scroll through the entire page to load all content.
                                Default: False.
         scroll_delay (float): Delay in seconds between scroll steps if scan_full_page is True.
@@ -1640,6 +1679,7 @@ def __init__(
         c4a_script: Union[str, List[str]] = None,
         js_only: bool = False,
         ignore_body_visibility: bool = True,
+        body_visibility_timeout: int = 30000,
         scan_full_page: bool = False,
         scroll_delay: float = 0.2,
         max_scroll_steps: Optional[int] = None,
@@ -1770,6 +1810,13 @@ def __init__(
         self.c4a_script = c4a_script
         self.js_only = js_only
         self.ignore_body_visibility = ignore_body_visibility
+        if (
+            not isinstance(body_visibility_timeout, (int, float))
+            or isinstance(body_visibility_timeout, bool)
+            or body_visibility_timeout <= 0
+        ):
+            raise ValueError("body_visibility_timeout must be a positive number")
+        self.body_visibility_timeout = body_visibility_timeout
         self.scan_full_page = scan_full_page
         self.scroll_delay = scroll_delay
         self.max_scroll_steps = max_scroll_steps
@@ -2137,6 +2184,7 @@ def to_dict(self):
             "js_code_before_wait": self.js_code_before_wait,
             "js_only": self.js_only,
             "ignore_body_visibility": self.ignore_body_visibility,
+            "body_visibility_timeout": self.body_visibility_timeout,
             "scan_full_page": self.scan_full_page,
             "scroll_delay": self.scroll_delay,
             "max_scroll_steps": self.max_scroll_steps,