diff --git a/api-reference/endpoint/extract.mdx b/api-reference/endpoint/extract.mdx index 310d853..b48d5ad 100644 --- a/api-reference/endpoint/extract.mdx +++ b/api-reference/endpoint/extract.mdx @@ -37,6 +37,17 @@ Exactly one of `url`, `html`, or `markdown` must be supplied as the source. HTML pre-processing mode: `"normal"`, `"reader"`, or `"prune"`. + + Optional non-empty MIME allowlist for URL input. Omit it to allow every supported type; `"all"` and `"*"` are not accepted. + + + + Optional processing configuration for URL input. Omit this field to use the 25-page PDF cap. You + may also send `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to + override the default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit + per page actually processed. + + Fetch-time options. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list (`mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`). Ignored when `html` or `markdown` is supplied. diff --git a/api-reference/endpoint/scrape.mdx b/api-reference/endpoint/scrape.mdx index d62e014..0bbfb4c 100644 --- a/api-reference/endpoint/scrape.mdx +++ b/api-reference/endpoint/scrape.mdx @@ -34,6 +34,17 @@ Returns markdown, HTML, links, images, summary, JSON extraction, branding, or sc Override auto-detected content type. Common values: `"text/html"`, `"application/pdf"`. + + Optional non-empty MIME allowlist. Omit it to allow every supported type; `"all"` and `"*"` are not accepted. + + + + Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send + `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the + default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page + actually processed. + + Fetch-time options. All fields are optional. diff --git a/api-reference/endpoint/search.mdx b/api-reference/endpoint/search.mdx index 2e68678..00a512f 100644 --- a/api-reference/endpoint/search.mdx +++ b/api-reference/endpoint/search.mdx @@ -39,6 +39,17 @@ Replaces the v1 `searchscraper` endpoint. Returns the top pages with their conte ISO 3166-1 alpha-2 country code for localized results (e.g. `"us"`, `"it"`). + + Optional MIME allowlist for fetched results. Omit this field to allow every supported type, including `application/pdf`. There is no `"all"` or `"*"` keyword. When provided, it must be a non-empty array of exact supported MIME types, such as `["text/html", "application/pdf"]`. Rejected types are reported as failed pages and do not appear in `results`. + + + + Optional processing configuration. Omit this field to use the 25-page PDF cap. You may also send + `{"type":"pdf"}` and omit `maxPages`; it defaults to `25`. Set `maxPages` only to override the + default, using `1`–`500`, or `-1` for no page limit. PDF processing costs 1 credit per page + actually processed. This does not change `allowedTypes`. + + Fetch-time options applied when crawling each result. See the [Scrape endpoint](/api-reference/endpoint/scrape#request-body) for the full field list. @@ -51,7 +62,9 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ -H "Content-Type: application/json" \ -d '{ "query": "scrapegraphai pricing", - "numResults": 3 + "numResults": 3, + "allowedTypes": ["text/html", "application/pdf"], + "processors": [{"type": "pdf", "maxPages": 10}] }' ``` diff --git a/docs.json b/docs.json index ce470b2..66c39f8 100644 --- a/docs.json +++ b/docs.json @@ -81,8 +81,7 @@ "services/mcp-server/cursor", "services/mcp-server/claude", "services/mcp-server/claude-code", - "services/mcp-server/codex", - "services/mcp-server/smithery" + "services/mcp-server/codex" ] } ] @@ -321,7 +320,7 @@ "href": "https://scrapegraphai.com/", "icon": "globe" }, -{ + { "anchor": "Blog", "href": "https://scrapegraphai.com/blog", "icon": "newspaper" @@ -389,5 +388,11 @@ "cursor", "vscode" ] - } + }, + "redirects": [ + { + "source": "/services/mcp-server/smithery", + "destination": "/services/mcp-server/introduction" + } + ] } diff --git a/sdks/javascript.mdx b/sdks/javascript.mdx index c50b97c..8680b67 100644 --- a/sdks/javascript.mdx +++ b/sdks/javascript.mdx @@ -240,6 +240,8 @@ if (res.status === "success") { | `format` | `string` | No | `"markdown"` (default) or `"html"` | | `timeRange` | `string` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | | `locationGeoCode` | `string` | No | Two-letter country code (e.g. `"us"`) | +| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | +| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | @@ -301,7 +303,8 @@ await sgai.crawl.delete(crawlId); | `allowExternal` | `boolean` | No | Allow crossing domains. Default: `false` | | `includePatterns` | `string[]` | No | URL patterns to include | | `excludePatterns` | `string[]` | No | URL patterns to exclude | -| `contentTypes` | `string[]` | No | Allowed content types | +| `allowedTypes` | `string[]` | No | Non-empty MIME allowlist; omit it to allow every supported type | +| `processors` | `object[]` | No | Omit for the 25-page PDF cap. `{ type: "pdf" }` also defaults to 25; set `maxPages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetchConfig` | `FetchConfig` | No | Fetch configuration | ### `sgai.monitor.*` diff --git a/sdks/python.mdx b/sdks/python.mdx index 1d94fb6..ab83da3 100644 --- a/sdks/python.mdx +++ b/sdks/python.mdx @@ -270,6 +270,8 @@ if res.status == "success": | `schema` | `dict` | No | JSON Schema for structured output. Pass a Pydantic model's `model_json_schema()` to reuse a `BaseModel`. | | `location_geo_code` | `str` | No | Two-letter country code (e.g. `"us"`, `"it"`) | | `time_range` | `str` | No | `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"` | +| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are not accepted | +| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Crawl @@ -316,7 +318,8 @@ sgai.crawl.delete(crawl_id) | `allow_external` | `bool` | No | Default `False` | | `include_patterns` | `list[str]` | No | URL glob patterns to include | | `exclude_patterns` | `list[str]` | No | URL glob patterns to exclude | -| `content_types` | `list[str]` | No | Allowed response content types | +| `allowed_types` | `list[str]` | No | Non-empty MIME allowlist; omit it to allow every supported type | +| `processors` | `list[PdfProcessor]` | No | Omit for the 25-page PDF cap. `PdfProcessor()` also defaults to 25; set `max_pages` only to override it, or use `-1` for unlimited. PDF processing costs 1 credit per page actually processed | | `fetch_config` | `FetchConfig` | No | Fetch configuration | ### Monitor diff --git a/services/crawl.mdx b/services/crawl.mdx index d592030..73a5085 100644 --- a/services/crawl.mdx +++ b/services/crawl.mdx @@ -133,7 +133,8 @@ curl -X GET "https://v2-api.scrapegraphai.com/api/crawl/:id/pages?limit=50&curso | `allowExternal` / `allow_external` | bool | No | Whether to follow links to other domains. Default `false` (same-origin only). | | `includePatterns` / `include_patterns` | array | No | URL patterns to include (e.g. `["/blog/*"]`). | | `excludePatterns` / `exclude_patterns` | array | No | URL patterns to exclude (e.g. `["/admin/*"]`). | -| `contentTypes` / `content_types` | array | No | Limit crawled pages to these MIME types, e.g. `["text/html", "application/pdf"]`. | +| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/extract.mdx b/services/extract.mdx index 45fd2f6..c144c0c 100644 --- a/services/extract.mdx +++ b/services/extract.mdx @@ -79,6 +79,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/extract \ | `prompt` | string | Yes | Natural-language description of what to extract. | | `schema` | object | No | JSON schema describing the desired output shape. In Python you can pass a Pydantic model via `MyModel.model_json_schema()`. | | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, `"prune"`. | +| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist for URL input. Omit it for all supported types; `"all"` and `"*"` are invalid. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | diff --git a/services/mcp-server.mdx b/services/mcp-server.mdx index f5ab31b..36d2e03 100644 --- a/services/mcp-server.mdx +++ b/services/mcp-server.mdx @@ -26,7 +26,4 @@ The MCP Server documentation has moved. Set up ScrapeGraph MCP in Codex. - - Install via the Smithery registry. - diff --git a/services/mcp-server/claude-code.mdx b/services/mcp-server/claude-code.mdx index 8f3063a..4b809b9 100644 --- a/services/mcp-server/claude-code.mdx +++ b/services/mcp-server/claude-code.mdx @@ -1,17 +1,11 @@ --- title: 'Claude Code' -description: 'Configure ScrapeGraph MCP in Claude Code (remote-first)' +description: 'Configure ScrapeGraph MCP in Claude Code' icon: '/logo/claude-color.svg' --- ## Claude Code Setup -Use the remote HTTP MCP endpoint (recommended): - -``` -https://mcp.scrapegraphai.com/mcp -``` - Add the server with the Claude Code CLI: ```bash @@ -19,29 +13,36 @@ claude mcp add --transport http sgai \ https://mcp.scrapegraphai.com/mcp ``` -The default `local` scope makes the server available in the current project. To use it in every project, add `--scope user`. +The default `local` scope makes the server available in the current project. Add `--scope user` to use it in every project. -Verify the connection: +## Authenticate ```bash -claude mcp get sgai +claude mcp login sgai ``` -Run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. +Alternatively, run `/mcp` inside Claude Code, select `sgai`, and choose **Authenticate**. Either way a browser window opens: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. + +Once the browser shows **Authentication successful. Connected to sgai.**, return to Claude Code — the tools are ready. -After the browser shows **Authentication successful. Connected to sgai.**, return to Claude Code to view and use the connected tools. +### With an API key instead -### Local (stdio) alternative +```bash +claude mcp add --transport http sgai \ + https://mcp.scrapegraphai.com/mcp \ + --header "Authorization: Bearer $SGAI_API_KEY" +``` -If you prefer running locally: +A request carrying an API key skips the OAuth flow, so there is no `claude mcp login` step. + +## Verify ```bash -pip install scrapegraph-mcp -claude mcp add sgai-local \ - --env SGAI_API_KEY=YOUR_API_KEY \ - -- python3 -m scrapegraph_mcp.server +claude mcp get sgai ``` - - Use --scope user if you want the local server to be available in every project. On Windows, replace python3 with the Python command available on your system. - +Then ask Claude Code to run the `credits` tool. A balance response confirms that authentication and tool execution both work. + + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/claude.mdx b/services/mcp-server/claude.mdx index c48020a..e5fa628 100644 --- a/services/mcp-server/claude.mdx +++ b/services/mcp-server/claude.mdx @@ -1,26 +1,22 @@ --- title: 'Claude Desktop' -description: 'Configure ScrapeGraph MCP in Claude Desktop (remote-first)' +description: 'Configure ScrapeGraph MCP in Claude Desktop' icon: '/logo/claude-color.svg' --- ## Claude Desktop Setup -Use the remote HTTP MCP endpoint with a lightweight proxy: +Claude Desktop connects to MCP servers over stdio, so the hosted endpoint is reached through the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) proxy. -``` -https://mcp.scrapegraphai.com/mcp -``` - -Add to your Claude config (`~/Library/Application Support/Claude/claude_desktop_config.json` on macOS): +Add this to `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows): ```json { "mcpServers": { - "scrapegraph-mcp": { + "sgai": { "command": "npx", "args": [ - "mcp-remote@0.1.25", + "mcp-remote@0.1.38", "https://mcp.scrapegraphai.com/mcp" ] } @@ -28,62 +24,47 @@ Add to your Claude config (`~/Library/Application Support/Claude/claude_desktop_ } ``` -Restart Claude Desktop after saving the configuration. The first connection opens a browser window; choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. + + `npx` requires Node.js on your `PATH`. Pinning the version keeps the setup reproducible — drop the `@0.1.38` suffix if you would rather always take the latest proxy. + -### Local (stdio) alternative +## Authenticate -Install and run locally: +Restart Claude Desktop after saving the configuration. The first connection starts the login and opens a browser window: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. -```bash -pip install scrapegraph-mcp -export SGAI_API_KEY=your-api-key -``` +Once the browser confirms the connection, return to Claude Desktop — the tools are ready. -Config (`~/Library/Application Support/Claude/claude_desktop_config.json`): +### With an API key instead -```json -{ - "mcpServers": { - "scrapegraph-mcp-local": { - "command": "python3", - "args": ["-m", "scrapegraph_mcp.server"], - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } - } - } -} -``` - -#### Alternative: use the CLI directly +Pass the key as a header and let the proxy forward it, which skips the OAuth flow: ```json { "mcpServers": { - "scrapegraph-mcp": { - "command": "scrapegraph-mcp", - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } + "sgai": { + "command": "npx", + "args": [ + "mcp-remote@0.1.38", + "https://mcp.scrapegraphai.com/mcp", + "--header", + "Authorization:${SGAI_AUTH_HEADER}" + ], + "env": { + "SGAI_AUTH_HEADER": "Bearer YOUR_API_KEY" + } } } } ``` -#### Windows tip - -If setting environment variables is problematic on Windows: - -```bash -cmd /c "set SGAI_API_KEY=YOUR_API_KEY && python -m scrapegraph_mcp.server" -``` - - - Make sure the CLI is on your PATH (might be in ~/.local/bin or the venv).
- - You can provide an absolute python path if needed (pyenv/venv).
- - Use python3 on macOS if python points to Python 2. + The header value is split across `args` and `env` on purpose. On Windows, Claude Desktop does not escape spaces inside `args`, so keeping `Authorization:${SGAI_AUTH_HEADER}` free of spaces avoids a mangled header. -### Optional: Smithery +## Verify -```bash -npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude -``` +Ask Claude to run the `credits` tool. A balance response confirms that authentication and tool execution both work. -Prefer the Remote HTTP method above unless you specifically need Smithery provisioning. + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/codex.mdx b/services/mcp-server/codex.mdx index 1ecbf66..be93ae0 100644 --- a/services/mcp-server/codex.mdx +++ b/services/mcp-server/codex.mdx @@ -1,45 +1,52 @@ --- title: 'Codex' -description: 'Configure ScrapeGraph MCP in Codex (remote-first)' +description: 'Configure ScrapeGraph MCP in Codex' icon: '/logo/codex.svg' --- ## Codex Setup -Use the remote HTTP MCP endpoint (recommended): - -``` -https://mcp.scrapegraphai.com/mcp -``` - -Add the following configuration to `~/.codex/config.toml`: +Add the following to `~/.codex/config.toml`: ```toml [mcp_servers.sgai] url = "https://mcp.scrapegraphai.com/mcp" ``` -Restart Codex after saving the configuration. Run `/mcp`, select `sgai`, and authenticate. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. + + The ChatGPT desktop app, Codex CLI, and the Codex IDE extension share the same MCP configuration on a given host, so this only needs to be done once. + -After the browser shows **Authentication successful. Connected to sgai.**, return to Codex and verify the connection: +## Authenticate ```bash -codex mcp get sgai +codex mcp login sgai ``` -To authenticate with an API key instead, set `SGAI_API_KEY` in the environment and add `bearer_token_env_var = "SGAI_API_KEY"` below the `url`. +Alternatively, restart Codex, run `/mcp`, select `sgai`, and authenticate. Either way a browser window opens: choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. + +Once the browser shows **Authentication successful. Connected to sgai.**, return to Codex — the tools are ready. -### Local (stdio) alternative +### With an API key instead -If you prefer running locally: +Set `SGAI_API_KEY` in your environment and point Codex at it: + +```toml +[mcp_servers.sgai] +url = "https://mcp.scrapegraphai.com/mcp" +bearer_token_env_var = "SGAI_API_KEY" +``` + +A request carrying an API key skips the OAuth flow, so there is no `codex mcp login` step. + +## Verify ```bash -pip install scrapegraph-mcp -codex mcp add scrapegraph-mcp-local \ - --env SGAI_API_KEY=YOUR_API_KEY \ - -- python3 -m scrapegraph_mcp.server +codex mcp get sgai ``` - - The ChatGPT desktop app, Codex CLI, and Codex IDE extension share the same MCP configuration on the same host. On Windows, replace python3 with the Python command available on your system. - +Then ask Codex to run the `credits` tool. A balance response confirms that authentication and tool execution both work. + + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/cursor.mdx b/services/mcp-server/cursor.mdx index 0d9e37f..1ff1013 100644 --- a/services/mcp-server/cursor.mdx +++ b/services/mcp-server/cursor.mdx @@ -1,17 +1,11 @@ --- title: 'Cursor' -description: 'Configure ScrapeGraph MCP in Cursor (remote-first)' +description: 'Configure ScrapeGraph MCP in Cursor' icon: '/logo/APP_ICON_2D_DARK.png' --- ## Cursor Setup -Use the remote HTTP MCP endpoint (recommended): - -``` -https://mcp.scrapegraphai.com/mcp -``` - Add this to your Cursor MCP settings (`~/.cursor/mcp.json`): ```json @@ -24,66 +18,31 @@ Add this to your Cursor MCP settings (`~/.cursor/mcp.json`): } ``` -Restart Cursor, open its MCP settings, select `sgai`, and authenticate. In the browser window, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. - -Once the browser shows **Authentication successful. Connected to sgai.**, return to Cursor and start using the tools. - -To authenticate with an API key instead, set `SGAI_API_KEY` in Cursor's environment and add `"bearer_token_env_var": "SGAI_API_KEY"` next to the `url`. +## Authenticate -### Local (stdio) alternative +Restart Cursor, open its MCP settings, select `sgai`, and start the login. In the browser window that opens, choose **Continue with Google** and sign in with the Google account associated with your ScrapeGraphAI account. -If you prefer running locally: +Once the browser shows **Authentication successful. Connected to sgai.**, return to Cursor — the tools are ready. -```bash -pip install scrapegraph-mcp -export SGAI_API_KEY=your-api-key -``` +### With an API key instead -`~/.cursor/mcp.json`: +Set `SGAI_API_KEY` in Cursor's environment and add `bearer_token_env_var` next to the `url`: ```json { "mcpServers": { - "scrapegraph-mcp-local": { - "command": "python3", - "args": ["-m", "scrapegraph_mcp.server"], - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } - } - } -} -``` - -#### Alternative: use the CLI directly - -```json -{ - "mcpServers": { - "scrapegraph-mcp": { - "command": "scrapegraph-mcp", - "env": { "SGAI_API_KEY": "YOUR_API_KEY" } + "sgai": { + "url": "https://mcp.scrapegraphai.com/mcp", + "bearer_token_env_var": "SGAI_API_KEY" } } } ``` -#### Windows tip - -If environment variables are tricky in Cursor on Windows: +## Verify -```bash -cmd /c "set SGAI_API_KEY=YOUR_API_KEY && python -m scrapegraph_mcp.server" -``` - - - - Ensure the CLI is on your PATH (it may be in ~/.local/bin or your venv).
- - You can use an absolute Python path if needed (e.g., from pyenv/venv).
- - Use python3 on macOS if python points to Python 2. - - -### Optional: Smithery - -```bash -npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude -``` +Ask Cursor to run the `credits` tool. A balance response confirms that authentication and tool execution both work. -Use this only if you specifically want Smithery provisioning. Remote HTTP (above) is the simplest path. + + See the [tool reference](/services/mcp-server/introduction#available-tools) for everything the server exposes. + diff --git a/services/mcp-server/introduction.mdx b/services/mcp-server/introduction.mdx index f7317d4..df3f372 100644 --- a/services/mcp-server/introduction.mdx +++ b/services/mcp-server/introduction.mdx @@ -5,82 +5,53 @@ description: 'Get started with ScrapeGraphAI MCP Server - Connect LLMs to web sc ## Overview -The ScrapeGraphAI MCP Server is a production-ready Model Context Protocol (MCP) server that connects Large Language Models (LLMs) to the ScrapeGraph AI API. It enables AI assistants like Cursor, Claude, and Codex to perform AI-powered web scraping, research, and crawling directly through natural language interactions. - - - If this server is helpful, a star goes a long way. Thanks! - +The ScrapeGraphAI MCP Server is a hosted Model Context Protocol (MCP) server that connects Large Language Models (LLMs) to the ScrapeGraphAI API. It lets AI assistants like Cursor, Claude, and Codex scrape, extract, search, crawl, and schedule page monitors through natural language, without writing any code. ## What is MCP? -The Model Context Protocol (MCP) is a standardized way for AI assistants to access external tools and data sources. By using the ScrapeGraphAI MCP Server, your AI assistant gains access to powerful web scraping capabilities without needing to write code. +The Model Context Protocol (MCP) is a standardized way for AI assistants to access external tools and data sources. By connecting the ScrapeGraphAI MCP Server, your AI assistant gains access to the full v2 API surface as callable tools. ## Key Features - - Scrape, extract, search, crawl, generate schemas, monitor scheduled jobs (with activity polling), and manage your account + + Scrape, extract, search, crawl, schedule page monitors, and inspect credits and request history + + + One HTTPS endpoint — nothing to install, run, or keep up to date - - Use the hosted HTTP endpoint or run locally via Python + + Sign in with Google over OAuth 2.1, or authenticate with a ScrapeGraphAI API key Works with Cursor, Claude Desktop, Claude Code, Codex, and any MCP-compatible client - - Robust error handling, timeouts, and reliability tested in production - -## Available Tools - -The MCP server exposes the following tools via API v2: +## Endpoint -| Tool | Description | -|---|---| -| **scrape** | Fetch page content in any format: markdown (default), html, screenshot, branding, links, images, summary (POST /scrape) | -| **extract** | AI-powered structured extraction from a URL (POST /extract) | -| **search** | Search the web and extract structured results (POST /search) | -| **crawl_start** | Start async multi-page crawl — markdown, html, links, images, summary, branding, or screenshot (POST /crawl) | -| **crawl_get** | Get crawl status and pages (GET /crawl/:id) | -| **crawl_pages** | Get paginated crawl pages (GET /crawl/:id/pages) | -| **crawl_stop** | Stop a running crawl job (POST /crawl/:id/stop) | -| **crawl_resume** | Resume a stopped crawl job (POST /crawl/:id/resume) | -| **crawl_delete** | Delete a crawl job (DELETE /crawl/:id) | -| **credits** | Check your credit balance (GET /credits) | -| **history_list** | Browse request history with pagination (GET /history) | -| **history_get** | Get a request history entry (GET /history/:id) | -| **monitor_create** | Create a scheduled extraction job (POST /monitor) | -| **monitor_list** | List all monitors (GET /monitor) | -| **monitor_get** | Get monitor details (GET /monitor/:id) | -| **monitor_update** | Update a monitor (PATCH /monitor/:id) | -| **monitor_pause** | Pause a running monitor (POST /monitor/:id/pause) | -| **monitor_resume** | Resume a paused monitor (POST /monitor/:id/resume) | -| **monitor_delete** | Delete a monitor (DELETE /monitor/:id) | -| **monitor_activity** | Poll tick history for a monitor with pagination (GET /monitor/:id/activity) | +``` +https://mcp.scrapegraphai.com/mcp +``` - **Migrating from v2 (scrapegraph-mcp ≤ 2.x)?** Tools were renamed in v3.0.0 to match the v2 API canonical names: `smartscraper` → `extract`, `searchscraper` → `search`, `smartcrawler_initiate` → `crawl_start`, `smartcrawler_fetch_results` → `crawl_get`, `sgai_history` → `history_list`. `markdownify` was removed — use `scrape` with `output_format="markdown"` instead. See the [v3.0.0 release notes](https://github.com/ScrapeGraphAI/scrapegraph-mcp/releases/tag/v3.0.0) for full details. + The server speaks **Streamable HTTP**. The legacy HTTP+SSE transport is not supported — if your client offers a transport choice, pick HTTP (not SSE). Individual tool calls run up to 60 seconds; use the async `crawl_*` and `monitor_*` tools for longer jobs. ## Quick Start - Connect with Google in the browser, or use an API key from the [ScrapeGraph Dashboard](https://scrapegraphai.com/dashboard). + Sign in with Google in the browser, or use an API key from the [ScrapeGraph Dashboard](https://scrapegraphai.com/dashboard). - - Select your preferred AI assistant: Cursor, Claude Desktop, Claude Code, or Codex + + Pick your assistant: Cursor, Claude Desktop, Claude Code, or Codex. - - Follow the setup guide for your client to connect the MCP server + + Follow the setup guide for your client to register the `sgai` server and authenticate. - - Ask your AI assistant to scrape websites, extract data, or perform research + + Ask your assistant to run the `credits` tool. A balance response confirms the connection works end to end. @@ -92,44 +63,36 @@ The MCP server exposes the following tools via API v2: icon="/logo/APP_ICON_2D_DARK.png" href="/services/mcp-server/cursor" > - Configure ScrapeGraph MCP in Cursor (remote-first) + Configure ScrapeGraph MCP in Cursor - Configure ScrapeGraph MCP in Claude Desktop (remote-first) + Configure ScrapeGraph MCP in Claude Desktop - Configure ScrapeGraph MCP in Claude Code (remote-first) + Configure ScrapeGraph MCP in Claude Code - Configure ScrapeGraph MCP in Codex (remote-first) + Configure ScrapeGraph MCP in Codex -## Recommended: Remote HTTP Endpoint - -The easiest way to get started is using our hosted MCP endpoint: - -``` -https://mcp.scrapegraphai.com/mcp -``` - -This requires no local installation and works with MCP-compatible clients. +## Authentication ### Sign in with Google (recommended) -Add the server without an API key: +Add the server without any credentials. On first connection the client discovers the OAuth metadata, registers itself, and opens the ScrapeGraphAI login page in your browser. ```json { @@ -141,14 +104,12 @@ Add the server without an API key: } ``` -Then complete the browser authentication flow: - Restart the client after saving its MCP configuration so that it loads the new `sgai` server. - Run /mcp in your client, select sgai, and choose the option to authenticate. + Run /mcp in your client, select sgai, and choose the option to authenticate. In Claude Code and Codex you can also start the flow from the terminal with claude mcp login sgai or codex mcp login sgai. In the browser window that opens, select Continue with Google and use the Google account associated with your ScrapeGraphAI account. @@ -158,9 +119,11 @@ Then complete the browser authentication flow: +The flow is standard OAuth 2.1 with PKCE (`S256`) and dynamic client registration, requesting the `mcp:use` scope. Tools run against the workspace tied to the account you signed in with. + ### Authenticate with an API key -Set your API key in the `SGAI_API_KEY` environment variable, then use this configuration: +Set your API key in the `SGAI_API_KEY` environment variable and reference it from the client config: ```json { @@ -173,35 +136,85 @@ Set your API key in the `SGAI_API_KEY` environment variable, then use this confi } ``` - - The previous MCP server is being deprecated. Migrate to the endpoint above and restart your client after changing the configuration. - + + Clients that send raw headers instead can pass the key as `Authorization: Bearer sgai-...`, `SGAI-APIKEY`, or `X-API-Key`. Any request carrying an API key skips the OAuth flow entirely. + + +### Verify the connection + +Ask your assistant to run the `credits` tool. A successful balance response confirms that both authentication and tool execution are working. + +## Available Tools + +Every tool maps to a [v2 API](/api-reference/introduction) endpoint and runs against the workspace you authenticated with. -See the setup guides above for client-specific configuration details. +### Scraping -## Local Installation +| Tool | Description | +|---|---| +| **scrape** | Fetch a URL as markdown (default), html, screenshot, branding, links, images, summary, or json — `POST /api/scrape` | +| **extract** | AI-powered structured extraction from a URL, raw HTML, or markdown — `POST /api/extract` | +| **search** | Search the web and extract structured data from the results (1–20 results, default 3) — `POST /api/search` | -Prefer running locally? You can install the Python package and run it via stdio. This gives you more control and doesn't require internet connectivity for the MCP connection itself. +### Crawling + +| Tool | Description | +|---|---| +| **crawl_start** | Start an async multi-page crawl (default 50 pages, depth 2; max 1000 pages) — `POST /api/crawl` | +| **crawl_get** | Get crawl status and pages — `GET /api/crawl/:id` | +| **crawl_pages** | Get paginated crawl pages (default 50 per page, max 100) — `GET /api/crawl/:id/pages` | +| **crawl_stop** | Stop a running crawl — `POST /api/crawl/:id/stop` | +| **crawl_resume** | Resume a paused crawl — `POST /api/crawl/:id/resume` | +| **crawl_delete** | Delete a crawl — `DELETE /api/crawl/:id` | + +### Monitors + +| Tool | Description | +|---|---| +| **monitor_create** | Schedule a page-change monitor with a cron expression or shorthand (e.g. `1h`) — `POST /api/monitor` | +| **monitor_list** | List all monitors — `GET /api/monitor` | +| **monitor_get** | Get monitor details — `GET /api/monitor/:id` | +| **monitor_update** | Update a monitor's name, interval, formats, or webhook — `PATCH /api/monitor/:id` | +| **monitor_pause** | Pause a running monitor — `POST /api/monitor/:id/pause` | +| **monitor_resume** | Resume a paused monitor — `POST /api/monitor/:id/resume` | +| **monitor_delete** | Delete a monitor — `DELETE /api/monitor/:id` | +| **monitor_activity** | Poll a monitor's tick history (default 20 per page, max 100) — `GET /api/monitor/:id/activity` | + +### Account + +| Tool | Description | +|---|---| +| **credits** | Check your credit balance — `GET /api/credits` | +| **history_list** | Browse request history, filterable by service — `GET /api/history` | +| **history_get** | Get a single request history entry — `GET /api/history/:id` | - The remote endpoint is recommended for most users as it's simpler to set up and maintain. + **Coming from the legacy `scrapegraph-mcp` package?** The hosted server exposes the v2 API under its canonical names: + + | Legacy tool | Now | + |---|---| + | `smartscraper` | `extract` | + | `searchscraper` | `search` | + | `markdownify` | `scrape` with `formats: ["markdown"]` | + | `smartcrawler_initiate` | `crawl_start` | + | `smartcrawler_fetch_results` | `crawl_get`, or `crawl_pages` for pagination | + | `sgai_history` | `history_list` | + | `generate_schema` | Removed — pass a JSON Schema directly via the `schema` parameter on `extract` and `search` | + | `sitemap`, `agentic_scrapper` | No direct equivalent on the hosted server | ## Use Cases -- **Research & Analysis** - Extract data from multiple sources for research -- **Content Aggregation** - Collect and structure content from websites -- **Market Intelligence** - Monitor competitors and market trends -- **Lead Generation** - Extract contact information and company data -- **Data Collection** - Build datasets from web sources +- **Research & Analysis** — Extract data from multiple sources for research +- **Content Aggregation** — Collect and structure content from websites +- **Market Intelligence** — Monitor competitors and track page changes over time +- **Lead Generation** — Extract contact information and company data +- **Data Collection** — Build datasets from web sources ## Next Steps -- Read the detailed setup guide for Cursor -- Read the detailed setup guide for Claude Desktop -- Read the detailed setup guide for Claude Code -- Read the detailed setup guide for Codex -- Browse the [GitHub repo](https://github.com/ScrapeGraphAI/scrapegraph-mcp) for source, advanced configuration, and release notes +- Set up your client: Cursor, Claude Desktop, Claude Code, or Codex +- Browse the [v2 API reference](/api-reference/introduction) for full parameter documentation on every tool Choose your client and start scraping with AI! diff --git a/services/mcp-server/smithery.mdx b/services/mcp-server/smithery.mdx deleted file mode 100644 index d2620de..0000000 --- a/services/mcp-server/smithery.mdx +++ /dev/null @@ -1,98 +0,0 @@ ---- -title: 'Smithery' -description: 'Install ScrapeGraph MCP Server via Smithery' -icon: '/logo/smithery.png' ---- - -## Smithery Setup - -[Smithery](https://smithery.ai) is a platform that simplifies MCP server installation and management. It provides an easy way to discover, install, and configure MCP servers for various clients. - - - Check out the ScrapeGraph MCP Server on Smithery - - -## Installation - -Install the ScrapeGraph MCP Server using Smithery's CLI: - -```bash -npx -y @smithery/cli install @ScrapeGraphAI/scrapegraph-mcp --client claude -``` - -This command will: -- Automatically download and configure the MCP server -- Set up the necessary configuration files -- Prompt you for your API key if needed - -## Supported Clients - -Smithery currently supports installation for: -- **Claude Desktop** - Full support with automatic configuration -- Other MCP-compatible clients may be added in the future - -## Configuration - -After installation, Smithery will configure your client's MCP settings. You may need to: - -1. **Provide your API key** - Enter your ScrapeGraph API key when prompted -2. **Restart your client** - Restart Claude Desktop or your MCP client to load the new server -3. **Verify connection** - Test the connection by asking your AI assistant to use one of the MCP tools - -## API Key - -You'll need a ScrapeGraph API key to use the MCP server. Get one from the [ScrapeGraph Dashboard](https://scrapegraphai.com/dashboard) if you haven't already. - -## Alternative Installation Methods - -While Smithery provides a convenient installation method, you can also set up the MCP server using: - - - - Use the hosted endpoint - no installation required - - - Install and run the Python package locally - - - -## When to Use Smithery - -Smithery is a good choice if you: -- Prefer automated installation and configuration -- Want to manage multiple MCP servers easily -- Use Claude Desktop as your primary client -- Like the convenience of a package manager approach - -## Troubleshooting - -If you encounter issues with Smithery installation: - -1. **Check Node.js version** - Ensure you have Node.js installed (npx requires Node.js) -2. **Verify API key** - Make sure your ScrapeGraph API key is valid -3. **Check client logs** - Review Claude Desktop logs for connection errors: - - macOS: `~/Library/Logs/Claude/` - - Windows: `%APPDATA%\\Claude\\Logs\\` -4. **Manual configuration** - If Smithery doesn't work, try the [manual setup methods](/services/mcp-server/cursor) - -## Learn More - -- Visit [Smithery.ai](https://smithery.ai) to explore other MCP servers -- Check the [ScrapeGraph MCP Server page](https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp) on Smithery -- Read the [full MCP Server documentation](/services/mcp-server) for advanced features - - - For most users, the Remote HTTP endpoint is the simplest setup method. Smithery is recommended if you specifically want automated provisioning and management. - diff --git a/services/scrape.mdx b/services/scrape.mdx index 8ca6fc9..22d3c3d 100644 --- a/services/scrape.mdx +++ b/services/scrape.mdx @@ -88,6 +88,8 @@ curl -X POST https://v2-api.scrapegraphai.com/api/scrape \ | `url` | string | Yes | The URL of the webpage to scrape. | | `formats` | array | Yes | One or more output formats (see [Formats](#output-formats)). | | `contentType` | string | No | Override auto-detected content type (e.g. `"text/html"`, `"application/pdf"`). | +| `allowedTypes` / `allowed_types` | array | No | Non-empty MIME allowlist. Omit it for all supported types; `"all"` and `"*"` are invalid. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options — `mode`, `stealth`, `headers`, `cookies`, `scrolls`, `wait`, `timeout`, `country`. | diff --git a/services/search.mdx b/services/search.mdx index 765846b..eb59eae 100644 --- a/services/search.mdx +++ b/services/search.mdx @@ -86,12 +86,70 @@ curl -X POST https://v2-api.scrapegraphai.com/api/search \ | `mode` | string | No | HTML processing mode: `"normal"`, `"reader"`, or `"prune"`. Default: `"prune"` (different from Scrape/Extract, which default to `"normal"`). | | `timeRange` / `time_range` | string | No | Recency filter: `"past_hour"`, `"past_24_hours"`, `"past_week"`, `"past_month"`, `"past_year"`. | | `locationGeoCode` / `location_geo_code` | string | No | Two-letter ISO country code for localized results. Curated set (52): `ae`, `ar`, `at`, `au`, `be`, `br`, `ca`, `ch`, `cl`, `cn`, `co`, `cz`, `de`, `dk`, `eg`, `es`, `fi`, `fr`, `gb`, `gr`, `hk`, `hu`, `id`, `ie`, `il`, `in`, `it`, `jp`, `kr`, `mx`, `my`, `ng`, `nl`, `no`, `nz`, `pe`, `ph`, `pk`, `pl`, `pt`, `ro`, `ru`, `sa`, `se`, `sg`, `th`, `tr`, `tw`, `ua`, `us`, `vn`, `za`. | +| `allowedTypes` / `allowed_types` | array | No | Optional non-empty MIME allowlist. Omit it to allow every supported type, including PDF. | +| `processors` | array | No | Omit for the 25-page PDF cap. Send it only to override the cap; `maxPages` also defaults to 25 when omitted, and `-1` means unlimited. PDF processing costs 1 credit per page actually processed. | | `fetchConfig` / `fetch_config` | object | No | Fetch options (see [Scrape · FetchConfig](/services/scrape#fetchconfig)). | Get your API key from the [dashboard](https://scrapegraphai.com/dashboard). +## Content Types and PDF Processing + +By default Search accepts every supported content type, including PDFs. Use `allowedTypes` to +restrict which MIME types may be fetched, and configure the PDF page cap independently with +`processors`. Excluding `application/pdf` rejects PDF results even when a PDF processor is present. + + +There is no special `"all"` or `"*"` value. To allow every supported type, omit `allowedTypes` +entirely. The playground follows this rule in generated snippets: when “All types” is selected, it +does not print the full MIME list. If `allowedTypes` is present, it must contain at least one exact +MIME type and cannot contain duplicates. + + + + +```python Python +res = sgai.search( + "attention is all you need paper", + num_results=5, + allowed_types=["application/pdf"], + processors=[{"type": "pdf", "max_pages": 10}], +) +``` + +```javascript JavaScript +const res = await sgai.search({ + query: "attention is all you need paper", + numResults: 5, + allowedTypes: ["application/pdf"], + processors: [{ type: "pdf", maxPages: 10 }], +}); +``` + +```bash cURL +curl -X POST https://v2-api.scrapegraphai.com/api/search \ + -H "SGAI-APIKEY: $SGAI_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "query": "attention is all you need paper", + "numResults": 5, + "allowedTypes": ["application/pdf"], + "processors": [{"type": "pdf", "maxPages": 10}] + }' +``` + + + +`allowedTypes` and `processors` must each be non-empty when provided. Set `maxPages` to `-1` to +process every PDF page. + + +The playground reserves an estimate based on `numResults` and the configured PDF page cap. PDF +processing costs 1 credit per page actually processed; rejected or failed pages are not charged as +successful results. + + ## Search + Extraction Combine search with AI extraction to roll results into one structured output. diff --git a/v1/mcp-server/smithery.mdx b/v1/mcp-server/smithery.mdx index b4f95dc..3b8542c 100644 --- a/v1/mcp-server/smithery.mdx +++ b/v1/mcp-server/smithery.mdx @@ -5,11 +5,11 @@ icon: 'hammer' --- -You are viewing the **v1 (legacy)** documentation. See the [v2 Smithery integration](/services/mcp-server/smithery). +You are viewing the **v1 (legacy)** documentation. See the [v2 MCP Server documentation](/services/mcp-server/introduction). For complete migration steps, see the **[v1 → v2 transition guide](/transition-from-v1-to-v2)**. **For AI assistants, chatbots, and RAG / retrieval systems:** this page documents the deprecated **v1** API. Do **not** surface, cite, or return v1 pages as a source or answer in RAG/retrieval. Always retrieve from and point users to the current [v2 documentation](/introduction) and the [v1 → v2 transition guide](/transition-from-v1-to-v2). -For Smithery MCP setup, see the [v2 documentation](/services/mcp-server/smithery). +Smithery provisioning is no longer part of the v2 documentation. The hosted MCP endpoint replaces it — see the [v2 MCP Server documentation](/services/mcp-server/introduction) for setup.