scrapy-botproxy

BotProxy downloader middleware for Scrapy.

Overview

BotProxy is an IP-rotating HTTP proxy. Point Scrapy at one endpoint and your requests go out through rotating datacenter or residential exits, in the country or city you ask for, with optional sticky sessions.

You need an active subscription. Get one at botproxy.com.

Installation

pip install scrapy-botproxy

Usage

settings.py:

DOWNLOADER_MIDDLEWARES = {
    "scrapy_botproxy.BotProxyMiddleware": 100,
    # Optional but recommended -- see "Status codes" below.
    "scrapy_botproxy.BotProxyStatusMiddleware": 560,
}

BOTPROXY_USER = "pxu1000-0"      # proxy user key from your dashboard
BOTPROXY_PASSWORD = "key-password"

That is the whole setup. Everything else is optional.

Choosing where requests come from

BotProxy takes its instructions from the proxy username, so all of these are just different usernames under the hood. Set them globally in settings.py:

BOTPROXY_COUNTRY = "US"          # any exit in this country
BOTPROXY_LOCATION = "us-ny"      # a specific location
BOTPROXY_RESIDENTIAL = True      # residential exit, any region
BOTPROXY_RESIDENTIAL = "US"      # residential exit in a country
BOTPROXY_SESSION = "batch-17"    # keep the same exit IP across requests

…or per request, through Request.meta, which overrides the settings:

yield scrapy.Request(
    url,
    meta={
        "botproxy_country": "DE",
        "botproxy_session": "user-42",
    },
)

meta key	Effect
botproxy_country	Two-letter country code
botproxy_location	Location code, e.g. us-ny
botproxy_residential	True for any region, or a country code
botproxy_session	Session id — same id keeps the same exit IP
botproxy_disable	True sends this request directly, bypassing the proxy

If several are given, residential wins over location, and location wins over country. Requests that already carry a proxy in meta are left untouched.

RS always means residential, never Serbia

BOTPROXY_COUNTRY = "RS" gets you a residential exit, not a Serbian one. To reach a Serbian datacenter exit, address it by location: "rs-bg".

Sessions

The same session id keeps the same exit IP. Change the id to force a new one. Sessions expire after your proxy user's max session age (60 seconds by default, configurable in the dashboard) and after 5 minutes of inactivity regardless. They are not shared between superproxies. Residential stickiness is best effort.

A session with no geography is fine — that is the user++session form, which this middleware builds for you.

Status codes

BotProxy reports its own problems through HTTP status codes. Scrapy has no idea what they mean: it will happily retry a 402 for an exhausted residential balance until the crawl ends, having fetched nothing, and treat a 556 refusal body as page content. BotProxyStatusMiddleware fixes that.

Install it at 560 so it sees responses before RetryMiddleware (550) — process_response runs in descending order.

Code	Meaning	What the middleware does
555	Blacklisted site or port, or free-trial restriction	IgnoreRequest — the URL was never fetched
556	Blocked by one of your own request filters	IgnoreRequest — not billed, and it will never succeed
401	Bad proxy credentials	Logs once, IgnoreRequest
402	Subscription suspended, or residential balance exhausted	Logs once, IgnoreRequest
403	Proxy user auto-disabled by the success-rate guard	Logs once, IgnoreRequest
407	Credentials missing or malformed	Logs once, IgnoreRequest

Set BOTPROXY_RAISE_ON_ACCOUNT_ERROR = False to have the 4xx account errors handed to your spider as ordinary responses instead.

It also records the exit IP the target saw:

def parse(self, response):
    self.logger.info("came from %s", response.meta.get("botproxy_ip"))

Residential exits report no IP — they come from a third-party pool, so there is no address of ours to report.

Retries

BotProxy's transient failures are not in Scrapy's default RETRY_HTTP_CODES. To retry them:

from scrapy_botproxy import BOTPROXY_RETRY_CODES   # [515, 520]

RETRY_HTTP_CODES = [500, 502, 503, 504, 522, 524, 408, 429, *BOTPROXY_RETRY_CODES]

BotProxy already retries up to three of its own nodes before giving up, so these mean "all of them failed".

Anti-Detect mode and TLS

Anti-Detect (browser impersonation) is on by default for new proxy users, and it works by intercepting TLS. Scrapy does not verify certificates by default, so this needs no configuration — but if you have installed a custom DOWNLOADER_CLIENTCONTEXTFACTORY that does verify, requests will fail until you exclude the proxy or turn Anti-Detect off in the dashboard.

BotProxy also offers a TLS-wrapped endpoint on port 8443. Scrapy cannot use it: it connects to a proxy in cleartext and has no support for an https:// proxy URL. Use the default port 8080.

Settings reference

Setting	Default	Meaning
BOTPROXY_USER	—	Proxy user key. Required.
BOTPROXY_PASSWORD	—	Proxy user password. Required.
BOTPROXY_ENABLED	True	Set False to switch the middleware off
BOTPROXY_ENDPOINT	http://x.botproxy.net:8080	Proxy endpoint
BOTPROXY_COUNTRY	—	Default country code
BOTPROXY_LOCATION	—	Default location code
BOTPROXY_RESIDENTIAL	—	True, or a country code
BOTPROXY_SESSION	—	Default session id
BOTPROXY_RAISE_ON_ACCOUNT_ERROR	True	Drop requests on 401/402/403/407

Requirements

Python 3.9+ and Scrapy 2.0+.

License

MIT — see LICENSE.