Scrapy Proxy: HttpProxyMiddleware Setup and Limits

Scrapy supports HTTP proxies out of the box. The built-in HttpProxyMiddleware is enabled by default at order 750 and reads the proxy key in each request's meta. You set request.meta["proxy"] or export http_proxy/https_proxy, and you need no extra package for a single proxy. Which proxy types work depends on the download handler that runs the request.

The Scrapy FAQ puts it in one line: "Yes. Support for HTTP proxies is provided through the HTTP Proxy downloader middleware." The settings reference lists "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750 in the default base list. You do not have to enable it.

How do you set a proxy in Scrapy?

You can set a proxy in Scrapy in two ways: a per-request proxy meta key, or environment variables. According to the downloader middleware docs, the middleware "sets the HTTP proxy to use for requests, by setting the proxy meta value for Request objects." The meta value takes the form http://some_proxy_server:port or http://username:password@some_proxy_server:port, so credentials go inside the URL.

1import scrapy2 3PROXY = "http://<list-username>:<list-password>@proxy.voidmob.com:10000"4 5class ExitCheckSpider(scrapy.Spider):6    name = "exit_check"7 8    def start_requests(self):9        yield scrapy.Request(10            "https://ipinfo.io/json",11            meta={"proxy": PROXY},12        )13 14    def parse(self, response):15        self.logger.info(response.text)

The environment route works for every request without code changes. The docs say that, "Like the Python standard library module urllib.request, it obeys the following environment variables": http_proxy, https_proxy and no_proxy.

1export http_proxy="http://<list-username>:<list-password>@proxy.voidmob.com:10000"2export https_proxy="$http_proxy"3scrapy crawl exit_check

When both are set, the meta key wins. The docs state that the meta value "will take precedence over http_proxy/https_proxy environment variables, and it will also ignore no_proxy environment variable." A host listed in no_proxy still goes through the proxy if its request carries a proxy meta value.

Proxy credentials are encoded with HTTPPROXY_AUTH_ENCODING, "The default encoding for proxy authentication on HttpProxyMiddleware". The default is latin-1. If a password holds characters outside that range, change the setting. If the gateway rejects you, work through 407 Proxy Authentication Required. For the host:port:user:pass versus URL question, see proxy format.

Which proxy types does each Scrapy download handler support?

Proxy support in Scrapy depends on the download handler, not on the middleware. The docs say: "Handling of this meta key needs to be implemented inside the download handler, so it's not guaranteed to be supported by all 3rd-party handlers. It's currently unsupported by H2DownloadHandler."

Two lines from the docs matter most here. The first is "Currently, HTTP11DownloadHandler supports HTTPS proxies only for HTTP destinations." An HTTPS proxy in that sentence means a proxy URL that starts with https://. That restriction applies to https:// proxy URLs, and the list endpoint above is an http:// URL. The second is "HttpxDownloadHandler supports SOCKS proxies while other built-in handlers don't." The FAQ confirms SOCKS works "when using HttpxDownloadHandler". VoidMob lists accept SOCKS5 on the same host, port and credentials with a socks5:// scheme. In Scrapy, that only applies once Httpx is the active handler. For the protocol trade-offs, read SOCKS5 vs HTTP vs HTTPS.

How do you add a custom proxy middleware in Scrapy?

A custom proxy middleware in Scrapy is a downloader middleware that writes request.meta["proxy"] before HttpProxyMiddleware reads it. The docs give the order rule: "the process_request() method of each middleware will be invoked in increasing middleware order". Any order below 750 runs first.

1# middlewares.py2import os3 4class GatewayProxyMiddleware:5    def process_request(self, request, spider=None):6        request.meta.setdefault("proxy", os.environ["PROXY_URL"])

1# settings.py2DOWNLOADER_MIDDLEWARES = {3    "myproject.middlewares.GatewayProxyMiddleware": 350,4}

Follow these rules when you register it:

• Register in DOWNLOADER_MIDDLEWARES. The docs say: "add it to the DOWNLOADER_MIDDLEWARES setting, which is a dict whose keys are the middleware class paths and their values are the middleware orders." That dict "is merged with DOWNLOADER_MIDDLEWARES_BASE".
• Leave the base setting alone. For DOWNLOADER_MIDDLEWARES_BASE, the docs say: "You should never modify this setting in your project, modify DOWNLOADER_MIDDLEWARES instead."
• Disable with None. "A component with priority None is disabled." Setting HttpProxyMiddleware to None switches off proxy handling entirely.
• Register each component once. "Be careful not to define the same component more than once, e.g. with different import path strings or defining both an import path and a type object."
• Scope settings to a spider when needed. Project settings live in "a settings module, usually a file called settings.py". To override them for one spider, set custom_settings on the spider class.

1class PricesSpider(scrapy.Spider):2    name = "prices"3    custom_settings = {4        "DOWNLOADER_MIDDLEWARES": {"myproject.middlewares.GatewayProxyMiddleware": 350},5    }

How does rotation work with a mobile proxy gateway in Scrapy?

A mobile proxy gateway rotates exit IPs on its side, so Scrapy only ever sees one proxy URL. VoidMob: mobile proxies on real 4G/5G devices, non-VoIP SMS verification and eSIM. On a VoidMob pay-per-GB package you create a proxy list in the dashboard. You get one login and password per list, plus a rotation setting. The list's connection string is the URL you put in the proxy meta key.

• Per request (the default) gives each request a fresh IP from the mobile pool. This mode fits independent page fetches.
• Sticky holds the same device while it stays online. This mode fits login, cart or multi-step flows. The sticky session explainer covers how long to hold one.
• Timed rotation holds an IP for a set number of seconds, then rotates.

You set the country, region or city as list fields, and you pick the country from the mobile proxy locations hub. Every exit sits behind the carrier's CGNAT and shares its IPv4 address with ordinary subscribers. Developers who want one credential for the whole pool can use flex mode at proxy.voidmob.com:10092, for example http://<username>_c_US_s_cart1:<password>@proxy.voidmob.com:10092. The username and password come from POST /v1/proxies/:id/flex_credentials, not from the dashboard. As of October 2026, shared data costs $3.99 for 1 GB and drops to $2.50 per GB on the 100 GB tier.

Rotation spreads independent requests across devices. It is not a way around a site's limits. If a site slows you down, slow down too.

What settings keep a proxied Scrapy crawl well behaved?

Three Scrapy settings matter on a proxied crawl:

• Timeout. DOWNLOAD_TIMEOUT is "The amount of time (in secs) that the downloader will wait before timing out". The default is 180. You can lower it per request with the download_timeout meta key, so one slow exit cannot stall a request for three minutes.
• User agent. The settings docs advise: "Set it to a value that identifies you, including a URL or an email address where website owners can reach you, e.g. "MyProject (+https://example.com/bot)", so that they can ask you to adjust your crawler rather than block it."
• Meta usage. The Request/Response docs say: "Use Request.meta only for data aimed at components, such as middlewares and extensions."

If requests fail at the gateway rather than at the target, mobile proxy not working walks through the common causes. You can also test the endpoint on its own with the Proxy Checker before you blame the spider. For non-Scrapy code, the same proxy URL works in Python Requests.