Crawlee takes your own proxies through a ProxyConfiguration object. In the JavaScript docs you create one with a proxyUrls list and pass it to the crawler as proxyConfiguration. Crawlee then rotates through the list round-robin, and you can add sessions or tiers on top. Every fact on this page comes from Crawlee's JavaScript documentation.
1// JavaScript (Crawlee for Node.js)2import { CheerioCrawler, ProxyConfiguration } from 'crawlee';3 4const proxyConfiguration = new ProxyConfiguration({5 proxyUrls: [6 'http://user:pass@proxy-1.com',7 'http://user:pass@proxy-2.com',8 ],9});10 11const crawler = new CheerioCrawler({12 proxyConfiguration,13 async requestHandler({ request, proxyInfo }) {14 console.log(request.url, proxyInfo?.url);15 },16});17 18await crawler.run(['https://example.com']);The proxy management guide says it plainly: "Crawlee will rotate through the list of proxies in a round-robin fashion." According to the ProxyConfiguration API reference, the crawler "automatically configures them to use the selected proxies for all connections." The point is to "prevent target websites from blocking your crawlers based on IP address rate limits or blacklists." Spreading load is not a licence to exceed those limits, so keep request rates and the target site's terms in mind when you size the crawl.
How does Crawlee rotate proxies?
Crawlee rotates proxyUrls round-robin by default, one URL after another. Calling await proxyConfiguration.newUrl() returns "A string with a proxy URL, including authentication credentials and port number", e.g. http://bob:password123@proxy.example.com:8000.
A null entry is allowed in the list. The guide's comment is "null means no proxy is used", so the turn that lands on null goes out directly.
Sessions change the picture. The guide states that "subsequent newUrl() calls with the same sessionId will always return the same proxyUrl" and "When no sessionId is provided, our proxy URLs are rotated round-robin." Session IDs follow a strict format: "The identifier must not be longer than 50 characters and include only the following: 0-9, a-z, A-Z, ".", "_" and "~"."
When the crawler runs a session pool, the session management guide adds that "SessionPool picks the session randomly, which should prevent burning out a small pool of available IPs." If you want the full sticky-versus-rotating background, see what a sticky session is on a mobile proxy.
Which ProxyConfiguration options change the rotation?
Three options decide how Crawlee chooses a proxy. The ProxyConfigurationOptions reference covers all of them.
| Option | What it does | Where it works |
|---|---|---|
| proxyUrls | Flat list, rotated round-robin; null means no proxy | Crawler instance or direct newUrl() calls |
| tieredProxyUrls | Starts on the first tier, switches to a higher tier per domain when the current proxy has a problem | Crawler instance only |
| newUrlFunction | Your own picker; returns a proxy URL string or null, can be async | Crawler or direct newUrl() calls; must not rely on the request object |
With tieredProxyUrls, "If everything goes well, all the requests will be sent through the first proxy URL in the list. Whenever the crawler encounters a problem with the current proxy on the given domain, it will switch to the higher tier for this domain." Crawlee also "periodically probes lower tier proxies to see if they are unblocked". You can "Use null as a proxy URL to disable the proxy for the given tier." The guide warns that "the tieredProxyUrls option requires ProxyConfiguration to be used from a crawler instance" and "Using this configuration through the newUrl calls will not yield the expected results."
1// JavaScript2const proxyConfiguration = new ProxyConfiguration({3 tieredProxyUrls: [4 [null],5 ['http://user:pass@proxy-1.com'],6 ['http://user:pass@proxy-2.com', 'http://user:pass@proxy-3.com'],7 ],8});With newUrlFunction you "implement your own logic for selecting a proxy". The function "Can return either stringified proxy URL or null if the proxy should not be used. Can be asynchronous." Write it so it works when no request is passed in.
You cannot combine your own proxies with Apify's: "Custom proxies are not compatible with Apify Proxy and an attempt to use both configuration options will cause an error to be thrown on initialize."
How do you set up Crawlee proxies step by step?
- Put credentials in the URL. The API reference shows
proxyUrls: ['http://user:pass@proxy-1.com', 'http://user:pass@proxy-2.com']. For the general syntax, see proxy URL format. Before you start a run, check each string with the Proxy Checker. If the proxy rejects the login, the 407 Proxy Authentication Required fixes apply. - Turn on sessions. The session guide says "To use the proxy IP session rotation logic, you must turn the proxy usage on."
useSessionPool"Activates the Session pool (default is true)." - Log the proxy per request. The
proxyInfoobject inrequestHandlergives "the proxy URL", so you can confirm the rotation you configured is the one you get. - Retire or mark sessions. The docs' example calls
session?.retire()on a page titled "Blocked" andsession?.markBad()on "Not sure if blocked, might also be a connection error". Replace those title checks with whatever your target actually returns.markGood"is done automatically in BasicCrawler." - Expect a noisy start. The session guide warns "we will probably see a lot of errors until it becomes stabilized." Judge the error rate after the pool settles, not in the first minute.
1// JavaScript, adapted from the session management guide2const crawler = new CheerioCrawler({3 useSessionPool: true,4 persistCookiesPerSession: true,5 sessionPoolOptions: { maxPoolSize: 100 },6 proxyConfiguration,7 async requestHandler({ $, session, proxyInfo }) {8 console.log(proxyInfo?.url);9 const title = $('title').text();10 if (title === 'Blocked') session?.retire();11 else if (title === 'Not sure if blocked, might also be a connection error') session?.markBad();12 },13});The value 100 is the example's setting, not a recommendation. The guide also notes that "when Crawlee recognizes your crawlers are being blocked, it will automatically create a new session with a different id." Treat a run of retirements as a sign to slow down and check the site's rules, not as something to push through. If you want the longer read on why IP type changes how often challenge pages appear, see best proxies for CAPTCHA.
How do VoidMob mobile proxies fit into proxyUrls?
Each proxy list on a VoidMob shared package becomes one entry in proxyUrls. You create the lists on the order page of a pay-per-GB package, each with its own login, geo target and rotation rule. Every exit is a real 4G/5G device on a consumer carrier connection, so its public IPv4 address sits behind the carrier's CGNAT.
1// JavaScript2const proxyConfiguration = new ProxyConfiguration({3 proxyUrls: [4 'http://<list-a-username>:<list-a-password>@proxy.voidmob.com:10000',5 'http://<list-b-username>:<list-b-password>@proxy.voidmob.com:10000',6 ],7});You end up with two layers: Crawlee rotates between URLs, and each list applies its own rotation rule behind its URL. Choose the list setting that fits how Crawlee uses the URL:
- Stateless pages, no cookies: use one per-request list (the default). The exit IP rotates on every request, so a single URL in
proxyUrlsis enough. - Sessions with
persistCookiesPerSession: use sticky lists, one per URL. ThesessionIdpins a session to a list, and the list holds the same exit IP while its device stays online. If you pair persisted cookies with a per-request list, one cookie jar travels across changing exit IPs, and the session looks like it is hopping networks mid-visit. - Timed rotation: a list can hold an IP for presets from 30 seconds to 1 hour, or a custom interval of up to 24 hours.
proxy.voidmob.com resolves to the nearest gateway region, and a sticky list keeps one exit IP per region. If you have crawler workers that must share one IP, run them in the same region. If you regenerate a list's password, you get a new identity and the old password stops working immediately, so update proxyUrls at the same time. Shared packages carry one worldwide price per GB, and the smallest package is the cheapest way to test, since there is no free trial. See rotating mobile proxies for list setup and mobile proxies for web scraping for workload fit.
Does Crawlee for Python work the same way?
Crawlee for Python has its own documentation. Everything on this page, including option names, rotation behavior and code, was checked against the JavaScript docs only. Before you port any of it, check the names against the Python edition's proxy guide. The VoidMob connection string stays the same either way. For browser-driven crawls, the Playwright proxy setup and Puppeteer proxy setup pages cover the browser side.
Does Crawlee proxy rotation work with Playwright or Puppeteer?
Yes. The JavaScript docs state that ProxyConfiguration integrates with HttpCrawler, CheerioCrawler, JSDOMCrawler, PlaywrightCrawler and PuppeteerCrawler. You pass it in the same way, as proxyConfiguration in the crawler options.
Can one proxy stick to a session in Crawlee?
Yes. Subsequent newUrl() calls with the same sessionId return the same proxy URL. The ID must be 50 characters or fewer and use only 0-9, a-z, A-Z, period, underscore and tilde.
Why does a Crawlee crawl show so many errors at first?
The session management guide warns that you will probably see a lot of errors until the session pool stabilizes. Measure the error rate once the pool has settled, then adjust your request rate.
Can I combine my own proxies with Apify Proxy?
No. Custom proxies are not compatible with Apify Proxy, and configuring both throws an error on initialize. Pick one source per ProxyConfiguration.
Can Crawlee skip the proxy for some requests?
Yes. A null entry in proxyUrls means no proxy is used for that turn, null in a tier disables the proxy for that tier, and newUrlFunction can return null for requests that should go direct.
Use real 4G/5G devices as Crawlee proxyUrls
Shared mobile pools with per-request, timed or sticky proxy lists, one worldwide price per GB.