Web Scraping With Python: Full 2026 Tutorial With Proxies

Web scraping with Python takes a few libraries and a few dozen lines of code to get a first scraper working. Most of the work comes after that: pages that load their data with JavaScript, rate limits, and sites that judge each request by its IP address and TLS fingerprint. Below, one scraper grows step by step against Scrape This Site, a public sandbox built for scraping practice, from a single request to a setup with retries, a browser-grade TLS handshake and proxies. Each snippet targets that site, so you can run it as you read.

Choose Your Python Web Scraping Libraries

Python has a library for each layer of a scraper. Most projects need two or three of them, not all six.

httpx is a close alternative to requests with async support, worth switching to once a scraper needs many requests in parallel.

Set Up Your Python Scraping Environment

Use Python 3.10 or newer, since current releases of requests, Playwright, Selenium and curl_cffi all require it. Each scraper gets its own virtual environment, which keeps its dependencies separate from the rest of your system.

1python -m venv .venv2source .venv/bin/activate   # Windows: .venv\Scripts\activate3pip install requests beautifulsoup4 lxml

Later steps add Playwright, Selenium and curl_cffi when they need them. Versions as of October 2026: requests 2.34, BeautifulSoup 4.15, Playwright 1.64, Selenium 4.51 and curl_cffi 0.16.

Scrape Web Data With Python: Your First Scraper

Scraping web data with Python starts with reading the page's HTML. Our first target is the Countries page of Scrape This Site, whose robots.txt allows everything under /pages/. Right-click a country and choose Inspect: each one sits in a div.country block, with the name in h3.country-name and the details in span.country-capital and span.country-population. Those class names become the selectors in the code.

1import requests2from bs4 import BeautifulSoup3 4URL = "https://www.scrapethissite.com/pages/simple/"5HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; my-scraper/1.0)"}6 7response = requests.get(URL, headers=HEADERS, timeout=15)8response.raise_for_status()9 10soup = BeautifulSoup(response.text, "lxml")11countries = []12 13for card in soup.select("div.country"):14  name = card.select_one("h3.country-name")15  capital = card.select_one("span.country-capital")16  population = card.select_one("span.country-population")17  countries.append({18      "name": name.get_text(strip=True) if name else None,19      "capital": capital.get_text(strip=True) if capital else None,20      "population": int(population.get_text(strip=True)) if population else None,21  })22 23print(len(countries), countries[:2])

Three habits in this code carry into every scraper you write. A timeout stops an unresponsive server from hanging the script forever. raise_for_status() fails loudly on a 403 or 429 instead of quietly parsing an error page. Those if name else None checks keep one missing element from crashing a run of hundreds.

Our User-Agent names the scraper honestly. Python's default announces itself as python-requests, which many sites refuse on sight; the anti-ban section below covers what to send to sites that expect a real browser.

Follow Pagination and Save the Data

The Hockey Teams page on Scrape This Site spreads its table across 24 pages, linked with a page_num parameter. Instead of hardcoding that number, the scraper reads the last page from the pagination links, so it keeps working if the table grows.

1import csv2import time3 4import requests5from bs4 import BeautifulSoup6 7BASE = "https://www.scrapethissite.com/pages/forms/"8FIELDS = ["team", "year", "wins", "losses", "ot_losses", "win_pct", "gf", "ga", "diff"]9 10session = requests.Session()11session.headers["User-Agent"] = "Mozilla/5.0 (compatible; my-scraper/1.0)"12 13def get_soup(page):14  resp = session.get(BASE, params={"page_num": page}, timeout=15)15  resp.raise_for_status()16  return BeautifulSoup(resp.text, "lxml")17 18first = get_soup(1)19last_page = max(20  int(a.get_text(strip=True))21  for a in first.select("a[href*='page_num=']")22  if a.get_text(strip=True).isdigit()23)24 25rows = []26for page in range(1, last_page + 1):27  soup = first if page == 1 else get_soup(page)28  for tr in soup.select("table.table tr"):29      cells = [td.get_text(strip=True) for td in tr.find_all("td")]30      if cells:31          rows.append(dict(zip(FIELDS, cells)))32  time.sleep(1.5)  # one page every 1.5 seconds33 34with open("hockey_teams.csv", "w", newline="", encoding="utf-8") as f:35  writer = csv.DictWriter(f, fieldnames=FIELDS)36  writer.writeheader()37  writer.writerows(rows)38 39print(f"Saved {len(rows)} rows from {last_page} pages")

Using requests.Session reuses one connection and one set of headers across all 24 pages, which is faster and looks more like a single visitor. Header rows have no td cells, so the if cells check skips them. For JSON output, json.dump(rows, f, indent=2) drops into the same spot; for data you will query later, SQLite ships with Python.

Scrape JavaScript Pages

On the Oscar Winning Films page, nothing loads until you click a year. Fetch it with requests and the table is empty, because the rows are added by JavaScript after the page loads.

1import requests2 3URL = "https://www.scrapethissite.com/pages/ajax-javascript/"4 5resp = requests.get(URL, params={"ajax": "true", "year": 2015}, timeout=15)6resp.raise_for_status()7 8for film in resp.json():9  print(film["title"].strip(), film["nominations"], film["awards"], film.get("best_picture", False))

Some pages build their data in the browser, sign their API calls or hide them behind tokens. Those need a real browser, and Playwright is the most direct way to drive one from Python:

1# pip install playwright && playwright install chromium2from playwright.sync_api import sync_playwright3 4with sync_playwright() as p:5  browser = p.chromium.launch(headless=True)6  page = browser.new_page()7  page.goto("https://www.scrapethissite.com/pages/ajax-javascript/")8  page.locator("a.year-link", has_text="2015").click()9  page.wait_for_selector("tr.film")10 11  for row in page.locator("tr.film").all():12      title = row.locator(".film-title").inner_text().strip()13      awards = row.locator(".film-awards").inner_text()14      print(title, awards)15 16  browser.close()

The JSON marks only the winner with best_picture, so the loop reads it with .get(). In the browser version, wait_for_selector waits until the rows exist instead of sleeping for a guessed number of seconds. Large retail sites combine JavaScript rendering with aggressive bot checks; our Amazon scraping setup covers scraping Amazon specifically.

Web Scraping With Selenium and Python

Selenium does the same job through WebDriver and is still common in existing codebases. Current releases ship Selenium Manager, which downloads the matching browser driver on first run, so there is no chromedriver to install by hand.

1# pip install selenium2from selenium import webdriver3from selenium.webdriver.common.by import By4from selenium.webdriver.support import expected_conditions as EC5from selenium.webdriver.support.ui import WebDriverWait6 7options = webdriver.ChromeOptions()8options.add_argument("--headless=new")9driver = webdriver.Chrome(options=options)  # Selenium Manager fetches the driver10 11try:12  driver.get("https://www.scrapethissite.com/pages/ajax-javascript/")13  driver.find_element(By.ID, "2015").click()14  WebDriverWait(driver, 10).until(15      EC.presence_of_element_located((By.CSS_SELECTOR, "tr.film"))16  )17  for row in driver.find_elements(By.CSS_SELECTOR, "tr.film"):18      title = row.find_element(By.CLASS_NAME, "film-title").text.strip()19      awards = row.find_element(By.CLASS_NAME, "film-awards").text20      print(title, awards)21finally:22  driver.quit()

Proxies are where Selenium gets awkward. Chrome accepts a --proxy-server argument but ignores any username and password written into it, and selenium-wire, the old workaround, has not had a release since October 2022. IP allowlist authentication is the clean fix: approve your server's IP with the proxy provider and pass the proxy address without credentials. With VoidMob's shared pay-per-GB proxies, create the list in IP Whitelist mode (up to 5 IPv4 addresses or /24-/32 subnets, set when the list is created) and pass --proxy-server=http://proxy.voidmob.com:10000. Our Selenium proxy setup guide covers the other options.

How to Scrape Without Getting Banned

Websites detect scrapers on several layers at once, and the parser is never one of them. They watch the request rate from each IP, the IP's type and reputation, headers that do not match a real browser, the TLS fingerprint of the connection and, on JavaScript-heavy pages, browser automation signals.

TLS is where many Python scrapers get caught even when everything else is right. requests negotiates TLS through Python's own stack, which produces a handshake no browser sends, and fingerprinting methods such as JA3 and JA4 read it before a single header arrives (our JA3 vs JA4 breakdown explains how). curl_cffi fixes that layer: impersonate="chrome" sends Chrome's handshake and header order and follows the newest Chrome version the library supports.

This function puts the polite habits in one place: it checks robots.txt, retries only temporary errors, and waits as long as the server asks.

1# pip install curl_cffi2import time3from urllib.robotparser import RobotFileParser4 5import curl_cffi6 7session = curl_cffi.Session(impersonate="chrome")  # browser TLS and headers8 9# RobotFileParser.read() fetches with Python's own User-Agent, which many sites10# answer with a 403, and the parser then treats every URL as disallowed.11# Fetch robots.txt through the same session and hand the text to the parser.12robots = RobotFileParser()13robots.parse(session.get("https://www.scrapethissite.com/robots.txt", timeout=20).text.splitlines())14 15RETRY_STATUSES = {429, 500, 502, 503, 504}16 17def polite_get(session, url, max_tries=4):18  if not robots.can_fetch("*", url):19      raise PermissionError(f"robots.txt disallows {url}")20  for attempt in range(max_tries):21      resp = session.get(url, timeout=20)22      if resp.status_code not in RETRY_STATUSES or attempt == max_tries - 1:23          break24      retry_after = resp.headers.get("Retry-After", "")25      wait = int(retry_after) if retry_after.isdigit() else 5 * 2 ** attempt26      time.sleep(min(wait, 120))27  resp.raise_for_status()28  return resp29 30page = polite_get(session, "https://www.scrapethissite.com/pages/simple/")31print(page.status_code, len(page.text))

Status 429 means slow down, so the retry honors the server's Retry-After value and otherwise backs off 5, 10, then 20 seconds. Blocks (403) are not retried, because hammering one only deepens it, and any status that is still an error after the last try raises through raise_for_status(). For robots.txt, the check uses Python's standard-library parser for the format defined in RFC 9309. The parser is fed through the same session because its built-in read() uses Python's own User-Agent, and a 403 on robots.txt makes it treat every URL as disallowed.

Why Python Requests Gets a 403 Forbidden

A 403 from a scraper that works in the browser usually means the site judged the request, not the URL. The usual causes, in order of how often they come up: python-requests as the User-Agent, a TLS handshake no browser sends, and an IP address that is rate limited or flagged as hosting. Fix them in that order: a browser-matching client first, pacing second, a different IP type last.

Is Web Scraping Legal?

Collecting publicly available pages is generally not unauthorized access under the US Computer Fraud and Abuse Act, which is how the Ninth Circuit ruled in hiQ v. LinkedIn in 2022. That same case shows the limits: hiQ later lost on breach of LinkedIn's user agreement. Terms of service, copyright and privacy laws such as the GDPR still apply, most of all to personal data and anything behind a login. This is general information, not legal advice.

Use a Proxy for Web Scraping in Python

A proxy sits between your script and the target, so the site sees the proxy's IP instead of yours. Practice sites and small one-off jobs rarely need one. Once a target starts rate limiting or scoring your IP, the choice comes down to two decisions (our mobile proxy comparison for scraping covers providers), and the first is the IP type:

• Datacenter proxies: unprotected sites and APIs. Fast and inexpensive, but hosting ranges are the first ones a protected site filters.
• Residential proxies: broad crawls of lightly protected sites, at a lower cost per GB than mobile.
• Mobile proxies: protected targets and logged-in sessions. Carrier IPs are shared with real phone users behind CGNAT, so blocking one risks locking out ordinary subscribers.

Rotation is the second decision, and a good provider lets you choose it per job. Per-request rotation gives every call a new IP, which suits independent pages such as listings and search results. Sticky sessions hold one IP through a multi-step flow, and a logged-in session on an account you own is steadiest on a dedicated mobile device.

With VoidMob's mobile proxies, every list connects through one gateway, proxy.voidmob.com on port 10000, and you set the rotation per list: per request, on a timer, or sticky. Each list has its own login and all of them draw from the same GB, so one package can serve both a per-request list for independent pages and a sticky list for multi-step flows. Dropping them into the scraper takes one dictionary per list:

1import requests2 3# One list set to per-request rotation, one set to sticky4ROTATING = "http://<rotating-list-username>:<rotating-list-password>@proxy.voidmob.com:10000"5STICKY = "http://<sticky-list-username>:<sticky-list-password>@proxy.voidmob.com:10000"6 7# Independent pages: every call leaves through a fresh exit IP8rotating = {"http": ROTATING, "https": ROTATING}9for _ in range(3):10  r = requests.get("https://api.ipify.org?format=json", proxies=rotating, timeout=30)11  print(r.json()["ip"])12 13# Multi-step flow: the sticky list keeps the same IP until you rotate it14with requests.Session() as s:15  s.proxies.update({"http": STICKY, "https": STICKY})16  s.get("https://www.scrapethissite.com/pages/forms/", timeout=30)

Playwright takes the username and password directly in its proxy option, which makes it the easier browser for authenticated proxies (our Playwright proxy guide covers the setup in more depth):

1from playwright.sync_api import sync_playwright2 3with sync_playwright() as p:4  browser = p.chromium.launch(proxy={5      "server": "http://proxy.voidmob.com:10000",6      "username": "<list-username>",7      "password": "<list-password>",8  })9  page = browser.new_page()10  page.goto("https://api.ipify.org?format=json")11  print(page.inner_text("body"))12  browser.close()

SOCKS5, special characters in passwords and the silent ways a requests proxy can fail are covered in our Python requests proxy guide. Per-GB plans bill every byte that passes through the proxy, so estimate a job's usage before choosing a plan:

Before It Runs Unattended

Before a scraper runs unattended, it should pass every line of this list:

• Timeouts on every request, and raise_for_status() before parsing.
• Selectors that tolerate missing elements.
• JSON endpoints checked before any browser gets launched.
• Pacing and retries that respect Retry-After.
• robots.txt checked in code, not by memory.
• Browser-matching TLS fingerprints on sites that read them.
• An IP type and rotation mode that fit the target, tested on a small run first.

Scale comes after the list, not before it: async httpx or Scrapy for thousands of pages, once the single-threaded version runs clean.

FAQ