Web Scraping with Proxies: A Responsible Guide
Proxies solve per-IP limits, not bad scraper design. Here is how to combine both responsibly.
Scraping at any scale hits the same problems: sites limit how many requests one address can make, and they treat some address types with more suspicion than others. Proxies address both — but they are only part of a good scraper.
Start with etiquette
- Check for an official API or data export first.
- Read robots.txt and the site’s terms.
- Identify your scraper honestly where required and keep request rates modest.
- Avoid collecting personal data without a legal basis.
Pick the right IP type
Datacenter IPs are cheapest for tolerant sites. Residential rotating IPs help on sites that filter hosting ranges. Static ISP IPs suit authenticated flows. See the residential vs datacenter guide for details.
Design for failure
Expect 429 (too many requests), 403 and timeouts. Retry with exponential backoff, switch to another IP on repeated blocks and log the status code per IP so patterns are visible.
import random, time, requests
PROXIES = ["http://USER:PASS@HOST1:PORT", "http://USER:PASS@HOST2:PORT"]
def fetch(url, tries=4):
for i in range(tries):
proxy = random.choice(PROXIES)
try:
r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)
if r.status_code == 200:
return r.text
if r.status_code in (403, 429):
time.sleep(2 ** i + random.random())
except requests.RequestException:
time.sleep(1 + random.random())
return NoneControl bandwidth
On GB-metered plans, block images, fonts and media and fetch JSON endpoints where possible. Measure a sample to forecast your monthly use.
Read the rules before you write the crawler
Start with robots.txt, but know what it is. The Robots Exclusion Protocol (RFC 9309) is a file served from the root of each origin that tells crawlers which paths they should not request, grouped by user-agent, with the most specific matching rule winning. If the file returns a 4xx status, crawlers may treat the site as having no restrictions; if it returns 5xx or cannot be reached, the standard says to assume everything is off limits until it works again. Crawl-delay is not part of the standard and not every crawler honors it, but if a site publishes one, treat it as a request to slow down.
robots.txt is a convention for crawlers, not a license. Terms of service can restrict automated access whatever the file says, and other rules may govern what you do with the data: copyright in the content, database rights in some jurisdictions, and data-protection law such as the GDPR in the EU or KVKK in Turkey if the data identifies people. This is general information, not legal advice; if the project is commercial, have someone qualified review it. Never bypass authentication, paywalls or access controls, and treat a CAPTCHA or an access-denied page as a signal to slow down or stop, not as an obstacle for tooling to defeat.
Proxy toolsHTTP Headers CheckerRequest a public URL and see its response headers, the redirect chain and whether common security headers are present.ExploreBuild the rate limiter before the rotation
A larger pool of addresses does not entitle you to a larger load. The server still carries every request, and limits may apply per account, per session cookie, per network block or per fingerprint, not only per IP. Set the budget per target host, not per proxy: a cap on in-flight requests (two to five is a sensible start, as an illustration), a minimum delay with random jitter between requests, and an adaptive rule that halves the rate on a 429 or 503 and raises it again in small steps after a run of successes, much like congestion control in TCP.
Politeness has a time cost that you should plan for and that no number of IPs removes. Fetching 100,000 pages at 2 requests per second takes 50,000 seconds, about 13.9 hours; at 5 requests per second it takes 20,000 seconds, about 5.6 hours. If that is too slow, the answer is a narrower scope, incremental updates or an agreement with the site, not more addresses.
Retries that help instead of hurt
Backoff with jitter keeps many workers from retrying in lockstep. “Full jitter” sleeps for a random time between zero and min(cap, base × 2^attempt), which spreads retries across the whole window. If a 429 or 503 response carries a Retry-After header, which holds either a number of seconds or an HTTP date, wait at least that long. Use separate connect and read timeouts, cap both attempts and total time per URL, and decide what each status means before you write the loop:
- Retry with backoff: timeouts, connection resets, 429, 502, 503 and 504. These are usually transient or a request to slow down.
- Retry once on a new connection or session: failures that look specific to one exit node, such as repeated proxy-side connection errors.
- Do not retry: 404 and 410 mean the page is gone, so record them. 401 and 403 are access decisions, not glitches, and repeating them from many addresses is a fast way to get a whole range blocked.
- Treat 407 as your own configuration problem: it is the proxy asking for credentials, not the website refusing you.
The function below combines these pieces: a robots.txt check that follows the status rules above, full-jitter backoff, Retry-After handling and a status policy. It is a template to adapt, not a finished crawler.
import random
import time
import urllib.robotparser
import requests
UA = "ExampleBot/1.0 (+https://example.com/bot-info)"
PROXY = "http://USER:PASS@HOST:PORT"
PROXIES = {"http": PROXY, "https": PROXY}
HEADERS = {"User-Agent": UA}
robots = urllib.robotparser.RobotFileParser()
resp = requests.get("https://example.org/robots.txt", headers=HEADERS, proxies=PROXIES, timeout=15)
if resp.status_code == 200:
robots.parse(resp.text.splitlines())
elif resp.status_code >= 500:
robots.disallow_all = True
else:
robots.allow_all = True
def backoff(attempt, base=1.0, cap=60.0):
return random.uniform(0, min(cap, base * 2 ** attempt))
def polite_get(url, tries=5):
if not robots.can_fetch(UA, url):
return None
for attempt in range(tries):
try:
r = requests.get(url, headers=HEADERS, proxies=PROXIES, timeout=(5, 20))
except requests.RequestException:
time.sleep(backoff(attempt))
continue
if r.status_code == 200:
return r.text
if r.status_code in (429, 503):
wait = r.headers.get("Retry-After", "")
time.sleep(min(float(wait), 300) if wait.isdigit() else backoff(attempt))
elif r.status_code in (500, 502, 504):
time.sleep(backoff(attempt))
else:
return None
return NoneTreat every proxy as a component with a health score
In a long run, individual exits go bad: a node gets rate-limited, a gateway restarts, a credential expires. Keep counters per proxy or per session (successes, 429s, 403s, timeouts, bytes) and separate proxy faults from target faults. Connection refused, proxy timeouts, 407 and gateway-generated 502s point at the proxy; 403 and 429 from the site point at the target and at your behavior. The proxy checker’s error codes (auth, refused, timeout, not_http_proxy, tls and others) use the same split, so you can verify a suspect proxy independently of your crawler.
Add a circuit breaker. After a set number of consecutive failures, for example five (illustrative), take that proxy out of rotation for a cooling-off period such as ten minutes, then probe it with a single request before restoring it. Discarding a proxy permanently on its first error wastes paid capacity, while hammering a failing one burns bandwidth on metered plans.
Make the crawl resumable and cheaper
- Persist a queue with each URL’s status, attempt count and last error, so a crash or a deployment resumes instead of refetching.
- Normalize and deduplicate URLs (tracking parameters, query-string order, trailing slashes) before they enter the queue.
- Use conditional requests: store the ETag or Last-Modified value and send If-None-Match or If-Modified-Since next time. A 304 response has no body, which saves data on GB-metered plans and load on the site.
- Accept compression; most HTTP clients handle gzip transparently, and it typically shrinks HTML and JSON considerably.
- Store raw responses before parsing, so a parser bug means reprocessing files rather than spending requests again.
- Validate content, not just status: a 200 that contains a block page or a login form is a failure, so require an expected element before you count a page as a success.
Keep each session internally consistent
When you use sticky sessions, give every session its own cookie jar and keep its headers stable for its lifetime: the same User-Agent, the same Accept-Language and so on. A cookie issued to one address and presented from another, or a header set that changes between consecutive requests from the same address, is a reason for a site to challenge you. The user-agent checker on this site shows what a browser exposes, including client hints, language and time zone, which helps you see what your own client’s headers say about it. Consistency means avoiding accidental contradictions in your own client; it does not mean imitating other people’s software where a site’s rules forbid it.
What to monitor in production
- Success rate per target and per proxy or session, with content validation applied.
- Latency percentiles (median and 95th), because averages hide the slow tail that stalls workers.
- Bytes per successful record and cost per 1,000 records, which turns proxy billing into a unit cost you can compare across IP types.
- The share of 429 and 403 responses over time; a rising trend means your rate or pattern needs attention before the blocks arrive.
- Data freshness: when each page was last fetched successfully compared with how often it needs refreshing.
Frequently asked questions
Why do I still get 403 responses on a fresh IP?
The address is only one signal. Sites also consider headers, the characteristics of your HTTP client, cookies, request rate and pattern, and whether the content requires an account. A 403 that repeats across many different IPs usually means the cause is not the address, so rotating further will not help. Slow down, re-read the site’s terms and check whether an official API exists.
What request rate is safe?
There is no universal number. Start low, for example one request per second per host as an illustration, watch the share of 429 responses and the latency trend, and raise the rate in small steps. If the site publishes a rate limit, an API quota or a Crawl-delay in robots.txt, that sets your ceiling.
Can I scrape data that sits behind a login?
Only with explicit permission or for data you are entitled to access, and within the site’s terms. Many terms prohibit automated access to accounts, and logged-in pages often contain personal data that data-protection law covers. Prefer an official API or export; if you must automate your own account, use a static IP so the account is not seen from many places at once.