Solutions

Proxies for reliable web scraping

Scrapers fail for predictable reasons: too many requests from one address, addresses that look automated, and sessions that break halfway. The right proxy type fixes the first two; good scraper design fixes the third.

Web Scraping
Residential ProxyFrom TRY 350 / month

The challenge

What makes this job hard

  • Per-IP rate limits

    Sites cap how many requests one address may send. Concentrate traffic on a single IP and you are throttled or blocked within minutes.

  • IP reputation scoring

    Defensive sites classify addresses as datacenter, residential or mobile and treat them differently from the first request.

  • Geo-dependent content

    Prices, availability and results change by visitor country, so a single-location scraper sees only one version of the web.

The approach

How the right proxies help

  1. Match IP type to target

    Use datacenter IPs for tolerant sites and APIs, residential rotating IPs for protected consumer sites, and ISP IPs where a stable identity matters.

  2. Rotate or stick on purpose

    Rotate per request for stateless fetching; use sticky sessions when a login, cart or paginated flow must keep one address.

  3. Target the right country

    Residential plans list country-level targeting so you collect the version of the page your audience sees.

In depth

The details that decide the outcome

Sizing a scraper: the IP budget and the bandwidth budget

Every scraping job has two independent limits. The IP budget is how many requests per minute one address may send before the target throttles it; you must measure that number, but assume 4 for an example site. The bandwidth budget is how many bytes you move, which matters only on metered plans. Take an illustrative workload of 24,000 pages a day, about 17 requests per minute. At 4 requests per minute per IP that needs roughly five addresses, so even the smallest IPv4 Datacenter Proxy pack, 30 IPs at 250 TRY a month, runs each address at about 0.6 requests per minute.

Residential billing flips the arithmetic, because you pay for gigabytes rather than addresses. The same 24,000 pages at an illustrative 150 KB each is 3.6 GB a day, or 108 GB over a 30-day period. That overshoots the 100 GB plan (3,500 TRY) by 8 percent and would push you to the 200 GB plan (7,000 TRY), so trimming images, fonts and trackers is worth more than it looks. Fitted into 100 GB, the cost is about 0.005 TRY per page, against roughly 0.0003 TRY on the 30-IP datacenter pack.

Very large, steady jobs flip the logic again. The unlimited rotating residential plans trade a fixed price for unlimited traffic: the monthly plan costs 75,000 TRY, which equals 2,500 GB at the 300 GB plan’s rate of 30 TRY per GB, or about 83 GB a day. Below that volume the GB plans are cheaper; above it, unlimited wins on paper. Terms for unlimited use, such as fair-use limits or concurrency caps, are not published on this site, so ask sales on WhatsApp before building a pipeline around that assumption.

SituationBinding limitPlan family to price first
Tolerant site, very high page countRequests per minute per IPIPv4 Datacenter Proxy packs (30 to 1,000 IPs)
Protected consumer site, moderate volumeGigabytes and block rateResidential Proxy plans (5 to 300 GB)
Protected site, steady very high volumeCost per GB at scaleUnlimited rotating residential (daily, weekly, monthly)
Mobile-network content or carrier-trusted endpointsAddress trust and quotaRotating mobile packs (5, 10, 100 GB) or unlimited monthly
Your own login behind a stable identitySession continuityISP Proxy (static Turkish IPs)

Session, retry and backoff design

Decide rotation by the unit of state, not by habit. A stateless fetch, such as a product page or a search result, can leave from a different IP every time. A flow with state, such as a login, a pagination token tied to a session or a cart, must stay on one address, so request a sticky session on a Residential Proxy plan or use a static ISP Proxy IP. Keep one cookie jar per exit IP, because replaying one address’s cookies from another is a common reason sessions are invalidated. With datacenter packs you do the rotation yourself, such as round-robin with a per-IP cooldown.

Treat failures by class. Connection errors, timeouts and a 407 response (proxy authentication required) point to the proxy path or your credentials, not to a block. A 403, a 429 or a 200 page carrying a challenge points to the target. For 429, honor a Retry-After header when one is present; otherwise back off exponentially with random jitter, for example around 2, 4, 8 and 16 seconds, and retry from a different IP instead of hammering the same one. Cap retries at three or four per URL, so a persistent block cannot burn through a GB plan.

A proxy changes only the network path. Your HTTP client’s TLS handshake, header order and protocol version still reach the target unchanged through a CONNECT tunnel, so a client that looks like a script still looks like a script on a residential IP. Keep headers coherent, with a real browser User-Agent and a matching Accept-Language, reuse connections with keep-alive to avoid a fresh handshake per request, and use a headless browser only where pages truly need JavaScript, because a full render usually costs several times the bytes of the raw HTML.

A 200 status code is not proof of success. Count a response only when it contains the fields you expected.

Monitoring the fleet and proving it works

Verify each endpoint before the first production run. The free proxy checker takes up to 10 proxies per request, detects HTTP or SOCKS5 automatically and reports the exit IP, latency and an error code, so you can screen a batch in one pass. The codes separate causes cleanly: auth means wrong credentials, refused or timeout means the endpoint or port is unreachable, and connect_denied or tls points at the tunnel itself. Then open the IP address checker in a browser that uses the proxy to confirm the address the target will see, and repeat it on a rotating endpoint to confirm the address changes.

In production, log one row per request: target domain, IP type, exit IP or a hash of it, status code, bytes, latency, retry count and a content-valid flag. From that table you can compute block rate per domain and per IP type and pull failing addresses from your list, for example those under 80 percent valid responses across at least 100 requests, a threshold you should tune to your target. Alert on trends rather than single failures: a block rate that doubles within an hour usually means a rule change on the target, not a network fault.

Rotation spreads load; it does not grant permission. If a site asks you to stop or offers an official feed, use it.

Playbook

A practical implementation plan

Four steps from a first test to stable, monitored operation.

  1. Look for a sanctioned source first

    Check for an official API, feed or bulk export, and read robots.txt and the terms before building anything. The fields you actually need often decide the whole design.

  2. Run a pilot per IP type

    Fetch 500 to 1,000 representative URLs on the cheapest candidate type and record valid-content rate, bytes per page and latency. Move up to the next type only if the results miss your threshold.

  3. Set limits per domain

    Define maximum concurrency, requests per minute and a cooldown per IP, with jittered delays and exponential backoff on 429 and 5xx responses.

  4. Scale in steps and review weekly

    Double volume only while the block rate stays under your threshold, and compare cost per thousand valid pages each week so a plan change is a data decision.

Worked scenarios

How this looks in practice

Illustrative examples — the numbers are assumptions to show the reasoning, not measured results.

Public directory crawl

A research team crawls 300,000 public directory pages over a week, about 1,800 an hour. They spread the work over the 100 IPs of an IPv4 Datacenter Proxy pack (400 TRY a month), roughly 18 requests per hour per IP, rotating round-robin with a cooldown. The rate is illustrative: the site’s real tolerance has to be measured in a pilot.

Protected retail category pages

A pilot against a retailer that rejects datacenter ranges returns valid pages for 40 percent of requests from datacenter IPs and 95 percent from residential IPs (illustrative figures). At about 200 KB per page, a 50 GB Residential Proxy plan (2,500 TRY) covers roughly 250,000 pages, or 0.01 TRY per page. The team uses residential only for this retailer and keeps datacenter IPs for the rest.

Authenticated export from a partner portal

A finance team downloads daily reports from a partner portal where it holds credentials and whose terms allow automated export. Because the portal ties sessions to an address, the job is pinned to one static IP from the 3-IP ISP Proxy plan (180 TRY a month) with a single cookie jar. Any other address would risk a re-authentication prompt on every run.

What to measure

Track these from the first day so problems show up as numbers, not surprises.

  • Valid-content rate per domain and IP type, counted by the presence of expected fields rather than by HTTP 200.
  • Block rate, meaning the share of requests ending in 403, 429 or a challenge page, tracked per IP type and per hour.
  • Cost per 1,000 valid pages, combining plan price, retries and wasted bytes.
  • Average and 95th-percentile bytes per page, which drives GB-plan sizing.
  • Retry ratio and 95th-percentile latency, because rising values signal throttling before hard blocks appear.

Common mistakes

Patterns that waste budget or get access blocked.

  • Counting HTTP 200 as success, so soft blocks and empty templates pollute the dataset unnoticed.
  • Rotating IPs in the middle of a flow that needs state, which invalidates tokens and forces re-authentication.
  • Retrying instantly on the same IP after a 429, which turns a rate limit into a block.
  • Downloading images, fonts and trackers on a GB-metered plan, multiplying cost with no data value.
  • Buying by pool size instead of measured per-IP tolerance, so capacity is either wasted or too thin.

Practical tips

  • Respect robots.txt and published terms; many sites offer an API.
  • Throttle per domain and add jitter instead of bursting.
  • Block images and media to cut bandwidth on GB-metered plans.
  • Log HTTP status codes per IP so you can spot blocks early.

Use proxies responsibly

Collect only data you are allowed to access, respect rate limits and privacy law, and never use proxies to attack, overload or defraud a service. Misuse can end your access.

Which proxy is best for scraping?

It depends on the target. Datacenter for tolerant sites, rotating residential for protected ones, ISP for stable sessions. The comparison table on the home page summarises the trade-offs.

How many proxies do I need?

Divide your required requests per minute by the safe rate per IP for the target. Start small, measure the block rate, then scale.

Is scraping legal?

It depends on the data, the site’s terms and your jurisdiction. Seek legal advice for commercial projects; proxies do not change legality.

Do I have to build rotation myself with datacenter packs?

Yes. The IPv4 Datacenter Proxy packs give you a list of static addresses with username and password authentication, and there is no automatic rotation. Rotate in your own code, for example round-robin with a per-IP cooldown. Rotating residential endpoints change the exit address for you.

Should I use HTTP or SOCKS5 for a scraper?

Most scraping libraries speak HTTP proxies natively, and HTTPS pages pass through as a CONNECT tunnel. SOCKS5 helps for non-HTTP traffic or tools that only support it. For datacenter and residential plans the protocol support comes from the provider’s listings and there is no selector at checkout, so confirm your protocol with sales on WhatsApp before building around it.

How do I scrape JavaScript-heavy pages without exhausting a GB plan?

First look for structured data embedded in the page, or for the JSON endpoint the page itself calls publicly, which is far lighter than a full render. If you need a headless browser, block images, fonts and media and cache static assets between runs. Measure bytes per page on a small sample before choosing a plan size.

Enterprise

Need volume, custom locations or a procurement-friendly setup?

Talk to sales about large IP counts, long contracts and dedicated onboarding. Tell us the target countries and traffic profile and we will propose a fit.