Research, analytics and machine-learning pipelines depend on large volumes of public data. Proxies make collection predictable by removing per-IP bottlenecks and providing the regional views your dataset needs.
A collection pipeline is easier to run when each source is described once in configuration and everything else follows from it. A source profile holds the domain, the pool tier, the maximum concurrent requests, the minimum delay between requests, the retry policy and the parser version. The scheduler feeds per-domain queues, workers pull from them under those limits, and a router attaches the proxy that the profile names. When a source starts blocking, you change one profile instead of editing code, and you can see exactly which pool is responsible for the cost.
Tiering then becomes a table lookup. Open, tolerant sources go to datacenter IPs, where traffic is listed as unlimited and the monthly bill is flat. Sources that serve IPv6 and need many addresses can use the IPv6 packs, with one important caveat: all addresses come from a single /64 subnet, and some defenses apply limits to the whole prefix rather than to each address, so 2,500 addresses may behave like one client. Protected sources go to rotating residential IPs billed per GB. Sources that need a login use static IPs so the session keeps one address.
Source type
Pool
Billing basis
Price range (TRY per month)
Tolerant, bulk
Datacenter IPv4
Flat; traffic listed as unlimited
30 IPs: 250 · 1,000 IPs: 2,000
IPv6-capable, address-hungry
IPv6 (one /64)
Flat; HTTP and HTTPS listed
100 IPs: 100 · 15,000 IPs: 2,250
Protected, varies by country
Residential rotating
Per GB, 30-day usage period
5 GB: 350 · 300 GB: 9,000
Login required
ISP static
Per IP; monthly up to 12 months
3 IPs: 180 · 1,000 IPs: 30,000
02
Budget arithmetic: requests, bytes and politeness
Start from the volume you need and work backwards. Illustrative example: 500,000 pages per day is about 5.8 requests per second on average. At an assumed 1.2 seconds per response, that needs roughly seven requests in flight at once, which any small worker pool handles. The number of IPs is then set by politeness, not by concurrency. If a source tolerates one request every ten seconds per address, you need about 58 addresses for that one source, so the 100-IP datacenter pack at 400 TRY per month covers it with headroom.
Now price the same job by bytes. Assume a fetched HTML document averages 40 KB: 500,000 pages are 20 GB per day, or about 600 GB per month. On per-GB residential plans at 30 to 70 TRY per GB, that is a very different bill from a flat datacenter plan, and it is why residential traffic should be reserved for sources that need it. Reduce bytes where terms allow: request compressed responses with Accept-Encoding, send If-None-Match or If-Modified-Since so unchanged pages return a tiny 304, and skip images, fonts and scripts you do not parse.
Politeness also protects you. A pool of 58 addresses sending one request every ten seconds still delivers 5.8 requests per second to the source, which is real load on a small site. Cap the per-domain aggregate rate, not just the per-IP rate, and spread large jobs across the day. Back off exponentially on 429 and 5xx responses, honor Retry-After when it is present, and pause a source entirely when its error rate crosses a threshold you define. Retries multiply cost on GB plans, so retry limits are a budget control as well as a courtesy.
Measure bytes per page on your own targets before choosing a plan; the figures above are assumptions, not typical values.
03
Compliance and data governance inside the pipeline
Compliance is easiest when it is enforced by the pipeline rather than remembered by people. Fetch and cache each domain’s robots.txt, refresh it daily, and make the fetcher refuse disallowed paths. Keep a reviewed record of each source’s terms and a flag for whether an official API or bulk export exists; if it does, use it, because it is cheaper, faster and unambiguous. If a source offers a way to identify your crawler or a contact address for bulk access, use that channel instead of working around the site.
Decide what you will not store. Public does not mean free of obligations: names, email addresses, photos and similar details can be personal data under laws such as the GDPR or Turkey’s KVKK, and collecting them can require a legal basis even when they are visible. Prefer pipelines that extract only the fields you need, drop the rest at parse time and keep a retention limit. This is general information, not legal advice; for commercial datasets involving people, get qualified advice before collection begins.
Finally, make every record traceable. Store the source URL, fetch time in UTC, HTTP status, pool tier, exit country and parser version with each row. Provenance lets you delete everything from a source if it objects, reproduce a number a stakeholder questions and spot when a region-specific sample is skewing the dataset. It also gives you evidence of good behavior: if a site owner contacts you, you can show request rates and respond promptly, and you should stop the job first and discuss afterwards.
Playbook
A practical implementation plan
Four steps from a first test to stable, monitored operation.
01
Inventory and classify sources
List every source with its robots.txt rules, terms, official API or export, and whether it publishes AAAA records (check with the DNS lookup tool). Drop or redesign anything you are not allowed to collect.
02
Assign tiers and budgets
Give each source a pool, a per-domain request ceiling and a daily byte budget, and write them into configuration rather than code.
03
Pilot at one percent
Run one percent of the planned volume for several days. Measure bytes per page, status mix and duplicates, then recompute the plan size from measured data.
04
Scale in steps with automatic backoff
Increase volume in stages, with alerts on 403 and 429 rates and an automatic pause per source, and review cost per thousand records each week.
Worked scenarios
How this looks in practice
Illustrative examples — the numbers are assumptions to show the reasoning, not measured results.
01
A tolerant directory, 120,000 pages a day
That is about 1.4 requests per second. At one request every five seconds per IP, seven addresses would carry it, so the 30-IP datacenter pack at 250 TRY per month is already ample. The limit that matters is the load on the directory, so the team caps the aggregate rate and spreads fetches over the day (illustrative figures).
02
Three countries, weekly sweeps
A team samples 5,000 listing pages in each of three countries every week through country-targeted residential IPs, fetching HTML only at an assumed 150 KB per page. One sweep is 3 × 5,000 × 150 KB, about 2.25 GB, so four sweeps use about 9 GB and the 10 GB plan at 650 TRY fits, with little margin for retries.
03
An IPv6-capable source
A source publishes AAAA records, confirmed with the DNS lookup tool. The team tests the 500-IP IPv6 pack at 300 TRY per month, but because all addresses share one /64 it applies a single token bucket to the whole pool, and it compares success rates with the 100-IP datacenter pack at 400 TRY before committing.
What to measure
Track these from the first day so problems show up as numbers, not surprises.
Success rate by status class per source: the share of 2xx responses against 403, 429 and 5xx, tracked hourly.
Bytes per successful record, which converts directly into GB-plan cost and shows when parsers or retries are wasting traffic.
Cost per 1,000 records in TRY, by pool tier, so you can see when moving a source to another pool pays off.
Data freshness: the age of the oldest record in each source’s dataset compared with the refresh target.
Duplicate and parse-failure rates, since silent layout changes and repeated fetches both inflate cost without adding data.
Common mistakes
Patterns that waste budget or get access blocked.
Sizing the IP count from concurrency instead of the source’s tolerance, which either overloads the source or wastes spend.
Treating the thousands of addresses in one IPv6 /64 as independent identities, when a defense that limits by prefix sees a single client.
Running unlimited retries on GB plans, where a block can turn into a retry storm that consumes the monthly quota in hours.
Storing no provenance, so you cannot explain, correct or delete a record when a source or a regulator asks.
Collecting personal data “because it is public”, which can still require a legal basis under data-protection law.
Practical tips
Store source URL, time, IP type and country with every record.
Back off automatically on 429 and 403 responses.
Prefer official APIs and bulk exports.
Keep personal data out of your pipeline unless you have a legal basis.
Use proxies responsibly
Collect public data lawfully and respectfully: honour robots.txt and terms, follow data-protection law, and avoid overloading services.
IPv6 for sources that serve IPv6 and need huge address counts; IPv4 for universal reach.
Do rotating proxies make collection legal?
No. Legality depends on what you collect and how, not on the proxy.
How should I budget bandwidth?
Sample real pages, measure bytes, then multiply. Blocking images and scripts can reduce usage dramatically.
Do I have to follow robots.txt?
robots.txt is a convention rather than a law, but it expresses the site owner’s wishes, and ignoring it can breach terms of use or weaken your position in a dispute. We recommend honoring it, and using an official API or export where one exists. This is general guidance, not legal advice.
How do I know whether a source works over IPv6?
Look up its AAAA records with the DNS lookup tool: with no AAAA record the site is IPv4-only, and an IPv6-only exit usually cannot reach it. A record is necessary but not sufficient, so run a small test through the pack. The provider lists HTTP and HTTPS for IPv6; SOCKS5 is not confirmed.
How do I avoid wasting GB when a long crawl restarts?
Persist your queue and a record of fetched URLs with their ETag or content hash, so a restart resumes instead of repeating. Make fetches idempotent and de-duplicate URLs before queueing. Every repeated page costs real money on residential GB plans, which also have a 30-day usage period, so plan your fetches to fit it.
Enterprise
Need volume, custom locations or a procurement-friendly setup?
Talk to sales about large IP counts, long contracts and dedicated onboarding. Tell us the target countries and traffic profile and we will propose a fit.