Web Scraping Best Practices: A Policy Checklist for Teams
Over the long run, reliable scraping and responsible scraping are the same thing. The teams whose data keeps arriving month after month are the ones that respect each site's rules, pace their requests and use suitable tools, not the ones pushing as hard as they can. This checklist is written so a team can adopt it as policy for collecting publicly available data, with proxies providing reliability and scale.
The whole policy rests on legitimate use: collect only publicly available data, stay within each website's terms of service, robots directives and applicable law, and remember that no proxy assures access. For choosing IPs, see how to choose proxies for web scraping.
We write these guides for e-commerce, marketing and agency teams that pay for proxies by the gigabyte.
Rule 1: Check the rules before the first request
Every project opens with a read of the target's terms of service and robots directives. Where the site offers an official API or data feed, use it: the data is cleaner and steadier than anything parsed from HTML. Limit collection to publicly available data, and when access is restricted, take that as the answer. Every other rule in this policy builds on this one.
Rule 2: Pace requests and spread them across IPs
Crawls that run too fast are the ones that get cut short. These habits keep the load on each target reasonable:
- Space requests out with randomised pauses, and set a ceiling on concurrency.
- Rotate through many IPs using rotating proxies, so every address carries a small share of the requests.
- Hold a sticky session through multi-step flows that a site expects one visitor to complete.
- Back off after an error rather than retrying at once.
Rule 3: Fit the proxy to each target, then cache
Choose the proxy per target: datacenter where defences are light, residential where a site is protected or the content depends on location. Then cache everything you can, so a page that has not changed is never downloaded twice. Caching lowers the load on the target and shrinks your bandwidth bill at the same time. The table sums up the core policy.
| Policy item | Why it matters |
|---|---|
| Read terms & robots first | Collection stays legitimate and in scope |
| Prefer official APIs | Cleaner, steadier data than scraped HTML |
| Throttle and randomise pacing | A reasonable load on every target |
| Rotate across many IPs | Few requests from any one address |
| Match proxy type to target | High success rate at a sensible cost |
| Cache unchanged pages | Lighter load, lower bandwidth bill |
| Timeouts, retries with backoff | Runs finish without retry floods |
Rule 4: Build a crawler that is dependable and considerate
A handful of engineering decisions make a crawler dependable and considerate together:
- Timeouts, plus retries with backoff, so temporary failures do not derail a run.
- Honest identification: name your crawler where appropriate, and publish a contact point if your collection runs at scale.
- Graceful error handling: log the problem, skip the page and carry on, rather than flooding the site with retries.
- Plain HTTP by default, with headless browsers kept for pages that genuinely need JavaScript, to save bandwidth.
What each error code means is covered in our proxy error code reference.
Equipping the team
Our proxies support HTTP/HTTPS and SOCKS5 with username/password or IP-whitelist authentication, cover 55+ countries and are delivered instantly. They run on ethically sourced pools with 99.9% uptime, and we keep no traffic logs. Once the policy is signed off, run the expected volume through the proxy calculator, check each type’s per-GB rate on the pricing page and order what the team needs.
Match the pool to the workflow
Every pool is billed per GB and run from the same dashboard, so one team can keep low-cost volume work and high-trust checks side by side.
High-volume collection from unprotected sites, automation, SEO tools
Market research, ad verification and geo-accurate checks on protected sites
Phone-first platforms, app and mobile ad testing
Hard targets where every failed request is costly
Covering several markets in one report? MIX packages combine multi-country residential traffic in a single order. Agencies buying on behalf of clients can apply for the reseller programme.
Get your team running
Per-GB pricing from $3.50 · 55+ countries · instant delivery · 30-day validity · no traffic logs · 24/7 human support
-
1
Size the job
Pick a proxy type in the proxy calculator and enter the gigabytes you expect to use to see the order total.
-
2
Order and pay
Choose the pool on the proxies page and pay by card through Stripe or in crypto through Cryptomus. Each order is billed per GB and does not renew on its own.
-
3
Share access
Credentials appear in the dashboard as soon as payment clears. Add them to each tool with a login or an IP whitelist, then send one test request before the real job starts.
Questions team leads ask
More playbooks for your team
What Are Proxies? A Primer for Business Teams
What a proxy server does for a business team: how the relay works, the main IP types and their per-GB prices,…
Read the playbook →Ad Verification Proxies: A Playbook for Ad-Ops Teams
How ad-ops teams check placements, creatives and landing pages from residential and mobile IPs per market, wh…
Read the playbook →Web Scraping Proxies: Picking a Pool for Each Data Job
The proxy pool a data team needs for each scraping target: datacenter for open sites, residential for protect…
Read the playbook →Datacenter vs Residential Proxies: Which Should Your Team Buy?
Decide workflow by workflow whether datacenter or residential proxies fit: cost, speed, trust and location ta…
Read the playbook →What Are Residential Proxies? When a Team Needs Home IPs
Residential proxies use real home IPs. Which team workflows need them, how a request travels, how they compar…
Read the playbook →Mobile vs Residential Proxies: Which Fits Your Team's Work?
Mobile proxies use carrier 4G/5G IPs; residential proxies use home broadband. Decide by platform which one yo…
Read the playbook →