When a price or rank monitoring job starts failing, the instinct is to buy more IPs. Usually the request pattern is the problem, not the proxy. A site sees bare, identical requests arriving at machine speed and reacts to them. Fix the pattern, meaning complete headers, a steady pace and the right IP type, and block rates usually fall.
Before you start: scope and terms
Agree as a team what the job collects and why. Stick to publicly available data such as prices, stock status and search positions. Read the site's terms and its robots.txt, never collect personal data, and use an official API or data feed where the site offers one, because it is usually more stable than collecting pages yourself.
1. Match the proxy type to the site
Sites without bot protection are fine on datacenter IPs. Sites behind Cloudflare or Akamai usually need residential. Mobile-first platforms call for mobile 4G/5G. Our comparison by team workflow goes through each case.
2. Send complete headers
Home-made monitoring scripts often send a single identifying header, such as User-Agent: python-requests/2.31.0, while a real browser sends ten or more. Copy the full set from a real browser request so the site receives what it normally expects:
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate, br
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Upgrade-Insecure-Requests: 13. Keep one user agent per IP
A user agent that changes on every request from the same IP is a warning sign for any site that checks. Give each IP one user agent and keep it for as long as you use that IP.
4. Pace the job
No site expects 100 requests a second from a single address. Add a random delay of 0.5-2 seconds between requests, honour Retry-After headers, and back off exponentially when you receive 429 or 503 responses. A slower job that finishes is worth more than a quick one that gets cut off halfway.
5. Use a real browser only where the page needs one
Some sites only return useful content once JavaScript has run. For those pages, a browser automation tool does the rendering:
- Playwright (Python and Node): a common choice for new projects.
- Puppeteer (Node): dependable for Chrome-based jobs.
- Selenium: older, with support for many languages.
Rendering a full page uses far more data than fetching its HTML, and data is what a per-GB plan charges for. Render only the pages that need it, and run those jobs on residential IPs.
6. Keep browser profiles separate and consistent
For signed-in work, such as a client dashboard, a seller account or a social profile, each account should run in its own browser profile with its own cookies and settings, so accounts never mix. A multi-account browser handles this per profile; see our guide to multi-account browsers.
7. Treat a CAPTCHA as feedback
Even a well-behaved job meets the occasional CAPTCHA on a strict site. Read it as a signal:
- Slow down and lower concurrency. Challenges are often triggered by timing.
- Check whether the IP appears on a public blocklist with our blacklist checker; an address with a history of abuse is worth replacing.
- If challenges keep coming, ask whether the site wants automated access at all, and look for an official API.
8. Track block rates as a team metric
Log every blocked request. Track the block rate per IP, per user agent and per target page, and review it alongside the rest of your reporting. A sudden rise means something changed: the site updated its defences, a batch of IPs picked up a poor reputation, or your user agent went out of date. Catching it early is cheaper than re-running a week of missing data.
9. Keep the job proportionate
Don't put heavy load on someone else's servers, and think before you publish anything you collect. Most legal trouble around data collection comes from going too hard at a site or from publishing data the team had no right to use.
