Scrapy proxies for business teams
Datacenter proxies for price-intelligence and catalogue teams running Scrapy: a low cost per request for large crawls of public pages on sites without heavy bot protection, billed per GB.
What teams use Scrapy proxies for
These pages look at software a team may already run and where a proxy fits around it. The proxy supplies only the connection; what the tool does still has to fit the target site's terms and the law that applies to you.
Scrapy in a data team: ownership, crawl reporting and proxy budgets
Scrapy is an open-source Python crawling framework, and many data teams run price, catalogue and listing collection on it. For a manager the questions are practical: who owns each spider, where the data lands, what each run costs in proxy traffic, and whether every source is public and allowed. Scrapy's built-in stats and feed exports answer most of them without extra tooling.
Ask for the stats after every run
Each Scrapy run ends with stats such as item_scraped_count, response counts per status code and downloader/response_bytes. Have engineers send them to a shared dashboard: falling item counts point to a broken spider, and rising 403 or 429 counts to a source asking for less traffic.
Budget proxy spend per source
Response bytes per run, multiplied by the schedule, give a monthly per-GB estimate for each source before anything is bought. Sources that look the same from everywhere go on datacenter; location-dependent ones, such as regional prices, justify country-targeted residential.
Data delivery through feed exports
Scrapy's FEEDS setting writes items as CSV, JSON, JSON Lines or XML to local files or storage such as S3, so analysts receive data without touching spiders. Agree a schema and a retention period for each feed, especially where pages contain names or reviews.
Compliance defaults to keep
New Scrapy projects obey robots.txt out of the box; make switching that off a reviewed decision, never a quick fix. Keep sources public and logins out of scope unless the account belongs to your company.
The plan we suggest for Scrapy
We suggest Datacenter here. It costs less per GB than our other plans and loads public pages quickly wherever a site accepts datacenter traffic. Price and catalogue teams need crawls that finish on schedule at a predictable cost. Scrapy sends many requests at once, and on lightly protected public pages datacenter proxies keep the price and latency of each request low, so a nightly run can finish before the morning report.
Built for concurrency
Scrapy's asynchronous engine keeps many requests in flight at once. Datacenter IPs keep pace at a lower cost per request than residential, which adds up when a crawl covers a large catalogue every night.
Drop-in middleware
Set the proxy on request.meta['proxy'] or in a small downloader middleware, using username/password or an IP-whitelisted endpoint. Your engineers keep their spiders as they are and change only the connection settings.
Rotate to dodge rate limits
Rotation spreads a crawl across many addresses so no single IP carries all of it. Pair it with AutoThrottle and the retry middleware so the spider slows down when a site does, and stay within each site's published limits.
Geo-targeted crawls
Prices, stock and promotions often vary by market. Pick IPs by country, or by city in major markets, and the spider records the localised page that customers there actually see.
Matching each job to a plan
Most teams run more than one kind of job. Data is bought per plan, so volume checks can sit on a lower-cost plan while the jobs that need a household or carrier connection get one.
Datacenter
Suggested hereOur entry-level rate per GB. Suits high-volume checks on your own sites and on public pages that accept datacenter traffic, such as rank tracking and catalogue sweeps.
Datacenter plan details →Residential
Household IPs whose owners consented to share their bandwidth. Use it when a report must show what local customers are shown, or when a site serves datacenter visitors something else.
Residential plan details →Residential Premium
Premium-grade residential IPs for demanding work. Shift any job that keeps failing on standard residential here, plus brand-protection sweeps and checks a client's reputation depends on.
Residential Premium plan details →Mobile 4G/5G
IPs from 4G/5G carrier networks, for apps, social platforms and mobile ad placements that need to be seen the way phone users see them.
Mobile 4G/5G plan details →Several markets, one order
MIX packages are ready-made residential pools covering several countries at once, so a multi-market report can run from a single order.
See MIX packages →Proxies for your own clients
The reseller programme is free to join. Once approved, you create a login for each client through the REST API and set your own client prices.
Reseller programme →Cost it before you commit
Pick a plan and a number of gigabytes in the proxy calculator to see the order total before anything is charged.
Open the calculator →Want the options side by side in more depth? The proxy comparison page sets them out.
Getting the team set up
Four steps take a team from order to first check, using the browsers and tools it already works with.
Size the order
Start on Datacenter, the plan suggested above, and choose how many gigabytes the work needs. The proxy calculator shows the total first.
Pay by card or crypto
Stripe handles card payments and Cryptomus handles crypto. Each order is a one-off, valid for 30 days, with no subscription to cancel.
Set the market
Pick the country on the order form, together with the protocol and sign-in method, so agree which market the work covers before checkout.
Give the team access
The dashboard shows the login details once the payment has gone through. Pass them to the colleagues who need them, or whitelist the office IP so nobody has to type a password.
Rotation or a sticky session?
- ▸Per-request rotation moves every request to a different IP. Use it for wide, read-only sweeps of public pages, such as price lists or search results.
- ▸Sticky sessions hold one IP for a window you choose, typically 1-30 minutes, for signed-in or multi-step tasks such as stepping through a client's checkout.
Both are switched in the dashboard, and the options on offer depend on the plan. Keep request rates modest either way.
Signing in
- ▸Username and password in the
host:port:user:passformat that browsers, browser-profile tools and monitoring software accept. - ▸IP whitelist: add your office or server address in the dashboard, and anything connecting from it gets in without credentials, which suits shared office lines and scheduled jobs.
Both work over HTTP, HTTPS and SOCKS5, and the protocol can be changed later in the dashboard at no extra charge.
Match the IP to the market you report on
Search rankings, prices, ads and even which products appear can change with the visitor's country. Set the IP to the market your client or campaign targets, and the screenshots and figures your team files will line up with what customers there are shown.
- ✓55+ countries to choose from.
- ✓City-level targeting where the network supports it.
- ✓One country per check keeps month-on-month results comparable.
Major markets on the network
When a report doesn't match what the client sees on the ground, the IP's country is one of the first things to check.
How the four plans compare
Every plan bills per GB. The highlighted row is the plan we suggest for this work.
| Plan | Trust level | Speed | Cost | From, per GB | Where it fits |
|---|---|---|---|---|---|
| Datacenter Suggested | Standard | ★★★★★ | $ | $3.50 | Bulk page checks, rank tracking and QA on your own sites |
| Residential | High | ★★★ | $$ | $3.99 | Local prices, search results, ads and marketplaces |
| Residential Premium | Very high | ★★★★ | $$$ | $6.50 | Demanding sites, brand protection and client work |
| Mobile 4G/5G | Very high | ★★★ | $$$$ | $4.50 | Apps built for phones and ads served on mobile networks |
The rate charged is the one shown for the live plan at checkout.
Keeping Scrapy work within the rules
We sell these proxies for legitimate business work: market and price research, ad verification, SEO, brand protection, QA and compliant automation, including running your own accounts inside each platform's rules. Whatever your team does through them has to stay within the terms of service of every site involved and the law that applies to you and your clients.
- ✓Collect only publicly available information, and pace requests so the site isn't put under strain.
- ✓No proxy can promise that an account escapes action by the platform. That depends mostly on how the account is used and whether it follows the terms.
- ✓Personal details in collected pages, such as names, contact details or reviews, bring data-protection duties with them. Agree with your legal team what you keep and for how long.
The full rules are in our acceptable use policy.
Scrapy proxies: questions from team leads
Refunds
Proxies that fail on arrival can be reported to support for up to 24 hours after delivery. Support swaps them, and where a working swap isn't possible, refunds the affected amount. Details are in the refund policy.
Support
People answer tickets 24/7. Open one from the Support page in your dashboard and include the order reference.
Logs
Our records cover billing and how much bandwidth each order used. The URLs, page content and destinations your team reaches are never logged.
Get your team started on Scrapy
Start with a small order, see how the work runs, and add data when the next round of checks is due.
