Python Requests through a proxy: a guide for data analysts

Analysts and marketing-operations teams often start their data work with a short Python script: fetch some public pages or an API, tidy the results, and load them into a spreadsheet or dashboard. The requests library is the usual tool, and pointing it at a proxy takes one dictionary. The proxy decides which market each request comes from, which matters whenever prices, availability or search results change by country.

Moving from a one-off test to a job that runs every morning takes a little more: credentials that parse correctly, sessions, rotation, retries and, sometimes, SOCKS5. Each is covered below with working code.

What analysts typically automate

• Public price and availability pages for products your business tracks.
• Status and content checks on your own websites in each market.
• Search visibility for client keywords, covered in more depth on our proxy for SEO page.

Install and gather credentials

pip install requests

Each order has four credentials, shown in the examples as HOST, PORT, USER and PASS.

The proxies dictionary

requests takes proxies as a dictionary keyed by scheme. The key says which traffic the entry applies to, and the value is the proxy URL with its login. Normally both keys point at the same proxy:

import requests

proxies = {
    "http": "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

r = requests.get("https://api.ipify.org?format=json", proxies=proxies, timeout=20)
print(r.json())

One detail regularly confuses people. The https key has a value that starts with http://, not https://, and that is correct. The value describes how requests talks to the proxy; for encrypted sites the proxy opens a tunnel with the CONNECT method. When the printed address is the proxy's, the connection works.

Passwords with special characters

A username or password containing @, : or / breaks the URL if pasted in as it is. Encode it first:

from urllib.parse import quote

user = quote("USER")
pwd = quote("P@SS/word")
proxy = f"http://{user}:{pwd}@HOST:PORT"
proxies = {"http": proxy, "https": proxy}

Sessions for multi-page jobs

Opening a new connection for every request wastes time. A Session keeps connections open and lets you set the proxy, headers and timeouts once for the whole job:

import requests

session = requests.Session()
session.proxies.update({
    "http": "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
})
session.headers.update({"User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)"})

for path in ["/", "/about", "/contact"]:
    resp = session.get(f"https://example.com{path}", timeout=20)
    print(path, resp.status_code)

Notice that the example's User-Agent names the script openly. That is good practice for a business: site owners can see what is visiting, and many teams add a contact address as well. Sessions also make multi-page jobs noticeably quicker.

Spreading load across several endpoints

If your order provides a list of sticky endpoints rather than one rotating gateway, cycle through them so no single address carries every request:

import requests, itertools

endpoints = [
    "http://USER:PASS@HOST:PORT1",
    "http://USER:PASS@HOST:PORT2",
    "http://USER:PASS@HOST:PORT3",
]
pool = itertools.cycle(endpoints)

def fetch(url):
    proxy = next(pool)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)

print(fetch("https://api.ipify.org?format=json").json())

With a rotating gateway this is unnecessary: keep one proxies dictionary, and successive requests exit from different addresses.

Retries that respect the server

Through any proxy, a request occasionally meets a timeout or an address that has dropped. A retry adapter handles that without hand-written loops:

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

session = requests.Session()
retry = Retry(total=4, backoff_factor=1.5,
              status_forcelist=[429, 500, 502, 503, 504])
session.mount("https://", HTTPAdapter(max_retries=retry))
session.mount("http://", HTTPAdapter(max_retries=retry))
session.proxies.update({"http": "http://USER:PASS@HOST:PORT",
                        "https": "http://USER:PASS@HOST:PORT"})

The backoff_factor setting lengthens the wait between attempts, so a site that has just answered 429 (too many requests) gets breathing space instead of an instant repeat.

SOCKS5 when you need it

For SOCKS5, install the extra dependency and change the scheme:

pip install "requests[socks]"

proxies = {
    "http": "socks5h://USER:PASS@HOST:PORT",
    "https": "socks5h://USER:PASS@HOST:PORT",
}

The h at the end of socks5h means the proxy resolves hostnames, so lookups do not leak through your local network. For data collection, that is nearly always the setting you want.

Which proxy type for each data source

Our proxy for scraping page goes further on larger collection jobs, and rates for each type are on the pricing page. Because Requests fetches only the document you ask for, without images or scripts, it uses far less data per page than a browser, which keeps per-GB costs down.

Before the job goes on a schedule

• Confirm the exit IP with an endpoint such as the ipify call above, or enter the credentials in our proxy checker.
• Read each site's robots.txt and its Terms of Service, and stay within both.
• Pause between requests with time.sleep() and keep the 429-aware retry policy in place.
• Collect only public data your business is permitted to use.
• Do not use proxies to get past logins or anti-fraud systems, or for any kind of fraud.

A dictionary gets the first request through. Sessions, rotation, retries and the right proxy type are what make the job dependable enough to feed a report every day.