Scraping Google results by market: a playbook for SEO teams

SEO agencies and in-house search teams spend a lot of time answering one question for clients: where do we rank, and in which market? Results change with country, city and language, so a check run from the office only shows the office's view. A small Python script, sent through proxies located in the client's market, gives a local snapshot of titles, snippets and positions. This playbook builds that script with requests and beautifulsoup4, keeps volumes low, and puts compliance at the centre.

Start with official data where it exists

For sites you or your clients own, Google Search Console reports clicks, impressions and average position by country, straight from Google. Licensed SERP data providers are another route for larger programmes. A script like the one below suits small, targeted spot checks: confirming a local result, capturing a snippet for a report, or looking at a market that your usual tool does not cover.

Ground rules before any code

Collect only results that are publicly visible without signing in. Read and respect the robots.txt file and the Terms of Service of any site you query, and do not try to defeat authentication or CAPTCHAs to reach anything you are not entitled to see. Keep the pace slow enough that the service is never degraded for others. Rank tracking for SEO research is a common practice, but staying within permitted use is your responsibility, and where an official API covers your need, use it. The wider principles are set out on our proxies for scraping page and the SEO and rank-tracking page.

Why location-matched proxies matter

Search results depend on where a query comes from, so the location of the proxy decides whose rankings you see: an address in the client's market returns that market's results. Spreading a small daily list of queries over a few addresses also keeps the traffic from any single connection light. Residential proxies suit this work because they connect through home broadband in the chosen country or city. Residential premium proxies are our premium tier of household IPs and are worth comparing when a client report depends on consistent results.

Setting up the script

Install the two libraries:

pip install requests beautifulsoup4

Every order comes as a HOST:PORT address with a login, which requests accepts as a dictionary:

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

Requesting a results page for one market

The function below requests a single results page through the proxy, with an ordinary browser User-Agent, a language header and a timeout. The hl parameter sets the interface language, which should match the market you are checking:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urlencode

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch_results(query, lang="en"):
    params = {"q": query, "hl": lang}
    url = "https://www.google.com/search?" + urlencode(params)
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=20)
    resp.raise_for_status()
    return resp.text

Pulling titles and links from the page

Google reshapes its results markup often, so the parser keys on structure rather than on specific class names: for each result block it takes the first link with an href and the heading text. Expect to adjust the selectors whenever the layout moves:

def parse_results(html):
    soup = BeautifulSoup(html, "html.parser")
    results = []
    for block in soup.select("div.g"):
        link = block.find("a", href=True)
        title = block.find("h3")
        if not link or not title:
            continue
        results.append({
            "title": title.get_text(strip=True),
            "url": link["href"],
        })
    return results

html = fetch_results("best running shoes")
for i, r in enumerate(parse_results(html), start=1):
    print(f"{i}. {r['title']} -> {r['url']}")

No results back? Either the layout has moved on or Google returned an interstitial page. Printing html[:2000] shows which of the two before you touch the selectors.

Pacing and spreading the queries

For more than a few keywords, pick a different endpoint for each query and wait several seconds, at random, between requests:

import random
import time

PROXY_POOL = [
    "http://USER:PASS@HOST1:PORT",
    "http://USER:PASS@HOST2:PORT",
    "http://USER:PASS@HOST3:PORT",
]

def pick_proxy():
    p = random.choice(PROXY_POOL)
    return {"http": p, "https": p}

def polite_fetch(query):
    proxies = pick_proxy()
    url = "https://www.google.com/search?" + urlencode({"q": query, "hl": "en"})
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=20)
    resp.raise_for_status()
    time.sleep(random.uniform(3, 8))  # pace yourself
    return resp.text

If your order is set to rotate the IP on every request, which you choose in the dashboard, one endpoint is enough and the gateway varies the exit for you. Whatever the setup, leave the random pause in place. It is the courtesy that matters most in the script, and it keeps your checks within a reasonable rate.

Failures: back off, do not push harder

Requests will sometimes fail. Retry with growing waits, and stop when non-200 responses keep coming:

def fetch_with_retry(query, attempts=3):
    for attempt in range(attempts):
        try:
            return polite_fetch(query)
        except requests.RequestException as e:
            wait = 2 ** attempt
            print(f"Attempt {attempt + 1} failed: {e}; waiting {wait}s")
            time.sleep(wait)
    return None

Repeated failures are a signal to slow down sharply or stop for the day. They usually mean the volume or pace is beyond what is reasonable, and pushing harder only makes things worse.

Prove the route before a client run

Look up the exit address in our my IP tool to prove the script leaves through the proxy, and put every endpoint on the list through the proxy checker. A short check up front prevents a whole report built on results from your own office connection.

Turning checks into client reporting

• Store each result with the date, market, language and query, so changes can be compared week by week.
• Describe results as a snapshot of one location at one time; personalisation and timing mean individual users may see something slightly different.
• Keep the keyword list focused on what the client actually needs reported, which keeps volumes and data costs down.

Used this way, a modest script becomes a dependable supplement to official data for SEO research and competitor analysis. Keep volumes modest, stay inside the terms of each service you query, and use official APIs wherever they exist.