Amazon listing monitoring in Python for e-commerce teams

E-commerce operators, brand managers and pricing analysts all keep an eye on Amazon. They check that their own listings show the right title and price in each marketplace, follow the public prices of competing products, and watch availability ahead of a promotion. For a focused list of products, a short Python job built with requests and beautifulsoup4 can collect those public fields every day and hand them to a spreadsheet or dashboard. This guide builds that job, with location-matched proxies, a modest pace and clear rules.

Pick the right route for the data

Official APIs are the sanctioned way to get catalogue data at scale, and they return cleaner data than any page parser. Use the script for light research that the APIs do not cover.

Rules for the team

Collect public product information only. Nothing behind a login, and no personal data about buyers, reviewers or sellers. Check the job against Amazon's robots.txt and Terms of Service, and keep within both. Do not attempt to get past CAPTCHAs or authentication, keep request rates low, and treat this as a research tool rather than a way to copy the catalogue. Responsibility for permitted use stays with your business.

Matching the proxy to the marketplace

Amazon runs separate marketplaces by country, and prices, delivery messages and stock depend on where the visitor is. Choose proxies in the country whose marketplace you are checking. Residential proxies come from consumer internet providers, so your team receives the same regional version of a product page that local customers receive; keep the request rate modest and stay within the site's terms. If results from standard residential addresses are inconsistent, compare residential premium proxies, our premium tier for sites where standard pools see more failed requests. For help pairing proxy types with tasks, see the proxy for scraping page.

Installing and setting the proxy

pip install requests beautifulsoup4 lxml

The requests library takes your HOST:PORT address and login as a dictionary:

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

Requesting a product by its ASIN

Every Amazon product has an ASIN, visible in its URL as /dp/ASIN. The function below requests one product page with the headers a standard desktop browser sends, so the server returns the regular desktop page in the language you ask for. Change the domain argument to check another marketplace:

import requests
from bs4 import BeautifulSoup

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
}

def fetch_product(asin, domain="www.amazon.com"):
    url = f"https://{domain}/dp/{asin}"
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=25)
    resp.raise_for_status()
    return resp.text

Extracting title, price, rating and stock

Product pages are not uniform: layouts differ between categories and shift over time. The parser therefore tries several known elements per field and is written to accept a missing value:

def parse_product(html):
    soup = BeautifulSoup(html, "lxml")

    def first_text(*selectors):
        for sel in selectors:
            el = soup.select_one(sel)
            if el and el.get_text(strip=True):
                return el.get_text(strip=True)
        return None

    title = first_text("#productTitle")
    price = first_text(
        "span.a-price span.a-offscreen",
        "#priceblock_ourprice",
        "#priceblock_dealprice",
    )
    rating = first_text("span.a-icon-alt")
    review_count = first_text("#acrCustomerReviewText")
    availability = first_text("#availability span")

    return {
        "title": title,
        "price": price,
        "rating": rating,
        "review_count": review_count,
        "availability": availability,
    }

html = fetch_product("B08N5WRWNW")
print(parse_product(html))

If every field returns None, the response was probably a robot-check page instead of the product. Treat that as a signal to slow down, shorten the list or check less often, not as an obstacle to work around. Look at the raw HTML before changing any selectors.

Working through a product list

For more than a handful of products, pick an endpoint for each request and pause for several seconds, at random, between products:

import random
import time

PROXY_POOL = [
    "http://USER:PASS@HOST1:PORT",
    "http://USER:PASS@HOST2:PORT",
    "http://USER:PASS@HOST3:PORT",
]

def scrape_many(asins):
    out = {}
    for asin in asins:
        p = random.choice(PROXY_POOL)
        prox = {"http": p, "https": p}
        try:
            url = f"https://www.amazon.com/dp/{asin}"
            resp = requests.get(url, headers=headers, proxies=prox, timeout=25)
            resp.raise_for_status()
            out[asin] = parse_product(resp.text)
        except requests.RequestException as e:
            out[asin] = {"error": str(e)}
        time.sleep(random.uniform(4, 10))  # be a good citizen
    return out

If your order rotates the address on every request, a single endpoint is enough, as the gateway varies the exit for you. Leave the random sleep in place regardless, because it keeps the job courteous.

Snapshots for price history

Price monitoring rarely needs fresh data every hour. Record each result with a timestamp, refresh on a schedule, and avoid fetching the same page twice in a day. A daily snapshot covers the bulk of pricing work and keeps the request count low, which respects the site and keeps your data use small:

import json
from datetime import datetime

def save_snapshot(asin, data, path="prices.jsonl"):
    record = {"asin": asin, "fetched_at": datetime.utcnow().isoformat(), **data}
    with open(path, "a", encoding="utf-8") as f:
        f.write(json.dumps(record) + "\n")

A file of dated records like this loads straight into a spreadsheet or BI tool, ready for a price-history chart.

Checks before each run

Use our my IP tool to confirm that requests go out through the proxy, and put each endpoint on the list through the proxy checker. This catches an expensive mistake: a full run that quietly used the office connection.

Notes for the manager signing it off

• Cost: the script fetches page HTML only, not images or scripts, so data use on a per-GB plan grows with the length of the list and how often it runs.
• Scope: a focused list of products, checked once a day, is the right scale for this approach.
• Growth: when the list grows or the business needs catalogue data, move to Amazon's official APIs rather than scaling the script.

Patience is what keeps this kind of monitoring sustainable: slow requests, daily snapshots and careful parsing, limited to public product data and Amazon's terms.