Social media managers, agency account teams and brand marketers turn to Instagram for campaign and competitive research: how public business profiles present themselves, which hashtags are gaining ground, how posts perform. Instagram controls its data closely. It provides an official API for authorised access and defends actively against automated collection. This guide follows the order a responsible team should: Meta's Graph API first, clear limits on what counts as acceptable public data, and only then a small, slow Python workflow with proxies for what the API does not cover.
What the Graph API covers
With the right permissions, Meta's Instagram Graph API returns structured data from Business and Creator accounts, such as:
- media and the comments on it,
- insights for accounts and posts,
- hashtag data, and more.
Access runs through an access token and documented endpoints, so the data arrives structured, the permissions are explicit, and nothing depends on a page layout. Where your team manages the account, or its owner has granted permission, that route almost always beats a page parser. Rule it out properly before anyone proposes scraping.
Four rules agreed before any job runs
Two facts shape every Instagram project. Its Terms of Use prohibit unauthorized automated data collection, and most of what makes the platform valuable is only visible after signing in. So the team signs up to four rules first:
- Scope: content anyone can see without an account. Automated logged-in accounts, and attempts to get past login or challenge screens, are excluded; both break Instagram's terms.
- Permissions: before anything is scheduled, someone checks the job against
robots.txt, the Terms of Use and the rate limits that apply. - Privacy: individuals' data stays out unless there is a lawful basis for it under GDPR, CCPA or whichever privacy law applies.
- Load: requests are few and slow, so the service carries no meaningful burden.
If a brief cannot be delivered inside those four rules, scraping is the wrong tool for it.
Where a public-page check fits
A typical legitimate case is basic metadata from public business profiles, for example the description a brand account shows to every visitor. For that, follow the same disciplined pattern as on any sensitive site: very few requests, long pauses, and proxies that spread those requests over several addresses. Residential proxies are the appropriate type, with our premium proxies (Residential Premium) adding stability. The proxies overview sets the four types side by side, and the proxy for scraping page adds guidance for this kind of task.
Setup
pip install requests beautifulsoup4The order's host and port (HOST:PORT) and its login slot into a requests dictionary:
proxies = {
"http": "http://USER:PASS@HOST:PORT",
"https": "http://USER:PASS@HOST:PORT",
}A conservative profile fetcher
Read the function as a set of brakes: a single request per call, standard browser headers, a long random pause after each page, and an exit on the first response other than 200. The job keeps only the Open Graph tags from the page head, which change less often than a rendered feed and are far lighter to parse.
import requests
import random
import time
from bs4 import BeautifulSoup
proxies = {
"http": "http://USER:PASS@HOST:PORT",
"https": "http://USER:PASS@HOST:PORT",
}
headers = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
}
def fetch_public_profile(username):
url = f"https://www.instagram.com/{username}/"
resp = requests.get(url, headers=headers, proxies=proxies, timeout=25)
if resp.status_code != 200:
print(f"Stopping: received status {resp.status_code}")
return None
time.sleep(random.uniform(8, 15)) # respectful pacing
return resp.textProfile metadata, nothing more
Those tags yield three fields, the profile's title, description and image, and nothing private:
def parse_profile_meta(html):
soup = BeautifulSoup(html, "html.parser")
def meta(prop):
tag = soup.find("meta", attrs={"property": prop})
return tag["content"] if tag and tag.has_attr("content") else None
return {
"title": meta("og:title"),
"description": meta("og:description"),
"image": meta("og:image"),
}
html = fetch_public_profile("example_business")
if html:
print(parse_profile_meta(html))Keep the job at that size. A request to list followers or gather details about individuals belongs in a Graph API conversation, or nowhere.
Back off early and cache everything
Every boundary is final on Instagram. If the response is a login wall, a challenge page or anything but a 200, the run is over: no retries, no fresh addresses. Schedule jobs rarely, keep the request count small, and append each result to a file so the same profile is never fetched twice:
import json
from datetime import datetime
def save_record(username, data, path="profiles.jsonl"):
row = {"username": username,
"fetched_at": datetime.utcnow().isoformat(),
**data}
with open(path, "a", encoding="utf-8") as f:
f.write(json.dumps(row) + "\n")Before the first run
Two checks belong before the first scheduled run. Opened through the proxy, our my IP tool should show the proxy's address rather than the office's, and every endpoint should pass the proxy checker. Both take a minute and catch the setup errors that matter.
Questions account managers ask
Can we report on a client's own account this way? Use the Graph API for that. With the client's permission it returns media, comments and insights directly, which a page parser cannot match.
Can we keep an eye on competitors? Only their public business profile details, at low volume, and only if your use stays within Instagram's terms. Follower lists and personal details are out of scope.
Instagram's own design points the same way as this guide: official access through the Graph API, small and impersonal public checks, and an immediate halt when anything pushes back. The code above is a narrow instrument for a narrow job; it does not license collection at scale.
