LinkedIn company research with Python: a compliance-first guide

Recruiters, sales teams and market analysts look to LinkedIn for the same kinds of information: company pages, public job postings and the public summaries organisations publish about themselves. LinkedIn is also a legally sensitive place to collect data from. Its User Agreement prohibits unauthorized scraping, and the platform works actively against automation. This guide is therefore ordered the way a careful team should work: first what you may collect, then the official channels, and only after that a narrow Python approach for genuinely public pages.

Match the team's need to an official channel

These channels come with explicit authorisation, a stable interface and cleaner data than any page parser. Weigh them properly before anyone writes a scraper. They usually cover the need, and they spare the business both the fragility of HTML parsing and the compliance exposure that comes with it.

In scope and out of scope

A project that cannot fit the left-hand column belongs in an official channel or nowhere.

The narrow case for public pages

A few legitimate questions can remain after that review, such as reading a company's public "About" page or a publicly listed job posting for a market study. The technique is the same as for any careful collection job: small volume, a slow pace, and proxies to spread the requests. Residential proxies are the appropriate type here, with our premium proxies (Residential Premium) adding resilience. Whichever you use, the proxy only carries the request; it does not change what LinkedIn's terms allow. Compare all types on the proxies overview, and see the proxy for scraping page for task guidance.

Install the libraries and set the proxy

pip install requests beautifulsoup4

Your order supplies a host and port, written HOST:PORT, plus a login. In requests those become a small dictionary with one entry per scheme:

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

One page at a time, with a hard stop

Notice what the function refuses to do. It never loops and never retries, and it returns nothing as soon as LinkedIn answers with anything other than 200. After a successful page it sleeps for a long, random interval before another call can begin.

import requests
import random
import time
from bs4 import BeautifulSoup

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/120.0 Safari/537.36"
    ),
    "Accept-Language": "en-US,en;q=0.9",
}

def fetch_public_page(url):
    resp = requests.get(url, headers=headers, proxies=proxies, timeout=25)
    if resp.status_code != 200:
        print(f"Stopping: received status {resp.status_code}")
        return None
    time.sleep(random.uniform(8, 15))  # generous, respectful pacing
    return resp.text

Reading company details from page metadata

Company pages describe themselves in Open Graph meta tags, the same data a chat app uses to build a link preview when someone shares the page. Reading those tags gives you the name, description and URL without depending on the visible layout, which changes far more often:

def parse_company_meta(html):
    soup = BeautifulSoup(html, "html.parser")

    def meta(prop):
        tag = soup.find("meta", attrs={"property": prop})
        return tag["content"] if tag and tag.has_attr("content") else None

    return {
        "name": meta("og:title"),
        "description": meta("og:description"),
        "url": meta("og:url"),
    }

html = fetch_public_page("https://www.linkedin.com/company/example-co/")
if html:
    print(parse_company_meta(html))

That is the whole scope: descriptive facts about an organisation. A request to add selectors for named individuals is the moment to stop and ask whether that data should be collected at all.

Stop signals

Treat every obstacle as a final answer. A login wall, a challenge page or a status other than 200 ends the job, and retrying harder or switching addresses faster would turn a polite job into an abusive one. Schedule runs rarely, keep the total small, and store results so nothing is requested twice. Ending early is how the job stays respectful of the platform and the law.

Confirm the route before any run

Before each run, open our my IP tool through the same proxy to confirm requests leave by that route, and test every endpoint in the proxy checker. Two minutes here confirms that the office's own address stays out of the job.

Sign-off checklist for managers

• Has the team checked whether an official LinkedIn channel already covers the need?
• Is every page in scope visible without signing in?
• Is personal data excluded, or covered by a lawful basis?
• Does the job stop at the first login wall, challenge page or error?

The pattern that holds up over time is simple: official channels first, public-page work kept small and descriptive, and no logged-in or personal data at scale. There are good reasons behind LinkedIn's strict rules, and work done through sanctioned routes nearly always lasts longer.