Python for SEO: 6 Working Scripts for Audits, Sitemaps and Search Console

You do not need to become a developer to use Python for SEO. You need a working environment, four or five libraries, and a handful of scripts you understand well enough to change. That is what this guide gives you.

The scripts build on each other. Script 3 produces a URL list that scripts 1 and 2 can check. Script 4 exports query data that script 6 analyses. Script 5 replaces 1 and 2 when you want to crawl instead of checking a fixed list. Each one writes a CSV you can open in Excel or Google Sheets.

Setup: Python 3, a virtual environment and the packages

Install a current Python 3 (3.10 or newer) from python.org, or with Homebrew on a Mac (brew install python). On Windows, tick Add python.exe to PATH in the installer. Then create a project folder with its own virtual environment, so these packages never conflict with anything else on your machine:

mkdir seo-scripts && cd seo-scripts
python3 -m venv .venv

# macOS / Linux
source .venv/bin/activate
# Windows (PowerShell)
.venv\Scripts\Activate.ps1

pip install requests beautifulsoup4 pandas advertools google-api-python-client google-auth

Your prompt now starts with (.venv). Run source .venv/bin/activate (or the Windows equivalent) each time you open a new terminal in this folder. Save each script below as a .py file and run it with python filename.py. If you prefer to work interactively, pip install jupyterlab and run jupyter lab; every script works in a notebook cell too.

What each package does: requests fetches URLs, beautifulsoup4 parses HTML, pandas handles tables, advertools wraps the Scrapy crawler and has SEO helpers, and google-api-python-client with google-auth talks to Search Console.

Put this at the top of every script that fetches pages. Identifying your crawler is polite and makes it easy for a site owner (or your own server logs) to recognise it:

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; MySEOAudit/1.0; +https://www.example.com/contact)"}

1. Bulk status code and redirect chain checker

Put one URL per line in urls.txt. The script requests each one, follows redirects, and records the first status code, the final status code, the final URL, the number of hops and the full chain. Five URLs are fetched in parallel.

import requests
import pandas as pd
from concurrent.futures import ThreadPoolExecutor

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; MySEOAudit/1.0; +https://www.example.com/contact)"}

def check(url):
    try:
        r = requests.get(url, headers=HEADERS, timeout=15, allow_redirects=True, stream=True)
        r.close()  # stream=True + close: we only need headers, not the body
        chain = " -> ".join(f"{h.status_code} {h.url}" for h in r.history)
        return {
            "url": url,
            "first_status": r.history[0].status_code if r.history else r.status_code,
            "final_status": r.status_code,
            "final_url": r.url,
            "hops": len(r.history),
            "chain": chain,
        }
    except requests.RequestException as e:  # timeouts, DNS errors, redirect loops
        return {"url": url, "first_status": "ERROR", "final_status": "",
                "final_url": "", "hops": "", "chain": type(e).__name__}

with open("urls.txt") as f:
    urls = [line.strip() for line in f if line.strip()]

with ThreadPoolExecutor(max_workers=5) as pool:
    results = list(pool.map(check, urls))

df = pd.DataFrame(results)
df.to_csv("status_report.csv", index=False)
print(df["first_status"].value_counts())
print(df[df["hops"].apply(lambda h: isinstance(h, int) and h > 1)])  # chains of 2+ hops

Read the output this way. first_status 301 or 308 with final_status 200 is a normal redirect; update internal links and sitemaps to the final URL. hops of 2 or more is a chain; point the first URL straight at the last one. A 302 or 307 on a URL that moved permanently should become a 301. ERROR with TooManyRedirects is a redirect loop. Google's documentation says Googlebot follows up to 10 redirect hops, but every hop costs crawl time, so aim for one.

Some servers answer bots differently from browsers. If you see unexpected 403s, open the URL in a browser before you report it: it may be a firewall rule blocking your script, not a real error.

2. Extract title, meta description, canonical and H1 to CSV

Same urls.txt input. This time the script parses the HTML with BeautifulSoup and pulls the tags that most often explain a ranking, snippet or indexing problem, plus the X-Robots-Tag HTTP header, which can carry a noindex that never appears in the HTML.

import requests
import pandas as pd
from bs4 import BeautifulSoup
from concurrent.futures import ThreadPoolExecutor

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; MySEOAudit/1.0; +https://www.example.com/contact)"}

def meta_content(soup, name):
    tag = soup.find("meta", attrs={"name": lambda v: v and v.lower() == name})
    return tag.get("content", "").strip() if tag else ""

def extract(url):
    try:
        r = requests.get(url, headers=HEADERS, timeout=20)
    except requests.RequestException as e:
        return {"url": url, "status": "ERROR " + type(e).__name__}
    soup = BeautifulSoup(r.content, "html.parser")  # bytes, so bs4 reads the page's charset
    title_tag = soup.find("title")
    title = title_tag.get_text(strip=True) if title_tag else ""
    canonical = soup.find("link", rel="canonical")
    h1s = [h.get_text(" ", strip=True) for h in soup.find_all("h1")]
    return {
        "url": url,
        "status": r.status_code,
        "final_url": r.url,
        "title": title,
        "title_length": len(title),
        "meta_description": meta_content(soup, "description"),
        "meta_robots": meta_content(soup, "robots"),
        "x_robots_tag": r.headers.get("X-Robots-Tag", ""),
        "canonical": canonical.get("href", "") if canonical else "",
        "h1_count": len(h1s),
        "h1": " | ".join(h1s),
    }

urls = [u.strip() for u in open("urls.txt") if u.strip()]
with ThreadPoolExecutor(max_workers=5) as pool:
    df = pd.DataFrame(list(pool.map(extract, urls))).fillna("")  # error rows have blanks

df.to_csv("onpage_report.csv", index=False)
print("Missing titles:", (df["title"] == "").sum())
print("Duplicate titles:", (df["title"].ne("") & df["title"].duplicated(keep=False)).sum())
print("Canonical points elsewhere:", ((df["canonical"] != "") & (df["canonical"] != df["final_url"])).sum())
print("noindex:", df["meta_robots"].str.contains("noindex", case=False, na=False).sum())

The summary lines at the end are the four checks worth doing on every export. A canonical that points elsewhere is fine for parameter or duplicate URLs and a problem on pages you want ranked. Note that relative canonicals (/page) will show as "elsewhere" in this comparison; use urllib.parse.urljoin(final_url, canonical) to normalise them if your site uses them.

This reads the raw HTML the server sends. If titles or canonicals come back empty on a site built with React, Vue or Angular, the tags are probably added by JavaScript. Check in a browser before reporting them as missing, and see the mistakes section for what to do instead.

3. Parse an XML sitemap (and sitemap index) into a URL list

Large sites split their sitemap into a sitemap index that lists child sitemaps, sometimes gzipped. This script handles plain sitemaps, sitemap indexes and .xml.gz files, and returns every URL with its lastmod.

import gzip
import xml.etree.ElementTree as ET
import requests
import pandas as pd

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; MySEOAudit/1.0; +https://www.example.com/contact)"}
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def fetch_xml(url):
    r = requests.get(url, headers=HEADERS, timeout=30)
    r.raise_for_status()
    data = r.content
    if data[:2] == b"\x1f\x8b":          # gzip magic bytes
        data = gzip.decompress(data)
    return ET.fromstring(data)

def sitemap_urls(url, seen=None):
    seen = set() if seen is None else seen
    if url in seen:
        return []
    seen.add(url)
    root = fetch_xml(url)
    rows = []
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            rows += sitemap_urls(loc.text.strip(), seen)
    else:
        for u in root.findall("sm:url", NS):
            rows.append({
                "url": u.findtext("sm:loc", default="", namespaces=NS).strip(),
                "lastmod": u.findtext("sm:lastmod", default="", namespaces=NS),
                "sitemap": url,
            })
    return rows

df = pd.DataFrame(sitemap_urls("https://www.example.com/sitemap.xml"))
df.to_csv("sitemap_urls.csv", index=False)
df["url"].to_csv("urls.txt", index=False, header=False)   # feeds scripts 1 and 2
print(len(df), "URLs in", df["sitemap"].nunique(), "sitemaps")

The last line writes urls.txt, so you can run script 1 next and find every sitemap URL that redirects or errors. A sitemap should only list canonical URLs that return 200; anything else wastes crawl attention and sends Google mixed signals. The sitemaps.org protocol caps each file at 50,000 URLs and 50 MB uncompressed.

If you only need the data, advertools has a one-liner that does the same recursion: adv.sitemap_to_df("https://www.example.com/robots.txt") reads every sitemap declared in robots.txt. For sitemaps from sites you do not control, the defusedxml package is a safer drop-in parser than the standard library.

4. Pull every Search Console row with the API

The Search Console interface exports 1,000 rows. The API returns up to 25,000 rows per request and lets you page through more with startRow. A service account is the simplest way to authenticate a script that runs unattended.

One-time setup: in Google Cloud Console, create a project, enable the Google Search Console API, then go to IAM & Admin → Service Accounts, create one, and add a JSON key. Save it as service-account.json next to your script and never commit it to Git. Copy the service account's email address (it ends in iam.gserviceaccount.com), then in Search Console go to Settings → Users and permissions → Add user and add that email with Restricted permission. Without this step every call returns 403.

from datetime import date, timedelta
import pandas as pd
from google.oauth2 import service_account
from googleapiclient.discovery import build

SITE = "sc-domain:example.com"     # or "https://www.example.com/" for a URL-prefix property
KEY_FILE = "service-account.json"
SCOPES = ["https://www.googleapis.com/auth/webmasters.readonly"]

creds = service_account.Credentials.from_service_account_file(KEY_FILE, scopes=SCOPES)
service = build("searchconsole", "v1", credentials=creds)

end = date.today() - timedelta(days=3)       # recent days are still incomplete
start = end - timedelta(days=89)             # last 90 days

def query_all(dimensions, row_limit=25000):
    rows, start_row = [], 0
    while True:
        body = {
            "startDate": start.isoformat(),
            "endDate": end.isoformat(),
            "dimensions": dimensions,
            "type": "web",
            "rowLimit": row_limit,
            "startRow": start_row,
        }
        resp = service.searchanalytics().query(siteUrl=SITE, body=body).execute()
        batch = resp.get("rows", [])
        rows.extend(batch)
        if len(batch) < row_limit:
            break
        start_row += row_limit
    return rows

rows = query_all(["query", "page"])
df = pd.DataFrame([
    {"query": r["keys"][0], "page": r["keys"][1], "clicks": r["clicks"],
     "impressions": r["impressions"], "ctr": r["ctr"], "position": r["position"]}
    for r in rows
])
df.to_csv("gsc_queries.csv", index=False)
print(len(df), "rows,", df["clicks"].sum(), "clicks")

Change SITE to the property exactly as Search Console shows it: domain properties use sc-domain:, URL-prefix properties need the full URL with the trailing slash. Add "date" to the dimensions for a daily trend, or "country" and "device" for splits.

Your totals will be lower than the Performance report's chart. Google omits anonymised queries (rare queries withheld for privacy) from query-level rows, so a query-and-page export never adds up to the site total. That is expected, not a bug in the script.

5. Crawl a small site with advertools and analyse it in pandas

When you do not have a URL list, crawl. advertools.crawl runs Scrapy under the hood, follows internal links, obeys robots.txt by default, and writes one JSON line per page with the title, meta description, headings, canonical, status, depth, links and response headers already extracted.

import os
import advertools as adv
import pandas as pd

OUT = "crawl.jl"
if os.path.exists(OUT):
    os.remove(OUT)      # advertools appends to an existing file

adv.crawl(
    "https://www.example.com/",
    OUT,
    follow_links=True,
    custom_settings={
        "CLOSESPIDER_PAGECOUNT": 500,          # stop after 500 pages
        "DOWNLOAD_DELAY": 1,                   # one request per second
        "USER_AGENT": "Mozilla/5.0 (compatible; MySEOAudit/1.0; +https://www.example.com/contact)",
        "LOG_FILE": "crawl.log",
    },
)

df = pd.read_json(OUT, lines=True)
pages = df.reindex(columns=["url", "status", "depth", "title", "meta_desc", "h1", "canonical"])

# Inlinks: advertools joins multiple values with "@@"
links = df[["url", "links_url"]].dropna()
links = links.assign(target=links["links_url"].str.split("@@")).explode("target")
inlinks = links.groupby("target")["url"].nunique()
pages["inlinks"] = pages["url"].map(inlinks).fillna(0).astype(int)

checks = {
    "not_200": pages[pages["status"] != 200],
    "missing_title": pages[pages["title"].isna()],
    "duplicate_title": pages[pages["title"].notna() & pages.duplicated("title", keep=False)],
    "missing_meta_desc": pages[pages["meta_desc"].isna()],
    "multiple_h1": pages[pages["h1"].fillna("").str.contains("@@")],
    "canonical_elsewhere": pages[pages["canonical"].notna() & (pages["canonical"] != pages["url"])],
    "deep_pages": pages[pages["depth"] > 3],
    "few_inlinks": pages[pages["inlinks"] <= 1],
}
for name, frame in checks.items():
    print(f"{name:20} {len(frame)}")
    frame.to_csv(f"crawl_{name}.csv", index=False)

Each check writes its own CSV, so you hand a developer a file of exactly the URLs to fix. depth is the number of clicks from the start URL; important pages buried four or more clicks deep, or with only one internal link pointing at them, are the usual candidates for better internal linking.

Keep the page cap and delay on. A 500-page crawl at one request per second takes under ten minutes and will not trouble any normal server. For sites in the tens of thousands of pages, or anything that needs JavaScript rendering, a dedicated crawler such as Screaming Frog or Sitebulb is faster to operate.

6. N-gram analysis and query clusters from Search Console

Thousands of queries are hard to read one by one. Breaking them into words and phrases (n-grams) and summing clicks and impressions per phrase shows which themes drive traffic and which ones get impressions but no clicks. This script uses the gsc_queries.csv file from script 4.

import re
import pandas as pd

df = pd.read_csv("gsc_queries.csv")
STOP = {"a", "an", "and", "the", "for", "to", "of", "in", "on", "with", "is", "how",
        "what", "best", "vs", "or", "my", "do", "i", "can", "you", "your"}

def ngrams(text, n):
    words = re.findall(r"[a-z0-9']+", str(text).lower())
    grams = [" ".join(words[i:i + n]) for i in range(len(words) - n + 1)]
    return [g for g in grams if not (n == 1 and g in STOP)]

# One row per query (the export has one row per query + page)
q = df.groupby("query", as_index=False)[["clicks", "impressions"]].sum()

parts = []
for n in (1, 2, 3):
    t = q.assign(ngram=q["query"].map(lambda s: ngrams(s, n)), n=n).explode("ngram")
    parts.append(t.dropna(subset=["ngram"]).drop_duplicates(["query", "ngram"]))
grams = pd.concat(parts)

summary = (grams.groupby(["n", "ngram"])
           .agg(queries=("query", "nunique"), clicks=("clicks", "sum"),
                impressions=("impressions", "sum"))
           .reset_index())
summary["ctr"] = (summary["clicks"] / summary["impressions"]).round(4)
summary = summary[summary["queries"] >= 3].sort_values("impressions", ascending=False)
summary.to_csv("gsc_ngrams.csv", index=False)
print(summary[summary["n"] == 2].head(20))

# Query clusters by ranking page: which queries does each URL win,
# and which queries are split across several URLs?
by_page = (df.sort_values("clicks", ascending=False)
             .groupby("page")
             .agg(queries=("query", "nunique"), clicks=("clicks", "sum"),
                  top_queries=("query", lambda s: " | ".join(s.head(5))))
             .sort_values("clicks", ascending=False))
by_page.to_csv("gsc_page_clusters.csv")

split = (df[df["impressions"] >= 10]
         .groupby("query")
         .agg(pages=("page", "nunique"), impressions=("impressions", "sum"))
         .query("pages > 1")
         .sort_values("impressions", ascending=False))
split.to_csv("gsc_queries_on_multiple_pages.csv")

Three files come out. gsc_ngrams.csv shows themes: a two-word phrase with high impressions and low CTR across many queries is a topic where you appear but rarely win the click, often worth a dedicated page or a better title. gsc_page_clusters.csv groups queries by the URL that ranks for them, which is the most practical cluster definition because Google already decided those queries belong together. gsc_queries_on_multiple_pages.csv lists queries where two or more of your URLs collect impressions; review these for keyword cannibalization, keeping in mind that some overlap is normal for broad queries.

Edit STOP to add your brand name and generic words for your niche, otherwise they dominate the single-word list. For semantic clustering (grouping queries that mean the same thing without sharing words), the next step is sentence embeddings with a library such as sentence-transformers, which is beyond a starter script.

When to use Python, Apps Script or a crawler

If you live in Google Sheets and your lists are small, start with Apps Script for SEO, which covers the same status, on-page and Search Console jobs without a local install. Move to Python when you hit the six-minute runtime limit or need to join datasets. When the data outgrows your laptop, BigQuery takes over.

Writing SEO scripts with AI, safely

ChatGPT, Claude and Gemini write usable Python if you give them a precise spec: the input file and its columns, the output file and its columns, the library to use, and an example row. "Read urls.txt, one URL per line; for each, fetch with requests and a 15-second timeout; write url, status and canonical to out.csv" gets better code than "write an SEO audit script". Before you run what they give you, check these:

Timeouts and error handling. Every requests.get needs a timeout, and the loop should catch requests.RequestException so one bad URL does not kill the run.
No secrets in code. Keys and service-account files stay in separate files or environment variables, and out of anything you paste into a chat or commit to Git.
Real libraries and real methods. AI models sometimes invent function names or parameters. If pip install fails or an attribute does not exist, check the library's documentation rather than asking the model to guess again.
Sensible concurrency. Code that opens 50 threads against one site is a denial-of-service script. Keep it to a handful of workers and add a delay.
Test on ten URLs first. Read the output and spot-check three rows in a browser before running the full list.

Python is also the quickest way to check whether AI crawlers can reach your site. This snippet reads your robots.txt and reports, for each major AI user agent, whether it may fetch a given URL:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser("https://www.example.com/robots.txt")
rp.read()
for agent in ["Googlebot", "Google-Extended", "GPTBot", "OAI-SearchBot",
              "ChatGPT-User", "ClaudeBot", "PerplexityBot", "Bingbot"]:
    print(f"{agent:16}", rp.can_fetch(agent, "https://www.example.com/blog/"))

Robots.txt only tells you what the site allows; a CDN or firewall can still block these bots. And none of it tells you whether ChatGPT, Perplexity or Google's AI Overviews actually mention you. That depends on the answers themselves, which change by prompt and by day, and is what our AI visibility tracker measures.

Mistakes to avoid

No rate limiting. Fifty parallel requests can trigger a firewall block, skew your client's analytics or slow their site. Use three to five workers, or DOWNLOAD_DELAY in advertools, and back off on HTTP 429 or 503 responses. The Search Console API has its own per-minute quotas; if you get a 429 there, sleep and retry.
Using the default user agent. python-requests/2.x is blocked by many CDNs and tells the site owner nothing. Set a descriptive user agent with a contact URL. Do not impersonate Googlebot: sites can verify real Googlebot by reverse DNS, and a fake one is often blocked outright.
Ignoring robots.txt. advertools obeys robots.txt by default; a hand-written requests loop does not. Check it with RobotFileParser before crawling a site you do not own.
Treating raw HTML as what Google sees. requests and BeautifulSoup read the server response. Google renders JavaScript, so a tag missing from your output may still be visible to Google. Compare with the URL Inspection tool's rendered HTML, or render the page with Playwright. The reverse also matters: most AI crawlers do not run JavaScript, so content that only exists after rendering may be invisible to them.
Comparing URLs as plain strings. http:// vs https://, trailing slashes, uppercase letters and tracking parameters make identical pages look different. Normalise URLs before joining crawl data with Search Console data.
Rerunning the GSC pull for today's dates. The last two or three days of data are incomplete. Leave a buffer, as script 4 does, or your week-on-week comparisons will always show a fake drop.