Log File Analysis for SEO and AI Crawlers: GPTBot, ClaudeBot and Googlebot

Search Console's Crawl Stats report gives you a sample of Googlebot activity. It says nothing about Bing, nothing about AI crawlers, and it does not let you join crawl data to your own list of important URLs. Raw logs do all three. For answer engine optimization (AEO) they matter even more, because no AI company gives you a Search Console of its own. The log file is where you find out whether OpenAI, Anthropic and Perplexity are reading your key pages at all, and whether your firewall is quietly turning them away.

This guide covers the log format, where the files live on common stacks and CDNs, every AI user agent worth tracking and what each operator says it does, how to verify bots, a step-by-step analysis, the checks that matter for AEO and for classic SEO, and how to scale up with pandas and BigQuery.

Crawled vs cited: why AI search needs logs and answer data

Use server logs to examine requests and response status. Use separately captured answer evidence to examine attributed sources. Neither record alone establishes the other event.

Anatomy of a log line

Apache and nginx both ship a "combined" log format by default, and most CDNs can export something close to it. One request, one line. Here is a request from OpenAI's search crawler:

20.171.207.10 - - [14/Oct/2026:09:41:07 +0000] "GET /guides/returns-policy HTTP/1.1" 200 24518 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"

20.171.207.10                       client IP (who connected; use this to verify the bot)
- -                                 identity and auth user (almost always empty)
[14/Oct/2026:09:41:07 +0000]        timestamp with timezone offset
"GET /guides/returns-policy HTTP/1.1"
                                    method, path including query string, protocol
200                                 status code returned
24518                               response size in bytes
"-"                                 referrer (crawlers usually send none)
"Mozilla/5.0 ... OAI-SearchBot/1.4; +https://openai.com/searchbot"
                                    user agent string (what the client claims to be)

The IP in that example is illustrative. Check real IPs against the operator's published list, as covered below. Version numbers inside user agent strings change over time, so match on the token (OAI-SearchBot), not the full string.

In nginx terms the format is $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent". If you split a combined line on spaces, the path is field 7 and the status is field 9. The user agent contains spaces, so split on double quotes instead and it becomes field 6. Every command below relies on those positions; if your format adds fields (response time, host, upstream), check the positions on one line first.

Two fields worth adding if you control the config: request time ($request_time in nginx, %D in Apache) so you can see whether bots are hitting slow pages, and host if one server answers for several domains or subdomains.

Where to get your logs

If a CDN sits in front of your origin, pull the CDN's logs, not the origin's. Requests answered from the CDN cache never reach the origin, so origin logs undercount every bot, often heavily on static pages. The CDN is also where bot protection and WAF rules run, so a request blocked there never shows up at the origin at all. Ask for at least 30 days. Googlebot revisits low-priority pages slowly, and a one-week sample makes many healthy pages look ignored.

Logs can contain identifying information. Use only authorised records, minimise retained fields and apply the relevant access and sharing rules; filtering to bot user agents alone does not clear a log for public release.

AI crawler user agents and what each one does

AI companies run several agents, and they do different jobs. Lumping them together as "AI bots" leads to bad decisions, such as blocking the agent that gets you cited while trying to block training. The roles below are as each operator documents them in October 2026.

Three practical consequences. First, you will never see Google-Extended or Applebot-Extended in a log, so do not build a report that waits for them. Second, the three user-triggered fetchers follow what people ask the assistant about. A rise in ChatGPT-User, Claude-User or Perplexity-User hits on a page is a direct sign that assistants are reading it to answer someone. Third, a training crawler and a search crawler from the same company are separate decisions: blocking GPTBot does not remove you from ChatGPT search, and blocking OAI-SearchBot does.

Other tokens you may want in the list: CCBot (Common Crawl, whose dataset is widely used for training), OAI-AdsBot (OpenAI's checker for landing pages submitted as ChatGPT ads), Meta-ExternalAgent and Bytespider. Control of each agent lives in robots.txt, covered in our robots.txt guide. The log tells you whether the rules you wrote are being followed, and whether something else is blocking the bots you allowed.

Verify the bot is real

Anyone can send a request with Googlebot or GPTBot in the user agent, and scrapers do it constantly to get past bot rules. Before you draw any conclusion, check the IP. There are two methods, and which one applies depends on the operator.

Reverse DNS, then forward DNS. This is Google's documented method, and Bing and Apple support it too. Look up the hostname for the IP, check the domain, then look up that hostname and confirm it returns the original IP:

$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.

$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1

For Google, the hostname must end in googlebot.com, google.com or googleusercontent.com. For Bingbot it ends in search.msn.com, and for Applebot in applebot.apple.com. To check every IP that claimed to be Googlebot in one go:

grep -i 'googlebot' access.log | awk '{print $1}' | sort -u | while read ip; do
  name=$(host "$ip" | awk '{print $NF}')
  back=$(host "$name" 2>/dev/null | awk '/has address/ {print $NF}')
  [ "$back" = "$ip" ] && echo "OK   $ip $name" || echo "FAKE $ip $name"
done

Published IP ranges. AI operators publish JSON files of the IP ranges their agents use, and this is the method to use for AI bots. OpenAI publishes one file per agent (openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json). Perplexity publishes perplexity.com/perplexitybot.json and perplexity.com/perplexity-user.json. Anthropic publishes a list of its crawler IP addresses in its help center. Google publishes several JSON files for its crawlers and fetchers in the Search Central documentation, and Apple publishes applebot.json. Download the files, then match each logged IP against the CIDR ranges. In Python:

import ipaddress, json, urllib.request

def load_ranges(url):
    data = json.load(urllib.request.urlopen(url))
    return [ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix'))
            for p in data['prefixes']]

gptbot = load_ranges('https://openai.com/gptbot.json')

def is_real(ip, ranges):
    addr = ipaddress.ip_address(ip)
    return any(addr in net for net in ranges)

The ranges change, so refresh the files each time you run an analysis rather than caching them for months. If a file's structure differs from the one above, print it once and adjust the key names. Anthropic also notes that blocking by IP may not reliably opt a site out, and recommends robots.txt for control; use IP lists to verify, not to manage access.

How to analyze server log files, step by step

These commands work on macOS and Linux. Use zcat -f access.log* instead of a filename to include rotated and gzipped files. If you write a lot of these, our regex for SEO guide covers the patterns in more depth.

1. Collect at least 30 days of logs from the edge. Use CDN logs if you have a CDN, origin logs if you do not. Write down the date range and whether any days are missing; a gap during a release week will look like a crawl drop later.

2. Filter to bot user agents and count hits per bot.

grep -ioE 'googlebot|bingbot|gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user|applebot|ccbot' access.log \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

Then look for agents you have not classified yet. New AI agents appear every few months.

awk -F'"' '{print $6}' access.log | grep -iE 'bot|crawl|spider|fetch' | sort | uniq -c | sort -rn | head -40

3. Verify the IPs with the methods above, and drop or flag anything that fails. Every number after this step should be based on verified hits only.

4. Check status codes per bot. Swap the token to compare bots. A healthy profile is mostly 200s and 304s; a bot that gets mostly 403s is being blocked.

grep -i 'claudebot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn

And every 404, 403 and 5xx a bot received, by URL:

grep -iE 'googlebot|bingbot|gptbot|oai-searchbot|claudebot|perplexitybot' access.log \
  | awk '$9 ~ /^(403|404|410|429|5[0-9][0-9])$/ {print $9, $7}' | sort | uniq -c | sort -rn | head -50

5. Measure crawl frequency by site section. Group paths by their first directory to see where each bot spends its requests. A bot that spends most of its hits on /tag/ or /search/ while your /products/ section gets little is crawling the wrong things.

grep -i 'oai-searchbot' access.log | awk '{split($7, p, "/"); print "/" p[2]}' | sort | uniq -c | sort -rn | head -20

Hits per day show drops and spikes after a release or a CDN change:

grep -i 'googlebot' access.log | awk '{print substr($4, 2, 11)}' | uniq -c

6. Find wasted crawl and orphan pages. Wasted crawl is requests on URLs you do not want crawled: parameters, redirects, error pages. Start with query strings:

grep -i 'googlebot' access.log | awk '{print $7}' | grep -c '?'
grep -i 'googlebot' access.log | awk '{print $7}' | grep -oE '[?&][^=&]+=' | sort | uniq -c | sort -rn | head

Orphan pages need a join. Export the list of URLs a site crawler found by following internal links, then compare it with the URLs bots requested. URLs bots request that your own crawl did not find are orphans, still known from old links or sitemaps. URLs in your sitemap that no bot requested in 30 days are the opposite problem. The pandas section below shows the join.

7. Look for rendering problems. Most AI crawlers fetch HTML and do not run JavaScript the way a browser does, so a page whose content is injected client-side can return a 200 with almost nothing in it. Compare response sizes by bot and template. If OAI-SearchBot gets 4 KB on a product page that is 60 KB in a browser, fetch the raw HTML with curl and check whether the main content is there. Our JavaScript SEO guide covers the fixes.

grep -i 'oai-searchbot' access.log | awk '$9 == 200 {print $10, $7}' | sort -n | head -20

What to look for with AI crawlers (AEO checks)

Are 4xx and 5xx errors reaching bots? An AI crawler that gets a 500 leaves without the content. Compare error rates for AI bots against Googlebot on the same URLs; if AI bots see more errors, a rate limiter or bot rule is treating them differently.

Do they request /llms.txt? Check directly:

grep -i 'llms.txt' access.log | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn

Which pages do user-triggered fetchers hit? ChatGPT-User, Claude-User and Perplexity-User requests happen while an assistant is building a response. The pages they hit are pages people are asking about right now. Sort them by hits and compare with your list of pages you want cited; the overlap, or the lack of it, is a useful starting point for content work.

Are training and search crawlers handled the way you intended? If your policy is "allow search, block training", the log should show OAI-SearchBot and Claude-SearchBot getting 200s and GPTBot and ClaudeBot either absent or limited to robots.txt. Anything else means the rules are not doing what you think.

Classic SEO checks: crawl budget and Googlebot

Log analysis started as a Googlebot tool, and those uses still matter.

Crawl waste on parameters. If a large share of Googlebot hits carry query strings (sort orders, filters, session IDs, tracking tags), the crawler is spending requests on duplicates. Faceted navigation is the usual cause. Our crawl budget guide covers when this matters; for sites with a few thousand URLs it usually does not.

Server errors and rate limiting. A steady stream of 404s on old URLs points to internal links or sitemaps that still reference them. 5xx and 429 responses are more serious: Google's documentation says Googlebot slows its crawl rate when it sees them, and URLs that keep returning server errors can be dropped from the index.

Redirect hops. Count 301 and 302 responses to bots. Many hits on redirecting URLs mean your internal links point at old addresses.

Pages never crawled. Important pages with zero Googlebot hits in 30 days usually have weak internal linking or are buried deep in pagination.

Crawl changes after releases. A migration, a new template or a CDN change shows up in daily Googlebot hit counts within days, long before rankings move. Compare the two weeks before and after.

Response times. If you log request time, compare bot response times by template. Slow responses limit how much Googlebot will crawl and can cause AI fetchers with short timeouts to give up.

The same analysis in Python and pandas

Once you want to join logs to a sitemap or a crawl export, move to pandas. This parses combined-format lines, labels bots and builds a bot-by-status table.

import re
import pandas as pd

LINE = re.compile(
    r'(?P<ip>\S+) \S+ \S+ \[(?P<time>[^\]]+)\] '
    r'"(?P<method>\S+) (?P<url>\S+)[^"]*" (?P<status>\d{3}) (?P<bytes>\S+) '
    r'"[^"]*" "(?P<ua>[^"]*)"'
)
BOTS = (r'(googlebot|bingbot|gptbot|oai-searchbot|chatgpt-user|claudebot|'
        r'claude-searchbot|claude-user|perplexitybot|perplexity-user|applebot)')

with open('access.log', encoding='utf-8', errors='replace') as f:
    rows = [m.groupdict() for line in f if (m := LINE.match(line))]

df = pd.DataFrame(rows)
df['time'] = pd.to_datetime(df['time'], format='%d/%b/%Y:%H:%M:%S %z')
df['bot'] = df['ua'].str.lower().str.extract(BOTS, expand=False)
df['path'] = df['url'].str.split('?').str[0]
bots = df.dropna(subset=['bot'])

# Bot x status code
print(bots.pivot_table(index='bot', columns='status', values='url',
                       aggfunc='count', fill_value=0))

# Key pages with no hits from AI search crawlers (key_pages.csv has a 'path' column)
key = set(pd.read_csv('key_pages.csv')['path'])
ai_search = {'oai-searchbot', 'claude-searchbot', 'perplexitybot'}
seen = set(bots.loc[bots['bot'].isin(ai_search), 'path'])
print(sorted(key - seen))

# Orphans: paths bots requested that your site crawl did not find
crawl = set(pd.read_csv('site_crawl.csv')['path'])
print(len(set(bots['path']) - crawl), 'bot-requested paths not found by the site crawl')

Our Python for SEO guide covers setting this up if you have not used pandas before.

BigQuery for large log files

Past a few gigabytes, grep and pandas get slow. Many CDNs can push logs straight into BigQuery, or you can load parsed files yourself. Once the logs are a table, the same questions become SQL. This assumes a table with timestamp, path, status and user_agent columns:

SELECT
  REGEXP_EXTRACT(LOWER(user_agent),
    r'(googlebot|bingbot|gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|claude-user|perplexitybot|perplexity-user)') AS bot,
  DATE(timestamp) AS day,
  COUNTIF(status BETWEEN 200 AND 299) AS ok,
  COUNTIF(status IN (403, 429)) AS blocked,
  COUNTIF(status >= 500) AS server_errors,
  COUNT(DISTINCT path) AS unique_paths
FROM `project.logs.access`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY bot, day
HAVING bot IS NOT NULL
ORDER BY day, bot;

Join that against a table of key pages or a sitemap export to get never-crawled and orphan lists at any scale. DuckDB can analyse local Parquet or CSV files, but adapt functions and syntax to its SQL dialect. Our BigQuery for SEO guide covers loading and cost control.

Which log analyzers recognize GPTBot and ClaudeBot?

There is no single answer, because tools change their bot lists often. What to check when you pick one:

Does it let you add your own user agents? This matters more than the default list. New AI agents appear regularly, and a tool that only reports the bots it ships with will miss them. Most dedicated SEO log analyzers, Screaming Frog Log File Analyser among them, let you configure which user agents to track. General log platforms such as the ELK stack, Splunk or Datadog can filter on any string you give them.

Does it separate training, search and user-triggered agents? A single "AI bots" total hides the decisions that matter. You want GPTBot, OAI-SearchBot and ChatGPT-User as separate lines.

Does it verify IPs? Reporting on unverified user agents mixes real crawlers with scrapers. Check whether the tool uses reverse DNS for search engines and the published IP lists for AI operators.

Does it see the edge? A tool reading origin logs behind a CDN misses cached and blocked requests. Some CDNs, Cloudflare among them, show AI crawler activity per bot in their own dashboard, which is a quick first look even without exporting logs.

GetCited includes Log Insights under Crawlability in its technical SEO and AEO audit: you upload or connect server logs and see which AI crawlers and search bots hit which pages. It sits next to an AI Referral Impact view, so you can compare crawler activity with the visits AI engines actually send.

Findings and fixes

Crawls are not citations

A server request establishes a fetch attempt at a particular time. It does not show whether a later answer used or cited the page. Keep request logs and captured answer evidence as separate records.

Mistakes to avoid

Trusting the user agent. Unverified "Googlebot" or "GPTBot" traffic can be a large share of what claims to be those bots. Verify before you report numbers.
Analyzing origin logs behind a CDN. Cached and blocked responses never reach the origin, so you undercount every bot and miss the blocks.
Using a week of data. Crawl frequency for low-priority pages is measured in weeks. Use 30 days or more before calling a page "never crawled".
Treating all AI bots as one. Blocking GPTBot is a training decision. Blocking OAI-SearchBot removes you from ChatGPT search. Decide per agent.
Searching logs for Google-Extended. It is a robots.txt token, not a user agent. It never appears in a request.
Fixing only robots.txt. Many AI crawler blocks come from WAF rules and CDN bot settings. If the log shows 403s to AI agents, look there first.
Reading a crawl as a citation. A fetch means eligible, not cited. Pair logs with answer tracking.
Sharing raw logs. They contain user IPs. Filter to bots first.