AI Crawlers and robots.txt: Which Bots to Allow or Block
Training, search and user-triggered crawlers do different jobs. Here is what each does and what the most-cited websites block.
AI crawlers are the bots AI companies use to collect pages for model training, to build their AI search indexes, or to fetch a page live when a user asks about it. You control most of them in robots.txt by user agent: block the training crawlers if you do not want your content used to train models, and allow the search and user-triggered agents if you want to be cited in AI answers. This guide lists the main AI crawlers, gives rules you can copy, and shows how the websites AI engines cite most actually handle them.
The Three Kinds of AI Crawler
Most confusion about blocking AI bots comes from treating them as one group. They do three different jobs:
- Training crawlers collect pages in bulk to train or update models. Blocking them keeps your content out of future training data. Examples: GPTBot, ClaudeBot, CCBot.
- Search (indexing) crawlers build the index an AI search product uses to find and cite sources. Blocking them can keep your pages out of that product's answers. Examples: OAI-SearchBot, Claude-SearchBot, PerplexityBot.
- User-triggered fetchers visit a page because a user asked the assistant about it or shared the link. Examples: ChatGPT-User, Claude-User, Perplexity-User.
Two more names are robots.txt tokens rather than crawlers: Google-Extended and Applebot-Extended. They do not crawl anything themselves; they tell Google and Apple whether content their regular crawlers collect may be used for AI model training.
AI Crawler List: User Agents and What They Do
| User agent | Company | Type | What blocking it does |
|---|---|---|---|
| GPTBot | OpenAI | Training | Keeps content out of OpenAI model training |
| OAI-SearchBot | OpenAI | Search indexing | Can keep pages out of ChatGPT search results |
| ChatGPT-User | OpenAI | User-triggered | Stops ChatGPT fetching your pages when users ask |
| ClaudeBot | Anthropic | Training | Keeps content out of Anthropic model training |
| Claude-SearchBot | Anthropic | Search indexing | Can keep pages out of Claude's search results |
| Claude-User | Anthropic | User-triggered | Stops Claude fetching pages for user requests |
| PerplexityBot | Perplexity | Search indexing | Can keep pages out of Perplexity's results |
| Perplexity-User | Perplexity | User-triggered | Perplexity says this agent generally does not follow robots.txt, because a user requested the page |
| Google-Extended | Training token | Opts content out of Gemini model training; Google says it does not affect Search or ranking | |
| Applebot-Extended | Apple | Training token | Opts content out of Apple's AI training; Applebot still crawls for search |
| CCBot | Common Crawl | Dataset crawler | Keeps pages out of the open Common Crawl dataset many models train on |
| Bytespider | ByteDance | Crawler | Blocks ByteDance's crawler |
AI companies add and rename agents, so check each vendor's current crawler documentation before you deploy rules. Our robots.txt optimization guide covers the syntax and the common mistakes.
What the Most-Cited Websites Do: Our Data
GetCited platform data identified the 100 domains AI engines retrieved most often while answering tracked questions for 17 brands between July 19 and October 18, 2026. We checked their robots.txt files on October 19, 2026; 86 could be parsed.
- 22 of 86 (26%) block at least one AI crawler from the whole site. The other 64 do not block any of these crawlers at the root.
- The most blocked are the dataset and training crawlers: Bytespider (18), CCBot (15), ClaudeBot (14) and GPTBot (13).
- Search and user agents are blocked a little less often: ChatGPT-User, PerplexityBot and Claude-SearchBot by 11 each, OAI-SearchBot by 10.
- Google-Extended is blocked least (8).
- Among the 17 brands' own websites, 3 name AI crawlers in robots.txt and none blocks them.
The Reddit Example: Blocked, Yet Most Retrieved
Reddit's robots.txt blocks these AI crawlers, yet reddit.com was the single most-retrieved domain in our data, with 882 retrievals, ahead of YouTube (691). Robots.txt is not the only route content takes into AI answers: search indexes and content licensing agreements between platforms and AI companies matter too. A block does not guarantee you are absent from answers, and an open robots.txt does not guarantee you are cited.
Should You Block AI Crawlers?
Decide by goal, not by fear:
| Your goal | Training crawlers | Search crawlers | User-triggered fetchers |
|---|---|---|---|
| Maximum visibility in AI answers | Allow | Allow | Allow |
| Be cited in AI search, but not used for training | Block | Allow | Allow |
| Protect licensed or paywalled content | Block | Block on protected paths | Block on protected paths |
| Stay out of AI entirely | Block | Block | Block (some may not comply) |
For most brands that want to be recommended by AI, blocking search crawlers is the costly mistake: it can remove you from the engines that link to sources. In GetCited data, the engines that search the web, such as Google AI Mode and Perplexity, linked the brand's own site in about 26% of answers, while engines answering from memory almost never did.
robots.txt Rules You Can Copy
These are illustrative. Check current user-agent names before deploying.
Allow AI search and user fetches, block training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Block all AI crawlers:
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
Disallow: /
Protect one section only:
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /members/
Disallow: /research/
If you add nothing, crawlers follow your general User-agent: * rules.
What robots.txt Cannot Do
- It is a request, not access control. Compliant crawlers follow it; scrapers can ignore it. To actually block traffic, use your firewall or CDN bot rules.
- Watch your firewall too. CDN and bot-protection settings can block AI crawlers you meant to allow; check both.
- User agents can be faked. Verify real bots with the published IP ranges or reverse DNS before trusting a log line.
- It does not control what is already in a model or what reaches engines through search indexes and licensing.
- llms.txt is a different file with no access control at all. In our data, none of 13,772 URLs AI engines retrieved was an llms.txt file; see our llms.txt guide.
How to Check Your Setup
- Open
yourdomain.com/robots.txtand list every rule that applies to the agents above, includingUser-agent: *. - Check your server or CDN logs for AI user agents and the status codes they receive. Our guide to log file analysis for SEO and AI crawlers shows how to verify genuine bots.
- Check firewall and bot-management settings for AI crawler blocks.
- Run an AI readiness check to confirm crawlers can reach and read your key pages; see technical AI search optimization for what else to look at.
Check Whether AI Crawlers Can Reach Your Site
Run the free AEO checker to see if your robots.txt and pages are ready for AI engines.
Run the free checkerFrequently Asked Questions
What AI crawlers should I block in robots.txt?
It depends on your goal. To stay out of model training but remain citable in AI answers, block training crawlers such as GPTBot, ClaudeBot and CCBot (and the Google-Extended and Applebot-Extended tokens) and allow search crawlers such as OAI-SearchBot, Claude-SearchBot and PerplexityBot. To maximise AI visibility, allow them all.
Does blocking GPTBot remove my site from ChatGPT?
Not necessarily. GPTBot is OpenAI's training crawler. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages users ask about. Blocking GPTBot keeps content out of training; blocking OAI-SearchBot is what can keep pages out of ChatGPT search results.
Does Google-Extended affect AI Overviews or rankings?
Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls whether content may be used to train and ground Gemini models. Google's AI features in Search draw on Googlebot's crawl.
Do most websites block AI crawlers?
Not most. In GetCited platform data, 22 of the 86 most-cited websites we could check (26%) block at least one AI crawler from the whole site. The most blocked were Bytespider, CCBot, ClaudeBot and GPTBot.
If I block AI crawlers, will AI stop mentioning my brand?
Not necessarily. Reddit blocks these crawlers yet was the most-retrieved domain in our data, because content also reaches AI engines through search indexes and licensing deals, and models may already know your brand from earlier training.
How do I check if AI crawlers are visiting my site?
Look for AI user agents in your server or CDN logs, check the status codes they get, and verify that they are genuine using the IP ranges or reverse DNS each company publishes.