Robots.txt

Robots.txt is one of the oldest and simplest tools in SEO, and also one of the most quietly dangerous, because its simplicity hides a distinction that trips up almost everyone at some point. It is a plain text file, a few lines long, sitting at the root of your site, and it does exactly one kind of thing: it tells crawlers where they may and may not go. That sounds harmless, and used correctly it is genuinely useful. But the gap between what people think it does and what it actually does is wide enough that a single misplaced line can either accomplish nothing you intended or, in the worst case, wipe your entire site out of the crawl. It is worth understanding properly precisely because it looks too simple to bother with.

What it does

The genuine job of robots.txt is crawl management: telling well-behaved crawlers which areas of your site to stay out of, so they spend their effort where it counts. When a major search engine arrives at your site, it checks the robots.txt file first, reads your instructions, and, being a courteous visitor, follows them. If you have asked it not to crawl a certain section, it will not crawl that section. That is the whole mechanic, and for the search engines that matter it works reliably, because they choose to respect the standard.

This makes robots.txt a useful lever for steering crawler effort, especially on larger sites. You can keep crawlers away from great swathes of low-value or repetitive URLs so their limited attention flows to the pages you actually care about, which ties directly to crawl budget. You can also use the file to point crawlers to your sitemap, a small courtesy that helps them find your directory of important pages. Within its proper lane, crawl management, robots.txt is a sharp and handy tool. The trouble only begins when people ask it to do a job it was never built for.

Disallow is not noindex

Here is the single most important thing on this page, and the misunderstanding responsible for a huge share of robots.txt disasters: robots.txt controls crawling, not indexing. Those are two different things, and the difference is not a technicality; it is the whole game. Crawling is a search engine fetching and reading a page. Indexing is a search engine deciding to include a page in its results. Robots.txt only touches the first. It can stop a crawler from reading a page, but it cannot, by itself, stop that page from appearing in search.

This produces a genuinely counterintuitive result that catches people out constantly. If you disallow a page in robots.txt to keep it out of Google, you can end up with the opposite of what you wanted: the page still gets indexed, for example because other pages link to it, but because you blocked crawling, the search engine never read its content, so it shows up in the results as a bare, contentless listing. You have made the page worse, not hidden it. The correct tool for keeping a page out of the index is a noindex instruction on the page itself, which requires the page to be crawlable so the instruction can be seen. Blocking a page in robots.txt and adding noindex to it are, maddeningly, in conflict: if the crawler cannot reach the page, it cannot read the noindex. Getting this distinction right is most of what it takes to use robots.txt safely.

What it is good for

Used in its proper lane, robots.txt earns its place. The best use is keeping crawlers out of low-value or effectively infinite areas that would otherwise soak up crawling for no benefit: internal search result pages, the endless URL sprawl generated by faceted navigation and filters, and system or admin paths that have no business in the crawl at all. On a large site, steering crawlers away from these can meaningfully protect the crawl budget that should be spent on your real pages. Pointing crawlers to your sitemap through the file is another small, worthwhile use.

The common thread is that all of these are crawl-management jobs, keeping crawlers away from parts of the site that genuinely do not need crawling. When the goal is "do not waste crawling here," robots.txt is exactly right. When the goal is "keep this page out of search" or "keep this content private," it is exactly the wrong tool, and reaching for it there is how the disasters happen.

What not to block

The cautions are as important as the uses. Do not use robots.txt to hide sensitive or private information. The file itself is public, anyone can read your robots.txt, so disallowing a path can actually advertise the existence of the very thing you were trying to conceal, and it provides no security whatsoever. Genuinely private content needs real access control, not a polite note. Do not block the resources a page needs to render, such as the CSS and JavaScript files, because a search engine that cannot fetch those cannot see the page properly, which can quietly wreck how your pages are understood. And do not use it to try to deindex pages, for all the reasons above. Treat robots.txt as a scalpel for crawl management and nothing else, and you will avoid essentially all of its failure modes.

Here is how the topic sits in US search data.

The bare term is heavily contested by official documentation and tool pages, so the sensible openings are the qualified, practical variants, the seo angle and the disallow question, where the intent is clearer and the competition softer. A page that explains the crawling-versus-indexing distinction genuinely well has real value here, because that is exactly what most searchers are quietly confused about.

How it works, briefly

The mechanics are refreshingly simple, which is part of why the file is so easy to misuse without noticing. A robots.txt file is a list of rules, each aimed at a named crawler, the user agent, followed by instructions about which paths that crawler may or may not access. In plain terms, you can say "this particular crawler is asked not to visit these paths," or "all crawlers are asked to stay out of this section," or the reverse, explicitly allowing something within a broader block. You can also list where your sitemap lives so crawlers can find it.

Because the syntax is so terse, a tiny error has an outsized blast radius. The most infamous mistake in all of technical SEO is a single stray line that accidentally tells every crawler to stay out of the entire site, which, if it slips into production, can quietly remove a whole site from search. This is why any change to robots.txt deserves careful review and testing before it goes live, and why it is worth checking the live file periodically to confirm it still says what you think it says. A file this small should never be edited casually, precisely because it is small enough to feel harmless.

Robots.txt and AI crawlers

Robots.txt has taken on a prominent new role in the AI era, because it is the main polite mechanism for telling AI crawlers whether they may access your site. Just as you can name a search engine's crawler and ask it to stay out, you can name the crawlers that gather content for AI systems and disallow them, and the well-behaved ones will comply, just as courteous search crawlers always have. This has turned a sleepy old file into the front line of a real strategic question.

That question deserves a deliberate answer rather than an accidental one. Blocking AI crawlers gives you more control over how your content is used, but it also removes you from the AI answers those crawlers feed, which is an increasingly important place to be visible. There is no single right choice; it depends on what your content is for and how you want it used. The point is simply that this is now a genuine decision, made through robots.txt, and worth making on purpose. The same cautions apply as ever: it is a polite request that only the well-behaved obey, and it manages access, not indexing.

Mistakes to avoid

The classic errors are all versions of misunderstanding the tool.

Using disallow to deindex a page, and ending up with a bare, contentless listing instead of a hidden page.
Blocking a page and adding noindex to it, so the crawler can never read the noindex you rely on.
Trying to hide sensitive content in a public file that provides no security at all.
Blocking CSS and JavaScript, so the search engine cannot properly render your pages.
Editing the file casually, and letting a single stray line quietly remove the whole site from search.