Technical AEO: How to Make Your Site Readable by AI Engines

Technical AEO decides whether AI engines can reach, render and understand your site before any question of citation even comes up. If GPTBot can't fetch a page, if content only appears after JavaScript runs, or if structured data is missing or inconsistent, nothing else about your content or authority matters, because the crawler never gets far enough to read it.

This guide covers every layer of that gate: which AI crawlers actually visit sites, how to write robots.txt rules that allow or block them deliberately, why JavaScript rendering trips up more sites than teams expect, what structured data does and does not do, what llms.txt is actually for, how to structure pages so machines can parse them, and how to confirm crawl health through status codes, canonicals, sitemaps and server logs.

Treat this as the resource you keep open while you audit your own site. If you want the definitional grounding first, start with what AEO means and why it differs from SEO; if you want the broader context for why AI crawlability matters now, read how AI engines are reshaping search. Every section below ends with something concrete to check on your own domain this week.

What Technical AEO Is

Technical AEO means making pages reachable, renderable and understandable for AI engines. It covers the mechanics that sit underneath everything else: can a crawler access the URL, can it see the content without running a browser, and can it extract a clear structure from the HTML. This is distinct from writing good answers or building topical authority, which matter only once the technical layer works.

The term overlaps heavily with technical SEO, and the overlap is not an accident. Clean status codes, working sitemaps, fast page loads and sensible canonical tags all help both Googlebot and newer AI crawlers. Teams that already have solid technical SEO have a head start. However, the two disciplines diverge in specific places, and treating them as identical is where most sites get into trouble.

The divergence shows up in three areas. First, many AI crawlers do not render JavaScript the way Googlebot increasingly does, so content that depends on client-side rendering can be invisible to them even when it ranks fine in Google. Second, a new file format, llms.txt, has emerged specifically for AI systems and has no equivalent in traditional SEO.

Third, the crawler landscape itself is different and growing: GPTBot, ClaudeBot, PerplexityBot and others each have their own fetching behavior, access patterns and purposes, and robots.txt rules have to account for all of them individually rather than assuming one dominant crawler.

It helps to be blunt about what technical AEO does and does not achieve. Technical AEO is a gate, not a prize. Getting every technical box checked does not earn a citation in an AI answer. It earns the right to be considered. A technically flawless page with thin, generic content still will not get cited, because AI engines are selecting content on substance and trust signals once they can actually read it.

This framing changes where teams should spend their first hours: before rewriting content, confirm that AI crawlers can reach, render and parse the pages that matter most.

Which AI Crawlers Visit Your Site

Understanding technical AEO requires naming the actual crawlers involved, because generic advice to "make sure bots can crawl you" is meaningless without knowing which bots and what they each do. The current landscape includes GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Bingbot and Googlebot, and each behaves differently.

GPTBot is OpenAI's crawler used to gather training data for its models. OAI-SearchBot is a separate crawler OpenAI uses specifically to retrieve content for ChatGPT's search and browsing features, meaning content it indexes can surface directly in a conversational answer. ChatGPT-User is different again: it fires when a live user inside a ChatGPT session triggers a real-time fetch of a specific page, rather than during background crawling.

Anthropic runs a similar split. ClaudeBot crawls broadly, largely for training purposes, while Claude-SearchBot is the crawler tied to retrieval for live Claude responses. Perplexity runs PerplexityBot, which fetches content to ground its answers in named sources. Bingbot and Googlebot remain active as well, and Google increasingly feeds crawled content into AI Overviews and AI Mode, meaning traditional search crawling now has downstream AI visibility consequences too.

For Microsoft's assistant, see how to get cited on Microsoft Copilot.

It helps to group these into three functional categories rather than treating every bot as interchangeable:

• Training crawlers: fetch content to improve model knowledge over time, with no immediate citation event tied to a single visit (GPTBot, ClaudeBot in its broad crawling mode)
• Search or retrieval crawlers: fetch content specifically to ground a live or near-live answer, making their access directly tied to citation opportunities (OAI-SearchBot, Claude-SearchBot, PerplexityBot)
• User-triggered fetchers: activate only when a real user's query causes a real-time page request inside the AI product itself (ChatGPT-User)

A common point of confusion is Google-Extended and Applebot-Extended. Neither is a crawler. Both are robots.txt tokens that control whether content already fetched by Googlebot or Applebot may be used for those companies' AI models. Disallowing Google-Extended does not stop Googlebot from crawling or indexing your site. The expensive mistake is the opposite one: blocking Googlebot itself while trying to block AI, which takes the site out of search and out of Google's AI features with it.

Knowing which crawler is which matters because blocking rules need to be deliberate. A site that wants visibility in Perplexity answers but wants to restrict AI training use needs rules that treat PerplexityBot differently from GPTBot, and a blanket "block all bots starting with a capital letter" rule from a security tool can quietly take out all of them at once.

robots.txt Rules for AI Crawlers

Once the crawler landscape is clear, the next decision is what robots.txt should actually say. The file sits at the root of a domain and tells each named user agent what it may and may not fetch. For AI visibility, the goal is usually to allow the crawlers tied to citation and discovery while making deliberate, informed choices about training crawlers.

For the full list of AI crawlers, what each does and rules you can copy, see AI crawlers and robots.txt.

A basic allow rule for a search-oriented AI crawler looks like this:

User-agent: PerplexityBot
Allow: /

A site that wants to opt out of training use while still allowing retrieval crawling for live answers would instead target the broad training crawler specifically:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

This distinction, blocking the training crawler while allowing the search crawler from the same company, is exactly the kind of deliberate rule that generic SEO-era robots.txt files never anticipated, because the split between training and retrieval crawlers from a single vendor did not exist in the Googlebot-only era. For a deeper walkthrough of syntax, ordering rules and common mistakes, robots.txt optimization covers the file format in more detail than fits here.

The more damaging problems, however, rarely come from the robots.txt file itself. They come from infrastructure sitting in front of it. Firewalls, CDNs and bot-protection services frequently block AI crawlers by accident, and the site owner never sees it happen because the block occurs before the request even reaches the point where robots.txt is evaluated. A security vendor's default ruleset might classify an unfamiliar user agent as a scraper and serve it a CAPTCHA or a hard block, regardless of what the site's own robots.txt permits.

This happens more often with AI crawlers than with Googlebot because bot-protection vendors have had decades to build allowlists for search engines, while GPTBot, ClaudeBot and PerplexityBot are comparatively new entrants. Consequently, a default-configured firewall can be fully compliant with a permissive robots.txt file while still blocking every AI crawler at the network layer. Checking firewall and CDN logs for 403 responses against these specific user agent strings is therefore a separate and necessary step from reading the robots.txt file itself.

A short list of places this accidental blocking tends to hide:

1. CDN-level bot management rules configured before the robots.txt file was ever updated for AI crawlers
2. Rate-limiting rules that treat rapid AI crawler requests as abusive traffic
3. Web application firewalls with default "block unknown bots" settings left untouched since before 2023
4. Hosting-provider security add-ons that are enabled by default and rarely reviewed

The practical fix is to treat robots.txt and infrastructure-level bot rules as two separate checks, not one. Getting the file right is necessary but not sufficient. A full technical review needs to confirm, at the server or CDN log level, that the crawlers the robots.txt file allows are actually reaching the origin server rather than being silently dropped upstream.

JavaScript Rendering and AI Engines

Most technical AEO failures trace back to one problem: the crawler never saw the content at all. A page can look complete in Chrome and still be an empty shell to an AI crawler, because what you see in a browser is not what many bots receive. This gap between rendered and raw output is one of the most common reasons content never gets cited.

Raw HTML vs Rendered Page

When a browser loads your page, it downloads the raw HTML, then runs any JavaScript to build out the final content you see on screen. That final version is the rendered page. The raw HTML, by contrast, is what arrives before any scripts execute, and for many JavaScript-heavy sites built on frameworks like React or Vue, the raw HTML is close to blank: a shell div, a handful of script tags, and little else.

The distinction matters because what an AI crawler can use depends on which version of the page it actually reads. AI crawlers behave inconsistently here, and treating every bot as equivalent to Googlebot is a habit carried over from SEO.

Many AI Crawlers Fetch HTML Without Running JavaScript

This is the core fact teams miss: a meaningful share of AI crawlers do not execute JavaScript at all. They request the URL, read the raw HTML response, and move on. If your product descriptions, FAQ answers, or article body copy only appear after a client-side script runs, those crawlers see nothing where your most citable content lives.

This is a direct departure from how teams have handled Googlebot for years. Google has invested heavily in rendering JavaScript at scale, so SEO teams have grown comfortable deploying JavaScript-heavy sites without a fallback. AI crawlers, however, have not made the same investment, and assuming they have is one of the most expensive blind spots in technical AEO. The fix is not to abandon JavaScript frameworks; it is to confirm server-side rendering or static generation delivers real content in the initial HTML payload, rather than leaving it to execute client-side.

How to Test What a Crawler Sees

You do not need to guess. Fetch the raw HTML the same way a bot would, bypassing your browser's rendering engine entirely:

• Use curl or a similar command-line tool to request the URL and inspect the returned HTML directly.
• Disable JavaScript in a browser dev tools panel and reload the page to see what remains.
• Compare the raw response against the rendered DOM view to spot gaps in body content, headings, or key data points.
• Check whether critical text sits inside a noscript fallback or is genuinely absent from the HTML entirely.

For a deeper breakdown of specific tools that automate this comparison, see this guide on JavaScript SEO tools. Running a full technical AEO audit tool also surfaces these rendering gaps automatically across every page template on a site, rather than page by page.

Structured Data That Helps AI Understand Your Pages

Structured data does not make content more persuasive or more accurate. It makes content easier to parse correctly, which is a different job entirely. Schema markup gives AI engines explicit, machine-readable labels for what a piece of content is, who published it, and how its parts relate to each other.

What Schema Does and Does Not Do

Schema is a translation layer, not a ranking signal. Alongside speed and site structure, it is one of the technical foundations that decide whether AI systems can parse a page correctly in the first place.

Adding schema to a page does not compensate for thin content, and it does not guarantee a citation. What it does is remove ambiguity. Without schema, a crawler has to infer that a block of text is a product description, an FAQ answer, or an author byline, purely from context. With schema, that inference is replaced with an explicit label, which matters more for AI engines synthesizing answers than it ever did for traditional ranking algorithms.

The Schema Types That Matter Most for AEO

A handful of schema types do most of the work for AI citability:

• Organization, with a sameAs property linking to verified profiles (social accounts, Wikipedia, Crunchbase), helps establish entity identity across the web.
• Article, for blog posts and editorial content, marking up headline, author, and publish date.
• FAQPage, for question-and-answer content, which maps directly onto how AI engines often extract direct-answer blocks.
• Product, for ecommerce pages, specifying price, availability, and reviews in a consistent format.
• BreadcrumbList, which clarifies page hierarchy and helps a crawler understand where a page sits within a site's structure.

Organization schema with sameAs helps disambiguate your brand from similarly named entities. FAQPage schema, consequently, increases the odds that a clean, quotable answer block gets extracted cleanly, rather than pulled from the middle of a longer paragraph.

For exact JSON-LD examples and field-by-field requirements, see our schema markup checklist for AI engines and the broader guide to structured data. Structured data is also one of the 300+ automated checks inside a full technical AEO audit, where missing or inconsistent schema across page templates gets flagged alongside crawlability and rendering issues.

llms.txt: What It Is and What It Isn't

Few topics in technical AEO generate as much confusion as llms.txt. It has been discussed widely since 2024 as a proposed standard for guiding AI systems, and the confusion mostly comes from people assuming it works like robots.txt. It does not.

For what our data shows about whether AI engines actually use the file, see llms.txt: what it is, does it work, and do you need one.

What the File Is

llms.txt is a plain text file, placed at the root of a domain, intended to give AI systems a curated summary of a site: what it is, which pages matter most, and links to key content in a clean, structured format. The idea is to hand language models a concise map of a site rather than leaving them to infer structure from a full crawl.

In practice, this means a short Markdown-style file listing the site's purpose, its most important sections, and direct links to core pages, written for a model to read quickly rather than for a human visitor to browse. Our free llms.txt generator drafts one from your sitemap.

Why It Is Not a Ranking Switch

This is the critical correction most coverage of llms.txt gets wrong. Unlike robots.txt, there is no confirmed, universal mechanism guaranteeing that any major AI crawler reads llms.txt, respects it, or weights it when deciding what to cite. Adding the file does not function like flipping a visibility switch, and nothing about this pillar's framing should be read as a promise that it increases citation odds.

Treat llms.txt as a courtesy document, not infrastructure. It costs little to add and causes no harm, but it is not a substitute for the fundamentals covered elsewhere in this guide: crawlable robots.txt rules, renderable HTML, consistent schema, and clean page structure. A site with excellent llms.txt and broken JavaScript rendering will still fail to get cited, because llms.txt cannot fix a crawler that never reaches your content.

When It Is Worth Adding

llms.txt earns its place on a technical AEO checklist in specific situations:

1. Large sites with complex navigation, where a concise summary genuinely reduces ambiguity about which pages matter most.
2. Sites undergoing frequent structural changes, where a maintained llms.txt file gives a stable reference point.
3. Teams that have already resolved crawlability, rendering, and schema issues and are looking for a low-cost, low-risk addition.

For a startup or small site with a handful of pages, the file is close to unnecessary. The content hierarchy is already obvious from a normal crawl, and the time is better spent fixing a firewall rule blocking GPTBot or auditing JavaScript rendering across templates. A free AEO scanner can confirm whether the fundamentals are solid before llms.txt is worth prioritizing at all.

The honest position, and the one this pillar holds throughout, is that llms.txt is a minor, optional layer. The gate items, crawler access, rendering, structured data, and page structure, do the actual work.

Page Structure Machines Can Parse

AI engines do not read a page the way a human does. They parse it. That means structure carries meaning, not just style, and a page that looks fine visually can still confuse a crawler if the underlying markup is a mess. Technical AEO treats headings, semantic HTML and answer order as functional elements, not decoration. Get this wrong and the content underneath may never surface in an AI answer, no matter how good the writing is.

One Clear H1, Logical Headings

Every page should carry a single H1 that states the page topic in plain language. Multiple H1 tags, or none at all, force a crawler to guess what the page is actually about. Headings below that should follow a logical order: H2 for major sections, H3 for subsections, never skipping a level to jump from H2 straight to H4.

This is the same discipline covered in technical SEO work, and it matters more, not less, for AI parsing, since these systems lean on heading hierarchy to build an outline of the page before they decide what to cite.

Messy heading structure is one of the most common findings in a technical AEO review. Pages that bounce between heading levels, reuse the same H2 text twice, or bury the real topic three sections down tend to get parsed incorrectly even when a human reader would have no trouble following the page.

Semantic HTML Over Div Soup

Semantic tags such as article, section, nav and table tell a machine what a block of content is for. A page built entirely from generic div and span tags gives a crawler no shortcuts. It has to infer structure from scratch, which raises the odds of misreading the page or missing content entirely. This overlaps directly with the fundamentals covered in html and css best practice, and it is worth treating as a baseline requirement rather than a nice-to-have.

Answer-First Blocks, Lists and Tables

AI engines tend to favor content that states the answer early and plainly, then supports it. A page that opens with three paragraphs of preamble before reaching the actual answer asks more of the crawler than one that leads with the point. Lists and tables help here too, since they present discrete, parseable facts rather than a single dense paragraph a model has to unpack.

Structuring a page to answer first does not mean stripping out nuance. It means putting the direct answer in the first sentence or two of a section, and using the paragraphs that follow to add context, caveats and detail. In addition, breaking out comparisons, steps or criteria into an actual list or table, rather than describing them in prose, gives the crawler a cleaner object to extract and cite.

Crawl Health: Status Codes, Canonicals, Sitemaps and Speed

A page can have perfect structure and still never reach an AI engine's index if crawl health is broken underneath it. Status codes, canonical tags, sitemaps and page speed are the plumbing of technical AEO, and plumbing failures are invisible until something backs up. Most of the issues in this category are mundane rather than exotic, which is exactly why they get missed.

Clean 200s, No Redirect Chains

Every important page should return a clean 200 status code on direct request. Redirect chains, where one URL bounces through two or three hops before landing on the final page, waste crawl budget and introduce failure points. Some crawlers follow redirects reliably; others cap how many hops they will follow before giving up. A single unnecessary redirect might cost nothing. However, a chain of three or four, especially stacked across thousands of URLs, starts to meaningfully affect what gets crawled, a concept covered in more depth under crawl budget optimization.

Consistent Canonicals

Canonical tags tell a crawler which version of a page is the authoritative one when duplicate or near-duplicate URLs exist. Inconsistent canonicals, where a page points to itself in some templates and to a different URL in others, send mixed signals. The safest approach is a single, consistent canonical rule applied site-wide by template, not patched page by page after the fact.

XML Sitemaps With Real URLs

A sitemap is only useful if it reflects reality. Sitemaps that still list redirected, deleted or noindexed URLs actively mislead crawlers, sending them toward dead ends instead of live content. The fix is straightforward: regenerate sitemaps on a schedule tied to publishing, strip out anything that does not return a clean 200, and keep the file current rather than static. The mechanics of this are covered in XML sitemaps optimization.

Speed and Core Web Vitals in Brief

Page speed affects crawl health in two ways. Slow servers mean crawlers complete fewer page fetches within whatever time or resource budget they allocate to a site, and some crawlers deprioritize domains that respond slowly on a consistent basis. Core Web Vitals are a useful proxy for this, even though AI crawlers are not identical to Googlebot in how they weigh speed.

The practical guidance lives in web speed optimization tools, and the short version is that a server response under a few hundred milliseconds, with no render-blocking resources stacked in front of core content, gives every crawler an easier job.

Checking Whether AI Crawlers Actually Visit

Everything covered so far is theory until it is checked against real server data. The reliable way to know whether GPTBot, ClaudeBot, PerplexityBot or any other AI crawler is actually reaching a site is to look at server logs directly. Analytics tools built on JavaScript tracking will not show this, since many AI crawlers do not execute JavaScript and therefore never fire a tracking pixel.

Reading Server Logs for AI User Agents

Server logs record every request made to a server, including the user agent string that identifies what made the request. Filtering logs for known AI crawler user agents, such as GPTBot, ClaudeBot or PerplexityBot, shows exactly which pages those crawlers have requested and how often. This is the single most direct way to confirm crawler access, and it sidesteps every assumption built into robots.txt rules or structured data markup. The full methodology is laid out in logfile analysis for SEO, and the same approach applies directly to AI crawler traffic.

A log review typically surfaces one of three patterns: the crawler visits key pages regularly, the crawler visits the site but skips important sections, or the crawler never appears in the logs at all despite robots.txt allowing it. Each pattern points to a different fix, from content prioritization through to checking firewall rules that might be silently blocking requests before they ever reach the application layer.

Verifying Genuine Bots

User agent strings can be spoofed. Anyone can send a request claiming to be GPTBot without actually being GPTBot, which matters if a site is making access decisions based on user agent alone. Verifying that a request is a genuine AI crawler, rather than a scraper impersonating one, typically involves checking the IP address or range the request came from against published ranges, where the crawler operator provides them, or doing a reverse DNS lookup to confirm the requesting server belongs to the crawler operator's infrastructure.

This verification step matters most for sites making decisions that depend on crawler identity, such as serving different content to a verified bot versus blocking an unverified one. For most site owners the simpler, higher-value move is consistent: pull logs regularly, check for the presence of real AI crawler activity, and treat an absence of expected crawler visits as a signal to investigate robots.txt, firewall rules or redirect chains rather than an isolated one-off.

Crawler visits are not citations. Once crawlers reach your pages, whether AI engines actually name and cite you is a separate measurement, covered in our guide to AI visibility tracking.

A Technical AEO Checklist

Everything covered so far in this guide collapses into a single working list. Technical AEO is not a philosophy, it is a set of checks you can run against any page in under an hour. The table below groups those checks by category, explains why each one matters for AI crawlers specifically, and tells you how to test it yourself without waiting on a vendor report.

You do not need hundreds of checks to start. The 15 below catch the mundane failures described earlier in this guide: a firewall rule blocking GPTBot, content that only renders after JavaScript, a canonical pointing at the wrong URL.

The Core Checklist Table

How To Run This Checklist This Week

Do not try to fix everything on the first pass. Work in the order the table is built: access first, rendering second, structure third, and verification last. A page that blocks GPTBot in robots.txt gains nothing from perfect schema, so fixing access issues always comes before polishing structure.

Start with your five highest-traffic or highest-intent pages rather than the whole site. Run each one through the table manually, or run a technical AEO audit with 300+ automated checks if you want every check automated and prioritized by impact. Document what you find before changing anything, since you will want a clear before state to compare against once fixes ship.

Once access and rendering checks pass, move to structure: headings, answer-first blocks, and schema. These are the checks competitors' generic SEO-ported checklists treat as cosmetic, but for AI engines they directly affect whether a parser can isolate a citable passage. Last, confirm with server logs that the crawlers you unblocked are actually showing up, because permission in robots.txt is not proof of a visit.

Keep reading: Schema Markup for AI Engines, Technical SEO & AEO Audit product page, What Is AEO.

Frequently Asked Questions

What is technical AEO?

Technical AEO is the set of practices that make a website reachable, renderable and parseable by AI crawlers such as GPTBot, ClaudeBot and PerplexityBot. It covers robots.txt rules, JavaScript rendering, structured data, page structure and crawl health. Technical AEO does not guarantee citation in AI answers; it removes the technical barriers that would otherwise stop an AI engine from considering a page at all. Content quality and authority still decide whether a page gets cited once it's reachable.

How is technical AEO different from technical SEO?

Technical SEO and technical AEO overlap heavily, since both care about crawlability, site speed and clean HTML. The difference shows up in the details: technical SEO was built around Googlebot's behavior, while technical AEO accounts for crawlers that often skip JavaScript rendering entirely, that respect separate robots.txt tokens like Google-Extended, and that fetch content for different purposes, from AI training to real-time search. A site tuned only for Googlebot can still be invisible to several AI crawlers.

How do I check if AI crawlers are visiting my site?

Check your server logs for user-agent strings matching GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and similar names, then confirm the requesting IP ranges actually belong to the crawler rather than a spoofed agent. Log file analysis tools can filter this automatically across large sites. If none of these user agents appear in your logs over a period of weeks, that's a strong signal something in your robots.txt, firewall or CDN configuration is blocking access before the crawler ever reaches your pages.

Does blocking AI crawlers in robots.txt actually stop them?

Robots.txt is a voluntary instruction, not an enforcement mechanism, so a well-behaved crawler like GPTBot or ClaudeBot will respect a disallow rule and simply not fetch the page. Badly behaved or unofficial bots can ignore it entirely. For reliable blocking you still need robots.txt as the first layer, since it's what legitimate AI crawlers check, but firewall or CDN-level rules are the backstop for anything that doesn't comply voluntarily.

Is llms.txt required for AI visibility?

No AI engine has confirmed that llms.txt affects crawling, indexing or citation behavior, so it is not a required file and should not be treated as a ranking lever. It functions more like a curated map of your most important pages for any system that chooses to read it. Adding one is low-risk and occasionally useful for organizing a large site, but skipping it will not hurt your technical AEO if the fundamentals, like crawl access and rendering, are already solid.

How long does a technical AEO fix take to show results?

Timelines vary by the type of fix and the crawl frequency of the AI engines involved, since some crawlers revisit sites daily while others take longer between passes. A robots.txt correction or a firewall rule fix can take effect as soon as the crawler's next scheduled visit, while structured data or rendering fixes may need a few crawl cycles before the content is fully reprocessed. Checking server logs after a fix is the most direct way to confirm a crawler returned and successfully fetched the corrected page.

What's the difference between JavaScript rendering problems and a crawl block?

A crawl block means the crawler is denied access entirely, through robots.txt, a firewall rule or a bot-protection service, and it never reaches the page. A JavaScript rendering problem means the crawler does reach the page but only sees the raw HTML, which may be empty or incomplete if the actual content is injected by a script after the page loads. Both result in the AI engine missing your content, but diagnosing them requires different tools: log analysis for blocks, and rendered-versus-raw HTML comparison for rendering issues.

Should I run a technical AEO audit before fixing individual issues?

Running a structured audit first is generally more efficient than fixing issues one at a time as you find them, because many technical AEO problems share a root cause, like a single firewall rule blocking multiple crawlers across the whole site. A comprehensive check across crawl access, rendering, structured data and crawl health, such as a technical AEO audit with 300+ automated checks, surfaces the full list of issues at once so you can prioritize by impact rather than guessing which fix matters most.