JavaScript SEO Tools: How to Test What Google and AI Crawlers See

The rendering problem in three paragraphs

A server returns raw HTML: the response body you see with View Source or curl. The browser then runs JavaScript, which can add, remove or rewrite content, links, titles and meta tags. The result is the rendered DOM, which is what you see in the Elements panel. On a server-rendered site the two are nearly identical. On a client-rendered single-page app, the raw HTML can be an empty <div id="root"> and a script tag.

Google handles this in two passes. Googlebot fetches and processes the raw HTML first, then queues the page for rendering in its Web Rendering Service, which runs an evergreen version of Chromium. Google's documentation says the page may wait in that queue for a few seconds or longer. Links and content found only after rendering are discovered later than those in the raw HTML, and anything that fails during rendering (a blocked script, a timeout, an API error) is never seen.

Most AI crawlers do not render at all. They read the raw HTML and move on. If your product description, prices or FAQ only appear after JavaScript runs, ChatGPT, Claude and Perplexity are likely working from an empty page. That makes the raw HTML check below the most important test for AI visibility, and it is the one Google's own tools do not show by default.

For the background on how to build JavaScript sites that avoid these problems, see our guides on JavaScript for SEO and HTML and CSS for SEO. This page is about testing.

1. Google Search Console URL Inspection

The closest view you can get to what Google itself rendered. It only works for properties you have verified.

How to use it: paste the URL into the inspection bar at the top of Search Console. The first screen shows the indexed version: whether the URL is on Google, the canonical Google selected and the last crawl date. Click Test live URL, wait for the test, then View tested page. You get three tabs:

HTML is the rendered HTML after JavaScript ran. Search it (click into the panel and use Ctrl+F or Cmd+F) for a sentence from your main content, your canonical tag and your meta robots tag.
Screenshot shows the rendered page above the fold. A blank or half-built screenshot means rendering failed or ran out of time.
More info lists page resources that could not be loaded (often scripts blocked by robots.txt or failing API calls) and JavaScript console messages, including errors.

What it does not show: the raw HTML before rendering. Pair it with View Source or curl.

2. Rich Results Test

Google's public testing tool at search.google.com/test/rich-results renders the page with Googlebot and shows the same rendered HTML and screenshot as URL Inspection, but for any URL, with no Search Console access needed. That makes it the tool to use on a competitor's site, a staging URL that is publicly reachable, or a client site before you have access.

Choose Googlebot smartphone or desktop, run the test, then open View tested page for the HTML, screenshot and resource errors. It also validates structured data, which matters on JavaScript sites because schema injected by scripts or a tag manager only exists in the rendered DOM. Google retired the standalone Mobile-Friendly Test in December 2023, so this is now the public rendering test.

3. Chrome DevTools

Free, already installed, and fast enough to use on every page you look at.

Disable JavaScript. Open DevTools (F12 or Cmd+Option+I), press Ctrl+Shift+P or Cmd+Shift+P to open the Command Menu, type Disable JavaScript and press Enter, then reload. What remains is roughly what a non-rendering crawler gets. Turn it back on the same way.

Compare View Source with Elements. view-source:https://example.com/page shows the raw HTML. The Elements panel shows the live DOM after scripts. Search both for your H1, the first sentence of body copy, rel="canonical" and name="robots". Anything present in Elements but missing from View Source depends on JavaScript.

Network panel. Filter to Fetch/XHR to see which API calls supply your content. If the main content arrives from an API that is slow, rate-limited or blocked by robots.txt, Googlebot may render the page without it.

Network conditions. In the Network conditions drawer you can untick "Use browser default" and set the user agent to Googlebot, which exposes sites that serve crawlers a different version.

A free extension such as View Rendered Source puts raw HTML, rendered HTML and a line-by-line diff side by side, which saves switching between tabs.

4. Screaming Frog SEO Spider (JavaScript rendering mode)

Single-page tools tell you about one URL. A crawler in rendering mode tells you which templates across the site depend on JavaScript.

How to set it up: go to Configuration → Spider → Rendering and set Rendering to JavaScript. Under Configuration → Spider → Extraction, tick Store HTML and Store Rendered HTML. Set the window size to a mobile device to mirror Googlebot smartphone. Crawl as usual.

What to read: the JavaScript tab lists pages where content, links, titles, meta descriptions or canonicals only appear or change after rendering, with filters such as Contains JavaScript Links and Contains JavaScript Content. In the lower pane, View Source shows the raw and rendered HTML side by side, and the Rendered Page tab shows a screenshot. Running one crawl with rendering off and one with it on, then comparing word counts and link counts, is a quick way to size the problem.

JavaScript rendering is part of the paid licence; the free version crawls up to 500 URLs without it. Rendering crawls are slower and heavier, so start with a sample of each template.

5. Sitebulb

Sitebulb is a paid desktop and cloud crawler that can render with Chromium. Choose the JavaScript-rendering crawler when you set up the audit. Its Response vs Render report compares the raw response with the rendered DOM for every page and flags what changed: title, meta description, canonical, meta robots, internal links and content. That report is the clearest site-wide answer to "what depends on JavaScript here", and it is easier to hand to a developer than a raw crawl export.

6. The curl vs headless Chrome diff

The most direct test, and the one that matches what AI crawlers see. Fetch the raw HTML with curl, render the same URL in headless Chrome, and compare.

# Raw HTML, as a non-rendering crawler gets it
curl -sL -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  https://example.com/page > raw.html

Then render with Playwright (npm i playwright, then npx playwright install chromium):

// render.js: node render.js https://example.com/page
const { chromium } = require('playwright');
const fs = require('fs');

(async () => {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  const res = await page.goto(process.argv[2], { waitUntil: 'networkidle' });
  fs.writeFileSync('rendered.html', await page.content());
  console.log('HTTP status:', res.status());
  await browser.close();
})();

Now compare the things that matter for SEO:

for f in raw.html rendered.html; do
  echo "== $f"
  grep -o '<title>[^<]*' "$f"
  grep -oiE '<link[^>]*rel="canonical"[^>]*>' "$f"
  grep -oiE '<meta[^>]*name="robots"[^>]*>' "$f"
  echo "links: $(grep -oi '<a [^>]*href=' "$f" | wc -l)"
  echo "words: $(sed -e 's/<[^>]*>//g' "$f" | wc -w)"
done

Read the output this way. If the title, canonical and robots tags match and the raw word count is close to the rendered count, the page is safe for every crawler. If the raw file has a fraction of the words or links, Google can probably still index it after rendering, but non-rendering crawlers cannot. If the title or robots tag differ between the two, decide which one is correct and make the raw HTML say it. Wrap the three snippets in a loop over one URL per template and you have a regression test you can run before each release.

7. Lighthouse

Lighthouse runs in Chrome DevTools (the Lighthouse panel), in PageSpeed Insights, or from the command line with npx lighthouse https://example.com --only-categories=seo,performance. Its SEO category checks the rendered page for a title, meta description, a successful HTTP status code, crawlable links (anchors with a real href), a page not blocked from indexing, valid hreflang and canonical. The performance category shows how much JavaScript blocks the main thread, which affects Interaction to Next Paint and how quickly content appears.

Lighthouse audits the rendered page, so a passing SEO score does not prove the raw HTML is fine. Use it for crawlable-link and performance checks, not as a rendering test.

Comparison table

JavaScript SEO test checklist

Run these on one URL per template (home, category, product, article, search results, error page), using the tools above.

Links are real <a href> elements. Google follows links only from anchor tags with an href attribute. Buttons and divs with onclick handlers, and href="javascript:void(0)", are not followed. Check in the Screaming Frog JavaScript tab or by counting anchors in the diff.
Main content is in the initial HTML. The H1, body copy, product details and prices should appear in raw.html. This is the test that matters most for AI crawlers.
Meta tags are not injected late or changed. Title, meta description, canonical and meta robots should be in the raw HTML and should not change after rendering. Google's documentation warns that if the raw HTML contains noindex, Google may skip rendering, so JavaScript that removes it later will not help.
Lazy-loaded content loads without interaction. Googlebot does not scroll or click. Content that loads only on scroll events, "load more" buttons or tab clicks may be missed. Native loading="lazy" on images and IntersectionObserver-based loading are safe. Check the Rendered Page screenshot and the rendered HTML for content below the fold.
Status codes are real in SPAs. A client-side router returns HTTP 200 for every route, including ones that show "Page not found". Google treats those as soft 404s. Request a nonsense URL with curl and check the status: it should be 404. If the app cannot return a server status, redirect to a URL that does, or add noindex on error views, as Google's JavaScript SEO documentation suggests.
URLs use the History API, not fragments. Routes such as /#/products/shoes are not treated as separate pages.
Required resources are not blocked. Scripts and API endpoints needed to render content must not be disallowed in robots.txt. URL Inspection's More info tab lists any that are.
Structured data is present after render. Check it in the Rich Results Test, and prefer putting it in the raw HTML for non-Google crawlers.

AI crawlers and JavaScript

Google's AI Overviews and AI Mode draw on pages Googlebot has rendered, so Google's tools cover them. ChatGPT, Claude and Perplexity are different: their crawlers and fetchers generally read the raw response only. In late 2024 Vercel published an analysis of AI crawler traffic on its network which found that the major AI crawlers it observed, including OpenAI's and Anthropic's, downloaded JavaScript files but did not execute them.

The practical test is the curl half of the diff above. If your key pages pass it, AI crawlers can read them. Whether those engines then cite you is a separate question, and it changes by prompt and by week. Our AI visibility tracker measures that side, and the AEO scanner checks a page's raw HTML for the signals AI crawlers rely on.

Mistakes to avoid

Only using Google's tools. URL Inspection shows that Google can render the page. It says nothing about crawlers that never render.
Trusting a single screenshot. Rendering can succeed one time and time out the next if an API is slow. Test more than once and check More info for failed resources.
Testing only the home page. JavaScript problems are usually template-specific: product pages, filtered category pages and search results fail more often than the home page.
Reading Elements and calling it the source. The Elements panel is the live DOM after scripts. Use View Source or curl for the raw HTML.
Running a full rendering crawl first. Rendering every URL of a large site is slow and can put load on the server. Sample templates, then scale up.
Ignoring status codes in SPAs. Many "indexed but should not be" problems on JavaScript sites are soft 404s from routes that always return 200.