How to check what AI crawlers can see on your WordPress pages

A page can look perfect in your browser and still be half empty to ChatGPT, Claude or Perplexity. Your browser runs JavaScript. Most AI crawlers do not. If your answer sits in a tab, a slider or a list that loads on click, the crawler may never see it, and it cannot cite what it cannot read.

On the sites I run, I make sure the answers are in the plain HTML the server sends. Checking takes about twenty minutes per page template, and you only need a browser and a terminal.

Do AI crawlers render JavaScript?

Mostly, no. Vercel’s December 2024 study of AI crawler traffic found that OpenAI’s crawlers, ClaudeBot and PerplexityBot did not run JavaScript. GPTBot and ClaudeBot downloaded JavaScript files but never executed them. Gemini, which uses Googlebot’s infrastructure, and Applebot were the exceptions. The safe assumption: if a sentence is not in the HTML the server returns, ChatGPT, Claude and Perplexity probably cannot see it.

What do you need before you start?

• Chrome or another browser with developer tools.
• A terminal with curl (built into macOS, Linux and current Windows).
• One page from each template you care about: a post, a service page, a product or category page.
• For each page, one sentence that holds the main answer. This is your test phrase.

How do you test a page as an AI crawler?

You compare three views of the same page and look for your test phrase in each.

1. Open view source. Press Ctrl+U (Cmd+Option+U on a Mac) and search for your test phrase. It worked if the phrase appears as readable text, not only inside a script.
2. Fetch the page with an AI crawler’s user agent. Run the commands below, which use GPTBot’s user agent from OpenAI’s documentation. It worked if the status code is 200 and the size is close to the browser fetch.

curl -s -L -o gptbot.html -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://example.com/your-page/

curl -s -L -o browser.html -w "%{http_code} %{size_download}\n" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" \
  https://example.com/your-page/

grep -c "your test phrase" gptbot.html browser.html

3. Repeat the fetch for the other crawlers that matter to you. Swap in the tokens OAI-SearchBot (ChatGPT search), ClaudeBot, Claude-SearchBot and PerplexityBot. It worked if every fetch returns 200 and finds your phrase.
4. Turn JavaScript off and reload. In Chrome, open DevTools, press Ctrl+Shift+P (Cmd+Shift+P on a Mac), type “Disable JavaScript”, press Enter and reload with DevTools still open. It worked if the page still shows the answer, the headings and the links. Run the same command with “Enable JavaScript” when you are done.
5. Compare the three views. Note anything the browser shows that the source, the crawler fetch or the no-JavaScript view is missing. That list is your fix list.

One limit: a faked user agent only tests rules that look at the user agent. Firewalls that verify crawlers by IP address will block your curl request as an impostor, so check your security logs for the real bot before you change anything.

What should you fix on a WordPress site?

Fix whatever keeps the answer out of the first HTML response. On WordPress these are the usual causes:

• Content loaded after the page. Load-more lists, infinite scroll and widgets that pull text through AJAX or the REST API. Print the first items in the HTML and use paginated links for the rest.
• Tabs, accordions and sliders built by script. Text that is in the HTML and only hidden with CSS is fine. Panels that JavaScript builds are not. The core Details block keeps its text in the HTML, so it suits FAQs.
• Page builder widgets and display settings. A widget can look correct in the browser while its content is missing from the server-generated HTML. For example, a tab or carousel may load its text only after JavaScript runs. Inspect the page source, identify the widget responsible, and switch to a server-rendered option that includes the content in the initial HTML. Then fetch the page again and verify that the answer appears in the response.
• Cache and security rules that treat bots differently. A security plugin or CDN firewall may return a 403 or a challenge page to AI user agents, and a cache can serve bots a stale copy. Compare the status codes and sizes from step 2.

Common mistakes

• Checking in DevTools’ Elements panel. It shows the page after JavaScript ran, which is exactly what crawlers do not see.
• Testing while logged in. Logged-in users usually skip the cache, so you test a page no crawler gets.
• Forgetting robots.txt. A crawler you disallow never reads the page at all. And an llms.txt file does not replace readable HTML, as I explain in how to add an llms.txt file to WordPress.

How do you know the fix worked?

Clear the page cache and run the three checks again while logged out. The page passes when your test phrase is in view source, the grep count is at least 1 for every crawler fetch, the status codes match the browser fetch, and the no-JavaScript view still shows the answer, headings and internal links. Keep the commands in a text file and rerun them after theme or plugin updates.

Start with the one template that carries most of your answers, usually the post or service page, and get it passing before you look at the rest. Readable HTML is the base everything else in AI search sits on, which I wrote about in what AI search taught me about SEO, AEO, GEO and CRO.