How AI crawlers read HTML
AI crawlers quote the HTML they fetch. If your answer is created in the browser, it is late or missing from the systems most likely to cite you.
A fetch is not the same as a render
The first response is a file. A browser may then download scripts and build a page the server never sent. Search engines sometimes do that second step. Answer engines often stop at the file.
Plan for the file. If the sentence is not in it, do not expect a citation.
What bots keep
They keep the parts HTML already names.
- The title, the meta description, and the canonical link
- Headings in order, paragraphs, and lists
- Tables with header cells, quotes, and figure captions
- Link hrefs and image alt text
- JSON-LD, when it repeats a fact the page also shows
What bots miss
They miss words that are not in that file, and they mistrust words you hid.
- Text inserted when a component hydrates
- Navigation that only exists as an onclick handler
- Facts inside a widget that starts closed, if you care about the strictest parsers
- Copy that CSS removes with display:none
A pattern you can ship this week
Put the answer in the first paragraph under the heading. Use a list for steps and a table for a comparison. Link with an href whose anchor says the destination. Then view source and search for the sentence. If it is not there, neither is the citation.
The semantic HTML article covers the document shape. Generative engine optimization is the same work done on a client site.
Questions about AI crawlers and HTML
Does Googlebot run JavaScript?
It can, in a later rendering wave that spends crawl budget. Treat that wave as a bonus, not as the place your answer lives. Many AI crawlers never run the bundle.
Which bots is this written for?
Any crawler that extracts the fetched document: Googlebot, Bingbot, PerplexityBot, ChatGPT-User, and ClaudeBot are the ones we plan for.
Request a free AI-extraction audit
Send one public URL. A person reads the HTML your server returns and writes back what AI crawlers can quote, and what they never see.