AI readiness audit: can AI engines actually read your site?

It is 9:40 on a Thursday. The marketing lead at a B2B software company types her category's most common question into an assistant, hoping to see what the 42-page setup guide they published two weeks ago has earned. The answer runs four sentences and lists three sources underneath. None of them is her domain. That afternoon a one-line reply arrives from an engineer: a rule added to robots.txt six months earlier, during a crawl-load scare, had blocked GPTBot entirely. The guide was well written. The engine was never allowed to read it.

This piece is about how to find that one line. In order: the difference between a site that works and a site that gets read, the eight categories an AI readiness audit scans, how to verify robots.txt and per-agent access, what llms.txt does and does not do, how much of your page survives with JavaScript off, where schema markup stops helping, what makes a paragraph quotable, how freshness and brand identity criteria are checked, how to turn findings into an ordered backlog, and the traps people fall into when reading the score. Every item here is checkable; none of it is opinion.

Where does the gap between working and being read begin?

A page opens in your browser because it clears three separate gates: the server accepts your request, the browser executes JavaScript, and your eyes scan the text. A language model's collector behaves differently at all three. At the first gate its identity comes from the User-Agent header, and on most modern stacks that identity lands in a separate rule set. At the second gate, most collectors either do not run JavaScript or run it in a limited way; whatever the server returns as raw HTML is what exists. At the third gate it does not reconstruct the context your eye assembles for free — it infers what a mid-page sentence answers from the heading and structure around it.

So an audit's first question is not aesthetic: what HTTP status does the page return, for which agent, and how many words are in the body that comes back. A site can load fast, look sharp, and show healthy traffic in analytics while returning 403 to a collector. Bot rules on your WAF, rate limiting, geo-blocking and challenge pages all live in this category. On commercial pages the picture gets harsher still: the answer to why AI skips your product page is usually hidden in how that page responds to a request, not in how well it was written.

What exactly does an AI readiness audit scan?

An audit has to be a checklist, not an impression. Rocketly's AI Readiness Audit scans your site across eight categories against 70 criteria: crawlability, answerability, citability, brand identity, freshness, structured data, agent access and measurement. Those eight are not arbitrary — they follow the stages a sentence must clear before it can end up inside an engine's answer. A page that cannot be reached cannot be read, a paragraph that cannot be read cannot answer anything, and a passage with no clear provenance does not get cited.

The set of possible states matters as much as the criteria. Every criterion resolves to one of four: met, partial, not met, or could not check. Most tools skip the fourth, and its absence quietly corrupts the score, because counting an unchecked item as failed sends you off to fix a problem you do not have, while counting it as passed hides a real gap. Items we could not check are removed from the denominator — you see a ratio over what was genuinely checkable, and the number of unchecked items is stated separately.

In practice a first audit is a mix of four output types: access failures at the server level, formatting failures at the page level, missing files at the site level, and blind spots on the measurement side. Their weight is not equal. Access failures are usually singular but lock everything behind them; formatting failures spread across hundreds of pages yet get fixed once in a template.

What does robots.txt tell each bot, and how do you verify it?

robots.txt is a declaration, not an authorisation layer: binding for collectors that respect it, irrelevant to those that do not. It is still usually the highest-return line item in an audit, because failures here are silent — nothing tells you a rule has excluded you. Rocketly's llms.txt, robots and schema tools read the file agent by agent and report access status for GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and CCBot individually. The tools never write to your site; output is a draft, and where a field is missing you get the name of what is missing rather than an empty box.

Agent	What it does	What blocking it costs you
GPTBot	OpenAI's general crawler; collects content	Your pages stay outside that corpus
ChatGPT-User	Fetches a page live when a user asks for it	Even a pasted link cannot be read
ClaudeBot	Anthropic's content-gathering crawler	You are absent from that source pool
PerplexityBot	Crawler for an engine that links its sources	Your odds of being cited drop directly
Google-Extended	Controls use in Google's generative features	Affects generative use, not your ranking
CCBot	Feeds the Common Crawl archive	You fall out of many downstream datasets

Every row here is a policy call, not a technical requirement. Some publishers deliberately keep their content out of training corpora; others allow live fetching only. What matters is that the decision was made on purpose and that two different purposes are not conflated: bulk collection and on-demand retrieval run through separate agents. If you want the permission you granted written down properly, our notes on handling customer data in AI workflows cover what that documentation should look like.

Verification takes one request. Fetch the file with the relevant User-Agent header, read the body, then request an actual content page with the same agent and note the status code and body length. Three failures dominate: an unnoticed wildcard Disallow, a robots.txt sitting behind a redirect chain, and a file returning 404 instead of 200. A fourth is just as common — robots.txt allows the agent, the firewall does not.

Do you need llms.txt, and what belongs in it?

llms.txt is a machine-readable table of contents placed at your root: a short description, the main sections, and canonical links for each. It is not a ranking factor, it does not replace robots.txt, and no engine treats it as mandatory. Its value sits elsewhere: it removes the moment where a model has to guess which version of your content is canonical. When the same feature description lives at ten URLs, you get to say which one counts.

The checkable items are unambiguous. Is the file at the root, does it return 200, is it served as plain text, do all links inside it return 200, do the listed titles match the pages they point to, and when was it last updated. An llms.txt that fails those six earns nothing by existing. The same goes for listing a page you no longer publish or leaving a retired section in place — that turns the file into a trust liability.

On content, brevity wins. Twenty entries that each represent a genuinely distinct topic beat a hundred-line copy of your sitemap. The goal is not to describe the whole site but to mark the pages that answer the most questions.

How much of your page survives with JavaScript off?

The renderability test is simple: fetch the page raw and count the visible words in the HTML that comes back. If the browser shows 1,200 words and the raw body holds 90, then for answer generation yours is a 90-word page. Single-page app architectures, client-fetched product descriptions, copy tucked behind tabs and accordions, and lists loaded through infinite scroll are the usual causes of that gap.

The fix is rarely a rewrite of your architecture. Server-side rendering for critical routes, embedding key copy in the first HTML response, or leaving accordion content visually hidden but present in the markup usually covers it. The second thing to check is whether your text lives inside images: a spec table shipped as a screenshot exposes none of its numbers as text. How images are interpreted is a separate subject — our piece on visibility in multimodal search goes deeper into alt text and file naming criteria.

The third check is redirects. Language selection, region detection or consent-gated redirects can work flawlessly for humans and leave a collector stranded on an empty interstitial. Self-contradicting canonicals belong in the same bucket: if page A points to B as canonical while B points back to A, nothing tells the engine which address to remember.

Where does schema markup help, and where does it turn into noise?

Structured data labels page information explicitly for machines, and it helps exactly as far as it removes guesswork. The core types an audit looks for are few: Organization for the company, Product on commercial pages, FAQPage on question blocks, Article on posts, and BreadcrumbList for navigation. Presence is checked first, validity second, and agreement with the visible content third.

That third check is where most sites fail. Marking up a Q&A block that is not on the page, declaring a rating nobody can see, or stamping one Organization block across every template and forgetting to update the domain produces risk rather than gain. The second common failure is omission: an Organization block without name, domain, logo and official profile links is where the brand identity category bleeds the most points. If you are building those question blocks properly, we covered why question-and-answer content gets cited separately.

What actually makes a paragraph quotable?

Answerability and citability are two categories, not one. The first asks whether the page addresses a question at all; the second asks whether the answer travels. A portable paragraph stands on its own: explicit subject, defined terms, numbers with units, context carried inside the sentence. A line that opens with "as mentioned above" loses its meaning the moment it is lifted off the page, which is exactly why it never gets lifted.

The concrete checks are these: is the heading a real question or a marketing slogan, does the answer arrive in the first two sentences under it, are terms defined where they first appear, are comparisons presented as tables, and does the page state its own limits — date, version, scope. On comparison queries this formatting difference shows up directly in outcomes; we have written separately about the structure needed for winning "X or Y" style queries.

There is a second dimension: being cited is not the same as being mentioned. An engine can name you without linking your domain. Rocketly's GEO side measures those two separately — mention and citation never collapse into one number — because a brand that gets named while nobody visits its site would otherwise look like a success. If you want to know which signals earn the link, our analysis of which brand an AI actually cites works through the trust and authority side.

What do the freshness and brand identity criteria check?

Freshness is not about stamping a date; it is about the date being true. The audit looks for contradictions: are publish and modified dates present in markup, does the visible date match the marked-up one, did anything actually change on a page labelled updated, and are time-bound statements inside — versions, pricing policy, regulatory references — still valid. Bumping a date to look current does not work even in the short run; a contradictory date is a worse signal than no date at all.

On brand identity the question is simpler: can a model learn who you are from your site, and can it corroborate that elsewhere? An about page, a consistently spelled brand name, company details, contact and author pages are the visible half. The invisible half lives off your domain, in how external sources describe you. That is why how structured references like Wikipedia and Wikidata feed brand knowledge counts as a continuation of the technical audit, not a separate marketing topic.

Third-party sources sit in the same frame. The source intelligence view lists the domains feeding an engine's answer with ownership labels (you, competitor, third party, user-generated) and type labels (review platform, marketplace, directory, forum, video, news); when a domain is unfamiliar it says undetermined rather than inventing a type. Why forum and community content shows up so often in those lists is something we unpacked in why AI leans on Reddit and forums.

How content is produced also feeds freshness indirectly. Hundreds of bulk-generated near-duplicate pages erode both your crawl budget and your canonical clarity; what we wrote about how search engines treat AI-assisted content helps calibrate that balance.

How do you turn audit findings into an ordered backlog?

The 70-line table an audit produces is not a to-do list as it stands. The items depend on each other, and work done in the wrong order is wasted. In Rocketly, audit findings, citation gaps and demand opportunities collect into a single action plan, and the order is built on dependency rather than score. The principle is blunt: writing new content while bots cannot read your site is wasted effort.

The practical sequence usually runs: access first — robots rules, firewall exceptions, root files returning 200. Then rendering — critical copy present in raw HTML. Then structure — schema types, canonicals, heading hierarchy. Then formatting — question headings, answers in the first two sentences, comparison tables. Content production last, filling genuine topic gaps. Reverse that order and well-written new pages simply pile up on an unreadable site.

The second rule is to collapse one root cause into one line item. If the same competitor sits ahead of you on eight questions, that is one item, not eight cards; a template-level formatting fault across four hundred pages is one job, not four hundred. Otherwise the list grows without the site improving.

Which traps catch people reading the audit score?

The first trap is mistaking the score for visibility. A high readiness score says your site can be read; it does not say you are being mentioned. The two are measured against different things: the audit looks at your site, GEO measurement looks at what the engine produced. Readiness is a precondition for visibility, not a guarantee of it.

The second trap is treating missing data as zero. If measurement did not run on a given day, that day's value is not zero — it does not exist. Seeing that distinction on screen matters: the chart breaks at that point and the ratio shows a dash rather than 0%, and the number of unmeasured days is stated separately. The same logic drives the "could not check" state on the audit side.

The third trap is reading a trend out of a single measurement. Generative engines can phrase the same question differently across two runs; movement between two points is not a change. The change-tracking view tells you what moved and does not claim why. The fourth trap is leaving the measurement category empty: if you never tag visits from AI assistants as their own channel, you cannot connect audit work to business outcomes. We covered why that traffic behaves differently in our piece on how visitors from AI search convert.

Do not treat the audit as a one-off; template changes, new page types and infrastructure updates quietly reintroduce closed doors. If you want to see your own site's readiness against all 70 criteria, you can open a free account and run the audit.

FAQ

Can AI engines read my site without an llms.txt?

Yes. llms.txt is a map, not a permission slip; its absence does not close your site. Access is governed by robots.txt rules, server responses and firewall policy. Add llms.txt after you have cleared the access and rendering items, not before.

Does blocking every AI bot in robots.txt hurt me?

It depends what you are protecting. For a publisher licensing its archive, blocking bulk collectors is a coherent choice. But if you also block the agents that fetch a page when a user asks for it, you cut your own odds of being cited in answers. Evaluate the two purposes separately.

My audit score went up but I still do not appear in answers. Why?

Because readiness and visibility are different measurements. Readiness says your pages can be read and quoted. Appearing in an answer also depends on topical authority, how often you are referenced off-site, and how contested the question is. Track them on separate screens.

How often should the audit be repeated?

Always after a template or infrastructure change; otherwise monthly is enough. What matters is comparability, not frequency: run the same criteria set and the difference between two dates becomes readable. And make it a habit to confirm your root files still return 200 after every deployment.