AI SEO Audit: Crawlability, Retrieval, and Citations

Most technical SEO audits still focus on how well Google's crawler can access and index a page. That checklist is necessary, but it's no longer sufficient. AI systems that answer search queries directly, rather than sending users to a list of links, follow a different set of rules for reaching, reading, and citing your content.

An AI SEO audit looks at three separate stages: whether AI systems can crawl your site at all, whether they can retrieve and parse your content once they get there, and whether your pages end up cited in the answers those systems generate. Treating these as one problem, rather than three related but distinct ones, is where most technical SEO analysis for AI visibility goes wrong. This guide walks through each stage and how to prioritise fixes once you've found the gaps. If you'd rather have this checked for you, see our web audit service.

What an AI SEO Audit Actually Checks

A standard technical SEO audit and an AI SEO audit overlap in places, but they're not the same exercise. Both look at crawl access, site structure, and page speed. Where they diverge is in what happens after a system reaches your page.

Traditional search crawlers index a page and rank it based on hundreds of signals, then send a user to click through. AI systems that generate direct answers need to extract specific facts or passages from your page, then decide whether to cite you as the source. That extra step, extracting and citing content, is where a purely traditional audit stops looking.

This matters because a page can pass every conventional technical check, fast load time, clean HTML, no crawl errors, and still be functionally invisible to an AI system that can't parse its structure well enough to lift a usable answer from it.

Crawlability: Can AI Systems Reach Your Content at All

Crawlability is the most basic layer, and the easiest to check. Most AI systems that browse the web for real-time answers identify themselves with their own user agent strings, separate from Googlebot. If your robots.txt file blocks those user agents, either deliberately or by accident, you've made a decision about AI visibility without necessarily meaning to.

"PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."

Worth noting here: blocking AI crawlers isn't inherently wrong. Some businesses have legitimate reasons to restrict how their content is reused. The audit step isn't about deciding whether to allow access, it's about confirming that whatever your robots.txt file currently allows matches what you actually intend, and that different crawlers aren't lumped together when their actual purpose, like Perplexity's distinction between surfacing links and training foundation models, differs.

Beyond robots.txt, check whether your site depends on JavaScript-rendered content for anything that matters. Crawlers that don't execute JavaScript the way a browser does may only see an empty shell of a page, which means there's nothing for a retrieval step to work with later on. A practical technical review should verify whether key content exists in the initial HTML response, whether structured data matches visible content, and whether AI-crawler requests are being blocked or degraded by robots.txt, a CDN, or firewall rules.

Retrieval: How AI Systems Pull Content Once They Reach It

Reaching a page isn't the same as reading it well. Once an AI system accesses your content, it still needs to identify which parts are relevant to a given question. Pages with a clear heading structure, direct answers near the top of a section, and minimal clutter around the main text tend to retrieve more cleanly than pages that bury the relevant information inside long, unstructured paragraphs.

To put that in context, retrieval problems are harder to spot than crawlability problems because the page loads fine and looks normal to a human reader. The failure happens in how the content is structured, not whether it's accessible.

Citations: Whether AI Answers Actually Reference Your Pages

Citation is the final stage, and the one most businesses actually care about: does an AI-generated answer name your business or link back to your page as the source of a fact? This is a different question from ranking or even retrieval. A page can be crawled, retrieved, and still lose the citation to a competing source that phrased the same fact more clearly or with more specific supporting detail.

The pages that tend to get cited share a few structural traits. They state facts directly rather than implying them. They use specific numbers, dates, or named details rather than vague claims. And they avoid burying the answer under promotional language that a system would need to filter out before finding anything usable.

Why the Same Fact Can Get Cited From One Page and Ignored on Another

Two pages can cover the same topic in similar depth and still get treated differently by an AI system. If one page states a fact in a single, self-contained sentence and the other spreads the same fact across three paragraphs of surrounding context, the self-contained version is easier to extract cleanly and more likely to be quoted or cited. This isn't about writing shorter content overall, it's about making sure the specific facts you want cited are stated in a form that can stand on their own.

That said, your results may vary depending on how competitive the specific topic is and how many other sources are saying something similar. A well-structured page competing against dozens of equally well-structured pages won't automatically win the citation just for being clear.

Citation also depends on more than your own website. AI-generated answers frequently draw on independent, third-party sources such as reviews, directories, and press coverage alongside a business's own pages.

"Boost your brand's presence on third party websites."

This isn't a shortcut for purchasing low-quality links or fabricated reviews. It means improving factual, independent, and relevant evidence such as genuine reviews, complete listings, credible local coverage, and industry references.

An analysis of 21,311 commercial brand mentions across ChatGPT, Claude, and Perplexity found that brands were 6.5 times more likely to be mentioned through third-party content than through their own domains. This is a directional finding from one dataset, not a universal ratio for every prompt, platform, or industry.

Running a Technical SEO Analysis for AI Visibility

A useful technical SEO analysis for AI visibility starts by separating the three stages rather than testing everything at once. Check crawlability first, since a blocked or JavaScript-hidden page makes the other two stages irrelevant. Then check retrieval by looking at how cleanly your key facts are structured on the page. Citation is the hardest to test directly, since it depends on how AI systems choose sources, but you can still review whether your highest-value pages state their key facts clearly enough to be extracted.

Run this analysis on a small set of your most important pages first, rather than trying to audit an entire site at once. Pages that already perform well in traditional search are a reasonable starting point, since fixing crawlability or retrieval issues on pages that already have authority tends to show results faster than starting from pages with no existing visibility.

Common Issues an AI Website Audit Uncovers

Most AI website audit findings fall into a small number of recurring categories. Reviewing them side by side makes it easier to see where a given site's problems actually sit.

Fixing one category doesn't guarantee the others are fine. A site can have excellent crawl access and still fail at citation because its content buries facts inside dense paragraphs.

Prioritising Fixes After an AI Search Audit

Once an AI search audit has identified issues across all three stages, fix them in the order they occur, not the order they seem most interesting. A citation-focused rewrite is wasted effort if the page is still blocked from being crawled in the first place.

A reasonable priority order looks like this:

1. Fix crawl access issues first, since nothing downstream matters until AI systems can reach the page at all.
2. Address retrieval problems next, restructuring content so key facts are easy to extract cleanly.
3. Refine citation-worthy statements last, once the page is reachable and readable, sharpening vague claims into specific, self-contained facts.

Working through the stages in this order avoids spending time refining content that a crawler can't reach yet, which is a common and avoidable source of wasted audit follow-up work.

Use Your Own Crawl Logs

Crawl logs do not prove that an AI product will cite a page, but they reveal whether a crawler reached it, which URLs it requested, what status code it received, and whether access failures are technical or intentional.

A useful log review typically tracks:

• Requests by user-agent
• Unique URLs crawled
• 200, 301, 403, 404, 429, and 5xx response distribution
• Requests to high-value service, product, category, and article pages
• Blocks caused by robots.txt, CDN, WAF, or rate limits
• Changes compared with the previous 30-day period

FAQ