Agentic Navigability: How AI Agents Read Your Site
Agentic navigability is how well an AI agent can reach, crawl and understand a website by following its links. Agentic citability is how likely a page’s content is to be quoted in an AI answer. Navigability decides whether an agent can read your pages. Citability decides whether it uses them. GraphoRank scores the signals behind both on one 0–100 scale, called GraphRank.
If you searched for “LLM SEO” or “SEO for ChatGPT”, this is the part of the topic that lives on your own website, the part you control. Below: both terms in plain words, how agents and crawlers move through a site, and which claims are documented and which are only suspected. A checklist at the end is ordered by likely impact, and you can do most of it without a developer.
Key takeaways
- Navigability is access: can an agent reach and read your pages? Citability is use: is what it read worth quoting?
- “Blocking AI” is not one switch. Training bots and search bots are separate, so decide bot by bot.
- robots.txt is a request that well-behaved bots honour, not a lock.
- llms.txt is cheap, harmless and unproven. Structured data is hygiene, not a guarantee.
- Orphan pages, deep pages and competing pages are the structural problems worth checking, though no study has measured their effect on AI answers.
What do agentic navigability and agentic citability mean?
Agentic navigability is how well an AI agent can reach, crawl and understand a website by following its links.
An AI agent is software that uses an AI model to do a job for a person: research a question, compare suppliers, fill in a form. It often has to open several pages to do it. An AI crawler is the simpler cousin, a program that fetches pages so an AI product can learn from them or look them up later. Both move through your site the way a person does, from link to link.
An agent finds a site easy to navigate when every important page can be reached through links, the real content comes in a form a program can read, and nothing blocks the visit by mistake.
Agentic citability is how likely a page’s content is to be quoted in an AI answer.
When someone asks an AI assistant a question, it may build the answer from pages it finds and name some of them as sources. Those named sources are citations. A page is citable when an AI system can lift a clear, self-contained, trustworthy statement from it: a definition, an answer, a figure with a source.
The two are linked but different. Navigability is about access: can the agent find and read the page? Citability is about use: having read it, is the page worth quoting?
A page nobody can reach cannot be cited. A page that can be reached but says nothing clear gets read and passed over, which is why we treat them together.
How to read what follows. Some of it is documented by the companies that run the crawlers. Some is measured by researchers. Some is common-sense reasoning that nobody has measured yet. We label which is which, because on a young topic that difference matters more than the advice.
How do AI agents and AI crawlers actually navigate a website?
AI systems reach your content in four main ways: they follow links, they obey (or sometimes ignore) robots.txt, they look for guidance files such as llms.txt, and they read the page structure the browser builds. Here is what the primary sources say about each.
Link following
The basic move is the one search engines have always made: fetch a page, read its links, fetch those. A crawler that works this way can only find what is linked.
Your sitemap, the list of pages you hand to search engines, helps, but it is a list, not a route. A page that no other page links to is hard to discover by following links, and the next section comes back to that.
robots.txt and the three kinds of AI bot
robots.txt is a small text file at the root of your site that tells automated visitors which URLs they may fetch. AI companies now run several different bots with different jobs, and their documentation separates them:
- OpenAI documents three. GPTBot is for training. OAI-SearchBot powers ChatGPT search: sites that opt out of it “will not be shown in ChatGPT search answers”, and OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust. ChatGPT-User fetches pages when a person asks for it, and OpenAI notes that for this one “robots.txt rules may not apply”.
- Anthropic documents three as well: ClaudeBot for training, Claude-User for user-directed fetches and Claude-SearchBot for search. It says its bots respect “do not crawl” signals in robots.txt and supports the Crawl-delay setting.
- Perplexity documents PerplexityBot, which surfaces sites in search and is not used for training foundation models, and Perplexity-User, which supports user actions and “generally ignores robots.txt rules” because a person triggered the fetch.
- Google documents a control called Google-Extended. It governs whether your content is used for training future Gemini models and for grounding, and Google states it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”.
Decide bot by bot. Blocking GPTBot stops training use but, per OpenAI’s documentation, does not by itself remove you from ChatGPT search, which a different bot governs. A blanket block can hide you from the very products you want to appear in.
Two findings keep this honest, and both are caveats.
Behaviour does not always match documentation. A research preprint that analysed the web logs of one institution, covering 130 self-declared bots over 40 days, found that bots are less likely to comply with stricter directives and that certain categories, including AI search crawlers, rarely check robots.txt at all. It is a single-institution preprint, so treat it as a warning, not a census.
Cloudflare’s allegation against Perplexity. Based on observations across tens of thousands of domains and controlled test sites, Cloudflare published an allegation that Perplexity ran an undeclared crawler that impersonated a normal Chrome browser, and that test domains with a robots.txt disallow-all rule still had their content details returned. Cloudflare is a vendor with a commercial interest, and we have not verified Perplexity’s response. Perplexity’s own documentation, for its part, states that its user-triggered fetcher generally ignores robots.txt.
The rule of thumb: robots.txt is a request that well-behaved bots honour, not a lock.
How many sites say anything about AI bots at all? Only a small share, in the Web Almanac’s desktop data. These are shares of sites whose robots.txt names the bot, not shares that block it:
- 4.5% of desktop sites name GPTBot (2.9% in 2024) HTTP Archive Web Almanac 2025, SEO chapter
- 3.6% of desktop sites name ClaudeBot (1.9% in 2024) HTTP Archive Web Almanac 2025, SEO chapter
- 2.8% of desktop sites name PerplexityBot HTTP Archive Web Almanac 2025, SEO chapter
Most sites say nothing, which means the default is open.
llms.txt
llms.txt is a proposed Markdown file, served at the root of your site, that gives AI agents a short guide to what the site is and which pages matter. It is a community proposal, not an internet standard. Our llms.txt guide covers the format, and you can test yours with the free llms.txt checker.
The evidence is mixed, and the numbers disagree because they measure different populations:
| Study | What it measured | Result |
|---|---|---|
| HTTP Archive Web Almanac 2025 | Sites in its crawl | llms.txt on 2.13% of desktop and 2.10% of mobile sites |
| SE Ranking (Nov 2025) | Nearly 300,000 domains | 10.13% had the file; no correlation with AI citations |
| Common Crawl (July 2026 crawl) | Random sample of about 5.2 million /llms.txt URLs on crawlable hosts | 22.32% of attempted /llms.txt URLs returned HTTP 200, and 52.5% of those carried a text body (11.72% of sampled URLs) |
| Ahrefs (May 2026) | 137,210 domains with traffic in its analytics | 28% publish the file; 97% of those files received zero requests that month |
Read the numbers with care. Ahrefs warns that its customer base skews technical, so 28% is an upper bound, and that a request for the file is not proof of use. Common Crawl’s authors say their 11.7% should not be compared with the Web Almanac figure.
On the engine side, Google says: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search”, and that llms.txt “will neither harm nor help” Search visibility. Yet Chrome’s Lighthouse tool, which audits pages, includes an experimental “Agentic Browsing” category that checks whether llms.txt exists. The two statements answer different questions: Google Search visibility on one side, readiness for browser-based agents on the other.
Our reading: llms.txt is cheap, harmless and unproven. Publish one if it takes an afternoon, but don’t expect it to rescue a site with weak structure.
The accessibility tree
When a browser loads a page, it builds a simplified, text-like description for assistive technology such as screen readers: what is a heading, what is a button, what each link says. This is the accessibility tree.
Chrome’s Lighthouse documentation for agentic browsing states that “Agents rely on the accessibility tree as their primary data model”. Its audits cover names and labels, the integrity of the tree, whether interactive content is visible, layout shift, llms.txt presence and WebMCP tools (an experimental way for a site to offer actions to agents). The category is experimental and has no 0–100 score.
A 2026 benchmark of computer-use agents (A11y-CUA, a CHI ’26 paper) adds a related signal. It used 60 tasks and recordings from blind, low-vision and sighted users:
- 78.3% of tasks completed by a default agent A11y-CUA, CHI ’26 (arXiv)
- 41.67% in keyboard-only mode A11y-CUA, CHI ’26 (arXiv)
- 28.3% with a magnifier A11y-CUA, CHI ’26 (arXiv)
Be careful with that result. It concerns everyday desktop computer tasks and how agents cope with different interaction modes, not website structure, and it does not show that clean markup makes agents succeed more often. It suggests only that agents are sensitive to how an interface is exposed, which is what the accessibility tree describes.
In practice: use real headings, real links with descriptive text, labelled buttons and form fields. Accessibility already asks the same of you.
JavaScript rendering
Many modern sites build their content in the visitor’s browser with JavaScript instead of sending it ready-made in the page. A person never notices. A simple crawler may.
We found no primary document stating which AI crawlers run JavaScript, so we won’t say they do or don’t. The cautious rule: put your important text and links in the HTML the server sends, and treat anything that appears only after a script runs as a risk. GraphoRank’s own crawler renders JavaScript so it can see what a browser-based reader sees, and it records whether a page needed rendering to show its content.
Why does site structure decide what gets read?
An agent can only read what it can reach, and it judges importance from the signals your links send.
An open question. We found no published study that ties internal link structure (depth, orphans, in-content linking) to agent success or to AI citation rates. What follows is reasoning from how link-following works, not a measured effect. As far as we can tell it is also the largest open question in this field, and the reason GraphoRank exists.
Orphan pages
An orphan page is a page with no internal links pointing to it. A person can only reach it with the exact address. An agent that follows links never will.
Picture a small shop whose “Shipping and returns” page is linked from nowhere because a redesign dropped the footer link. The page exists, it is correct, it is in the sitemap. But when an agent researches “does this shop ship to Germany?”, no link leads to the answer. Orphans are the cheapest structural problem to fix: add a link.
Click depth
Click depth is how many clicks it takes to get from your home page to a given page. Say the guide that best explains your service sits five clicks deep, behind a category, a sub-category and a paginated archive. A person might never get there either.
The deeper a page sits, the more steps an agent needs and the fewer routes lead there. Keep the pages that matter close to the home page.
Links inside content versus links in menus
Menus, headers and footers repeat the same links on every page. A link that appears everywhere says little about any one page, because it comes with the template. A link inside a sentence says something specific: this page is relevant to this point, for this reason. The anchor text, the clickable words, tells the reader what is on the other side.
That is why GraphoRank separates links inside the main content from template links when it builds the graph of a site: the first group is what a reader extracting the main text of a page actually sees. As far as we know, no one has published a measurement of the effect on AI answers. We present it as our design choice, grounded in how a reader follows a thread of argument, not as a proven ranking factor.
Duplicate or competing pages on one topic
If five pages on your site each partly answer “how much does it cost?”, an agent has to choose, or may blend them, and none of the five becomes the page to quote.
Think of a company with a pricing page, a blog post about pricing, an old pricing page nobody removed, and a FAQ entry with a fourth set of numbers. The reader can no longer tell which one is current.
The fix: pick one page per question, make it complete, redirect or merge the rest, and link the other pages to it.
What makes content citable?
Content is citable when an AI system can extract a clear, accurate, attributable statement from it. Four things matter most in practice, and they are what good editors have always asked for.
- Clear entities and definitions. An entity is a named thing: a company, a product, a person, a place, a concept. Say what each one is in a plain sentence (“X is a …”), early, and use the same name every time. A definition that stands alone can be quoted alone.
- Answer-first passages. Open each section with the answer, then give the reasoning. Research on language models offers an indirect reason: a controlled study of long inputs (“Lost in the Middle”) found that models often perform best when the relevant information is at the beginning or end of what they are given and degrade for information in the middle. That study is about prompts, not about web pages being chosen as sources, so it is an analogy, not proof. Answer-first is also simply better for human readers.
- Sources, authorship and freshness. Name who wrote the page, show when it was published and updated, and link to primary sources for any number you give. Keep dates honest: a fresh date on a stale page helps no one.
- One clear page per topic. The previous section applies again: a page that owns its topic is easier to cite than five that overlap.
Structured data: what Google says, and what a study found
Structured data is code, usually in a format called JSON-LD, that labels the content of a page for machines: this is an article, this is its author, these are the questions and answers. The sources differ in tone, so here are both.
Google’s documentation says there are “no additional requirements to appear in AI Overviews or AI Mode”, that no special schema.org structured data is needed, and, in a later guide, that “Structured data isn’t required for generative AI search”. To appear, a page has to be indexed and eligible to be shown with a snippet. Google also explains the controls that limit it (nosnippet, data-nosnippet, max-snippet and noindex), and that traffic from AI features is counted under the “Web” search type in Search Console.
A separate preprint, the GEO-16 study, looked at what cited pages have in common:
- 1,702 citations audited GEO-16, arXiv preprint
- 1,100 unique URLs GEO-16, arXiv preprint
- 70 product-intent prompts on three AI answer engines GEO-16, arXiv preprint
It found that metadata and freshness, semantic HTML and structured data were the page-quality groups most associated with citation, and that pages scoring at least 0.70 on its scale with at least 12 pillar hits had substantially higher citation rates.
Limits of the study. The authors caution that it is observational, covers English-language B2B software pages only, and is a preprint.
The two sources do not contradict each other. Google describes what is required to be eligible in Google’s own features. The preprint describes what co-occurred with being cited across several engines. Neither shows that adding markup causes citations.
Our reading: structured data is not a trick, it is part of describing a page clearly. Add the types that match the content, keep them accurate, and don’t expect markup to rescue a thin page.
What the GEO research does and does not say
The 2024 paper that introduced the term Generative Engine Optimization reported that methods it tested could boost visibility in generative engine responses by up to 40%, with effectiveness varying by domain.
Scope. It was tested on engines of that period, and it measured visibility inside a benchmark, not traffic to a real business.
For the wider picture of the term itself, see what GEO is and the GEO guide.
How do you measure agentic navigability and citability?
You rebuild the site the way an agent sees it, as pages and links, and score what that picture shows. That is what GraphoRank does. It crawls the site while respecting robots.txt, reads each page the way a language model would ingest it, turns pages into nodes and internal links into edges, and produces one score from 0 to 100 called GraphRank. The how it works page shows the five stages, and the semantic graph page shows the map.
GraphRank reads four dimensions:
| Dimension | What it looks at | Leans toward |
|---|---|---|
| Structure | How pages connect: whether authority reaches your key pages, how deep your content sits, which pages are cut off entirely | Navigability |
| Technical health | Indexability, canonicals, redirect chains, broken links and sitemap consistency | Navigability |
| Meaning | Whether content holds together as topics: real clusters, pillars with genuine support, no gap between what a page promises and what it says | Citability |
| AI visibility | How easy pages are to read, extract and cite inside a generated answer | Citability |
The “leans toward” column is our editorial reading, not a split of the score into two numbers. Each dimension informs both ideas to some degree.
Every page in the graph also gets one of six roles: pillar, cluster, hub, money page (a page where visitors convert), orphan or noindex. Scores fall into five bands:
| Band | Score |
|---|---|
| At risk | 0–39 |
| Developing | 40–59 |
| Solid | 60–74 |
| Strong | 75–89 |
| Benchmark | 90–100 |
The exact formula is proprietary on purpose. A score with a public recipe can be gamed, and a benchmark that is easy to game stops being comparable across sites. What is public is what each dimension examines, as described above, and that every site is measured the same way on every scan.
The score is meant to lead to action. The recommendations list fixes ranked by how much they would move the score, such as orphans to reclaim, clusters to reconnect and money pages to surface. Re-scan afterwards and compare to see whether a change worked.
What should you check, and what does the evidence say?
| Area | Plain question | Evidence status |
|---|---|---|
| Link following | Can every important page be reached by links? | Mechanism is basic to crawling; no study measures the effect on AI answers |
| robots.txt | Have you chosen, bot by bot, what to allow? | Documented by OpenAI, Anthropic, Perplexity and Google; compliance varies |
| llms.txt | Is there a short, accurate guide to your site? | Unproven; Google says it neither helps nor harms Search visibility |
| Accessibility tree | Are headings, links and buttons real and labelled? | Documented by Chrome Lighthouse (experimental); related agent benchmark on desktop tasks |
| JavaScript | Is key text in the HTML the server sends? | No primary source found on AI crawlers; cautious rule only |
| Orphans and depth | Is every key page linked and close to home? | Reasoning, not measured |
| Content links | Do pages link to each other in the text? | Design choice, not measured |
| Competing pages | Is there one page per question? | Reasoning, not measured |
| Citable passages | Definitions, answer-first, sources, dates | Partly supported by research on engines and models; indirect |
| Structured data | Is it present and accurate? | Google: not required. Preprint: associated with citation |
Checklist: how to improve navigability and citability
The order is our judgment of likely impact for a typical small site, from cheapest and most fundamental to most refined. It is not a measured ranking.
- Make sure the pages that matter can be indexed. Check that they return a normal page, are not marked noindex by mistake and are not blocked in robots.txt. For Google’s AI features, the page must be indexed and eligible to be shown with a snippet.
- Decide your AI bot policy deliberately. Read the documentation of the bots you care about, then allow or block each one on purpose. Blocking a training bot is not the same as blocking a search bot.
- Find and fix orphan pages. List every page you want read and make sure at least one other page links to it, in the text where it makes sense.
- Bring key pages closer to the home page. Link your most important pages from the home page or from a strong hub page so that they are a click or two away.
- Link inside your content, with descriptive anchor text. When a paragraph mentions a topic you cover elsewhere, link there, using words that describe the destination, not “click here”.
- Put your essential text and links in the HTML the server sends. Test by viewing the page without JavaScript. If the main content disappears, fix that first.
- Use real headings, links and labels. One H1, sensible H2s, buttons and form fields with names. It helps people who use screen readers as well as agents.
- Give each question one page. Merge or redirect overlapping pages, and link the rest to the winner.
- Open sections with the answer and define your terms. State what each entity is in one plain sentence near the top.
- Show authorship, dates and sources. Name the author, show real publication and update dates and link to primary sources for facts and figures.
- Add FAQs where people really ask questions. Short, self-contained answers, written for readers first.
- Add structured data that matches the content. Article, Organization, FAQ and Person types are common starting points. Keep it accurate. It is hygiene, not a shortcut.
- Optionally, publish an llms.txt. Cheap and unproven; do it last.
- Re-check regularly. Sites change. A new page can become an orphan the day it is published.
You can run this checklist by hand on a small site. If you want it measured, the free GraphoRank audit scans the first 10 pages and shows the graph, the roles and the GraphRank. See pricing for larger sites.
How does this relate to SEO, AEO and GEO?
Search engine optimization (SEO) is about ranking in search results, answer engine optimization (AEO) is about being the answer a system gives, and generative engine optimization (GEO) is about being included in AI-generated responses. Our comparison of SEO, AEO and GEO goes through the differences, and the AI SEO guide covers the practical side.
Agentic navigability and citability are not a fourth label to learn. They are the two properties of your own site that all three disciplines rely on: can a machine reach and read it, and is what it reads worth quoting? For usage and traffic numbers, see AI search statistics.
FAQ
What is agentic navigability?
Agentic navigability is how well an AI agent can reach, crawl and understand a website by following its links. It depends on whether pages are reachable, linked from other content, readable without special tricks, and not blocked by mistake.
What is agentic citability?
Agentic citability is how likely a page's content is to be quoted in an AI answer. It depends on clear definitions, direct answers near the top of a section, named entities, sources, visible dates and authorship, and one clear page per topic.
What is the difference between agentic navigability and agentic citability?
Navigability is about access and understanding: can an agent find and read your pages? Citability is about use: once it has read them, will it quote them? A site can be easy to navigate and still not citable, and the reverse is rarely true.
Is agentic navigability the same as SEO for ChatGPT or LLM SEO?
They overlap, but they are not identical. SEO for ChatGPT and LLM SEO are search phrases for getting found and cited by AI assistants. Agentic navigability and citability name the two properties of your site that decide whether that can happen, and they can be measured.
Does blocking GPTBot remove my site from ChatGPT?
Not necessarily. OpenAI documents separate crawlers: GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for user-initiated fetches. According to OpenAI, sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
Do AI crawlers obey robots.txt?
Vendors say their main crawlers do, but behaviour varies. OpenAI and Perplexity document that user-initiated fetchers may ignore robots.txt, and an academic study found that some categories of bots, including AI search crawlers, rarely check it at all.
Do I need an llms.txt file?
No. Google states that you do not need new machine-readable or AI text files to appear in Google Search, and that llms.txt will neither harm nor help. It costs little to publish, but no study we found shows that it increases citations.
Do I need structured data to be cited by AI?
Google says structured data is not required for generative AI search. A preprint of 1,702 citations did find structured data, semantic HTML and metadata associated with more citations on English B2B SaaS pages. Treat it as good hygiene, not as a guarantee.
Why do orphan pages matter for AI agents?
An orphan page has no internal links pointing to it. An agent that discovers pages by following links has no path to it, so it can be missed even if it is in your sitemap. No published study has measured the size of this effect yet.
How do I measure agentic navigability?
GraphoRank crawls your site, rebuilds it as a graph of pages and links, and scores it from 0 to 100 as GraphRank across four dimensions: structure, meaning, AI visibility and technical health. A free audit covers the first 10 pages.
Is the GraphRank methodology public?
The four dimensions and what each one looks at are public. The exact formula is proprietary, so that the score stays comparable across sites and cannot be gamed. Every site is measured the same way on every scan.
Sources
- AI features and your website (Google Search Central) (October 10, 2026)
- Optimizing your website for generative AI features (Google Search Central) (October 10, 2026)
- Google crawlers: Google-Extended (Google Search Central) (October 10, 2026)
- OpenAI crawlers: GPTBot, OAI-SearchBot, ChatGPT-User (October 10, 2026)
- Does Anthropic crawl data from the web, and how can site owners block the crawler? (October 10, 2026)
- Perplexity crawlers: PerplexityBot and Perplexity-User (October 10, 2026)
- Cloudflare: Perplexity is using stealth, undeclared crawlers to evade no-crawl directives (October 10, 2026)
- Scrapers selectively respect robots.txt (Kim et al., arXiv) (October 10, 2026)
- HTTP Archive Web Almanac 2025: SEO chapter (October 10, 2026)
- SE Ranking: llms.txt and AI citations (October 10, 2026)
- Ahrefs: 137K sites, 97% of llms.txt files never get read (October 10, 2026)
- Common Crawl: a content analysis of llms.txt files from the July 2026 crawl archive (October 10, 2026)
- Chrome Lighthouse: Agentic Browsing category (October 10, 2026)
- A11y-CUA: accessibility gap in computer-use agents (arXiv) (October 10, 2026)
- AI Answer Engine Citation Behavior: the GEO-16 framework (arXiv) (October 10, 2026)
- GEO: Generative Engine Optimization (Aggarwal et al., KDD 2024, arXiv) (October 10, 2026)
- Lost in the Middle: how language models use long contexts (arXiv) (October 10, 2026)