Bot Access Rules: Which AI Crawlers Your robots.txt Lets In
How robots.txt decides which AI bots can reach your content, and why llms.txt has nothing to do with it.
What it measures
This check reads your robots.txt and works out which AI crawlers can reach your content. It evaluates the file the way a crawler does, following RFC 9309: per-agent groups, Allow overrides, longest-prefix matching, and wildcards. It then reports which named AI bots, such as GPTBot, ClaudeBot and PerplexityBot, are blocked from your site root.
Why it matters for AI
robots.txt is how you tell crawlers what they may fetch. Block an AI agent, whether by naming it or with a blanket "Disallow: /" under "User-agent: *", and it skips your content. Nothing you do to the page itself changes that.
| Status | Meaning | Impact |
|---|---|---|
| All agents allowed | No rule in robots.txt blocks a known AI crawler | AI crawlers can reach your content |
| Some agents blocked (for example, 3 of 8) | Some of the AI bots we check are disallowed by name | Those bots skip your site. Crawlability is capped at 50 |
| Every AI agent blocked (8 of 8) | Every AI bot we check is disallowed, even though search engines are still allowed | No AI crawler reads the page. Crawlability is capped at 10 |
| All crawlers blocked | A blanket Disallow: / under User-agent: * with no exceptions for AI bots | No crawler reads the page. Crawlability is capped at 10 |
robots.txt and llms.txt do different jobs
These two get conflated constantly, so to be plain about it: llms.txt has no blocking semantics. It cannot allow or deny a crawler, and no version of the spec has suggested it could. It is a curated index of what is worth reading, used once an agent already has access.
Access is robots.txt. Curation is llms.txt. We grade llms.txt as its own metric.
How to improve
- Read your robots.txt - visit yoursite.com/robots.txt and look at every Disallow rule
- Remove a blanket Disallow: / - under User-agent: * it blocks everything, AI crawlers included
- Decide about named AI bots deliberately - many sites inherit these rules from a plugin default without ever choosing them
- Block paths, not domains - disallow the directories you want kept out rather than the whole site
💡 Quick win
Open yoursite.com/robots.txt now and search for "Disallow: /" under "User-agent: *". That single line, if it is there, is why AI assistants never quote you.