GenReady scanned the 500 most popular domains on the internet for the eight agent-readiness signals our analyzer checks. The result: 24.7% set explicit AI crawler rules, 14% publish llms.txt, 2.7% serve markdown to agents, and exactly one domain, fraud-prevention company Forter, publishes MCP, A2A, and UCP files together.

The agent-ready web is loudly discussed and barely built. Here is the first data set, and what it means if you run a website.

Sweep run July 22, 2026, against the Tranco top-500 list. We re-run it before each report.

How did we measure this?

We fetched eight signals from each of the top 500 domains: AI crawler rules in robots.txt, llms.txt, markdown content negotiation, MCP server cards, A2A agent cards, Agent Skills indexes, UCP manifests, and x402 manifests. Each one gets a plain-language explainer in our agent readiness series, publishing on this blog over the coming weeks.

328 of the 500 domains serve a reachable website; the rest are infrastructure domains like CDN and DNS hosts. Percentages below use the 328.

We validated strictly. JSON files had to parse as JSON. 62 domains answer any well-known path with a 200 response, so we probed a nonsense path on every domain and discarded detections from those catch-alls, then manually verified every rare positive. The numbers below are conservative by design.

What has real adoption? The permission layer.

The clearest finding: the web adopted the "no" layer before the "yes" layer. 81 of 328 sites (24.7%) name AI crawlers explicitly in robots.txt. Sites learned to state terms to AI companies before they learned to serve them.

llms.txt follows at 46 sites (14%), including GitHub, Adobe, Shopify, PayPal, Samsung, and Dropbox. That tracks the "over 10% of major domains" figure from earlier this year. It is also a striking pairing with the evidence: Search Engine Land tested llms.txt on nine sites and found no citation or traffic lift on eight. Adoption is real even though measured benefits are not. It is cheap, so people ship it.

Who actually serves markdown to agents?

Nine sites (2.7%) return markdown when asked with an Accept: text/markdown header: Cloudflare, WordPress.org, Sentry, Linktree, Zendesk, GitLab, Shein, AddToAny, and Cloudflare's workers.dev.

Small number, telling names. These are infrastructure and developer companies plus one massive retailer, exactly the crowd that watches its agent traffic logs. Markdown negotiation is the check where adoption most lags value: it answers a request Claude Code and OpenCode already send, and 97% of top sites ignore it.

Where are the agent protocols?

Nearly nowhere, at the top of the web.

Four domains publish an MCP server card: Cloudflare, Sentry, Linktree, and Forter. One publishes an A2A agent card: Forter. One publishes a UCP manifest: Forter again. Zero publish an Agent Skills index. Zero publish an x402 manifest.

Forter deserves its moment. A fraud-and-identity company for digital commerce is the single most agent-ready domain in the top 500, with MCP, A2A, and UCP files all live. When agents transact, someone has to verify them, and Forter clearly intends to be in that room early.

The UCP number needs context. UCP counts 8,000+ verified stores, but roughly 99% run on Shopify, where it ships by default. Those are small and mid-size merchants, not top-500 domains. Adoption is happening bottom-up through platforms, not top-down through the web's giants.

The zeros need context too. x402 processed 169 million payments in its first year, but its 100,000 sellers are API and data products, mostly behind AWS and Coinbase infrastructure rather than the consumer domains that top the Tranco list.

What does this mean for your site?

Three takeaways from the data.

The early-mover window is wide open. If four MCP cards and nine markdown-serving sites is the state of the top 500, a mid-size site that ships the basics, schema markup, markdown serving, explicit robots rules, and the right discovery files, is ahead of roughly 97% of the web's biggest properties. That is rare positioning for about three hours of work.

Protocol news volume and protocol deployment are different things. WebMCP, NLWeb, and the commerce protocols generate headlines weekly; their footprint among top domains rounds to zero. Calibrate your urgency to what your site type actually needs, not to the announcement cycle. Our upcoming series covers exactly that call, protocol by protocol.

And the direction is one-way. A quarter of top sites writing AI crawler rules means the web's biggest properties take agent traffic seriously enough to regulate it. Serving that traffic well is the obvious next step, and the names already doing it, Cloudflare, GitLab, Shopify, are not companies known for wasted engineering.

We will re-run this sweep and publish the trend every quarter, with the tracker page updated monthly.

Where does your site stand against the top 500? Run a free GenReady scan — all eight signals, under 60 seconds.