Adoption of llms.txt nearly tripled in six months. SE Ranking found the file on 10.13% of domains in November 2025; by May 2026 Ahrefs found it on 28%. Then Ahrefs read the server logs. Of the roughly 38,000 domains carrying a valid file, 97% got zero requests for it that month.

Meanwhile a quieter standard is doing real work. Claude Code, Cursor, OpenCode and GitHub Copilot send Accept: text/markdown with their requests, asking your server for a cheaper version of the page.

One is a signpost most agents ignore. The other is a format agents ask for by name.

Findings verified September 9, 2026.

What is llms.txt supposed to do?

It is a plain text file listing your most important pages for AI systems, usually linking to a markdown copy of each one. Jeremy Howard of Answer.AI proposed it in September 2024. The pitch: crawlers waste effort on messy HTML, so hand them a curated map.

The catch is that the big AI crawlers never agreed to read it. Googlebot does not fetch it, and nobody has documented a major AI assistant changing its citations because of one. Our walkthrough of the format covers the file structure.

What does the evidence say?

Three independent studies, and not one found a citation lift.

SE Ranking modeled about 300,000 domains in November 2025, testing whether the file predicted getting cited in AI answers. It did not. Dropping llms.txt made the model more accurate.

Search Engine Land watched 10 sites for 90 days each. Eight saw no change in AI traffic, one insurance site fell 19.7%, and two rose. The author traced both gains to other work done at the same time: press coverage, rebuilt product pages, 27 new templates.

Then Ahrefs went to the server logs, which is the part we keep coming back to. Their June 2026 study covered 137,210 domains. Across a full month, 97% of the valid llms.txt files on them were never fetched. Not rarely. Never.

Who fetched the other 3%? Of those 22,000 requests, SEO audit tools alone made 21.7%. Add the four AI categories together and AI bots are the largest single bucket at 19.5%, but that is training crawlers and coding-agent infrastructure. The retrieval bots that answer a live question and produce a citation are 1.1%.

That gap is the point. The file is reaching the systems that train models and write code, not the ones answering questions. Claude Code pulled more of it than the big AI search crawlers did. OpenAI, Anthropic and Google all publish one for their own developer docs, and Mintlify argues for it from the docs-platform side.

So the finding narrows rather than reverses. Want Perplexity to cite your consumer site? Do not expect much. Do coding agents read your documentation? The file earns its keep.

What changed in llms.txt v2?

Howard published a second version of the spec on August 10, 2026. The change that matters is that agents can now find the file instead of guessing where it sits. Two standard link relations do it. rel="describedby" points at the llms.txt covering a page, and rel="alternate" type="text/markdown" points at that page's markdown version. Either can sit in the page HTML or in an HTTP response header you set once at your CDN.

Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

This is not theoretical. OpenAI's Codex CLI reads the alternate link to find markdown pages, and never sends a markdown request header at all.

v2 also says plainly that the links inside an llms.txt should point at content built for LLMs. Which means the two standards in this article's title were never really rivals.

What is markdown content negotiation?

One URL, two formats. A browser asks for HTML and gets your normal page. An agent sends the header Accept: text/markdown and gets clean markdown: same words, none of the layout machinery.

We fetched thirteen documentation pages twice on September 9, 2026, once as HTML and once as markdown. All thirteen honored the header, and the markdown responses came back 94 to 99% smaller, a median of 96%.

Read that number carefully. It measures the response, not the writing. The prose is about the same length either way. What disappears is scaffolding: framework JSON, scripts, markup the agent downloads and never reads. An agent that converts your HTML to markdown itself saves far less. Compare markdown against only the visible text of a server-rendered page and the gap is small: Next.js 8%, Cloudflare 11%.

The response size is still the number that counts, because agents pay per token to read. Cheap pages get read more often, and read to the end.

Which agents ask for markdown?

Checkly tested seven coding agents in February 2026 and found three sending the header. A live tracker keeps score, and as of September 2026 it reads:

AgentRequests markdown?HowClaude CodeYesAccept: text/markdown, text/html, */*CursorYesMarkdown first, HTML by wildcardOpenCodeYesExplicit q-values, markdown at 1.0GitHub Copilot (chat and CLI)Yes, added June 2026Markdown first, HTML at q=0.9Codex CLIPartialFollows rel="alternate" links insteadChatGPT and PerplexityNoHTML onlyGemini, Devin, WindsurfNoHTML only

Notice the split. Every agent that asks is a coding agent. Every consumer AI search product takes HTML and parses whatever arrives. Same divide the Ahrefs logs found, so know which side you are on before spending a day on either standard.

And the asking side is growing. Mintlify's July 2026 traffic report put direct markdown requests at 54.4% of machine-route volume, up from 25.1% in February.

How do you serve it?

Check your settings before writing any code. On a docs platform or behind Cloudflare you may already have it: Read the Docs turns it on for every hosted domain.

Use the header, not a .md URL suffix. Cloudflare's docs 404 on the suffix while the header works, and Cursor's docs hand back the ordinary HTML page for .md. A suffix is a guess the agent makes per site. The header either gets honored or does not.

On your own server the logic is one conditional. In Express:

app.get('/blog/:slug', (req, res) => {
  if (req.accepts(['html', 'text/markdown']) === 'text/markdown') {
    return res.type('text/markdown').send(getMarkdown(req.params.slug));
  }
  res.send(renderHtml(req.params.slug));
});

Keep 'html' first in that array. Express breaks ties by the client's ordering before its own, so the markdown agents above still get markdown, while a browser or a bare Accept: */* client falls through to HTML. Flip the order and every wildcard client gets markdown it never asked for.

Where does the markdown come from? If your content starts life as markdown, as most blogs and docs sites do, serving the source file is nearly free. If it lives in a CMS as HTML, convert it at publish time with a library like Turndown and cache the result, minus navigation, cookie banners and related-post widgets.

One warning from GenReady scans: send real markdown, not HTML with a text/markdown label on it. A mislabeled response is worse than none, because the agent budgeted for markdown and paid HTML prices.

Then read your server logs. Agents do not run JavaScript trackers, so none of this reaches normal analytics. Counting requests that carry Accept: text/markdown is the only honest measure of whether it worked.

So what should you do?

Serve markdown first, then wire llms.txt to it. Negotiation answers a request real agents already make, and it cuts what it costs them to read you by an order of magnitude. Once the markdown versions exist, an llms.txt takes ten more minutes and turns them into something an agent can navigate: a short list of your key pages, each linking to the markdown rather than the HTML. Add the describedby and alternate link relations while you are in there. Two lines of configuration, and Codex CLI already reads them.

You can watch this working today. Ask nextjs.org for the markdown version of its installation page and the file opens with frontmatter carrying docs_index: /docs/llms.txt. The markdown page points back at the index that lists it, so an agent landing on either one has a route to the other. That is the whole argument in one line, shipped by somebody else.

Markdown, then index, then the links tying them together. Expect the index on its own to move your citations and 137,000 domains of log data will disappoint you.

Both sit in the content layer of the wider stack, next to schema.org. For where they fit among all 17 protocols, see the agent readiness stack guide.

Does your server speak markdown when an agent asks? Run a free GenReady scan and find out in under 60 seconds.