llms.txt: What It Is and How to Write One (2026)

llms.txt is a Markdown file you put at the root of your domain, at /llms.txt, listing your most important pages with a one-line description of each. Jeremy Howard published the proposal on September 3, 2024, and the idea was simple. Context windows are small and websites are full of navigation, popups, and markup, so hand a model a clean map instead of making it chew through HTML. It is a convention, not a standard anyone enforces.

Two years on, the file has a reputation problem. Half the internet will tell you it is essential for AI search. Google’s own documentation tells you not to bother. Both camps are arguing about the wrong question, because llms.txt turned out to have a real user, and it is not the one anybody was optimizing for.

This guide covers the spec, a working example you can copy, how to write one in about ten minutes, and the measured data on who is actually fetching these files in 2026. I publish one, I checked what it does, and I am going to tell you the unflattering half.

What is llms.txt?

llms.txt is a Markdown file served at the root of a website that gives large language models a curated, machine-friendly index of that site’s most useful pages.

The reasoning behind it is sound. A documentation page rendered as HTML is mostly not documentation. It is nav bars, sidebars, cookie banners, scripts, and framework wrapper divs. If a model has to spend most of a fetch on that, it burns context on nothing. A Markdown index costs a fraction of the tokens and points straight at the pages that matter.

The official proposal at llmstxt.org is deliberately narrow. It is a file format, not a permissions system. It does not tell anyone what they may crawl, it does not block anything, and it carries no enforcement of any kind. Nobody has to read it and nobody has promised to.

That last point is where most of the confusion starts. robots.txt works because crawlers agreed to obey it. sitemap.xml works because search engines agreed to consume it. llms.txt has no such agreement behind it. It is a good idea that shipped without a counterparty.

What does an llms.txt file look like?

The format is Markdown with a fixed shape. Only the first element is required.

Cloudflare publishes a clean, spec-compliant example. Here is the shape of their developer docs llms.txt, trimmed down:

# Cloudflare Developer Documentation

Explore guides and tutorials to start building on Cloudflare's platform.

> Each product below links to its own llms.txt, which contains a full index
> of that product's documentation pages.

## Application performance

- [Argo Smart Routing](https://developers.cloudflare.com/argo-smart-routing/llms.txt): Route web traffic across the fastest, most reliable network paths
- [Cache / CDN](https://developers.cloudflare.com/cache/llms.txt): Make websites faster by caching content across our global server network

Two things worth stealing from that. Cloudflare nests the format, so each product gets its own llms.txt and the root file is a directory of directories. And every link carries a description, which is the part people skip and the part that does the actual work. A bare list of URLs tells a model nothing about which one to open.

Not everyone follows the spec. Anthropic’s developer docs serve a 600-line file with prose headings the spec does not define, and it works fine, because there is no validator with teeth. Treat the format as a strong convention rather than a contract.

How do you write an llms.txt file?

Ten minutes, five steps.

1. List the pages you would send a new hire: Not every page. The ones that explain what the thing is, how to install it, and how to do the three most common jobs. Fifteen to fifty links is a normal range. A file with 400 links is a sitemap with extra steps.

2. Write the H1 and the blockquote: The H1 is your product or site name. The blockquote is one or two sentences that let a model understand what you are before it reads anything else. Be concrete. “MCP-native tools that connect Claude and Cursor to the data they are missing” beats “a platform for modern teams.”

3. Group the links under H2 headings: Group by product, or by job, or by section of your docs. The heading is context, so name it something a model can reason about. “Getting started” and “API reference” are useful. “Resources” is not.

4. Add a description to every link: Format is [Page title](https://example.com/page/): what this page covers. If your pages already have meta descriptions, reuse them. This is the highest-value part of the file and it is the one most published examples get lazy about.

5. Generate it, do not hand-maintain it: A hand-written llms.txt goes stale in a month. Ours is generated from the docs navigation and each page’s frontmatter description at build time, so it cannot drift from the real doc set. If your site is static, this is twenty lines of build code. If it is WordPress, several plugins do it.

Serve it as text/plain at https://yourdomain.com/llms.txt. That is the whole job.

What is llms-full.txt?

llms-full.txt is the second file you will see referenced, and it is worth understanding because it does a genuinely different job.

llms.txt is an index. It lists pages and the agent then fetches the ones it wants, which means several round trips. llms-full.txt is the whole documentation set flattened into one large Markdown file, so an agent can pull everything in a single request and keep it in context.

Only llms.txt appears in the original proposal. llms-full.txt grew out of documentation platforms, Mintlify’s implementation in particular, and spread because it is what an agent actually wants when you point it at a docs site. Anthropic’s docs use exactly this pattern, a slim index linking to a much larger full export.

The tradeoff is size. A full export of a large docs site can run to hundreds of thousands of tokens, which is more than most context windows and more than most people want to pay for. Publish both if you have docs worth reading. The index is for discovery and the full file is for the agent that has decided it needs everything.

Does llms.txt help you rank or get cited in AI search?

No. This is the part the marketing posts get wrong, and the evidence is not ambiguous.

Google’s own AI features documentation says it plainly. “You don’t need to create new machine readable files, AI text files, or markup to appear in these features.” There are no extra requirements for AI Overviews or AI Mode beyond being indexed, being snippet-eligible, and doing normal SEO. Google’s John Mueller has separately called the file purely speculative and compared it to the keywords meta tag, noting the awkward fact that the file has existed for years and none of the AI systems use it.

Then there is the measurement. Ahrefs analyzed server-log data from 137,210 domains and published the results in June 2026. Of the domains studied, 28% published an llms.txt file. In May 2026, 97% of those files received zero requests. Roughly 1,100 domains saw any traffic to the file at all.

So the honest summary is that the file is widely published, almost never read, and explicitly disclaimed by the one search engine that matters most. If you added llms.txt hoping to appear in more AI answers, you did not get what you paid for, and the correct next move is to spend that effort on answer engine optimization instead, which is about the structure of the pages themselves.

So who actually reads llms.txt?

Here is the interesting half, and it is the reason I still publish one.

The Ahrefs study did not just count zero. It named the requests that did land. Once you break the traffic down, a clear pattern shows up.

Read that table twice. The single largest consumer of llms.txt is SEO tooling checking whether you have one, which is a snake eating its own tail. And the second-largest named AI fetcher is Claude Code, which is not a search engine discovering your brand. It is a developer sitting in a terminal, pointing an agent at your documentation, and asking it to read the API reference.

That is the real user. Not GPTBot building an index of the web. A coding agent that has been told to go learn a specific library, right now, on behalf of someone who already knows your product exists.

Every IDE agent has converged on this behavior. Point Cursor, Windsurf, Claude Code, or Copilot at a docs site and the first thing it reaches for is /llms.txt and /llms-full.txt, because that is the cheapest possible way to load a doc set. The file is a documentation delivery format that got marketed as an SEO tactic, and the mismatch is why every study of it reads like a failure.

Should you publish an llms.txt file?

My take, and it is not the one you will read on most SEO blogs.

Publish one if you have documentation. If developers point AI tools at your docs, llms.txt is the difference between an agent reading a clean index and an agent scraping four rendered HTML pages to find your auth flow. That is a real user experience improvement for the exact person most likely to become a customer, and it costs a build script.

Skip it if you run a marketing site and want AI visibility. A ten-page site about your agency does not need a machine-readable index of ten pages. Nothing will fetch it, and the hour you spend on it is an hour not spent on the work that does move citations, which is publishing pages that answer questions cleanly enough to be quoted. Our guide on how to rank in ChatGPT covers what actually moves that number.

Never treat it as a substitute for the fundamentals. llms.txt cannot make thin content citable, it cannot fix a page that renders as an empty shell without JavaScript, and it does not grant crawl permission you did not already give in robots.txt. It is a convenience layer on top of a site that already works.

The uncomfortable version, since I promised one. A lot of llms.txt guidance in 2026 exists because “add this file” is an easy thing to sell and an easy thing to check off. It has the shape of an SEO deliverable, it takes minutes, and nobody can prove it did nothing. That combination is exactly how the keywords meta tag survived for a decade after it stopped mattering. Publish the file for the coding agents. Do not put it in a strategy deck.

How to check whether yours is doing anything

Three checks, in order of how much they tell you.

Check the file parses: Fetch https://yourdomain.com/llms.txt in a browser. It should return plain text, start with a single # heading, and contain absolute URLs. Relative links are the most common mistake and they break the file for any agent that fetched it from somewhere else.

Check your server logs: This is the only measurement that answers the real question. Grep your access logs for llms.txt and look at the user agents. If you see coding-agent traffic, the file is doing its job. If you see nothing but audit tools for three months, you now know exactly how much it is worth to you.

Check the pages behind it, not the file: If your goal was AI visibility, measure the outcome rather than the tactic. Ask your target questions in ChatGPT, Claude, and Perplexity and record whether you are named. The free AI Visibility Checker scores that 0 to 100 and names the brands recommended in your place, which is a far more useful number than whether a file exists.

For the ongoing version of that check, ContextBolt SEO runs it from inside your agent. You connect one MCP URL and ask it, in plain language, whether AI tools are mentioning you and how your share of voice is moving against named competitors. It is $35 a month, month to month, and it also does the keyword, rank, and backlink research you would otherwise open a $129 tool for. The best AEO tools comparison covers the alternatives honestly if you want the full field first.

llms.txt is a good file with a bad reputation, earned by the people who oversold it. It is not a ranking signal and it never was. It is how you hand a coding agent your documentation without making it read your nav bar, and judged as that, it works.