llms.txt vs robots.txt: What Each One Does

robots.txt and llms.txt do two completely different jobs, and neither one replaces the other. robots.txt is an access file. It tells a crawler which URLs it may fetch. llms.txt is a content file. It tells a model which pages are worth reading once it is already allowed in. One is a door. The other is a map.

That distinction sounds obvious written down. It is not how these files get discussed. Search for the comparison and you will find a stack of posts framing them as rivals, as if llms.txt were the modern replacement for a file from 1994 and you had to pick a side.

You do not have to pick. They barely touch. The more interesting question, and the one almost nobody in that stack is asking, is which of the two is getting the standards work that will actually decide how AI companies use your content. It is not the new one.

What is the difference between llms.txt and robots.txt?

Here is the whole comparison in one table. Everything after this is detail.

That last row is the fastest way to feel the difference. Delete your robots.txt and you have changed what crawlers believe they are allowed to take. Delete your llms.txt and, for 97% of sites that publish one, nothing measurable changes at all.

What does robots.txt do?

robots.txt is a plain-text file at the root of a domain that tells automated clients which parts of the site they are asked not to request.

Martijn Koster wrote the original convention in 1994. It ran on nothing but goodwill for twenty-eight years until the IETF published it as RFC 9309, the Robots Exclusion Protocol, in September 2022. That RFC is standards track. It has formal parsing rules, error handling, and caching guidance, which is why a crawler written today and a crawler written in 2005 read your file the same way.

The syntax is four directives. User-agent names the bot a rule applies to, Disallow and Allow set the paths, and Sitemap points at your sitemap. That is the entire language. It has no concept of what your pages contain, no concept of quality, and no way to say “read this one first.”

It also does less than most people think. Google’s own robots.txt documentation says the file is “used mainly to avoid overloading your site with requests” and states plainly that it “is not a mechanism for keeping a web page out of Google.” A blocked URL that other sites link to can still appear in results. If you want a page gone from search, the tool is a noindex tag or a password, not a Disallow line.

For AI specifically, robots.txt is where the action is. GPTBot, ClaudeBot, Google-Extended, PerplexityBot, OAI-SearchBot and the rest are all controlled from this file, and each one does a different job. Our AI crawlers list covers every user-agent string and what blocking each one actually costs you, because the costs are not symmetrical.

What does llms.txt do?

llms.txt is a Markdown file at the root of a domain that lists a site’s most important pages with a one-line description of each, so a language model can read a clean index instead of crawling rendered HTML.

Jeremy Howard published the proposal at llmstxt.org on September 3, 2024. The reasoning is sound. A docs page rendered as HTML is mostly nav bars, cookie banners, and framework wrapper divs, so handing a model a Markdown index costs a fraction of the tokens and points straight at what matters.

The proposal is explicit about scope, and this is the part that settles the whole comparison. It states that llms.txt is “designed to coexist with current web standards” and can “complement robots.txt by providing context for allowed content.” Note the last two words. llms.txt describes content you have already allowed. It does not decide what is allowed.

The spec is equally clear about when the file gets used. It says llms.txt “will mainly be useful for inference,” meaning the moment a user asks an assistant a question, “as opposed to for training.” The author is not claiming this file governs training data. Nobody reading the source document would think it did.

The measured reality is thinner still. Ahrefs analyzed server logs across 137,210 domains and found that in May 2026, 97% of published llms.txt files received zero requests. The full picture, including who the remaining 3% actually are, is in our guide to writing an llms.txt file.

Does llms.txt replace robots.txt?

No, and the proposal’s own site says so. There are three reasons the replacement framing cannot work.

llms.txt has no permission syntax: There is no Disallow. There is no Allow. There is no User-agent line. The format is an H1, an optional blockquote, and H2 sections of links. You could not express “stay out of /admin/” in it if you wanted to, because the vocabulary does not exist.

The two files are read at different moments: robots.txt is fetched before anything else, as the first request of a crawl, and it decides whether the rest of the crawl happens. llms.txt is fetched during a session by an agent that already has access and is deciding what to open. One gates, the other guides.

Compliance is not comparable: Every major crawler has read robots.txt for three decades. llms.txt got zero requests on 97% of the domains that published it. Even if it had permission semantics, almost nothing is looking at the file to obey them.

Here is the mistake that follows from getting this wrong, and I have seen it on real sites. You block ClaudeBot in robots.txt because you read a post about AI scraping. Then you spend an afternoon writing a beautiful llms.txt because you read a different post about AI visibility. You have now locked the door and left a map on the doorstep. Anything that respects the lock never reads the map, and anything that ignores the lock never needed it.

Which file controls AI training?

robots.txt does, to the extent anything does. And the more useful version of this answer is that robots.txt is being actively extended to control it better, while llms.txt is not being extended at all.

Two efforts matter right now.

Cloudflare’s Content Signals Policy: launched September 24, 2025, this adds a machine-readable vocabulary to robots.txt that separates uses rather than just access. Before it, robots.txt could mostly say crawl or do not crawl. Now a publisher can allow the fetch and still object to what happens afterward. Cloudflare’s managed robots.txt documentation shows the directive as Content-signal: search=yes, ai-train=no, use=reference. At launch, Cloudflare said more than 3.8 million domains use its managed robots.txt service.

The IETF AIPREF working group: chartered in February 2025 to standardize exactly this. Its stated problem is that “AI vendors use a confusing array of non-standard signals in the robots.txt file (defined by RFC 9309) and elsewhere,” and its charter announcement describes building a common vocabulary plus a way of attaching it “by formats similar to robots.txt.” The vocabulary draft, revised April 28, 2026, defines two categories with y and n values.

Read that table and notice what is missing. Every one of these signals attaches to robots.txt. Not one of them attaches to llms.txt. The standards bodies, the CDN with 3.8 million domains behind it, and the working group with an August 2026 milestone are all building on the file from 1994.

That is my uncomfortable take on this whole topic. The SEO industry spent two years writing about llms.txt because it is new, easy to publish, and impossible to disprove. Meanwhile the file that will actually decide whether a model is allowed to train on your work is the boring one that was already sitting in your web root. The interesting file and the important file are not the same file.

Do robots.txt and llms.txt actually stop anything?

Neither one is a wall. This is worth saying plainly because both are routinely sold as protection.

robots.txt is a request. A crawler that ignores it has broken a convention, not a lock, and the page is still served. That is why Google’s documentation is careful to tell you it will not keep a page out of search results, and why Cloudflare’s own announcement frames Content Signals as expressing preferences and encouraging companies to respect them rather than as a technical control.

llms.txt is not even a request. It is a document. Its worst case is that nobody reads it.

If you need something enforced, it has to happen at the edge before the response is sent. That means a firewall rule, bot management, rate limiting, or authentication. Everything discussed above is a sign on the lawn. Signs are useful, because most large crawlers do read and honor them and because a documented preference matters if you ever end up arguing about it. Just do not confuse a sign with a fence.

Do you need both files?

Almost certainly one, probably not both. Match your situation to the row.

That last row is the one people get wrong most often. Google’s AI features documentation states that you do not need to create new machine-readable files or AI text files to appear in AI Overviews or AI Mode. If AI visibility is the goal, the work is in the pages themselves, which is what answer engine optimization covers, and what ranking in ChatGPT actually turns on.

What should you do this week?

Three checks, twenty minutes.

1. Read your robots.txt as if you were a crawler: Fetch yourdomain.com/robots.txt and look at it fresh. Most sites carry rules a developer added years ago for reasons nobody remembers. Check whether you are blocking GPTBot or Google-Extended by accident, and whether you meant to.

2. Decide the training question deliberately: Pick a position and write it down. If you want the search visibility but not the training, Content Signals gives you a way to say that in one line. If you have no strong view, leaving robots.txt permissive is a real choice, not a failure to choose.

3. Only publish llms.txt if you have docs: If developers point Cursor or Claude Code at your documentation, the file earns its keep. If you run a marketing site, skip it and spend the hour on a page that answers a question well enough to be quoted.

Once those are set, the only question left is whether any of it moved the number you care about, which is whether AI tools name you when someone asks about your category. ContextBolt SEO answers that from inside your agent. You connect one MCP URL, ask in plain language whether ChatGPT and Claude are mentioning you, and it reports your share of voice against named competitors alongside normal keyword, rank, and backlink research. It is $35 a month, month to month, and it saves every lookup to your project as markdown so the research does not vanish when the chat does.

The short version of all this. robots.txt is the door and it is thirty years old, standardized, universally read, and quietly becoming the place where AI permissions get decided. llms.txt is the map, it is two years old, mostly unread, and genuinely useful to the handful of coding agents that fetch it. Keep the door. Draw the map only if someone is coming inside.