ai.txt: The Permissions File That Is Not llms.txt

You went looking for ai.txt and half the results were about llms.txt. That is not a search engine mistake. It is the whole story of this file.

ai.txt is a plain text file some sites put at their root, next to robots.txt. The version you will actually find on live sites was generated by a tool from Spawning in 2023, and it looks like this.

# Spawning AI
# Prevent datasets from using the following file types

User-Agent: *
Disallow: *.txt
Disallow: *.pdf
Disallow: *.jpg
Disallow: *.png
Disallow: *.mp3
Disallow: *.mp4
Disallow: *.js
Disallow: /
Disallow: *

That is a permissions file. It says what AI companies may take for training. llms.txt, the file it keeps getting mixed up with, is a map. It says what an AI reader should read first. Same folder, same two years, opposite jobs. And the name ai.txt has since been claimed by at least four other proposals, so the file you find depends on who wrote it.

Where you are seeing this

If you arrived here from a mention rather than a search, it came from one of these places.

• A site’s root folder, next to robots.txt, on a site that ran Spawning’s generator. Try https://example.com/ai.txt on an artist’s portfolio and there is a fair chance you will find one.
• An SEO audit report with a row that checks for ai.txt alongside llms.txt, usually flagged as missing.
• A GitHub repository named ai.txt, of which there are several, each describing a different file.
• The IETF datatracker, where an individual draft titled AI.TXT was posted on June 12, 2026.
• A platform help page. Spawning ships install guides for WordPress, Squarespace, Shopify, Wix and Webflow.

All five are talking about a root-level text file. They are not all talking about the same one.

What ai.txt actually is

The ai.txt with real deployments is Spawning’s. Spawning is the company behind Have I Been Trained, the tool artists used to check whether their work was in the LAION dataset. On May 30, 2023 it announced ai.txt as a way for websites to set permissions for AI training, and shipped a generator that writes the file for you.

The generator is five toggles. Text, images, audio, video and code, each set to block or allow. By default everything is blocked. Flip a toggle to allow and the generator drops that media type’s extensions from the Disallow list. Download the file, put it at the root of your site, done.

Three design choices explain what the file is for.

It is written per file type, not per path. robots.txt says “do not crawl /archive/”. ai.txt says “do not train on *.jpg”. That is a rights holder’s view of a website, not a webmaster’s. A photographer does not care which folder the image lives in. They care that it is an image.

It is meant to be read at download time, not crawl time. Spawning’s own framing is that the file is checked when a piece of media is downloaded, so a dataset that already holds a link to your image can still be told no when it goes to fetch it. That is a different moment from robots.txt, which is read before a crawler walks the site.

It targets one law. The EU’s Digital Single Market directive lets commercial text and data mining go ahead unless the rights holder has reserved those rights in a machine-readable way. Spawning’s generator page says the file is built to be that reservation. Everything about ai.txt makes more sense once you know it is a legal opt-out wearing a robots.txt costume.

Why it keeps getting confused with llms.txt

Both files live at the site root. Both are plain text. Both appeared in the same window, ai.txt in May 2023 and llms.txt in September 2024. Both have “AI” in the name and “txt” on the end. Google does not yet have a big enough corpus for ai.txt to keep the two apart, so it pads the results page with the neighbor.

The jobs are opposite, and this table is the whole distinction.

If you want the door, that is robots.txt and its newer extensions. If you want the map, that is llms.txt. ai.txt is a sign on the door written for one specific visitor. The difference between the door and the map is covered in llms.txt vs robots.txt and this post does not repeat it.

Five files share the name

Here is the part that makes ai.txt genuinely confusing, and it is not Google’s fault. The filename has been claimed at least five times for different jobs.

Two of those matter for a reader who lands here today.

The aitxt.ing format is why the llms.txt confusion is now partly justified. Its own page draws the line as llms.txt answering “where are the docs” and ai.txt answering “what is this, what are its limits, and how should you treat it”. That is a content map with opinions, and it competes with llms.txt for the same reader. So the sentence “ai.txt is a permissions file” is true of the file on most sites and false of the one the newest proposal describes.

The IETF draft is not what its address suggests. draft-car-ai-txt-wellknown-00 is an individual submission by one author, dated June 12, 2026, expiring December 14, 2026. The datatracker lists no stream and no working group, and the draft’s own text says it is not endorsed by the IETF and has no formal standing. It also says plainly that the file is advisory and that servers must enforce rate limits and access control on their own. It is a proposal that happens to be hosted at ietf.org, which is a different thing from a standard.

Who actually reads ai.txt today

This is the question that decides whether the file is worth your afternoon, so it was checked against each vendor’s own crawler documentation rather than against anyone’s blog.

OpenAI’s bots page documents GPTBot, OAI-SearchBot and ChatGPT-User. The controls it describes are robots.txt rules and a published IP range. It does not mention ai.txt. Google’s crawler list explains that Google-Extended is the token you use in robots.txt to manage whether crawled content trains Gemini. It does not mention ai.txt. Anthropic’s support page names ClaudeBot, Claude-User and Claude-SearchBot and says they honor robots.txt. It does not mention ai.txt. Perplexity’s bot documentation is the same story for PerplexityBot.

So the four companies whose crawlers most site owners are worried about have a documented control, and it is robots.txt in every case. The full list of their user agents, with what each one fetches, is in our AI crawlers list.

What ai.txt does have is Spawning. Its generator page says the company’s API communicates ai.txt permissions to “a growing list of AI researchers and partners”, and the launch post named Hugging Face and Stability AI. That is a real channel. It is also one company’s channel, and that company’s main domain now serves a data marketplace called Source.Plus, so the generator lives on a subdomain with a 2023 copyright line.

Meanwhile the standards work went somewhere else. The IETF chartered a working group called AIPREF to standardize how sites express AI preferences, and its charter points at the Robots Exclusion Protocol, not at a new file. The W3C’s TDMRep protocol does the EU text-and-data-mining reservation, the exact legal job ai.txt was built for, with a JSON file at /.well-known/tdmrep.json and a tdm-reservation flag. And Cloudflare’s Content Signals put training, search and AI-input permissions inside robots.txt itself, which is covered in the comparison post below. The standards bodies and a major CDN all extended robots.txt or picked a well-known path of their own. Of the efforts that set training permissions, Spawning’s file and the individual IETF draft are the two that kept the ai.txt name, and neither has a major crawler behind it.

Should you publish an ai.txt file?

Here is the take, and it is the one I would give a client. ai.txt is a 2023 answer to a question the EU asked, wearing a 2026 name the agent crowd wants for something else, and neither crowd’s crawlers read it.

That said, publishing one costs nothing and does no harm. Decide by what you are actually trying to do.

If you want to reserve training rights, especially in the EU: Publish TDMRep at /.well-known/tdmrep.json and add Content Signals to robots.txt. Those are the mechanisms with a standards body or a major CDN behind them. Add Spawning’s ai.txt as well if you like. It is one more machine-readable reservation, and a rights claim is stronger with three signals than one. Just do not stop at ai.txt and believe the job is done.

If you want AI agents to read your site well: That is not ai.txt at all, unless you mean the aitxt.ing format. It is llms.txt, and even that file’s honest traffic data shows its single largest source of requests is SEO audit tools checking whether it exists.

If you want to block AI crawlers: robots.txt, with the user agents from the crawlers list, and read should you block AI crawlers first, because the block usually costs more than it protects.

If you want to be cited by ChatGPT or Gemini: No file does this. Not ai.txt, not llms.txt, not robots.txt. Citation is decided by whether your page is the one an answer engine reaches for on a question, and that is measured, not declared.

How to check what your site has

Four requests tell you which of these files exist on any domain. Run them against your own site, then against a competitor you suspect has thought about this.

Copy this prompt

Fetch these four URLs for example.com and tell me which exist:
https://example.com/robots.txt
https://example.com/ai.txt
https://example.com/llms.txt
https://example.com/.well-known/tdmrep.json
For each one that exists, summarize what it permits or forbids
for AI crawlers and AI training, in plain English.

Any agent with web access can run that. Two things to read in the result.

A 200 with an HTML body is a miss. Plenty of hosts serve the homepage for any unknown path, so ai.txt returns 200 and a page full of <div> tags. A Spawning-format file starts with a # comment or a User-Agent line; if the body opens with <div> tags, there is no file. That is exactly how Spawning’s own domain behaves today.

Case matters. One portfolio checked while writing this served /ai.txt correctly and returned a 404 for /AI.txt. Spawning’s generator writes the lowercase name. Use it.

Then look at what you found next to what you meant. If robots.txt says nothing about GPTBot and ai.txt says deny everything, you have written the opt-out in the file the big crawlers do not read and left the file they all read silent.

What none of these files can do

Every file on this page answers “what may you take” or “what should you read”. Not one of them answers “did you use it”.

A model can fetch your page, honor every permission you set, and still answer the question without pointing anyone at you. A crawler that respects ai.txt is still a crawler. A perfectly written llms.txt still shows up in your logs more often as an audit tool checking it exists than as any AI fetcher. The thing that actually moves traffic, being the source an answer engine names, leaves no trace in any of these files and no trace in your access log either. How that choice gets made is its own subject, in how AI assistants choose citations.

ContextBolt SEO is ours, so weigh the bias. But this is the job it was built for. Ask, in plain English, whether ChatGPT or Gemini names your site on the questions that matter to you, and the tool asks the engine live and reports back with every source it cited. Ask how often your brand appears across AI answers, broken down by platform, and who is cited alongside you. Every answer saves automatically to your SEO Board, dated, so you can watch it move after you change something. The live answers are real; the visibility shares are estimates from a large index, decision-useful and directionally right. It starts with a 7-day free trial, then $35 a month.

Here is the question to start with, because it is the one this page is about.

Copy this prompt

Ask ChatGPT "what is ai.txt and should I publish one" and tell me
whether example.com is named, cited as a source, or absent from
the answer. List every source it cited. Then ask Gemini the same
question and compare.

Two engines, two credits each, and the answer is either you or somebody else. If it is somebody else, you now know which page to beat, which is a better afternoon than tuning a file the big crawlers skip.

So put the permissions where the crawlers read them, put the map where the agents look, and publish ai.txt if it makes your rights position clearer. Then stop tuning files and measure the thing none of them can report, which is whether the answer engines cite you. That measurement is one question away on the ContextBolt SEO trial.