If you opened the robots.txt of a site that runs through Cloudflare and found a line like Content-signal: search=yes, ai-train=no, use=reference, that’s a Content Signal. It’s Cloudflare’s vocabulary for telling a crawler what it may do with your pages after it fetches them: appear in search, feed AI answers, or train models.
Cloudflare launched it on September 24, 2025, with its managed robots.txt already switched on for more than 3.8 million domains. The question people ask is whether it does anything. Mostly, no. But the managed file doesn’t ship the line alone, and one of the lines under it does change where your site can appear in AI answers.
- It is a preference, not a block.
Content-Signalsits inside a robots.txt group and says yes or no tosearch,ai-input(feeding AI answers) andai-train(training models). Leave a value out and you neither grant nor refuse it. - Google does not read it. Google’s robots.txt spec supports four fields and ignores the rest. OpenAI, Anthropic and Perplexity do not mention it in their crawler documentation.
- Cloudflare’s managed file does more than one line. Under the signal it disallows eight named crawlers, including GPTBot, ClaudeBot and Google-Extended.
- One of those lines costs you AI visibility. Disallowing Google-Extended takes your pages out of grounding in Gemini Apps. It does not touch Google Search or AI Overviews.
What do the three signals mean?
The definitions come from the policy text Cloudflare publishes at contentsignals.org, shortened.
search: building a search index and returning links and short excerpts. Cloudflare’s text explicitly excludes AI-generated search summaries. This is the classic blue link.ai-input: feeding content into AI models at answer time, such as retrieval-augmented generation or grounding. This is being read to answer somebody’s question right now.ai-train: training or fine-tuning AI models. This is becoming part of a model.
Each value is yes or no. A yes means a crawler may collect your content for that use, and a no means it may not. A signal you leave out means you “neither grant nor restrict” that use, which is not the same as saying yes.
In Cloudflare’s vocabulary an AI summary on a results page is ai-input, not search. Google doesn’t offer that split. AI Overviews are fed by Googlebot, the same crawler as your blue link, and Google’s controls for AI features are nosnippet, data-nosnippet, max-snippet and noindex, every one of which also shrinks or removes your normal listing.
How is Content-Signal written?
The signal lives inside a User-Agent group, next to Allow and Disallow. Values are comma-separated pairs. Give one crawler its own group to set a different preference for it, and put a path first to scope the signal to part of your site, as in Cloudflare’s own example Content-Signal: /blog/ ai-train=no, search=yes, ai-input=no.
You’ll see both Content-Signal and Content-signal. Google’s spec says robots.txt field names are case-insensitive, so they’re the same line.
One trap before you write your own. Google’s robots.txt spec says only one group is valid for a given crawler, the most specific one that matches it. A crawler with its own group never reads the signal in your User-Agent: * group. If you write a signal for everyone and then give one bot its own rules, repeat the signal inside that bot’s group.
What is use=?
On July 1, 2026 Cloudflare started testing a fourth field, use, which describes what a crawler may keep after it visits. From least to most permissive:
use=immediate: interact, but store and reuse nothing.use=reference: index, excerpt and link back.use=full: summarize and reproduce.
Every site that already had the managed robots.txt switched on got use=reference added to its line. No AI company documents reading it.
Who honors Content Signals?
Google: its robots.txt specification supports user-agent, allow, disallow and sitemap and says other fields are not supported. In July 2026 Google’s John Mueller answered a thread on r/TechSEO about the directive, and Search Engine Roundtable reported his reply. As far as he knew, he wrote, it has “no effects whatsoever for any crawler or LLM”. He added that it was made up by a CDN and adds maintenance to your file.
OpenAI, Anthropic and Perplexity: none of their crawler pages mention Content Signals. OpenAI’s bots page, Anthropic’s crawler page and Perplexity’s crawler page all describe control through named user-agent tokens in robots.txt.
Cloudflare: the signal has teeth here, and only on Cloudflare’s network. In the same July 2026 announcement Cloudflare said it is starting to track content use for every bot in its BotBase directory, and that a bot found abusing these signals loses its Verified status and is no longer allowed.
The IETF: the AI Preferences working group is drafting a standards-track version with different words. Its vocabulary draft, last revised September 13, 2026, names three categories, train-ai, ai-use and search, set to y or n. Its companion draft attaches them to robots.txt as a Content-Usage rule, such as Content-Usage: train-ai=n. Both are still Internet-Drafts.
So the likely future is a renamed version of the same line, and what enforces your preference today is the user-agent rule beside it.
What does Cloudflare write into your robots.txt?
Turn on the Cloudflare setting called Set your preference to block training in robots.txt, under Security Settings, and Cloudflare prepends its block to your existing file, or creates one if you had none. On a Free plan domain with no robots.txt of its own, Cloudflare serves the policy text instead, a block of # comments defining the three signals with no yes or no values.
The managed block, per Cloudflare’s robots.txt documentation (last updated August 3, 2026), is the signal line above in a User-Agent: * group with Allow: /, followed by a plain Disallow: / group for each of eight crawlers: Amazonbot, Applebot-Extended, Bytespider, CCBot, ClaudeBot, Google-Extended, GPTBot and meta-externalagent.
The signal is the headline, and the eight groups are the working part. They use the syntax OpenAI, Google and Anthropic all document honoring. Three details:
ai-inputis not set. Cloudflare’s default says yes to search and no to training, and says nothing about AI answers, which neither grants nor refuses them.- The comment block claims legal weight. The
#lines state that any restriction you express is an express reservation of rights under Article 4 of the EU’s 2019 copyright directive, the provision that lets rights holders opt out of commercial text and data mining. The other machine-readable reservations are covered in ai.txt. - The spec site hedges that claim itself. contentsignals.org says courts and regulators may conclude robots.txt files do not impose enforceable legal obligations. The policy is released under a CC0 license, so anyone can copy it.
Search Console’s robots.txt report may also flag the line as Syntax not understood. Cloudflare’s documentation says it has seen no effect on crawl rates or SEO from those reports. The generator at contentsignals.org writes the block for you from four preset policies, from Disallow All to allowing everything.
Does Content-Signal affect SEO?
The signal line doesn’t. Google ignores fields it doesn’t support, so it changes nothing about how Googlebot crawls or ranks you.
The Disallow lines don’t touch Google Search either. Google’s crawler documentation says Google-Extended does not affect a site’s inclusion in Google Search and is not a ranking signal. But several of them change what happens in AI answers outside Google Search.
| Line in the managed file | Google Search and AI Overviews | AI answers everywhere else |
|---|---|---|
| The signal line | Nothing. Google does not read the field | Nothing any vendor documents |
Google-Extended disallowed | Nothing, per Google | Your pages stop being used for Gemini training and for grounding answers in Gemini Apps and Vertex AI |
GPTBot disallowed | Nothing | Out of OpenAI training. ChatGPT search runs on OAI-SearchBot, which the file does not name |
ClaudeBot disallowed | Nothing | Out of Anthropic training. Claude-SearchBot and Claude-User are not in the file |
The other five groups are covered one by one in our AI crawlers list.
The line the internet argues about has no documented effect. The line that can cost you an AI citation is User-agent: Google-Extended, and it arrives in the same click as the signal. If Gemini naming your product is important to your business, that group deserves more thought than the signal above it.
What is the difference between ai-input and ai-train?
ai-train=no is the easy call for most sites. Training takes your pages and gives nothing back in the moment, and there’s no link in a model’s weights.
ai-input covers the use that produces a citation. An AI answer that grounds itself on your page is the answer that can name you and link to you. Write ai-input=no and you’re asking to be left out of exactly the answers that send people your way.
No vendor reads the value today, so writing it does nothing beyond the rights notice. The rules a crawler does obey are per vendor:
- Stay in classic search,
search=yes: allow Googlebot and Bingbot, which is the default. - Stay out of model training,
ai-train=no: disallow GPTBot, ClaudeBot and CCBot. Google-Extended too, but see below. - Stay out of AI answers,
ai-input=no: disallow OAI-SearchBot, Claude-SearchBot, PerplexityBot and Google-Extended. For AI Overviews there’s onlynosnippetornoindex, and both also cut your normal listing. - Stay in AI answers,
ai-input=yes: allow those search crawlers and leave Google-Extended out of your file.
Google has no token for training alone. Google-Extended covers Gemini training and Gemini grounding together, so ai-train=no with ai-input=yes can’t be enforced through robots.txt on Google’s side. OpenAI, Anthropic and Perplexity do separate the jobs. OpenAI’s page says a site can disallow GPTBot and still appear in ChatGPT search through OAI-SearchBot, and Perplexity says PerplexityBot is not used to crawl content for AI foundation models.
The fetchers that act when a user asks about a specific page, ChatGPT-User and Perplexity-User, sit partly outside robots.txt. OpenAI says its rules might not cover visits a user starts, and Perplexity says its user fetcher generally ignores them. Whether any of this is worth blocking is a separate decision, covered in should you block AI crawlers.
What should you put in your robots.txt?
The position most publishers hold is yes to search, yes to AI answers, no to training. With the signal stating it and the user-agent rules enforcing it:
User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
- The signal sits in the
*group, and that’s enough here. GPTBot, ClaudeBot and CCBot skip that group, but they’re told to stay out entirely, so a signal adds nothing. Give any crawler a group with partial rules and repeat the signal there. - Stacking user-agent lines is standard. Google’s spec shows several
user-agentlines sharing one set of rules. - Google-Extended is left out deliberately. Add a
User-agent: Google-Extendedgroup withDisallow: /only if staying out of Gemini training outranks being cited by Gemini, because the token refuses both. - If Cloudflare manages your robots.txt, you already have most of this. Open
https://example.com/robots.txton your own domain and look forUser-agent: Google-Extended. If it’s there and you care about Gemini citations, make that a decision rather than a default.
The difference between this file and llms.txt is covered in llms.txt vs robots.txt.
How do you check who cites you before saying no?
Every rule that keeps a crawler out of AI answers is paid for in citations, and a robots.txt file can’t tell you how many you hold today. You have to ask the engines: ask ChatGPT, Gemini, Claude and Perplexity the questions a buyer would ask, and note whether your site is named, cited as a source or absent. A tracker will often fold the answers into an AI visibility score, but for this decision you want each engine’s answer on its own.
Read the result by engine. If Gemini cites you, think hard before the Google-Extended line, because that’s the citation it removes. If ChatGPT or Claude cite you and you want out of training, you can have both: disallow GPTBot and ClaudeBot, and allow OAI-SearchBot and Claude-SearchBot. If nobody cites you, the signal choice is academic for now, and how AI assistants choose citations is where to start.
ContextBolt SEO is ours, so weigh the bias, but this is the job it was built for. Ask in plain English and it asks each engine live, then reports whether you were named, cited or absent, with every source the engine used. Three buyer questions across the four engines is twelve live checks, at two credits each 24 credits, out of the 1,000 a month the plan includes. AI engines don’t answer the same way every time, so a live check tells you whether you showed up that time. Every result saves to your SEO Dashboard, dated, so you can check again a month after you change your robots.txt. It’s a 7-day free trial, then $35 a month.