For most sites, the answer is no. Block the crawlers that train models if the extraction bothers you, and leave the ones that fetch pages to answer a question and cite you.
Something is being taken, and a blanket robots.txt rule is not the fix. The question is narrower than the feeling: does the line you were about to paste stop the taking, and what else does it stop on the way out? For most small sites the answers are no, and quite a lot.
- For most sites the answer is no. Block training crawlers if extraction bothers you. Leave search and citation crawlers open. Worth knowing what you would be giving up first, and ContextBolt SEO shows you whether AI answers name you today.
- ”AI crawlers” is four different jobs, training, search, agent fetches and archiving, each with its own user agent.
- Blocking Google-Extended does not remove you from AI Overviews. Those run on Googlebot and have no separate token.
- robots.txt is a request, not a fence. Cloudflare caught Perplexity routing around blocks with undeclared crawlers in August 2025.
- From September 15, 2026 Cloudflare decides this for new domains by default, free tier included.
Should you block AI crawlers?
Not as a blanket rule, and not with robots.txt as your only measure. The split that holds up is this. Block the crawlers that take your content to train a model, if that bothers you, because you lose nothing measurable by doing it. The exception is Google-Extended, which also feeds Gemini’s answers (covered below). Leave open the crawlers that fetch your content to answer a question and cite you, because that is the route into the place people increasingly search.
The two groups have different names, and the companies that run them document them separately. A blanket Disallow treats them as one category, which is the mistake.
The counter-argument is real. If you are an ad-funded publisher, a model that answers in place of sending a click is dismantling your revenue, and “at least you got cited” does not pay a newsroom. That is a licensing fight, and blocking is leverage in it. But a 40-page SaaS site has no licensing position. It has an obscurity problem.
What are you blocking when you block an AI crawler?
“AI crawler” bundles four jobs that have little to do with each other.
| Job | What it does | Example bots | Cost of blocking |
|---|---|---|---|
| Training | Fetches pages to train a foundation model | GPTBot, ClaudeBot, CCBot | Nothing measurable today |
| Search | Builds the index an assistant searches at answer time | OAI-SearchBot, Claude-SearchBot, PerplexityBot | You leave the citation pool |
| Agent | Fetches one page because a user asked for it | ChatGPT-User, Claude-User, Perplexity-User | A user who named you gets nothing |
| Archive | Preserves pages for a public corpus | CCBot, ia_archiver | Nothing today, something later |
OpenAI states the separation in its documentation: each setting is independent of the others. A site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training. Anthropic and Perplexity run the same split.
Most people are angry about the Training row, then paste a rule that hits all four. The full user-agent table, checked against vendor docs, is in our AI crawlers list.
Does blocking Google-Extended keep you out of AI Overviews?
No. Google-Extended is a control for training and grounding in Gemini Apps and the Vertex AI API. AI Overviews and AI Mode are surfaces inside Google Search, and they are fed by Googlebot. There is no AI Overviews crawler to block.
Google’s guidance says robots.txt directives for Googlebot are the control for how your site gets crawled for Search. The levers for limiting what appears are nosnippet, data-nosnippet, max-snippet and noindex, and every one also reduces or removes your normal search listing. No setting keeps your blue link in Google and takes your content out of the AI Overview above it.
So adding User-agent: Google-Extended with Disallow: / does not take you out of the AI Overview that quotes you. It opts you out of Gemini training and out of grounding answers in Gemini Apps, and the second half can cost you citations in Gemini.
Does robots.txt stop AI crawlers?
Only the ones that decide to obey it. robots.txt has never been an access control. It is a note on the door.
In August 2025 Cloudflare published the clearest evidence. Perplexity was using undeclared crawlers that presented as an ordinary browser and rotated IP addresses and networks to reach sites that had blocked its named bot. Cloudflare tested it with brand-new domains that blocked all automated access, asked Perplexity about them, and got the content back. It de-listed Perplexity as a verified bot.
So “block” covers two actions. A Disallow line is a preference. A firewall rule or WAF policy is enforcement, and it is the one almost nobody does. Enforcement also needs to know who it is talking to, which is what Web Bot Auth is being built to provide.
The newer files have the same gap. Cloudflare’s Content Signals Policy, launched September 24, 2025, added search, ai-input and ai-train signals to robots.txt, and no AI company has publicly committed to honoring it. Content Signals in robots.txt covers what each signal means and llms.txt vs robots.txt covers the other file people reach for.
What changes on September 15, 2026?
This may not be your decision at all. On July 1, 2026 Cloudflare announced it is splitting AI traffic into Search, Agent and Training, and setting new defaults on September 15, 2026. For all new domains onboarding to Cloudflare, Training and Agent crawlers are blocked by default on pages that display ads. Search stays allowed. The free tier is included.
Multi-purpose crawlers get judged on all of their behaviors, so Googlebot, Applebot and Bingbot are blocked for any customer who chose to block Training. Someone flipping one switch labeled “block AI training” can take out Googlebot. If you run through Cloudflare, open the AI crawler settings and read what is set today, because it may be a default someone at a CDN chose for you.
Do the crawl numbers say to block?
The number everyone quotes is the crawl-to-refer ratio: how many pages a company’s crawlers fetch from you for every one visitor it sends back. In the week of June 19 to 26, 2025, Cloudflare Radar had Anthropic at roughly 70,900 crawls per referral, against a Google search ratio in the single digits.
Those ratios moved fast. Through 2026 the same dashboard shows Anthropic in the low thousands and OpenAI in the low hundreds, because the AI companies started shipping products that link out. A policy written on the 2025 figures runs on numbers that are wrong by a factor of thirty.
The ratio is also a publisher metric. It assumes a crawl costs you something and a referral is worth something, which holds if you sell ads by the impression. If you sell a $35 product to people who find you through a recommendation, a crawl costs a few kilobytes and a citation is worth more than a hundred anonymous pageviews. The crawl cost worth being annoyed about is wasted fetching: Cloudflare noted that more than half of AI crawl traffic goes on re-fetching pages that have not changed. That is a better argument for rate limiting than for blocking.
Who should block AI crawlers, and who should not?
| Your situation | Training crawlers | Search crawlers | Why |
|---|---|---|---|
| Ad-funded publisher | Block | Negotiate | You have something to license and impressions to lose |
| Paywalled or subscription content | Block | Allow the paywalled snippet only | Citation drives subscriptions, full text does not |
| Small SaaS or product site | Allow or block, little difference | Allow | Your risk is obscurity, not extraction |
| Personal blog or portfolio | Your call, no cost either way | Allow | Being named in an answer is the whole upside |
| Ecommerce | Allow | Allow | Assistants increasingly route product research |
| Original research or proprietary data | Block | Allow | The citation is the point, the corpus is not |
The Block column tracks how much revenue you make from your content directly. The Allow column is almost never the right thing to close. And if your content has been online for a few years, it is already in the training corpora: blocking today governs future crawls of future writing. It is not a recall.
What should you do before editing robots.txt?
- Look at your logs: Search your access log or CDN bot analytics for the user agents in the crawlers list. You may find these bots are not a live issue on your site.
- Check what your CDN already decided: Behind Cloudflare, open the AI crawler settings and read them rather than trusting your memory.
- Split the rule if you block: Name the training bots, and leave
OAI-SearchBot,Claude-SearchBotandPerplexityBotalone unless you have a licensing position to defend. Our free GEO audit reads your robots.txt and prints the verdict for fifteen AI crawlers, split by whether each one trains a model, fetches pages to cite them, or reads a page because a person asked. - Measure the outcome: Crawl volume is an input. The outcome is whether assistants name you when someone asks about your category.
For that last step, ContextBolt SEO checks inside your agent whether ChatGPT, Gemini, Claude and Perplexity mention your brand and how your share of voice compares to your competitors, with every check saved to your SEO Dashboard. It’s $35 a month after a 7-day free trial.