ContextBolt SEO Free for 7 days. Keywords, SERPs, backlinks and AI visibility, inside Claude. SEO data inside Claude. Start free trial
Guide · Should You Block AI Crawlers?

Should You Block AI Crawlers? An Honest Answer

For most sites, the answer is no. Block the crawlers that train models if the extraction bothers you, and leave the ones that fetch pages to answer a question and cite you.

Something is being taken, and a blanket robots.txt rule is not the fix. The question is narrower than the feeling: does the line you were about to paste stop the taking, and what else does it stop on the way out? For most small sites the answers are no, and quite a lot.

Quick answer
  • For most sites the answer is no. Block training crawlers if extraction bothers you. Leave search and citation crawlers open. Worth knowing what you would be giving up first, and ContextBolt SEO shows you whether AI answers name you today.
  • ”AI crawlers” is four different jobs, training, search, agent fetches and archiving, each with its own user agent.
  • Blocking Google-Extended does not remove you from AI Overviews. Those run on Googlebot and have no separate token.
  • robots.txt is a request, not a fence. Cloudflare caught Perplexity routing around blocks with undeclared crawlers in August 2025.
  • From September 15, 2026 Cloudflare decides this for new domains by default, free tier included.

Should you block AI crawlers?

Not as a blanket rule, and not with robots.txt as your only measure. The split that holds up is this. Block the crawlers that take your content to train a model, if that bothers you, because you lose nothing measurable by doing it. The exception is Google-Extended, which also feeds Gemini’s answers (covered below). Leave open the crawlers that fetch your content to answer a question and cite you, because that is the route into the place people increasingly search.

The two groups have different names, and the companies that run them document them separately. A blanket Disallow treats them as one category, which is the mistake.

The counter-argument is real. If you are an ad-funded publisher, a model that answers in place of sending a click is dismantling your revenue, and “at least you got cited” does not pay a newsroom. That is a licensing fight, and blocking is leverage in it. But a 40-page SaaS site has no licensing position. It has an obscurity problem.

What are you blocking when you block an AI crawler?

“AI crawler” bundles four jobs that have little to do with each other.

JobWhat it doesExample botsCost of blocking
TrainingFetches pages to train a foundation modelGPTBot, ClaudeBot, CCBotNothing measurable today
SearchBuilds the index an assistant searches at answer timeOAI-SearchBot, Claude-SearchBot, PerplexityBotYou leave the citation pool
AgentFetches one page because a user asked for itChatGPT-User, Claude-User, Perplexity-UserA user who named you gets nothing
ArchivePreserves pages for a public corpusCCBot, ia_archiverNothing today, something later

OpenAI states the separation in its documentation: each setting is independent of the others. A site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training. Anthropic and Perplexity run the same split.

Most people are angry about the Training row, then paste a rule that hits all four. The full user-agent table, checked against vendor docs, is in our AI crawlers list.

Does blocking Google-Extended keep you out of AI Overviews?

No. Google-Extended is a control for training and grounding in Gemini Apps and the Vertex AI API. AI Overviews and AI Mode are surfaces inside Google Search, and they are fed by Googlebot. There is no AI Overviews crawler to block.

Google’s guidance says robots.txt directives for Googlebot are the control for how your site gets crawled for Search. The levers for limiting what appears are nosnippet, data-nosnippet, max-snippet and noindex, and every one also reduces or removes your normal search listing. No setting keeps your blue link in Google and takes your content out of the AI Overview above it.

So adding User-agent: Google-Extended with Disallow: / does not take you out of the AI Overview that quotes you. It opts you out of Gemini training and out of grounding answers in Gemini Apps, and the second half can cost you citations in Gemini.

Does robots.txt stop AI crawlers?

Only the ones that decide to obey it. robots.txt has never been an access control. It is a note on the door.

In August 2025 Cloudflare published the clearest evidence. Perplexity was using undeclared crawlers that presented as an ordinary browser and rotated IP addresses and networks to reach sites that had blocked its named bot. Cloudflare tested it with brand-new domains that blocked all automated access, asked Perplexity about them, and got the content back. It de-listed Perplexity as a verified bot.

So “block” covers two actions. A Disallow line is a preference. A firewall rule or WAF policy is enforcement, and it is the one almost nobody does. Enforcement also needs to know who it is talking to, which is what Web Bot Auth is being built to provide.

The newer files have the same gap. Cloudflare’s Content Signals Policy, launched September 24, 2025, added search, ai-input and ai-train signals to robots.txt, and no AI company has publicly committed to honoring it. Content Signals in robots.txt covers what each signal means and llms.txt vs robots.txt covers the other file people reach for.

What changes on September 15, 2026?

This may not be your decision at all. On July 1, 2026 Cloudflare announced it is splitting AI traffic into Search, Agent and Training, and setting new defaults on September 15, 2026. For all new domains onboarding to Cloudflare, Training and Agent crawlers are blocked by default on pages that display ads. Search stays allowed. The free tier is included.

Multi-purpose crawlers get judged on all of their behaviors, so Googlebot, Applebot and Bingbot are blocked for any customer who chose to block Training. Someone flipping one switch labeled “block AI training” can take out Googlebot. If you run through Cloudflare, open the AI crawler settings and read what is set today, because it may be a default someone at a CDN chose for you.

Do the crawl numbers say to block?

The number everyone quotes is the crawl-to-refer ratio: how many pages a company’s crawlers fetch from you for every one visitor it sends back. In the week of June 19 to 26, 2025, Cloudflare Radar had Anthropic at roughly 70,900 crawls per referral, against a Google search ratio in the single digits.

Those ratios moved fast. Through 2026 the same dashboard shows Anthropic in the low thousands and OpenAI in the low hundreds, because the AI companies started shipping products that link out. A policy written on the 2025 figures runs on numbers that are wrong by a factor of thirty.

The ratio is also a publisher metric. It assumes a crawl costs you something and a referral is worth something, which holds if you sell ads by the impression. If you sell a $35 product to people who find you through a recommendation, a crawl costs a few kilobytes and a citation is worth more than a hundred anonymous pageviews. The crawl cost worth being annoyed about is wasted fetching: Cloudflare noted that more than half of AI crawl traffic goes on re-fetching pages that have not changed. That is a better argument for rate limiting than for blocking.

Who should block AI crawlers, and who should not?

Your situationTraining crawlersSearch crawlersWhy
Ad-funded publisherBlockNegotiateYou have something to license and impressions to lose
Paywalled or subscription contentBlockAllow the paywalled snippet onlyCitation drives subscriptions, full text does not
Small SaaS or product siteAllow or block, little differenceAllowYour risk is obscurity, not extraction
Personal blog or portfolioYour call, no cost either wayAllowBeing named in an answer is the whole upside
EcommerceAllowAllowAssistants increasingly route product research
Original research or proprietary dataBlockAllowThe citation is the point, the corpus is not

The Block column tracks how much revenue you make from your content directly. The Allow column is almost never the right thing to close. And if your content has been online for a few years, it is already in the training corpora: blocking today governs future crawls of future writing. It is not a recall.

What should you do before editing robots.txt?

  1. Look at your logs: Search your access log or CDN bot analytics for the user agents in the crawlers list. You may find these bots are not a live issue on your site.
  2. Check what your CDN already decided: Behind Cloudflare, open the AI crawler settings and read them rather than trusting your memory.
  3. Split the rule if you block: Name the training bots, and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot alone unless you have a licensing position to defend. Our free GEO audit reads your robots.txt and prints the verdict for fifteen AI crawlers, split by whether each one trains a model, fetches pages to cite them, or reads a page because a person asked.
  4. Measure the outcome: Crawl volume is an input. The outcome is whether assistants name you when someone asks about your category.

For that last step, ContextBolt SEO checks inside your agent whether ChatGPT, Gemini, Claude and Perplexity mention your brand and how your share of voice compares to your competitors, with every check saved to your SEO Dashboard. It’s $35 a month after a 7-day free trial.

Where next

Frequently asked questions

Something else on your mind? Get in touch.

Should I block AI crawlers?

For most small sites, no. Blocking training crawlers is defensible if the extraction bothers you. Blocking search and citation crawlers removes you from the answer engines people now use to find things. Those are separate decisions with separate user agents, and one blanket rule answers both of them badly.

Does blocking Google-Extended stop AI Overviews?

No. AI Overviews and AI Mode are surfaces inside Google Search, fed by Googlebot, with no separate crawler token. Google's documentation says robots.txt directives for Googlebot are the control. Google-Extended governs training and grounding in Gemini Apps and the Vertex AI API, which is a different thing entirely.

What changes for AI crawlers on September 15, 2026?

Cloudflare sets new defaults. Training and Agent crawlers get blocked on pages that display ads for all new domains onboarding to Cloudflare, including the free tier, while Search crawlers stay allowed. Multi-purpose crawlers like Googlebot and Bingbot are blocked for anyone who chose to block Training.

Does robots.txt actually stop AI crawlers?

Only for crawlers that choose to obey it. In August 2025 Cloudflare found Perplexity using undeclared crawlers that rotated IP addresses and networks to reach sites that had blocked its named bot, then de-listed it as a verified bot. Real enforcement needs a firewall rule, not a text file.

Is blocking AI crawlers bad for SEO?

Blocking training crawlers has no effect on Google rankings. Blocking a search crawler like OAI-SearchBot removes you from ChatGPT's search results without moving a single Google position, so nothing in Search Console will tell you it happened. That is what makes it an easy mistake to miss.