Should You Block AI Crawlers? An Honest Answer

Somebody read an article about AI companies scraping the web, and now there is a pull request adding six Disallow lines to your robots.txt. It feels responsible. It takes four minutes. And it is probably the only technical SEO decision anyone on your team will make this year without looking at a single number first.

Nobody blocks Googlebot on a hunch. You would want traffic data, a rank check, an argument. But GPTBot gets blocked on a feeling, and the feeling is usually that something is being taken.

Something is being taken. That part is true and I am not going to talk you out of it. The question is narrower than the feeling. Does the thing you are about to paste into robots.txt stop the taking, and what else does it stop on the way out? For most small sites the honest answers are no and quite a lot.

Should you block AI crawlers?

For most sites, no. Not as a blanket rule, and not in robots.txt as the only measure.

Here is the split that actually holds up. Block the crawlers that take your content to train a model, if that bothers you, because you lose nothing measurable by doing it. Leave open the crawlers that fetch your content to answer a question and cite you, because that is the only route into the surface where people are increasingly searching.

Those two groups have different names. They are documented separately by the companies that run them. Treating them as one category is the mistake, and a blanket Disallow is that mistake in one line.

The counter-argument is real and I want to state it fairly. If you are a publisher with an ad-funded business, a model that answers in place instead of sending a click is dismantling your revenue, and “at least you got cited” does not pay a newsroom. That is a licensing fight, and blocking is leverage in it. But that argument belongs to sites with something to license. A 40-page SaaS site does not have a licensing position. It has an obscurity problem.

What are you actually blocking?

“AI crawler” bundles four jobs that have almost nothing to do with each other.

OpenAI states the separation in its own documentation. Each setting is independent of the others, and OpenAI gives the exact use case as an example. A site can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep its content out of training. Anthropic and Perplexity run the same split.

So the first question is not whether to block. It is which of those four columns you were actually angry about. Most people are angry about column one and then paste a rule that hits all four.

The full user-agent table, with every string checked against vendor docs, is in our AI crawlers list. This post is the decision. That one is the reference.

Does blocking Google-Extended keep you out of AI Overviews?

No, and this is the single most expensive misunderstanding in the whole topic.

Google-Extended is a control for training and grounding in Gemini Apps and the Vertex AI API. AI Overviews and AI Mode are not separate products. They are surfaces inside Google Search, and they are fed by Googlebot. There is no AI Overviews crawler to block.

Google’s own guidance is blunt about the consequence. robots.txt directives for Googlebot are the control for how your site gets crawled for Search, and the levers for limiting what appears are nosnippet, data-nosnippet, max-snippet and noindex. Every one of those also reduces or removes your normal search listing.

Read that again, because it is the trade nobody spells out. There is no setting that keeps your blue link in Google and takes your content out of the AI Overview above it. You can have both or neither.

Which means the person who added User-agent: Google-Extended / Disallow: / to feel protected from AI answers protected themselves from nothing they were worried about. The AI answer that quotes them is still there. They opted out of Gemini training instead, which is a fine thing to want and a completely different thing.

What do the crawl numbers actually say?

The number everyone quotes is the crawl-to-refer ratio. Crawl-to-refer ratio is how many pages a company’s crawlers fetch from you for every one visitor it sends back.

Cloudflare put it on the map. In the week of June 19 to 26, 2025, its Radar data had Anthropic at roughly 70,900 crawls per referral, against a Google search ratio in the single digits. That number went around the industry and a lot of blocking decisions were made on it.

Here is the part that got left behind. Those ratios moved fast. Through 2026 the same Cloudflare Radar dashboard shows Anthropic in the low thousands and OpenAI in the low hundreds, an improvement of roughly an order of magnitude in a year, because the AI companies started shipping products that link out. If your blocking policy was written on the 2025 figures and has not been looked at since, it is running on numbers that are wrong by a factor of thirty.

There is a second reason to be careful with this metric. It is a publisher metric. It assumes a crawl costs you something and a referral is worth something, which is exactly true if you sell ads by the impression. If you sell a $35 product to people who find you through a recommendation, a crawl costs you a few kilobytes of bandwidth and a citation is worth more than a hundred anonymous pageviews. Same ratio, opposite conclusion. Run the number for your own business model rather than borrowing a newspaper’s.

The one crawl figure worth being annoyed about is efficiency. When Cloudflare announced its 2026 changes it noted that more than half of AI crawl traffic is spent re-fetching pages that have not changed. That is a genuine cost imposed on you for nothing, and it is a much better argument for rate limiting than for blocking.

Does robots.txt actually stop anything?

Only for the crawlers that decide to obey it. robots.txt has never been an access control. It is a note on the door.

In August 2025 Cloudflare published the clearest evidence of what that means in practice. Perplexity was using undeclared crawlers that presented as an ordinary browser, rotating IP addresses and networks, to reach content on sites that had blocked its named bot. Cloudflare tested it with brand-new domains that had never been indexed and blocked all automated access, then asked Perplexity about them and got the content back. Cloudflare de-listed Perplexity as a verified bot over it.

So there are two different actions hiding under the word “block”. Writing a Disallow line is a preference. Adding a firewall rule or a WAF policy is enforcement. If you genuinely need the content not fetched, only the second one does that, and it is the one almost nobody does.

This is the same gap that makes the newer files disappointing. Cloudflare’s Content Signals Policy, launched September 24, 2025, added three plain-language signals to robots.txt for search, ai-input and ai-train. It is a good idea and it is well written. Nearly a year on, no AI company has publicly committed to honoring it. Publishing signals nobody has agreed to read is not the same as having a policy, which is the same trap people fall into with llms.txt.

What changes on September 15, 2026?

This one may not be your decision at all, which is why it belongs in a post about whether to make it.

On July 1, 2026 Cloudflare announced that it is splitting AI traffic into three categories, Search, Agent and Training, and setting new defaults on September 15, 2026. For all new domains onboarding to Cloudflare, Training and Agent crawlers get blocked by default on pages that display ads. Search stays allowed. The free tier is included.

The detail that will catch people is how multi-purpose crawlers are handled. A crawler that does both Search and Training gets judged on all of its behaviors, so Googlebot, Applebot and Bingbot are blocked for any customer who chose to block Training. Somebody flipping one switch labeled “block AI training” can take out Googlebot. That is a setting worth checking rather than assuming.

If you run through Cloudflare, go and look at what is set today. There is a real chance the AI crawler policy you think you have is a default somebody at a CDN chose for you, and it is about to change again.

Who should block, and who should not

The pattern in that table is worth saying out loud. The Block column tracks how much revenue you make from your content directly. The Allow column is almost never the right thing to close.

And one uncomfortable point for anyone blocking training crawlers to protect what they have already published. If your content has been online for a few years, it is in the training corpora. Common Crawl has been archiving the web since 2011 and its archives feed most open datasets. Blocking today governs future crawls of future writing. It is not a recall.

What to do this week

Four steps, in order, and the first one is not editing a file.

Look at your logs first: Grep your access log or your CDN bot analytics for the user agents in the crawlers list. You may find these bots are not actually a live issue on your site, which ends the discussion cheaply.

Check what your CDN already decided: If you are behind Cloudflare, open the AI crawler settings and read them rather than trusting your memory of them. The September 15 defaults change things again.

Split the rule if you block: Name the training bots. Leave OAI-SearchBot, Claude-SearchBot and PerplexityBot alone unless you have a licensing position you are defending.

Measure the thing you actually care about: Crawl volume is an input, not an outcome. The outcome is whether assistants name you when someone asks about your category. That is a different measurement, and most sites have never run it once. ContextBolt SEO checks it inside your agent for $35 a month, so you can ask whether ChatGPT and Perplexity mention your brand and how your share of voice compares to the competitors you worry about. Deciding to block without that baseline is deciding blind.

The honest answer

Blocking AI crawlers is emotionally satisfying and mostly beside the point for small sites. The extraction you are angry about mostly already happened, the file you are editing is a request rather than a fence, and the rule you are pasting probably closes the one door that was worth leaving open.

If you are big enough that AI companies should be paying you, block, and use it as leverage. If you are not, the question you should be asking is not how to keep the models out. It is how they choose what to cite, and whether anything on your site is written well enough to be picked.