An AI crawler is a bot that fetches your pages on behalf of an AI company. There are dozens of them now, they arrive under names like GPTBot and ClaudeBot and meta-externalagent, and most site owners meet them for the first time in a server log, months after they started visiting.
The lists you find are mostly tables of names. A name is not useful on its own. What you need to know is which job each bot is doing, because these bots do three completely different things, and the block that protects your work from one of them quietly deletes you from the other two.
This page is that list, with every user-agent string checked against the vendor’s own documentation on July 26, 2026, not copied from another blog post. It covers what each crawler does, which ones ignore robots.txt, what blocking each one actually costs, and how to see which of them are already hitting your site.
- AI crawlers do three different jobs. They collect training data, they build a search index, or they fetch one page because a user just asked about it.
- Blocking the training bot is not the same as going invisible. GPTBot is training. OAI-SearchBot is what puts you in ChatGPT search results.
- User-triggered fetchers largely ignore robots.txt on purpose, including ChatGPT-User and Perplexity-User, because a person requested the page.
- Name-based blocking is a treadmill. The community block list now carries 164 user-agent tokens and grows every month.
- Only about 14% of the top 10,000 domains have any AI directive in robots.txt at all, so most sites have made this decision by accident.
What is an AI crawler?
An AI crawler is an automated program operated by an AI company that requests web pages and identifies itself with a named user-agent string, so site owners can allow or disallow it in robots.txt.
That definition sounds narrow, and the narrowness is the point. A crawler that does not declare a name cannot be managed with robots.txt, which is why the undeclared traffic is the part of this problem nobody can put in a table.
The three jobs below are the entire framework. Once you can place a bot in one of them, every decision about it gets easy.
Training crawlers collect text to train foundation models. They send you nothing back. GPTBot, ClaudeBot, Google-Extended, meta-externalagent, Bytespider, and CCBot live here.
Search and index crawlers build the index that AI answers cite from. Blocking one of these is the same decision as blocking Googlebot in 2010. OAI-SearchBot, PerplexityBot, Claude-SearchBot, MistralAI-Index, and meta-webindexer live here.
User-triggered fetchers grab one page in real time because a person pasted a link or asked a question about it. ChatGPT-User, Perplexity-User, Claude-User, and meta-externalfetcher live here, and this is the group that mostly does not honor robots.txt.
The complete AI crawlers list for 2026
Every token below was verified against the operator’s own documentation. Tokens are case-insensitive in robots.txt but must be spelled exactly.
| User agent | Operator | Job | What blocking it costs you |
|---|---|---|---|
GPTBot | OpenAI | Training | Nothing in ChatGPT search. Your text is not used to train OpenAI models. |
OAI-SearchBot | OpenAI | Search index | You stop appearing in ChatGPT search results. |
ChatGPT-User | OpenAI | User action | A user who pastes your link may get nothing back. |
OAI-AdsBot | OpenAI | Ad landing pages | Only visits pages submitted as ads. Not used for training. |
ClaudeBot | Anthropic | Training | Your content is not collected for Claude model training. |
Claude-SearchBot | Anthropic | Search quality | You are less likely to be surfaced in Claude’s search results. |
Claude-User | Anthropic | User action | Claude cannot open your page when a user asks about it. |
Googlebot | Search, including all Search features | You leave Google Search, including AI Overviews. | |
Google-Extended | Gemini training and grounding | You stay in Search. Your content stops powering Gemini Apps and Vertex AI. | |
GoogleOther | One-off research and development crawls | Little to nothing. | |
Google-CloudVertexBot | Builds Vertex AI agents, at a site owner’s request | Only relevant if you are building on Vertex. | |
PerplexityBot | Perplexity | Search index | You stop appearing in Perplexity results. |
Perplexity-User | Perplexity | User action | Documented as generally ignoring robots.txt. |
Applebot | Apple | Search plus training source | You leave Spotlight, Siri, and Safari suggestions. |
Applebot-Extended | Apple | Training opt-out signal only | Nothing in search. Apple stops training on your content. |
meta-externalagent | Meta | Training and direct indexing | Your content is not used for Meta’s foundation models. |
meta-webindexer | Meta | Meta AI search results | You stop appearing in Meta AI search. |
meta-externalfetcher | Meta | User action and agentic browsing | Meta’s agents cannot open your pages on request. |
Amazonbot | Amazon | Products, and may train Amazon AI models | You leave Alexa answers and Amazon’s AI surfaces. |
MistralAI-Index | Mistral | Search index for Vibe | You stop appearing in Mistral’s search. |
MistralAI-User | Mistral | User action | Mistral cannot fetch your page when asked. |
DuckAssistBot | DuckDuckGo | Real-time retrieval for DuckAssist | You lose DuckAssist citations. Organic rankings are unaffected. |
Bytespider | ByteDance | Undocumented, widely reported as training | Unknown, because ByteDance publishes nothing. |
CCBot | Common Crawl | Open research dataset | You leave the dataset that many other trainers start from. |
Four details in that table are worth pulling out, because they are the ones people get wrong.
Applebot-Extended does not crawl anything. Apple’s Applebot documentation is explicit that it is a signal about how already-crawled data may be used, not a separate bot fetching pages. You will never see it in a log file. Blocking it keeps you in Spotlight, Siri, and Safari while opting out of Apple foundation model training.
Applebot follows Googlebot’s rules if you never mention it. From the same page, if your robots.txt does not name Applebot but does name Googlebot, Apple’s crawler follows the Googlebot instructions. That single line means a lot of sites have an Apple policy they never wrote.
Meta has five bots and only one of them is the training one. The Meta crawler documentation separates facebookexternalhit (link previews), meta-webindexer (Meta AI search), meta-externalads, meta-externalagent (training), and meta-externalfetcher (agents). Blocking all five to stop training also kills your link previews on Facebook and Instagram.
Google-Extended does not touch AI Overviews. Google’s common crawlers documentation scopes it to training Gemini models and powering Gemini Apps and Vertex AI, and puts “all Google Search features” under Googlebot. AI Overviews are a Search feature. There is no directive that keeps you in Google Search and out of AI Overviews.
Which AI crawlers ignore robots.txt?
Two categories, and the difference between them matters.
The first category ignores robots.txt openly and tells you so. OpenAI’s crawler documentation notes that because ChatGPT-User actions are user-initiated, robots.txt may not apply. Perplexity’s bot documentation says Perplexity-User generally ignores robots.txt because a user requested the fetch, and points you at IP allowlisting or a WAF instead. That is a defensible position. Nobody thinks robots.txt should stop a browser, and a fetch triggered by a person clicking a link sits closer to a browser than to a crawler.
The second category ignores robots.txt without saying so. In August 2025 Cloudflare published an investigation finding that Perplexity was using undeclared crawlers to evade no-crawl directives, running 3 to 6 million requests a day under a generic Chrome user agent with rotating IPs across tens of thousands of domains, alongside its declared 20 to 25 million. Cloudflare removed Perplexity from its verified bot list and shipped detection signatures.
Bytespider belongs to a third, quieter category. ByteDance publishes no crawler documentation page at all. There is no vendor URL to cite for its purpose, its robots.txt stance, or its IP ranges. Every fact you will read about Bytespider comes from someone else’s log analysis. That is the most important thing to know about it.
The practical takeaway is that robots.txt is a sign, not a lock. It works on the bots that want to be good citizens, which is genuinely most of the traffic. For anything else you need a firewall rule, and you should stop treating a robots.txt entry as a decision that has been enforced.
How to block AI crawlers in robots.txt
The syntax is unglamorous. One User-agent line per token, then Disallow: / underneath.
# Training crawlers, blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /
# Search crawlers, allowed on purpose
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /That file is a real position, not a default. It says you do not want to be training data and you do want to be cited. Most sites that “block AI” have accidentally taken the opposite position, or a scrambled version of both.
Here is the honest problem with the whole approach. The community-maintained ai.robots.txt block list currently carries 164 user-agent tokens, and it grows every month as new labs, agent frameworks, and coding tools ship crawlers. Blocking by name means auditing that list forever, and any operator who does not want to be on it simply does not declare a name. You are maintaining a list of the bots polite enough to tell you who they are.
If you want AI crawlers actually stopped rather than politely asked, that is a network-level job. Cloudflare, Fastly, and most WAFs now ship managed rules for it, and Cloudflare began blocking AI crawlers by default for new domains in July 2025.
Should you block AI crawlers?
Here is the take that gets me argued with. Most small sites are optimizing the wrong side of this trade.
The case for blocking is real. Cloudflare’s analysis of AI crawler traffic found the crawl-to-refer ratios are brutal, with Anthropic near 50,000 pages crawled for every 1 referral sent back in early August 2025, OpenAI at 887 to 1, and Perplexity at 118 to 1. If you are the New York Times, that is an extraction problem worth a legal department.
If you are a 40-page site nobody has heard of, it is not the same problem. Your risk is not that a model trains on your writing. Your risk is that in three years an assistant answers a question you could have owned, and names a competitor because your pages were never in the index it draws from. Blocking OAI-SearchBot or PerplexityBot to protect content that nobody is currently finding is a trade of a real future upside for a theoretical present harm.
So split the decision. Block training if the extraction bothers you, and it reasonably might. Leave the search and citation crawlers open, because that is the channel every answer engine optimization tactic is trying to get into. Deciding both with one blanket rule is the actual mistake, and it is the most common one.
The Cloudflare robots.txt survey makes the scale of the accident clear. Of 3,816 domains sampled from the top 10,000 in June 2025, only about 14% had any allow or disallow directive aimed at AI bots. GPTBot was both the most-blocked token, at 312 domains, and the most explicitly allowed, at 61. Nearly nine in ten large sites have not written a line about this at all.
How to see which AI crawlers are hitting your site
You cannot make this call sensibly on someone else’s data. Three ways to get your own, cheapest first.
Read your server logs: Grep your access log for the tokens in the table above. On most hosts this is one command and it is free. You will find out in a minute whether these bots are a live issue for you or a hypothetical one.
Check your CDN analytics: Cloudflare, Fastly, and Vercel all break out bot traffic by name now, with a lot less work than parsing raw logs. Cloudflare’s bot analytics will show you AI crawlers as their own category.
Check whether it is turning into citations: Crawl volume is an input, not a result. What you want to know is whether the models actually name you, which is a different measurement entirely. We tested the AEO tools that do this, and ContextBolt SEO runs the same check inside your agent for $35 a month, with a 7-day free trial. Ask it whether ChatGPT and Perplexity mention your brand, and how your share of voice compares to the competitors you actually worry about.
While you are in there, one adjacent question worth settling. A lot of sites publish an llms.txt file expecting it to help with all of this. It does not, and the measured fetch data on those files is worth reading before you spend an afternoon on one.
What this list cannot tell you
Two limits, stated plainly, because reference pages that pretend to be complete are how bad robots.txt files get written.
First, this list will be out of date. Not wrong today, out of date later. New crawlers ship most months, tokens get renamed, and vendors change what a bot is for without changing its name. Applebot-Extended did not exist before June 2024. OAI-AdsBot is newer than that. Check the vendor doc links above before you act on anything here, and treat any AI crawler list without a verification date, including this one in six months, as folklore.
Second, the declared bots are the easy part. Everything in the table above is traffic from an operator who chose to be identifiable. The traffic that worries people most, the scrapers running through residential proxies under a Chrome user agent, is by definition not in any list. If your goal is control rather than a policy statement, the log file and the firewall are where the work is. The robots.txt file is where you write down what you decided.