Complete List of AI Crawlers: GPTBot, ClaudeBot, PerplexityBot and Others
Every AI answer engine has to see your content before it can cite it. If you’ve never checked which bots hit your server logs, you’re deciding blind, and that decision shapes whether ChatGPT, Claude, Perplexity, and Gemini even know your brand exists. AI crawlers aren’t one thing. Some scrape your site to train a future model. Others fetch a single page in real time when a user asks a question. A few, Bytespider chief among them, just ignore your rules. Get the distinctions wrong and you either leak content into training sets you never agreed to, or block the one bot that would have gotten you cited in an AI Overview. Bookmark this one instead of reading it once: every major AI crawler, what it does, and the exact robots.txt line that controls it.
The Master AI Crawler Table
Here’s the full lineup: what each bot does with your content, and how to control it.
| Bot Name | Company | User-Agent Token | Purpose | First Seen | Typical robots.txt Rule |
|---|---|---|---|---|---|
| GPTBot | OpenAI | GPTBot | Collects content for training GPT models | Aug 2023 | Disallow: / to opt out of training |
| OAI-SearchBot | OpenAI | OAI-SearchBot | Builds the index behind ChatGPT Search citations; now shares crawl data with GPTBot | 2024 | Allow: / if you want ChatGPT citations |
| ChatGPT-User | OpenAI | ChatGPT-User | Live fetch for user clicks, Custom GPTs, and GPT Actions | 2023 | No longer reliably governed by robots.txt |
| ClaudeBot | Anthropic | ClaudeBot | Crawls public content for Claude model training | 2024 | Disallow: / to opt out of training |
| Claude-User | Anthropic | Claude-User | Fetches a page when a Claude user asks a question | Formalized as separate agent, Feb 2026 | Allow: / for retrieval visibility |
| Claude-SearchBot | Anthropic | Claude-SearchBot | Indexes content to improve Claude’s search results | Formalized as separate agent, Feb 2026 | Allow: / for search visibility |
| PerplexityBot | Perplexity | PerplexityBot | Crawls and indexes pages for Perplexity’s answer engine | 2024 | Allow: / for AI search visibility |
| Perplexity-User | Perplexity | Perplexity-User | User-triggered fetch; Perplexity calls it “an agent, not a bot” | 2024 | Not considered bound by robots.txt |
| Google-Extended | Google-Extended | Opt-out token for Gemini and AI Overviews training | 2023 | Token only, no HTTP requests | |
| CCBot | Common Crawl | CCBot | Bulk web crawl feeding many outside training sets | 2008 (AI use since ~2019) | Disallow: / to exclude from datasets |
| Bytespider | ByteDance | Bytespider | Trains ByteDance’s Doubao and related models | 2021 | Disallow: /, enforcement needs a WAF |
| cohere-ai | Cohere | cohere-ai | Purpose undisclosed; unconfirmed by tracking services | Unknown | Disallow: /, block alongside cohere-training-data-crawler |
| Applebot-Extended | Apple | Applebot-Extended | Flags already-crawled content as eligible for Apple Intelligence training | 2024 | Disallow: / to opt out of training |
| Meta-ExternalAgent | Meta | Meta-ExternalAgent | Crawls public pages for Llama training and indexing | 2024 | Disallow: / to opt out of training |
| Meta-ExternalFetcher | Meta | Meta-ExternalFetcher | Live fetch when a Meta AI user references a URL | 2024 | Usually left open |
Detailed Profiles: The Nine Bots You Need to Know
GPTBot
GPTBot is OpenAI’s training crawler, and the one most site owners recognize first. OpenAI launched it in August 2023 to collect public content for future GPT training runs. The user-agent string and official IP ranges are documented on OpenAI’s developer platform. GPTBot handles training only. Two other agents, OAI-SearchBot and ChatGPT-User, cover search indexing and live fetches, and each needs its own robots.txt line.
Two changes from OpenAI’s December 2025 documentation update matter here. OAI-SearchBot and GPTBot now share crawl results, so a site that allows both won’t get double-crawled. And ChatGPT-User no longer reliably follows robots.txt: OpenAI’s own wording now separates “OAI-SearchBot and GPTBot robots.txt tags” from ChatGPT-User’s behavior, so disallowing it doesn’t guarantee it stays out.
There’s an older problem too, and it hasn’t gone away. Sites that explicitly allow GPTBot still get it blocked at the firewall, because default WAF rules on Cloudflare, Akamai, and AWS often rate-limit unfamiliar bot traffic before it ever reaches robots.txt logic.
ClaudeBot
ClaudeBot is Anthropic’s training crawler, and it now sits alongside two purpose-built siblings rather than standing alone. Anthropic formalized this three-agent structure in a documentation update around February 20, 2026. ClaudeBot collects content for training. Claude-User fetches pages when a Claude user asks a question. Claude-SearchBot crawls to improve Claude’s search results. Each is controlled independently, so blocking one doesn’t block the others.
There’s also a fourth, narrower agent: claude-code, which fires when a developer’s Claude Code CLI fetches a URL on their behalf. That one’s separate from the three consumer-facing bots.
One quirk worth knowing before you try to hard-block Claude: IP-based blocking won’t help. Anthropic says blocking its bots by IP may also stop them from reading your robots.txt file in the first place, and the company doesn’t publish fixed IP ranges. Allow Claude-User and Claude-SearchBot if you want citations. Disallow ClaudeBot if you want to keep your content out of training.
PerplexityBot
PerplexityBot crawls and indexes public pages to power Perplexity’s real time answer engine, which makes it one of the more consequential bots for AI search visibility. But two different bots share the name space here, and mixing them up matters. PerplexityBot is the indexing crawler, and it does respect robots.txt. Perplexity-User is the separate, user-triggered fetcher. Perplexity has called it “an agent, not a bot,” and says it isn’t bound by robots.txt at all, a position that’s put the company in real disputes with publishers.
On top of that, Cloudflare reported on August 4, 2025 that Perplexity was also running undeclared crawlers that rotated user-agents, IPs, and ASNs specifically to reach disallowed content. So: PerplexityBot itself behaves the way it’s documented to. Perplexity’s broader access model doesn’t. If you need a hard guarantee against Perplexity-linked traffic, back your Disallow rule with a server-level block.
Google-Extended
Google-Extended isn’t a crawler in the usual sense. It’s a robots.txt token that never appears in your access logs, because Google routes AI-training access through its existing Googlebot infrastructure and just checks the Google-Extended directive as an opt-out flag. It makes no separate HTTP requests of its own. Disallowing it opts your content out of Gemini and AI Overviews training without touching your Google Search rankings, since the two systems are decoupled by design.
CCBot
CCBot is the crawler behind Common Crawl, the nonprofit open web archive that has become one of the most important upstream data sources in AI training. It doesn’t serve one company. Common Crawl’s bulk crawl of the public web feeds downstream training pipelines at OpenAI, Anthropic, and various academic research groups, among others. Because so many models train on Common Crawl derivatives, blocking CCBot is one of the highest-leverage single moves a publisher can make for a broad AI-training opt-out. It does nothing for the direct crawlers, though. You still need to block those separately.
Bytespider
Bytespider is ByteDance’s crawler, feeding training data into Doubao and the company’s other foundation models. It’s also the bot most frequently flagged as unreliable. Bytespider publishes no official documentation and has repeatedly been observed fetching paths marked Disallow, so a robots.txt rule alone won’t stop it. If it’s hammering your logs after you’ve already disallowed it, add an edge or WAF-level block instead of going back to check the robots.txt file again.
cohere-ai
cohere-ai has a real documentation gap. Crawler-tracking services classify it as unconfirmed, since Cohere hasn’t disclosed what triggers it or what it does with the content it fetches. The bot Cohere has actually documented for training-data collection uses a different token: cohere-training-data-crawler. Most site owners just block both. Treating the ambiguous one as harmless by default is the riskier bet.
Applebot-Extended
Applebot-Extended is Apple’s opt-out signal for Apple Intelligence training, introduced at WWDC in June 2024 alongside Apple Intelligence itself. Unlike Google-Extended, it isn’t purely silent. Rather than crawling the web on its own, it reviews content the standard Applebot has already indexed and flags what’s eligible for AI training. Some site owners do see the distinct Applebot-Extended string in their logs, even though the underlying fetch piggybacks on Applebot’s crawl. The control works the same as Google-Extended: disallow it to opt content out of Apple Intelligence training while keeping standard Applebot access for Siri and Spotlight search.
Meta-ExternalAgent
Meta-ExternalAgent is Meta’s training crawler for the Llama model family, paired with a separate live-fetch agent, Meta-ExternalFetcher, for user-triggered requests inside Meta AI. Meta documents both through its developer platform. Meta-ExternalAgent crawls public pages for Llama training and indexing. Meta-ExternalFetcher fetches a page on the spot when a Meta AI user asks about a specific URL. As with OpenAI’s and Anthropic’s split agents, blocking the training crawler doesn’t touch the retrieval one. Control each independently.
Other AI Bots Worth Watching
The nine above cover the crawlers with the most volume and the clearest documentation, but the list keeps growing. A few more worth adding to your log-monitoring regex:
- Amazonbot. Amazon’s crawler, feeding Alexa answers and Amazon’s broader AI initiatives.
- xAI-Bot (Grok). xAI’s crawler for Grok. Unlike the bots above, it publishes no official documentation page or IP range file, so its robots.txt compliance is unverified rather than confirmed.
- DuckAssistBot. DuckDuckGo’s retrieval crawler for its AI-assisted answers.
- MistralAI-User. Mistral’s user-triggered fetcher, parallel to ChatGPT-User and Claude-User.
These four move fast and lack the documentation depth of the core nine. Treat any claim about their behavior as provisional until you check the vendor’s current docs.
The robots.txt Mega-Snippet
Three ready-to-use configurations, depending on your AI visibility strategy. Copy the block that matches your policy into /robots.txt.
Option 1 (Block all AI crawlers, maximum content protection):
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
Disallow: /
Option 2 (Allow all AI crawlers, maximum AI search visibility):
User-agent: GPTBot
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
Allow: /
Option 3 (Selective access: allow search and retrieval, block training-only bots):
# Allow the bots that power AI search citations
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-ExternalFetcher
Allow: /
# Block the bots that only feed model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Meta-ExternalAgent
Disallow: /
Most brands pursuing AI Overview visibility land on Option 3. It keeps your content eligible for citation in live AI answers while opting out of having it baked permanently into a model’s training weights.
How to Detect AI Crawlers in Your Logs
You don’t need special software to see which AI bots are visiting your site. Your server or CDN logs already have the answer.
- Grep your access logs for known tokens. A single regex like
GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Bytespider|cohere-ai|Applebot-Extended|Meta-ExternalAgent|Amazonbot|xAI-Bot|DuckAssistBot|MistralAI-Useragainst your raw logs will surface most declared AI traffic in seconds. - Check status codes, not just hits. A bot showing up with repeated 403s or 429s means your WAF is silently blocking a crawler your robots.txt explicitly allows. That’s a common, easy-to-miss misconfiguration.
- Validate by IP when it matters. User-agent strings can be spoofed. OpenAI, Google, and Common Crawl publish machine-readable IP range files, so cross-check suspicious high-volume visitors against those before assuming they’re the real bot.
- Watch for undeclared traffic. Some AI-adjacent scraping doesn’t announce itself at all. If you see unusual traffic patterns with no recognizable AI user-agent, look closer at your CDN’s bot-management analytics instead of relying on robots.txt logs alone.
- Re-check quarterly, not annually. New AI crawlers show up several times a year, and existing ones periodically split into multiple specialized agents, the way Anthropic and OpenAI have both done. A user-agent list from six months ago can already be out of date.
Turn This List Into an Action Plan
Knowing which bots exist is only half the job. What matters is whether the ones that count for AI visibility can actually see and cite your content today. Cross-referencing your robots.txt, your WAF rules, and your live server logs against a fifteen-bot list by hand is easy to get wrong, and the mistakes tend to be quiet ones that cost you citations for months before anyone notices. ICODA’s AI Visibility checker runs that audit for you. It scans your site’s current configuration against every major AI crawler, flags where a bot you meant to allow is being silently blocked, and shows you exactly where you stand with ChatGPT, Claude, Perplexity, and Gemini.
Crawler access is only the technical layer, though. Getting cited also depends on how your content is structured, what schema it carries, and whether it signals the expertise these systems weigh before quoting a source. If you want that full picture covered rather than just the robots.txt piece, ICODA’s AI SEO services handle the strategy end to end.
Frequently Asked Questions
Robots.txt only works on bots that choose to respect it, and that’s the catch. GPTBot, ClaudeBot, and PerplexityBot all say they follow it, but Cloudflare caught Perplexity rotating user-agents and IPs specifically to get around blocks it had already hit. So treat robots.txt as your first layer, not your only one β if a bot matters enough to block, back it up with a WAF rule too.
Because GPTBot and OAI-SearchBot are two different bots doing two different jobs. GPTBot pulls content for training; OAI-SearchBot and ChatGPT-User handle live search and page fetches, and blocking one doesn’t touch the others. If you want out of training but still want citations, disallow GPTBot and leave OAI-SearchBot and ChatGPT-User alone.
Blocking it still matters because PerplexityBot, the actual indexing crawler, does follow robots.txt β it’s the separate “Perplexity-User” fetcher and the undeclared scrapers that don’t. Perplexity’s own spokesperson dismissed Cloudflare’s findings, but the pattern held across tens of thousands of domains. If you want a real guarantee against Perplexity-linked traffic, pair your Disallow rule with a server-level block instead of trusting robots.txt alone.
No, and this is the one where people worry for nothing. Google-Extended only controls whether your content trains Gemini and AI Overviews β it’s not even a real crawler, just a flag Google checks against its normal Googlebot traffic. Your Search rankings run through Googlebot, which is a completely separate system, so disallowing Google-Extended doesn’t touch them.
Your firewall is overriding your robots.txt, and this happens more than people think. Default WAF rules on Cloudflare, Akamai, and AWS often flag unfamiliar bot traffic as suspicious and rate-limit or 403 it before your robots.txt rules ever come into play. Check your access logs for status codes, not just hits β a bot showing repeated 403s despite an “Allow” rule means your infrastructure is the actual blocker.
Not really, and it can backfire. Anthropic doesn’t publish fixed IP ranges for its bots, and blocking by IP can also stop ClaudeBot from reading your robots.txt in the first place β which means you lose the one control mechanism that’s actually documented to work. Stick to the user-agent rule in robots.txt; it’s the version Anthropic says it honors.
It’s not pointless, but it does mean robots.txt is only half the fix. Most AI crawlers still can’t execute JS reliably, so if your content loads client-side, a bot can get full access and still see a blank page. Server-side rendering or at least exposing the core content in static HTML matters as much as your crawler permissions do.
Block it, because nobody β including crawler-tracking services β can confirm what it actually does. Cohere hasn’t disclosed what triggers this bot or what happens to the content it grabs, and it’s separate from cohere-training-data-crawler, which Cohere has documented for training. Given the lack of transparency, most site owners block both rather than assume the undocumented one is harmless.
Share with
Rate the article
