If you’ve checked your server logs lately, you’ve probably seen GPTBot show up. Fewer people notice its quieter sibling, OAI-SearchBot, sitting right next to it, doing a different job. Confusing the two is the most common mistake in AI robots.txt setups, and it’s quietly costing sites their ChatGPT citations without anyone realizing why.
The short version:
- OAI-SearchBot is OpenAI’s search crawler. It builds the index behind ChatGPT search and decides whether your pages can be shown and cited. Current user agent:
OAI-SearchBot/1.4. - GPTBot collects public content that may be used to train OpenAI’s models. Current user agent:
GPTBot/1.4. - ChatGPT-User opens a page when a person asks ChatGPT or a Custom GPT to read it.
- Each one is set separately in robots.txt. Disallow GPTBot to opt out of training. Allow OAI-SearchBot to stay eligible for ChatGPT search answers.
The two crawlers split cleanly along one line: training versus search. Get that line wrong in robots.txt, and you can lose citations you never meant to give up.
What Is OAI-SearchBot?
OAI-SearchBot is OpenAI’s web crawler for ChatGPT search. It fetches and indexes pages so ChatGPT can find them, show them in search answers and link to them as sources. It does not collect training data. That is GPTBot’s job.
“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers.”
OpenAI, Overview of OpenAI Crawlers
The same page adds that opted-out sites can still appear as navigational links, so a block removes you from answers, not from every mention. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges.
- Operator: OpenAI
- robots.txt token:
OAI-SearchBot - Current user agent:
OAI-SearchBot/1.4(the version number may change) - IP ranges:
openai.com/searchbot.json - Used for training: No. OpenAI describes it as the crawler for search.
- Follows robots.txt: Yes, according to OpenAI. Some third-party bot directories list a different user agent and say the bot ignores robots.txt; OpenAI’s documentation is the reference.
- Time to take effect: about 24 hours after a robots.txt change
Allowing it makes your pages eligible for ChatGPT search answers. It does not guarantee a citation, because that depends on how well a page answers the question. Our guide to ChatGPT SEO covers that side. For an outside view of the bot, Cloudflare Radar lists OAI-SearchBot as a verified bot in its Search category and charts the HTTP traffic and response codes it receives. Treat third-party directories as a secondary source: Radar still shows an older OAI-SearchBot/1.0 user-agent string.
GPTBot vs OAI-SearchBot: Two Different OpenAI Crawlers
GPTBot and OAI-SearchBot are separate crawlers with separate jobs: GPTBot gathers content for training OpenAI’s models, while OAI-SearchBot indexes pages so ChatGPT can retrieve and cite them in search answers. Treating them as one “OpenAI bot” is where most robots.txt files go wrong.
OpenAI runs a small family of user agents, and the split between training and search is deliberate. According to OpenAI’s own crawler documentation, each setting is independent: a webmaster can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of model training. That’s the whole strategic puzzle most SEO teams miss.
Anthropic runs the same three-way split with ClaudeBot, Claude-User, and Claude-SearchBot.
There are actually four OpenAI agents worth knowing, not two:
- GPTBot: bulk crawler feeding OpenAI’s training pipeline.
- OAI-SearchBot: indexes pages specifically for ChatGPT’s search and citation features.
- ChatGPT-User: fires only when a live user asks ChatGPT to fetch a specific URL in real time.
- OAI-AdsBot: checks the landing pages of ads submitted to ChatGPT for policy compliance. It only visits pages submitted as ads, and per OpenAI’s documentation, its data isn’t used for model training either.
Most guides published in 2023–2024 only mention GPTBot, because OAI-SearchBot didn’t exist yet. If your robots.txt hasn’t been touched since then, there’s a good chance it’s silently blocking the exact crawler responsible for getting you cited in ChatGPT today.
What Each Crawler Does, and Why It Matters
GPTBot collects web content that may be used to train and improve OpenAI’s generative AI models. OAI-SearchBot builds the real-time index that powers ChatGPT’s search and citation features, and it isn’t used for training at all. The practical difference: one affects what the model knows long-term, the other affects whether you show up right now.
GPTBot (current user agent: GPTBot/1.4, full string: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot) exists to feed OpenAI’s training pipeline, and nothing else. Block it, and your content won’t train future models. It has no effect on ChatGPT search; that’s a separate system.
OAI-SearchBot (current user agent: OAI-SearchBot/1.4; its full string is longer than GPTBot’s, using a Chrome-style prefix: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot) is younger, more targeted, and crawls to keep a fresh index for real-time answers. Blocking it doesn’t protect your IP from training. It removes you from ChatGPT’s answer surface, full stop. If your goal is AI search visibility, this is the crawler that matters. Version numbers on both agents change periodically, so match on the GPTBot or OAI-SearchBot substring rather than pinning to an exact version string.
ChatGPT-User (user agent: ChatGPT-User/1.0) is the odd one out. It’s not automated crawling at all. It fires only when someone types your URL into ChatGPT and asks the model to read it, or interacts with a Custom GPT via GPT Actions. OpenAI is direct about this: because these fetches are user-initiated, robots.txt rules may not even apply to them the way they do to automated crawlers. OpenAI changed this wording in a December 2025 documentation update, which also tied ChatGPT-User to Custom GPTs and GPT Actions (Search Engine Roundtable covered the change). Blocking it is rarely useful. It can break a fetch the user explicitly asked for, and it isn’t the control point for search opt-outs anyway; that’s OAI-SearchBot’s job.
| Crawler | User Agent (core string) | Purpose | Used for Training? | Affects ChatGPT Citations? |
|---|---|---|---|---|
| GPTBot | GPTBot/1.4 | Bulk crawl for model training | Yes | No (indirect only) |
| OAI-SearchBot | OAI-SearchBot/1.4 | Real-time search indexing | No | Yes, directly |
| ChatGPT-User | ChatGPT-User/1.0 | User-triggered page fetch | No | No, robots.txt may not even apply |
| OAI-AdsBot | OAI-AdsBot/1.0 | Checks ad landing pages for policy compliance | No | No |
The takeaway: these are independent levers, not one switch. Pull them separately, and you control how OpenAI’s ecosystem interacts with your site.
robots.txt Configuration: Allow One, Block the Other
You configure GPTBot and OAI-SearchBot independently in the same robots.txt file, using separate User-agent blocks. Allowing one and disallowing the other has no effect on the other’s behavior. This is the config most content-driven sites should be running in 2026: stay eligible for citations, opt out of training.
Recommended setup for most publishers, visible in ChatGPT search, opted out of training:
# Allow ChatGPT search citations
User-agent: OAI-SearchBot
Allow: /
# Allow user-triggered fetches (someone pastes your URL into ChatGPT)
User-agent: ChatGPT-User
Allow: /
# Opt out of training data collection
User-agent: GPTBot
Disallow: /
Maximum AI visibility: allow both crawlers, accept that content may be used for training too:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Full opt-out: no citations, no training, typically reserved for paywalled or regulated content:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
Hybrid, path-level control. Many enterprise sites don’t make an all-or-nothing call. They open the blog and documentation, and lock down account areas and internal tools:
User-agent: GPTBot
Disallow: /account/
Disallow: /checkout/
Disallow: /internal-kb/
Allow: /
User-agent: OAI-SearchBot
Disallow: /account/
Disallow: /checkout/
Disallow: /internal-kb/
Allow: /
A few things worth knowing before you ship any of these:
- robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers honor it, but it carries no legal weight on its own.
- Expect roughly 24 hours for OpenAI’s systems to reflect a robots.txt change in search-facing behavior.
- Directory-level rules only work if they sit under the correct
User-agentblock. A strayAllow: /under the wrong agent undoes your intent. - On WordPress, SEO plugins and some hosts generate a virtual robots.txt, so a file you edit by hand can be overwritten. Edit it in the plugin’s own editor (Yoast SEO: Tools, File editor; Rank Math: General Settings, Edit robots.txt), then open yoursite.com/robots.txt to confirm the live version.
OpenAI isn’t the only operator with split agents. The complete list of AI crawlers has the robots.txt token for every major bot, including PerplexityBot, Google-Extended, and Bytespider.
How to Allow OAI-SearchBot: robots.txt, Firewall and CDN
Allowing OAI-SearchBot in robots.txt only helps if its requests reach your site. Three layers can stop it, and robots.txt is the last one. A CDN rule or a server rule blocks the request before robots.txt is ever read, and it leaves no trace in the file.
ICODA’s October 2026 crawl of 3,590 measurable casino, affiliate and crypto sites shows how this plays out. 19.2% had an AI-specific server block, against 3.5% with an AI-specific robots.txt block. Of the 751 sites that blocked at least one AI crawler, 621 (83%) sat behind Cloudflare. On 58.1% of the server blocks, OpenAI’s search and user agents still got through, so many of these blocks catch the training crawlers only. The sample covers those three niches, so read the shares as a picture of them, not of the whole web. The method is in Who Lets AI In?
To allowlist OAI-SearchBot (some teams say whitelist), work through the layers in this order:
- In robots.txt, give it its own group:
User-agent: OAI-SearchBotfollowed byAllow: /. - In your firewall, WAF or CDN, allow requests from the ranges OpenAI publishes at
openai.com/searchbot.json. Match on the IP, not on the user-agent string alone, because anyone can fake a user agent. - In Cloudflare or a similar service, look for settings that block AI bots as a category, and make an exception for OAI-SearchBot.
- Check security plugins, rate limits and geo-blocks. A rule that throttles unknown bots can answer OAI-SearchBot with a 403 or 429.
- Wait about 24 hours, then look in your server logs for OAI-SearchBot requests that got a 200.
Verification: IP Ranges and Crawl Patterns
You verify a crawler claiming to be GPTBot or OAI-SearchBot by checking the request against OpenAI’s published IP ranges, not by trusting the user-agent string alone. User agents can be spoofed; IP ranges are much harder to fake. OpenAI publishes machine-readable JSON files for each agent specifically so this check can be automated.
The published ranges live at predictable URLs:
openai.com/gptbot.json(GPTBot’s ranges)openai.com/searchbot.json(OAI-SearchBot’s ranges)openai.com/chatgpt-user.json(ChatGPT-User’s ranges)
For log searches, these are the full user-agent strings as OpenAI’s documentation lists them. Version numbers can change, so match on the product name.
GPTBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
OAI-SearchBot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
ChatGPT-User
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
OAI-AdsBot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbotNot every OpenAI visit carries one of these names. ChatGPT agent, which browses sites on a user’s behalf, has no crawler token of its own. It looks like an ordinary browser, so it will not show up in a search for the four strings above. OpenAI signs its requests instead: look for a Signature-Agent header pointing to https://chatgpt.com, and verify the signature against the keys at chatgpt.com/.well-known/http-message-signatures-directory. OpenAI’s ChatGPT agent allowlisting guide has the details.
A practical verification workflow looks like this:
- Search server access logs for the relevant substring (
GPTBot,OAI-SearchBot,ChatGPT-User). - Pull the requesting IP from each matching log line.
- Cross-reference that IP against the matching JSON range file.
- Flag any request that claims to be an OpenAI crawler but doesn’t match a published range. That’s a spoofed or unrelated bot, not OpenAI.
One detail that trips up log audits: when GPTBot or OAI-SearchBot fetch the robots.txt file itself, OpenAI’s documentation notes they may append an extra robots.txt marker to the user-agent string, specifically so that logs without full path data can still tell a robots.txt check apart from a regular page fetch. If you’re grepping logs by user-agent alone, account for that marker or you may double-count or miscategorize requests.
On crawl patterns: OpenAI has stated that if a site allows both GPTBot and OAI-SearchBot, it may reuse the results of a single crawl for both purposes rather than crawling the same pages twice. Identical crawl footprints for both agents don’t necessarily mean two separate passes. Don’t read crawl volume as a proxy for which agent is “more aggressive.” Check which agent is actually hitting your logs, and match those IPs against the published ranges.
Impact on ChatGPT Citations
Blocking OAI-SearchBot removes your content from ChatGPT’s search answers, even if GPTBot already crawled and trained on the same pages elsewhere. Per OpenAI’s documentation, an opted-out site won’t show up in cited search answers, though it can still surface as a plain navigational link. The opt-out affects citation-grade inclusion, not every mention. That breaks an old assumption: “if the AI already knows about my site, I’m covered.” In 2026, knowing and citing are separate systems.
A few consequences worth internalizing:
- A site can be “known” to the model and still invisible in search answers. Training data and the live citation index don’t talk to each other.
- Allowing GPTBot has no bearing on whether you’re cited today. Training affects a future model version, not this week’s ChatGPT search results.
- A robots.txt file inherited from a 2023 template is a common, invisible cause of missing citations. OAI-SearchBot didn’t exist when many of those files were written, so it often gets caught by a blanket “block all OpenAI bots” rule that predates it.
- Path-level blocking still applies. If OAI-SearchBot is allowed site-wide but blocked on your best long-form content, that content won’t surface in ChatGPT answers, no matter how strong it is.
Access is a prerequisite, not a guarantee. OAI-SearchBot can crawl every page you have and ChatGPT may still not cite any of them. Whether a page gets cited depends on how well it answers the question; our ChatGPT SEO guide covers what moves that.
If your content strategy depends on AI search visibility, and for most competitive niches in 2026, it increasingly does, OAI-SearchBot access is not optional. It’s the single configuration line that determines whether your best work has a chance to be cited at all.
Check Your Own AI Visibility Setup
Reading a checklist is one thing; knowing whether your own site is configured correctly is another. Misconfigured robots.txt rules, orphaned “block all AI” templates, and path-level mistakes are common enough that guessing isn’t a reliable strategy.
ICODA’s AI Visibility tool checks how AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other major ones) reach your site, along with schema markup and llms.txt, and shows where the gaps are. If GPTBot and OAI-SearchBot have been one undifferentiated line in your robots.txt file, it’s worth five minutes to confirm you’re not accidentally opting out of citations you want.
Frequently Asked Questions
OAI-SearchBot is OpenAI’s crawler for ChatGPT search. It indexes web pages so ChatGPT can find them, show them in search answers and cite them as sources. It does not collect training data; GPTBot does that. The robots.txt token is OAI-SearchBot, the current user agent is OAI-SearchBot/1.4, and the IP ranges are published at openai.com/searchbot.json.
No. ChatGPT is a chatbot, and it does not crawl the web on its own. OpenAI runs separate agents around it: GPTBot collects content that may be used for training, OAI-SearchBot indexes pages for ChatGPT search, and ChatGPT-User opens a page only when a person asks ChatGPT or a Custom GPT to. OpenAI says ChatGPT-User is not used to crawl the web automatically.
From three places. Training data is the large body of text a model learns from. It includes public pages that OpenAI’s GPTBot crawls, which is the part people call scraping, and a site can opt out of it in robots.txt. Live search pulls current pages from an index, such as the one OAI-SearchBot builds for ChatGPT, and that is where citations come from. Direct fetches happen when a user pastes a URL and the chatbot opens it. Only the last two can send a reader to your site.
Yes. Disallow GPTBot to signal that your content should not be used for training, and allow OAI-SearchBot so your pages stay eligible for ChatGPT search answers. OpenAI’s documentation says each setting is independent. One detail: if you allow both bots, OpenAI may use the results of a single crawl for both purposes, so the opt-out only holds while GPTBot stays disallowed. The separation rests on OpenAI’s own statement.
Do three things. Add a group with User-agent: OAI-SearchBot and Allow: / to robots.txt. Allow OpenAI’s published IP ranges (openai.com/searchbot.json) in your firewall, CDN and security plugins, because blocks there never show up in robots.txt. Then wait about 24 hours and check your logs for OAI-SearchBot requests that return 200.
ChatGPT agent has no crawler token of its own. It browses like an ordinary Chrome browser, so a filter on GPTBot or OAI-SearchBot will not catch it. OpenAI identifies its traffic with signed requests instead: look for a Signature-Agent header pointing to https://chatgpt.com and verify the signature against the keys at chatgpt.com/.well-known/http-message-signatures-directory.
It most likely refers to oaiusercontent.com, an OpenAI domain that serves files and images generated by or uploaded to ChatGPT. It is a file-hosting domain, not a crawler, and it has no robots.txt token. The OAI in OAI-SearchBot and OAI-AdsBot stands for OpenAI, and those crawlers identify themselves in the user-agent string.
Check the IP address, not just the user-agent string, which anyone can fake. For Googlebot, run a reverse DNS lookup on the IP, confirm the name ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup to confirm it returns the same IP. For OpenAI, compare the IP with the matching file on openai.com: gptbot.json, searchbot.json, chatgpt-user.json or adsbot.json.
Crawling public pages is not illegal in itself, and OpenAI’s crawlers identify themselves. Whether using collected content for AI training is lawful depends on the country, copyright rules and the site’s terms, and courts are still deciding such cases. robots.txt is a voluntary standard, not a lock and not a legal contract. OpenAI says OAI-SearchBot and GPTBot follow it, and you can check that by comparing crawler IPs with OpenAI’s published ranges. This is not legal advice.
No. Blocking GPTBot only stops future crawls; it does not remove what was already collected. Training happens in cycles, so content gathered earlier may already be inside a trained model. robots.txt cannot take old content out retroactively. It only keeps new content from going in.
No. GPTBot and Googlebot are separate crawlers run by separate companies, so a GPTBot rule does not affect your Google rankings. The one way to get burned is a broad User-agent: * block that also catches Googlebot. Check that GPTBot has its own group and that no wildcard rule disallows everything.
Share with
Rate the article
