OpenAI's crawler documentation says GPTBot and OAI-SearchBot are separate settings. In OpenAI's words, "each setting is independent of the others", so a site can allow OAI-SearchBot and appear in ChatGPT search while disallowing GPTBot so its content isn't used for training. Anthropic gives you the same kind of split. Google handles it differently. By the end, you'll know how to block AI crawlers in robots.txt one bot at a time: training stays off, AI search stays on, and you'll know which rules the bots might not follow.
What each bot does before you block it
Anthropic says it runs three robots, and OpenAI lists four user agents on its crawler page. Most bad robots.txt files come from blocking by company name when you should block by job. Here is what each token controls, based on each vendor's own description.
| Token | Vendor | Job | If you disallow it |
|---|---|---|---|
| GPTBot | OpenAI | Model training | Content excluded from training |
| OAI-SearchBot | OpenAI | ChatGPT search | Missing from ChatGPT search answers |
| ChatGPT-User | OpenAI | User-triggered visits | robots.txt rules may not apply |
| OAI-AdsBot | OpenAI | Ad landing page checks | Only visits submitted ad pages |
| ClaudeBot | Anthropic | Model training | Future content excluded from training |
| Claude-SearchBot | Anthropic | Search indexing | Less visibility in Claude search |
| Claude-User | Anthropic | User-directed fetches | No retrieval for user queries |
| Google-Extended | Gemini training and grounding | Opts out of both |
Two of these you can usually ignore. OAI-AdsBot only visits pages submitted as ads, and OpenAI says the data it collects isn't used to train foundation models.
Step 1: Split OpenAI's training bot from its search bot
GPTBot crawls content that may be used in training OpenAI's foundation models. OAI-SearchBot is the bot that surfaces sites in ChatGPT's search features. If you opt out of OAI-SearchBot, your site won't be shown in ChatGPT search answers, though it can still appear as a navigational link. So if you want to stay out of training and still show up in search, disallow GPTBot and allow OAI-SearchBot.
If you allow both bots, OpenAI says it may use the results from just one crawl for both use cases. That means a single fetch you see in your logs doesn't tell you which bot's permission was used.
Failure mode: expecting changes to take effect right away. OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust. Wait a full day before deciding a rule didn't work.
Step 2: Don't count on robots.txt to stop ChatGPT-User
ChatGPT uses this agent when a user asks ChatGPT or a Custom GPT a question and it visits a web page to answer. It also covers GPT Actions. OpenAI says ChatGPT-User isn't used for automatic crawling, and that because these visits are user-initiated, "robots.txt rules may not apply." It also has no effect on search eligibility: OpenAI tells site owners to use OAI-SearchBot to manage search opt-outs. Adding a Disallow line for ChatGPT-User doesn't hurt, but don't treat it as a guarantee.
Step 3: Configure Anthropic's three bots, one subdomain at a time
Anthropic says that restricting ClaudeBot signals that the site's future materials should be excluded from its training datasets. Claude-SearchBot and Claude-User serve users directly. Disabling Claude-User stops Anthropic from retrieving your content in response to a user query, and disabling Claude-SearchBot stops your content from being indexed for search. Either one can make your site less visible in Claude.
Failure mode: forgetting subdomains. Anthropic tells you to put the rule in the robots.txt in your top-level directory and to do this for every subdomain you want to opt out. A rule on example.com does nothing for docs.example.com or shop.example.com.
Failure mode: blocking IPs instead. It's tempting to firewall Anthropic's IP ranges. Anthropic warns that this may not work correctly or persistently, because it stops its bots from reading your robots.txt in the first place. Use the IP list only to confirm that a request actually came from Anthropic.
If you're worried about server load, Anthropic supports the non-standard Crawl-delay extension, and its example uses Crawl-delay: 1.
Step 4: Google-Extended works differently from the others
OpenAI and Anthropic both separate training from search. Google-Extended combines two things into one switch. Google's crawler page calls it a standalone product token that lets publishers control whether content Google crawls can be used to train future Gemini models (the models behind the Gemini apps and the Vertex AI API for Gemini). The same token also controls grounding, which means feeding content from Google's search index to the model at prompt time, in the Gemini apps and in Grounding with Google Search on Vertex AI.
In practice, disallowing Google-Extended takes your content out of Gemini training and also out of Gemini's grounded answers. You can't keep one and drop the other. Google-Extended also has no user agent of its own: Google crawls with its existing user agent strings and uses the token in a control capacity. And it does not impact a site's inclusion in Google Search, nor is it a ranking signal, so blocking it leaves your Google Search presence alone. The real choice is between staying out of Gemini training and appearing in Gemini's grounded answers. On the upside, Google says its common crawlers always respect robots.txt rules during automatic crawling.
Google's crawler page also says Google intends to support the shared AIPREF training preference signal, written as user-agent: __usage_ai-training, in early 2027.
Step 5: Deploy the file
This file blocks training at all three companies and leaves OpenAI's and Anthropic's search and user-fetch bots allowed:
# OpenAI: no training, yes to ChatGPT search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
# Anthropic: no training, yes to search and user fetches
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
Crawl-delay: 1
User-agent: Claude-User
Allow: /
# Google: opts out of Gemini training AND grounding
User-agent: Google-Extended
Disallow: /
Upload it to the root of every host you want covered. Then open /robots.txt in a browser to make sure the live version is the one you just saved and not a cached copy.
Step 6: Check your logs
OpenAI's search bot identifies itself as OAI-SearchBot/1.4, and its user-agent string includes a robots.txt marker. OpenAI says the marker helps you tell robots.txt requests apart from other requests when your logs don't record paths. ChatGPT-User shows up as ChatGPT-User/1.0. OpenAI and Anthropic both publish IP lists, so you can check that traffic claiming to be one of these bots really is.
The main thing to take away: your training and search settings are separate for OpenAI and Anthropic, and combined for Google. Set each token based on what it actually does. Recheck the file whenever a vendor updates its documentation, since OpenAI offers an RSS feed for changes to its crawler page.
Sources
- developers.openai.com – "Overview of OpenAI Crawlers"
- developers.google.com – "Google's common crawlers | Google crawling infrastructure"
- support.claude.com – "Does Anthropic crawl data from the web, and how can site owners block the crawler?" (2026)