Skip to main content

tutorial

How to Block AI Crawlers in robots.txt Without Losing AI Search Visibility

By AInews Desk
How to Block AI Crawlers in robots.txt Without Losing AI Search Visibility
7 min read
1211 words

OpenAI's crawler documentation says GPTBot and OAI-SearchBot are separate settings. In OpenAI's words, "each setting is independent of the others", so a site can allow OAI-SearchBot and appear in ChatGPT search while disallowing GPTBot so its content isn't used for training. Anthropic gives you the same kind of split. Google handles it differently. By the end, you'll know how to block AI crawlers in robots.txt one bot at a time: training stays off, AI search stays on, and you'll know which rules the bots might not follow.

What each bot does before you block it

Anthropic says it runs three robots, and OpenAI lists four user agents on its crawler page. Most bad robots.txt files come from blocking by company name when you should block by job. Here is what each token controls, based on each vendor's own description.

TokenVendorJobIf you disallow it
GPTBotOpenAIModel trainingContent excluded from training
OAI-SearchBotOpenAIChatGPT searchMissing from ChatGPT search answers
ChatGPT-UserOpenAIUser-triggered visitsrobots.txt rules may not apply
OAI-AdsBotOpenAIAd landing page checksOnly visits submitted ad pages
ClaudeBotAnthropicModel trainingFuture content excluded from training
Claude-SearchBotAnthropicSearch indexingLess visibility in Claude search
Claude-UserAnthropicUser-directed fetchesNo retrieval for user queries
Google-ExtendedGoogleGemini training and groundingOpts out of both

Two of these you can usually ignore. OAI-AdsBot only visits pages submitted as ads, and OpenAI says the data it collects isn't used to train foundation models.

Step 1: Split OpenAI's training bot from its search bot

GPTBot crawls content that may be used in training OpenAI's foundation models. OAI-SearchBot is the bot that surfaces sites in ChatGPT's search features. If you opt out of OAI-SearchBot, your site won't be shown in ChatGPT search answers, though it can still appear as a navigational link. So if you want to stay out of training and still show up in search, disallow GPTBot and allow OAI-SearchBot.

If you allow both bots, OpenAI says it may use the results from just one crawl for both use cases. That means a single fetch you see in your logs doesn't tell you which bot's permission was used.

Failure mode: expecting changes to take effect right away. OpenAI says it can take about 24 hours after a robots.txt update for its search systems to adjust. Wait a full day before deciding a rule didn't work.

Step 2: Don't count on robots.txt to stop ChatGPT-User

ChatGPT uses this agent when a user asks ChatGPT or a Custom GPT a question and it visits a web page to answer. It also covers GPT Actions. OpenAI says ChatGPT-User isn't used for automatic crawling, and that because these visits are user-initiated, "robots.txt rules may not apply." It also has no effect on search eligibility: OpenAI tells site owners to use OAI-SearchBot to manage search opt-outs. Adding a Disallow line for ChatGPT-User doesn't hurt, but don't treat it as a guarantee.

Step 3: Configure Anthropic's three bots, one subdomain at a time

Anthropic says that restricting ClaudeBot signals that the site's future materials should be excluded from its training datasets. Claude-SearchBot and Claude-User serve users directly. Disabling Claude-User stops Anthropic from retrieving your content in response to a user query, and disabling Claude-SearchBot stops your content from being indexed for search. Either one can make your site less visible in Claude.

Failure mode: forgetting subdomains. Anthropic tells you to put the rule in the robots.txt in your top-level directory and to do this for every subdomain you want to opt out. A rule on example.com does nothing for docs.example.com or shop.example.com.

Failure mode: blocking IPs instead. It's tempting to firewall Anthropic's IP ranges. Anthropic warns that this may not work correctly or persistently, because it stops its bots from reading your robots.txt in the first place. Use the IP list only to confirm that a request actually came from Anthropic.

If you're worried about server load, Anthropic supports the non-standard Crawl-delay extension, and its example uses Crawl-delay: 1.

Step 4: Google-Extended works differently from the others

OpenAI and Anthropic both separate training from search. Google-Extended combines two things into one switch. Google's crawler page calls it a standalone product token that lets publishers control whether content Google crawls can be used to train future Gemini models (the models behind the Gemini apps and the Vertex AI API for Gemini). The same token also controls grounding, which means feeding content from Google's search index to the model at prompt time, in the Gemini apps and in Grounding with Google Search on Vertex AI.

In practice, disallowing Google-Extended takes your content out of Gemini training and also out of Gemini's grounded answers. You can't keep one and drop the other. Google-Extended also has no user agent of its own: Google crawls with its existing user agent strings and uses the token in a control capacity. And it does not impact a site's inclusion in Google Search, nor is it a ranking signal, so blocking it leaves your Google Search presence alone. The real choice is between staying out of Gemini training and appearing in Gemini's grounded answers. On the upside, Google says its common crawlers always respect robots.txt rules during automatic crawling.

Google's crawler page also says Google intends to support the shared AIPREF training preference signal, written as user-agent: __usage_ai-training, in early 2027.

Step 5: Deploy the file

This file blocks training at all three companies and leaves OpenAI's and Anthropic's search and user-fetch bots allowed:

# OpenAI: no training, yes to ChatGPT search
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

# Anthropic: no training, yes to search and user fetches
User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /
Crawl-delay: 1

User-agent: Claude-User
Allow: /

# Google: opts out of Gemini training AND grounding
User-agent: Google-Extended
Disallow: /

Upload it to the root of every host you want covered. Then open /robots.txt in a browser to make sure the live version is the one you just saved and not a cached copy.

Step 6: Check your logs

OpenAI's search bot identifies itself as OAI-SearchBot/1.4, and its user-agent string includes a robots.txt marker. OpenAI says the marker helps you tell robots.txt requests apart from other requests when your logs don't record paths. ChatGPT-User shows up as ChatGPT-User/1.0. OpenAI and Anthropic both publish IP lists, so you can check that traffic claiming to be one of these bots really is.

The main thing to take away: your training and search settings are separate for OpenAI and Anthropic, and combined for Google. Set each token based on what it actually does. Recheck the file whenever a vendor updates its documentation, since OpenAI offers an RSS feed for changes to its crawler page.

Sources

7 min read
•
1211 words

Tags

block AI crawlers robots.txtGPTBot robots.txt blockingClaudeBot robots.txt configurationGoogle-Extended user-agent blocking

Related Articles