Which AI crawlers to allow in robots.txt
Each AI company runs more than one crawler, and they do different jobs: some collect training data, some build a search index, some fetch a page because a person asked for it. Block the wrong one and you disappear from AI answers without meaning to. This is the current list, taken from each company's own documentation, and what to do with it.
Three jobs, not one "AI bot"
Almost every crawler falls into one of three roles:
- Training crawlers collect pages that may be used to train future models. Blocking one keeps your content out of training. It does not remove you from search answers.
- Search crawlers build the index an assistant searches when it answers with web access. Blocking one is how you disappear from those answers.
- User-requested fetchers load a page because someone asked the assistant about it or pasted its link. They do not crawl on their own, and some companies say robots.txt may not apply to them.
The list
| Company | robots.txt name | Job | If you block it |
|---|---|---|---|
| OpenAI | OAI-SearchBot | Search | Not shown in ChatGPT search answers, apart from navigational links |
| OpenAI | GPTBot | Training | Content not used to train OpenAI's models |
| OpenAI | ChatGPT-User | User-requested | OpenAI says robots.txt rules may not apply |
| Anthropic | Claude-SearchBot | Search | May reduce visibility in Claude's search answers |
| Anthropic | ClaudeBot | Training | Future content excluded from training |
| Anthropic | Claude-User | User-requested | Claude cannot fetch your pages when asked to |
| Perplexity | PerplexityBot | Search | Pages not surfaced in Perplexity's search results |
| Perplexity | Perplexity-User | User-requested | Perplexity says it generally ignores robots.txt |
Googlebot | Search, including AI Overviews | Google can no longer crawl your pages for Search | |
Google-Extended | Gemini training and grounding | Not used to train Gemini or to ground Gemini app answers; Search unaffected |
Sources: OpenAI, Anthropic, Perplexity, Google. Older guides still list Claude-Web and anthropic-ai; Anthropic no longer uses them.
Two things to know about Google
Google-Extended is not a crawler. Nothing visits your site under that name. It is a switch in robots.txt that tells Google whether content Googlebot has already fetched may be used to train Gemini and to ground answers in the Gemini apps. Google says it has no effect on inclusion or ranking in Search.
So it does not control AI Overviews or AI Mode. Those are part of Google Search and are built from Googlebot's index, which Google-Extended has no say over.
A robots.txt for being recommended
If the goal is to appear in AI answers, allow the search and user-requested crawlers. A group can list several User-agent lines that share the rules beneath them:
# Appear in AI answers: search and user-requested fetches
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
# Training: your choice. These lines opt out of it.
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /Leave out the training group if you are happy for future models to learn from your site. Being in training data is one way a model comes to know your business without having to search for it. Think twice before disallowing Google-Extended: it covers grounding in the Gemini apps as well as training, so blocking it costs you visibility there.
Anything robots.txt does not mention is allowed by default, so a file that never names these crawlers already lets them in. Naming a crawler in its own group also overrides a general User-agent: * block for that crawler, which is how you let search crawlers through a site that is otherwise closed.
When robots.txt says yes and the crawler still cannot get in
Bot protection often turns AI crawlers away regardless of robots.txt: a CDN challenge page, a firewall rule, or rate limiting that answers with errors. Perplexity's documentation tells sites behind a web application firewall to allow its crawlers explicitly. OpenAI, Anthropic and Perplexity publish the IP addresses their crawlers use, so you can let the real ones through without trusting anything that merely claims the name.
The way to know is your server log. Search it for the names in the table and check the status codes they get back. Noksaro's crawler report reads an uploaded log and counts visits and errors per crawler, per day.
Changes are not instant. OpenAI and Perplexity both say a robots.txt update can take about a day to reach their systems.
See it working before you pay for it.
An account costs nothing and opens the demo: real sites we measure every week, with the whole product running on them — the questions, the rivals, the channels, the steps and the trend lines a first check cannot have.
See the demo