In robots.txt, sort AI crawlers by the job they do. Allow the search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot) if you want AI assistants to cite you: block one and you drop out of that assistant's answers. Blocking training (GPTBot, ClaudeBot, CCBot, Applebot-Extended) is a separate decision and doesn't affect AI search. Google-Extended sits in between: it never affects Google Search, but it also decides whether Gemini apps can use your pages in their answers.
You ask ChatGPT or Perplexity for the best roofer, accountant or law firm in your town. Three competitors come up. You don’t, even though you rank on page one of Google. One of the first things we check in that situation is the site’s robots.txt rules for AI crawlers, and the CDN or firewall in front of it. A line copied from a “block the AI bots” article, or a switch flipped in a dashboard, can shut out the bots behind AI answers along with the training ones.
The answer is rarely “allow everything” or “block everything”. The big AI companies now run separate bots for separate jobs, and robots.txt lets you answer each one differently. Here’s who they are, what each one does, and the setup most businesses settle on.
Which AI crawlers read your robots.txt?
Every name below comes from the operator’s own documentation, checked on November 3, 2026. The token is the name you put after User-agent: in robots.txt. Operators add and rename bots, so check the linked page before you edit your file.
| Crawler | What it does |
|---|---|
| GPTBot OpenAI |
Training. Blocking it opts you out of model training. |
| OAI-SearchBot OpenAI |
Search index for ChatGPT search. Blocked sites don’t appear in its answers. |
| ChatGPT-User OpenAI |
User-triggered. Visits a page for a person using ChatGPT. OpenAI says robots.txt rules may not apply. |
| ClaudeBot Anthropic |
Training. Blocking it excludes your future pages from model training. |
| Claude-SearchBot Anthropic |
Search index. Improves Claude’s search results. |
| Claude-User Anthropic |
User-triggered. Fetches a page when someone asks Claude a question. |
| PerplexityBot Perplexity |
Search index. Perplexity says it isn’t used to train AI models. |
| Perplexity-User Perplexity |
User-triggered. Perplexity says it generally ignores robots.txt. |
| Googlebot |
Search index for Google Search, including AI Overviews and AI Mode. |
| Google-Extended |
Control token, not a crawler. Covers Gemini training and grounding. No effect on Google Search. |
| Google-Agent |
User-triggered. Agents browsing for a user. Google says these fetchers generally ignore robots.txt. |
| Applebot Apple |
Search for Spotlight, Siri and Safari. Apple says the data may also help train its models. |
| Applebot-Extended Apple |
Control token, not a crawler. Opts you out of training Apple’s foundation models. |
| CCBot Common Crawl |
Builds a free public copy of the web for researchers and companies. |
Google-Extended doesn’t touch Google Search. Google says it doesn’t have its own user agent, it isn’t used as a ranking signal, and it doesn’t affect whether your site is included in Search (Google for Developers). It does cover grounding, which is Gemini apps pulling content from the Search index to answer a prompt, so blocking it can keep your pages out of Gemini’s answers while Search stays the same. AI Overviews and AI Mode are part of Search, so they run on Googlebot. A page only needs to be indexed and eligible for a snippet to appear as a link in them (Google Search Central). To limit them, Google points to nosnippet, data-nosnippet, max-snippet or noindex, and those change your regular listings too.
Applebot-Extended works the same way. Apple says it doesn’t crawl pages. It only decides how Apple may use what Applebot collects, and it isn’t considered in ranking for Search (Apple).
CCBot is easy to forget. Common Crawl gives its archive away free, and many AI developers have trained on it. If you want out of training, block CCBot as well as the big names.
A Mozilla Foundation study of 47 large language models published between 2019 and October 2023 found that at least 64% were trained on Common Crawl data (Mozilla Foundation).
Three kinds of AI bots, and what blocking each one costs
Sort the bots by job, not by company. That’s the decision you’re really making.
- Training crawlers copy pages to build future models. Blocking them doesn’t affect search or citations.
- Search crawlers build the index AI answers draw from and link to. Block one and you drop out of that assistant’s answers.
- User-triggered fetchers open one page because a person asked, such as a customer pasting your link into ChatGPT. Block them and the assistant can’t read you at that moment.
Each setting is independent. OpenAI says so directly: you can allow OAI-SearchBot and block GPTBot (OpenAI). Perplexity says the same about its two bots.
What most businesses decide
If you sell services or products and want AI assistants to recommend you, the sensible default is:
- Allow every search crawler. OpenAI and Perplexity both recommend allowing theirs if you want to appear in their search results. Never block Googlebot.
- Allow user-triggered fetchers. They only visit when a person is asking about something, often you.
- Decide on training as a separate business question.
Training has no single right answer. Publishers, course creators and anyone whose content is the product often block it. A local service business usually has less to protect, and some owners allow training so models learn their brand. Nobody can show how much that helps, so we treat it as the owner’s call and write it down, so the next redesign doesn’t reverse it by accident.
robots.txt only decides who gets in the door. Our guide to answer engine optimization covers how AI answers choose sources.
An example robots.txt for AI crawlers
This file allows AI search and user fetches, opts out of training, and leaves Googlebot in the general group. If you want Gemini apps to use your pages in their answers, delete the Google-Extended line. Swap /account/ for the private folders on your own site.
# AI search: index and cite us
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /account/
# Fetches when a person asks
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Disallow: /account/
# Model training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
# Everyone else, including Googlebot
User-agent: *
Disallow: /account/
Sitemap: https://www.example.com/sitemap.xmlOne rule catches people out. A crawler obeys only one group: the one that names it most specifically. It ignores the rest, including the * group (Google for Developers). So once you give OAI-SearchBot its own group, it no longer sees the Disallow: /account/ you wrote for everyone. Repeat any rule you need in every named group.
The accidental blocks: CDNs and firewalls
Your robots.txt can be perfect and AI search crawlers can still be turned away. A CDN or firewall with its own bot settings blocks the request before robots.txt matters.
Cloudflare is the best-known example. It has offered a one-click “Block AI bots” option for a while. On July 1, 2026 it split AI traffic into three categories, Search, Agent and Training, available on every plan including Free, and announced two changes starting September 15, 2026 (Cloudflare):
- For new domains, Training and Agent bots are blocked by default on pages that show ads. Agent bots are the ones that visit a page for a person, such as ChatGPT-User.
- Crawlers that do both search and training are judged on both. Cloudflare says Googlebot, Applebot and Bingbot are blocked for customers who chose to block Training, either with the new options or the older “Block AI bots” setting.
If your site is on Cloudflare and anyone ever switched on "Block AI bots" or blocked the Training category, check now whether Googlebot is being blocked. Cloudflare's own announcement says multi-purpose crawlers such as Googlebot are blocked under that setting from September 15, 2026, unless the owner opted out in the Security settings. Losing Googlebot means losing Google Search and AI Overviews.
Cloudflare can also write rules into your robots.txt for you. Its managed robots.txt adds Disallow lines for bots such as GPTBot, ClaudeBot, CCBot and Google-Extended, plus a “Content-signal” line that states your preferences for search and training (Cloudflare Docs). Those lines aren’t in the file on your server, so they’re easy to miss.
Other CDNs, hosting firewalls and WordPress security plugins have similar bot controls. If AI assistants can’t seem to see your site, check every layer between the bot and the file.
How to check what’s really happening
- Read the live file. Open yourdomain.com/robots.txt in a browser. That’s what crawlers get, including anything a CDN added.
- Test each AI crawler against it. Our free validator lists which crawlers and AI bots can read your site and flags rules that block the ones behind AI answers.
- Check your CDN and firewall. In Cloudflare, look at AI Crawl Control and the bot settings under Security. Blocks there are enforced with firewall rules and return a 403 or 402 error (Cloudflare Docs).
- Search your server logs. Ask your host for 30 days of access logs and search them for OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. A 200 status means the bot got the page. A 402, 403 or 429 means something blocked it. User agent names can be faked, so for a serious check match the IP address against the ranges OpenAI, Anthropic and Perplexity publish.
For a wider view, our Site AI Readiness Check reads your robots.txt crawler by crawler, separates answer engines from training-only bots, and checks whether AI tools can tell who you are. If you’re reading a full audit report, our guide on reading a technical SEO audit explains where crawl blocks usually show up.
robots.txt is a request, not a lock
Google is plain about this: robots.txt instructions can’t enforce crawler behavior, and it’s up to each crawler to obey them (Google Search Central). The large operators in the table say their crawlers honor it. User-triggered fetchers are the exception: Google says its own generally ignore robots.txt because a person asked for the page (Google for Developers). And nothing stops a less careful bot from ignoring it.
- It isn’t privacy. robots.txt is public, so listing a secret folder in it tells everyone where to look. Put private content behind a login.
- It doesn’t remove pages from search. Google says a blocked URL can still be indexed if other sites link to it. Use noindex for that, and leave the page crawlable so Google can see the tag.
- Real blocking happens at the server or CDN. If a bot ignores your file, a firewall rule is the tool, which is why those rules need checking.
Blocking also isn’t retroactive. Opting out of GPTBot or ClaudeBot today covers what they collect from now on, and it can’t pull your pages back out of an archive someone already copied.
What to do this week
If robots.txt looks fine and Google still isn’t crawling the pages that matter, read our guide to crawl budget.
FAQs
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI’s training crawler. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages when a person asks. OpenAI says each setting is independent, so you can block GPTBot and still appear in ChatGPT search answers.
Does blocking Google-Extended keep my site out of AI Overviews?
No. Google-Extended covers Gemini training and grounding, and Google says it has no effect on Google Search. AI Overviews and AI Mode use Googlebot and the normal Search index. Limiting them takes snippet controls or noindex, which also change your regular listings.
Should I block ClaudeBot?
Only if you don’t want Anthropic to train on your pages. ClaudeBot collects training data. Claude-SearchBot and Claude-User are the bots that help Claude find and read your site when people ask, and Anthropic says blocking those may reduce your visibility in Claude’s answers.
Do AI crawlers actually follow robots.txt?
The big operators all say their training and search crawlers follow it. User-triggered fetchers are different: OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity and Google say theirs generally ignore it. Smaller or unknown bots may ignore it completely.
How long does a robots.txt change take to work?
Usually about a day. Google generally caches robots.txt for up to 24 hours, OpenAI says its search systems take about 24 hours to adjust, and Perplexity says up to 24 hours. A CDN or firewall rule takes effect as soon as it’s saved.
Sources
- Overview of OpenAI crawlers, OpenAI
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic
- Perplexity crawlers, Perplexity
- Google’s common crawlers, Google for Developers
- Google’s user-triggered fetchers, Google for Developers
- AI features and your website, Google Search Central
- Introduction to robots.txt, Google Search Central
- How Google interprets the robots.txt specification, Google for Developers
- About Applebot, Apple Support
- Frequently asked questions, Common Crawl
- How Common Crawl’s data infrastructure shaped the battle royale over generative AI, Mozilla Foundation, February 2024
- Your site, your rules: new AI traffic options for all customers, Cloudflare, July 2026
- Managed robots.txt, Cloudflare Docs
- Manage AI crawlers, Cloudflare Docs
