You know what the real question is? Site owners want to know which names block AI crawlers in robots.txt. They also want to know if it costs them search traffic. The answer is short. To block AI training, add a Disallow rule for GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. None of those rules removes your site from Google Search.
AI search bots are a separate choice. OAI-SearchBot, Claude-SearchBot and PerplexityBot fetch pages so AI tools can cite and link to you. Block them too, and you lose that visibility. We went through the official crawler page of every AI company that publishes one, so you don’t have to. Every user agent, rule and limit here comes from those pages.
- Know the AI Crawler User Agents to Use in robots.txt
- Decide Which AI Crawlers You Actually Want to Block
- Copy a robots.txt File That Blocks AI Training Crawlers
- Block Every AI Crawler in robots.txt, Search Bots Included
- Block AI Crawlers From Only Part of Your Site
- Write AI Crawler Rules in robots.txt That Bots Will Follow
- Know What robots.txt Cannot Do Against AI Crawlers
- Turn On Cloudflare’s AI Crawler Blocking for Stricter Control
- Test Your robots.txt Rules for AI Crawlers
- Answer Common Questions About Blocking AI Crawlers
- Choose an AI Crawler Policy That Protects Your Traffic
Know the AI Crawler User Agents to Use in robots.txt
We compiled the user agent tokens each AI company publishes, along with what each bot does. The company name links to its official crawler page.
| Company | robots.txt token | What the bot does | Follows robots.txt |
|---|---|---|---|
| OpenAI | GPTBot | Collects content that may train OpenAI models | Yes |
| OpenAI | OAI-SearchBot | Surfaces sites in ChatGPT search features | Yes |
| OpenAI | ChatGPT-User | Visits pages when a ChatGPT user asks | Rules may not apply |
| Anthropic | ClaudeBot | Collects content that may train Claude models | Yes |
| Anthropic | Claude-SearchBot | Indexes content for Claude search results | Yes |
| Anthropic | Claude-User | Fetches pages when a Claude user asks | Yes |
| Google-Extended | Controls Gemini training and grounding use | Yes (control token only) | |
| Apple | Applebot-Extended | Controls Apple model training use | Yes (control token only) |
| Perplexity | PerplexityBot | Indexes content for Perplexity search | Yes |
| Common Crawl | CCBot | Builds an open web crawl dataset | Yes |
| Meta | meta-externalagent | Trains AI models and indexes content | Yes |
| Meta | meta-externalfetcher | Fetches links a user asks for | May bypass robots.txt |
Two tokens on that list never show up in your server logs. Google-Extended has no separate HTTP user agent string. Google crawls with its usual user agents and uses the token only as a control. Applebot-Extended works the same way. It does not crawl webpages, and it only decides how Apple may use what Applebot collects.
Decide Which AI Crawlers You Actually Want to Block
AI bots fall into three groups, and each group needs its own decision. In our research, we found that most confusion comes from treating them as one.
Training Crawlers Collect Content for AI Models
GPTBot, ClaudeBot, CCBot and meta-externalagent crawl the web to gather content for model training. Google-Extended and Applebot-Extended are control tokens with the same purpose. Blocking this group says you don’t want your content used to train future models. It does not affect your normal search rankings.
AI Search Crawlers Decide Whether AI Tools Can Cite You
OAI-SearchBot, Claude-SearchBot and PerplexityBot index pages for AI search answers. If you block OAI-SearchBot, your pages drop out of ChatGPT search answers. They can still appear as plain navigational links. Blocking Claude-SearchBot stops Anthropic from indexing your content for search. So most businesses that want AI visibility keep this group open. Our guides on SEO for ChatGPT citations and earning LLM citations explain what these bots feed into.
User-Triggered Fetchers Act for One Person
ChatGPT-User, Claude-User and meta-externalfetcher visit a page because a person asked a question or gave a task. They don’t crawl the web on their own. OpenAI states that robots.txt rules may not apply to ChatGPT-User, because a user starts the action. Meta states that meta-externalfetcher may bypass robots.txt for the same reason. Anthropic’s bots honor robots.txt, and blocking Claude-User may reduce your visibility for user-directed web search.
Leave Link Preview Bots Open
Meta runs more crawlers than the two AI bots in the table. FacebookExternalHit fetches pages that people share on Facebook, Instagram or Messenger. It pulls the title, description and thumbnail for the link preview. Block it, and your shared links lose their preview cards. Meta also notes that this crawler might bypass robots.txt during security or integrity checks.
Meta-WebIndexer is another one to think about. It analyzes content to improve Meta AI search results. Allowing it lets Meta cite and link to your pages in Meta AI answers. So treat it like the other AI search bots.
Copy a robots.txt File That Blocks AI Training Crawlers
This is the version we recommend for most sites that want to opt out of training. It blocks the six training crawlers and control tokens, and it leaves search bots alone. Paste it into the robots.txt file at the root of your domain, for example example.com/robots.txt.
# Block AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# Everyone else, including Googlebot
User-agent: *
Allow: /Keep your existing rules for other bots in the same file. If you already have a User-agent: * group, leave it as it is and add the AI groups above it. Each AI group stands on its own, so the order of groups does not change the result.
Blocking GPTBot tells OpenAI that your content should not be used for training. Blocking ClaudeBot tells Anthropic to exclude your site’s future materials from training datasets. Note the word future. A new rule does not pull back content collected before it existed.
Block Every AI Crawler in robots.txt, Search Bots Included
Some publishers want no AI access at all. Paywalled news, paid courses and member-only research are common cases. This version adds the search bots and the user-triggered fetchers to the training list.
# Training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /
# AI search crawlers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /
# User-triggered fetchers
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: meta-externalfetcher
Disallow: /This file stacks several User-agent lines on top of one rule. The robots.txt standard allows that, and it keeps the file short. Each named bot gets the same Disallow rule.
Expect a real cost. Your pages stop being indexed for ChatGPT, Claude and Perplexity search answers. Also, the fetcher rules work only as far as each company honors them, as the table above shows. Before you go this far, measure how often AI tools mention you today. One of the AI citation tracking tools we reviewed will show you.
Block AI Crawlers From Only Part of Your Site
You don’t have to choose all or nothing. Many sites keep blog posts open for AI search and close paid or private areas. Use path rules for that.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /members/
Disallow: /courses/
Disallow: /research/
User-agent: OAI-SearchBot
User-agent: PerplexityBot
Disallow: /members/Here GPTBot, ClaudeBot and CCBot can still read the blog but not the paid sections. The two search bots can index everything except the member area. Perplexity, Meta and Apple show path rules like this in their official docs.
Google decides between conflicting rules by length. The longest matching path wins, and when two rules tie, Google uses the least restrictive one. So Allow: /research/free/ beats Disallow: /research/ for any URL in the free folder.
Write AI Crawler Rules in robots.txt That Bots Will Follow
Small syntax errors cause most failed blocks. These rules come from Google’s robots.txt specification and the AI companies’ own pages.
- Each crawler reads one group. A crawler obeys only the most specific group that names it. If you add a GPTBot group, GPTBot ignores your
User-agent: *group completely. Copy any rules you still want into the GPTBot group. - Case does not matter in the token. Google matches user agent names without regard to case, so GPTBot and gptbot work the same.
- Each host needs its own file. Rules apply only to the host, protocol and port where the file lives. A blog on blog.example.com needs its own robots.txt. Anthropic also asks site owners to apply the block on each subdomain.
- Plain UTF-8 text at the root. The file must sit at /robots.txt. Google ignores anything past 500 KiB.
- Crawl-delay is not universal. Google ignores it. Anthropic supports it for ClaudeBot, so
Crawl-delay: 1slows that bot without blocking it.
Google-Extended deserves a closer look, because it worries SEO teams most. Google states that Google-Extended does not affect a site’s inclusion in Google Search, and it is not a ranking signal. It controls whether crawled content trains future Gemini models and whether it grounds answers in Gemini Apps and Vertex AI.
Know What robots.txt Cannot Do Against AI Crawlers
Robots.txt is a request. Its directives are voluntary, and crawlers can ignore them. The named AI companies publish their tokens and follow them. Even so, the file itself enforces nothing.
Fake AI Crawlers Can Copy a Bot Name
Any scraper can claim to be GPTBot or CCBot in its user agent string. Common Crawl warns that it knows of crawlers falsely identifying themselves as CCBot. So check the IP address too. OpenAI publishes JSON lists for GPTBot, OAI-SearchBot and ChatGPT-User. Anthropic publishes its list at claude.com/crawling/bots.json. Common Crawl runs CCBot on dedicated IP ranges with reverse DNS that resolves to crawl.commoncrawl.org.
Blocked Pages Can Still Show Up in Search Results
Robots.txt controls crawling. It does not hide a page. Google can still index a blocked URL from links that point to it, without its content. For Google’s AI features in Search, Google points site owners to nosnippet, data-nosnippet, max-snippet and noindex. Our Google AI Overviews SEO guide covers how those controls fit with ranking work.
IP Blocks Can Break Your Opt-Out
Blocking an AI company’s IP ranges at the firewall feels safer, but it has a side effect. Anthropic notes that IP blocking may not work reliably and can stop its crawler from reading your robots.txt at all. Then it cannot see your opt-out. Use robots.txt as the main signal, and add firewall rules only for traffic that ignores it.
Turn On Cloudflare’s AI Crawler Blocking for Stricter Control
If your site runs through Cloudflare, you can block AI bots at the network edge. That covers bots that skip robots.txt, since Cloudflare refuses the request before it reaches your server. The setting is called AI Crawler and Agent access policies.
- In the Cloudflare dashboard, go to Security > Settings.
- Filter by Bot traffic, then select AI Crawler and Agent access policies.
- Pick an action for each behavior: Search, Agents and Training.
- Click Save.
Each behavior offers Block, Block on pages with ads, or Allow. Training also has a Disallow option, which tells crawlers through robots.txt not to use your content. Every policy applies to verified bots with that behavior and to matching unverified bots. For new domains, all three behaviors are set to Allow by default, so nothing changes until you choose.
One detail matters for SEO. Bot Preference Sync can enforce Disallow at the edge. Even so, it skips crawlers used for search and training at once. So a mixed-purpose crawler is not blocked by that setting.
Test Your robots.txt Rules for AI Crawlers
Never assume a robots.txt change worked. Run these checks after every edit.
- Open the live file. Visit yourdomain.com/robots.txt in a private browser window. Confirm the AI groups appear and nothing else got lost.
- Check the Search Console robots.txt report. It lists the robots.txt files Google found for your top hosts, the fetch status and any parsing issues. It works for domain-level properties.
- Request a recrawl if you changed something important. The same report lets you ask Google to fetch the file again.
- Read your server logs. Search for each bot token and compare the source IPs with the lists each company publishes.
- Wait a day. Google generally caches robots.txt for up to 24 hours. Meta’s crawlers may cache it for up to 24 hours too. OpenAI notes about 24 hours for search results to adjust.
On Shopify, add AI rules as plain text in the robots.txt.liquid template, outside the loop that prints Shopify’s default groups. Shopify’s developer docs show a custom User-agent block placed exactly this way. Then confirm the output at /robots.txt, since a broken template can affect every crawler.
Find AI Crawler Visits in Your Server Logs
Your access logs show which AI bots already visit. Each official bot sends a recognizable user agent string. These are the published formats for four of them. Version numbers may change over time.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
CCBot/2.0 (https://commoncrawl.org/faq/)
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)On a Linux server, a single grep on the bot token shows every hit. For example, grep -i "gptbot" access.log lists GPTBot requests. Then compare each source IP with the company’s published list. A request that names GPTBot from an IP outside OpenAI’s list is almost certainly not from OpenAI.
Answer Common Questions About Blocking AI Crawlers
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot handles training. ChatGPT search uses OAI-SearchBot, so keep that one open if you want to appear in ChatGPT answers. If you allow GPTBot and OAI-SearchBot, OpenAI may use one crawl for the two purposes.
Will blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended has no effect on inclusion in Google Search and is not a ranking signal. Googlebot keeps crawling your site as usual.
Does blocking Applebot-Extended remove my site from Siri and Spotlight?
No. Pages that disallow Applebot-Extended can still appear in Apple’s search results, and Apple does not use those rules for ranking.
How long does a robots.txt change take to work for AI bots?
Plan on about a day. Google, Meta and OpenAI all point to a window of roughly 24 hours. Use the Search Console robots.txt report to confirm Google has fetched the new version.
Can I slow AI crawlers down without blocking them?
Partly. Anthropic supports Crawl-delay for ClaudeBot. Google ignores that rule, so it has no effect on Googlebot. For other bots, Cloudflare’s policies give finer control than robots.txt.
Choose an AI Crawler Policy That Protects Your Traffic
For most business sites, our recommendation is to block the training crawlers and keep the AI search bots open. Copy the first robots.txt example on this page, test it, and stop there. You keep control over training, and ChatGPT, Claude and Perplexity can still cite and link to you.
Block everything only when your content is the product you sell. Even then, start tracking how AI tools mention you before the change, so you can measure the loss. Our post on checking whether your site appears in AI Overviews gives you a baseline for Google. The most cited domains in ChatGPT study adds context on who wins those citations today. Still weighing AI search against classic SEO? Read our breakdown of SEO vs AEO differences next.
