About 6% of all websites block OpenAI’s GPTBot in robots.txt, but about half of major news publishers do. That gap is the main story in AI crawler blocking. Most of the web still lets AI bots in, while the sites with the most original content are closing the door.
We researched the largest studies we could find, from Ahrefs, Cloudflare, HasData, the Reuters Institute and the Data Provenance Initiative. We also checked the live robots.txt files of 133 major websites ourselves in November 2026, testing each one against 18 AI bots. In our research, we found four trends. Publishers block far more than other sites. Training bots get blocked more often than search bots from the same company. AI bots crawl much more than they send back. And a robots.txt rule often does not match what a server actually does.
- AI Crawler Blocking Statistics at a Glance
- SEO Sandwitch Study: AI Bot Rules on 133 Major Websites
- How Many Websites Block AI Crawlers
- AI Crawler Blocking Statistics for News Publishers
- Robots.txt Rules vs Real AI Crawler Blocking
- AI Crawler Activity Statistics: Crawling vs Referrals
- Cloudflare’s Role in AI Crawler Blocking
- How AI Crawler Blocking Affects AI Training Data
- Does Blocking AI Crawlers Hurt AI Search Visibility?
- AI Crawler Blocking Statistics: Frequently Asked Questions
- My Recommendation
AI Crawler Blocking Statistics at a Glance
- 5.89% of about 140 million websites block GPTBot in robots.txt, the highest rate of any AI bot.
- 50.5% of 1,148 news publishers blocked GPTBot in July 2026, against 7.9% of 9,746 top websites.
- 46.7% of top sites that disallow GPTBot in robots.txt still served it a page in a live test.
- 52% of crawler requests on Cloudflare’s network were for AI training in June 2026.
- 147,755 pages crawled for every visitor Anthropic referred, on average, from January to July of last year.
- 0.1% of referral traffic came from AI sources across about 35,000 websites in Ahrefs Analytics.
- 47% of 133 major websites in our November 2026 check block at least one AI bot in robots.txt.
- 15 of 40 sites that block GPTBot in our check still allow OAI-SearchBot, so ChatGPT search can still reach them.
- 5% of SEO and marketing sites block GPTBot, against 60% of news publishers in our check.
SEO Sandwitch Study: AI Bot Rules on 133 Major Websites
Large studies, such as the Ahrefs data later in this article, cover millions of sites, and most of those sites are small. So we ran our own check on the sites people actually read and buy from. In November 2026, we pulled the live robots.txt files of 145 well-known websites. 133 of them returned a readable file: 45 news publishers, 37 SEO and marketing sites, 28 ecommerce and retail sites and 23 tech platforms and SaaS sites. Then we tested every file against 18 AI user agents.
How We Tested the Robots.txt Files
- Rule matching: Each bot was matched to the group that names it. If no group named it, the bot fell back to the
User-agent: *group, which is how RFC 9309 tells crawlers to read the file. - What counts as a block: A bot counted as blocked only when
Disallow: /applied to it and noAllow: /rule undid it. Rules that close only some folders were not counted. - What the test does not show: A robots.txt file is a request, not a lock. Firewalls and Cloudflare rules can block bots that the file allows, and some bots ignore the file.
Main Findings From Our Robots.txt Check
- 47% of the 133 sites (62) block at least one AI bot site-wide, and 54% (72) name at least one AI bot in robots.txt.
- Bytespider is the most blocked bot at 39%, followed by CCBot at 38% and ClaudeBot at 36%. GPTBot is blocked by 30%.
- 89% of news publishers (40 of 45) block at least one AI bot, against 11% of SEO and marketing sites (4 of 37).
- 15 of the 40 sites that block GPTBot still allow OAI-SearchBot. In other words, they opt out of training but stay open to ChatGPT search.
- Only 5 sites block all 18 bots: USA Today, Daily Mail, Politico, news.com.au and Reddit.
- 6 sites (4.5%) use the Content-Signal line in robots.txt. Only The Atlantic and MarketingProfs use it to say no to AI training.
Major sites block AI bots far more often than the web as a whole. GPTBot is blocked by 30% of the sites we checked, about five times the 5.89% rate Ahrefs found across 140 million sites. That gap makes sense. Large sites have more original content to protect, and they have teams who maintain robots.txt.
AI Crawler Blocking by Site Type
| Site type | Sites checked | Block at least one AI bot | Block GPTBot | Block ClaudeBot | Name any AI bot |
|---|---|---|---|---|---|
| News publishers | 45 | 40 (89%) | 27 (60%) | 34 (76%) | 41 (91%) |
| Tech platforms and SaaS | 23 | 10 (43%) | 8 (35%) | 8 (35%) | 10 (43%) |
| Ecommerce and retail | 28 | 8 (29%) | 3 (11%) | 4 (14%) | 12 (43%) |
| SEO and marketing sites | 37 | 4 (11%) | 2 (5%) | 2 (5%) | 9 (24%) |
| All sites | 133 | 62 (47%) | 40 (30%) | 48 (36%) | 72 (54%) |
News publishers sell their content, so they have the most to lose when AI tools answer questions without a click. The median news site in our check names 10 AI bots in its robots.txt file. By contrast, the median SEO and marketing site names none.
SEO and marketing brands go the other way. They want to be cited in AI answers, and some say so openly. Four of the six Content-Signal users, Neil Patel, SparkToro, BrightLocal and Hootsuite, set ai-train=yes, search=yes and ai-input=yes. That is an open invitation to AI tools.
Training Bots Get Blocked More Than Search Bots
Sites treat bots from the same company differently, based on what each bot does. GPTBot collects training data and is blocked by 30% of sites. OAI-SearchBot powers ChatGPT search and is blocked by 19%. Anthropic shows the same split: 36% block ClaudeBot, while 23% block Claude-SearchBot.
This split is deliberate. Reuters, Forbes, the Los Angeles Times, TechCrunch, CBS News, LinkedIn, Canva and eBay all block GPTBot but leave OAI-SearchBot open. Altogether, 18 of the 48 sites that block ClaudeBot still let Claude-SearchBot in.
Unusual AI Bot Rules We Found
- Amazon blocks rival AI bots, not its own: The amazon.com robots.txt blocks GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot. However, Amazonbot is not blocked.
- Some sites name AI bots only to let them in: 11 sites, including Moz, Squarespace, SE Ranking, Samsung and Best Buy, list AI bots by name but block none of them site-wide.
- Single-bot blocks: GitHub blocks only Bytespider. IKEA and SparkToro block only Amazonbot.
- Big news sites that block nothing: Fox News, The Independent, NDTV, Time and Sky News do not block any of the 18 bots site-wide.
- Wildcard blocks catch AI bots by default: Reddit, X, Pinterest, Reuters and news.com.au block GPTBot through a site-wide
User-agent: *rule instead of naming it.
How Many Websites Block AI Crawlers
Ahrefs checked the robots.txt files of about 140 million websites. GPTBot came out as the most blocked AI bot, at 5.89%. When Ahrefs counted all 461 million robots.txt files across subdomains, GPTBot’s block rate rose to 7.3%.
| AI bot | Operator | Block rate | Explicit block rate |
|---|---|---|---|
| GPTBot | OpenAI | 5.89% | 0.50% |
| CCBot | Common Crawl | 5.85% | 0.49% |
| Amazonbot | Amazon | 5.78% | 0.34% |
| Bytespider | ByteDance | 5.74% | 0.33% |
| ClaudeBot | Anthropic | 5.74% | 0.28% |
| Google-Extended | 5.71% | 0.30% | |
| PerplexityBot | Perplexity | 5.61% | 0.15% |
| OAI-SearchBot | OpenAI | 5.54% | 0.08% |
The “explicit” column is the share of sites that name the bot. The bigger number also counts sites that block every bot at once. So only about 1 in 200 websites has written a rule just for GPTBot.
- ClaudeBot had the fastest growth. Its block rate rose 32.67% in a year, ahead of every other AI bot Ahrefs tracked.
- Search bots are blocked less. OAI-SearchBot, which powers ChatGPT search results, has an explicit block rate of only 0.08%, about a sixth of GPTBot’s.
- Some industries block much more. Arts and entertainment sites block at about 45%, and law and government sites at about 42%.
- More AI bots exist now. The number of active AI bots Ahrefs tracked doubled from 10 in August 2023 to 21 in December 2024.
AI Crawler Blocking Statistics for News Publishers
News sites block AI crawlers far more than the rest of the web. HasData tested 9,746 top websites and 1,148 news publishers in July 2026, then ran the same checks again on September 16, 2026.
| Measure | Top websites (July) | Top websites (Sept) | Publishers (July) | Publishers (Sept) |
|---|---|---|---|---|
| Block any AI crawler in robots.txt | 10.3% | 8.3% | 56.4% | 53.0% |
| Block GPTBot in robots.txt | 7.9% | 5.7% | 50.5% | 46.9% |
| GPTBot actually served a page | 67.8% | 68.6% | 54.2% | 47.2% |
| Publish an llms.txt file | 7.9% | 9.3% | 3.2% | 4.5% |
Publishers block GPTBot at more than six times the rate of other top sites. Yet the robots.txt numbers dropped between the two checks. HasData traced most of that drop to Cloudflare. GPTBot bans in robots.txt among sites behind Cloudflare fell from 17.1% to 9.9%, while sites not on Cloudflare stayed flat at about 18.7%.
The drop does not mean publishers opened up. Live blocking went the other way: GPTBot was served a page on 47.2% of publisher sites in September, down from 54.2% in July. More sites now block at the server instead of in robots.txt.
The pattern is not new. The Reuters Institute found that 48% of the 15 biggest news sites in each of 10 countries blocked OpenAI’s crawlers by the end of 2023, and 24% blocked Google’s AI crawler. US news sites led at 79%, while Mexico and Poland were at 20%.
Robots.txt Rules vs Real AI Crawler Blocking
A Disallow line in robots.txt is a request, not a lock. HasData compared what sites say in robots.txt with what their servers do when GPTBot visits.
- Paper-only bans are common. Among top sites that disallow GPTBot, 46.7% still served it a normal page. HasData notes this figure rests on a small base of 44 to 45 sites, so treat it as a signal, not a precise rate.
- Silent blocking is growing. 115 sites blocked GPTBot at the server without any robots.txt rule in July. By September, that number had grown to 178.
- CDNs behave very differently. Fastly hard-blocked GPTBot on 61.5% of tested sites, Akamai on 34.5%, Cloudflare on 16.9% and CloudFront on 14.4%. Cloudflare showed a challenge page instead on 24.7%.
- Other AI bots get blocked more than browsers. In a test of 250 publishers, a normal browser got through 77.2% of the time. GPTBot got through 44.4%, ClaudeBot 36.8% and CCBot 37.6%.
This is why checking robots.txt alone gives you an incomplete picture. If you want to know what an AI bot sees on your site, test it with the bot’s user agent and look at your server logs. Our guide to blocking AI crawlers shows the exact robots.txt lines for each bot.
AI Crawler Activity Statistics: Crawling vs Referrals
Sites block AI crawlers because the trade feels one-sided. Cloudflare measured how many pages each company crawls for every visitor it sends back.
| Company | Pages crawled per referred visit (Jan to Jul average) | Change January to July |
|---|---|---|
| Anthropic | 147,754.7 | -86.7% |
| OpenAI | 1,437.8 | -10.4% |
| Perplexity | 172.4 | +256.7% |
| Microsoft | 42.3 | +5.7% |
| 11.8 | +43% |
Anthropic’s ratio improved sharply over those months but stayed far above everyone else. Google crawled about 12 pages for each visit it sent.
- Training dominates AI crawling. Training made up about 80% of AI crawling over that 12-month window. Search was 18% and user-triggered fetches were 2%.
- Training keeps rising. By June 2026, AI training accounted for 52% of all crawler requests on Cloudflare’s network, up from 22% in spring of last year.
- AI referrals are still small. Across about 35,000 sites in Ahrefs Analytics, AI sources sent 0.1% of referral traffic. Search sent 43.8%.
For more on what AI tools send back, see our AI website traffic statistics. Our look at whether GEO will replace SEO covers what this means for search traffic.
Cloudflare’s Role in AI Crawler Blocking
Cloudflare sits in front of about 26% of top websites in the HasData sample, so its default settings move the numbers.
- One-click blocking: More than 1 million Cloudflare customers had turned on the one-click AI bot block, a feature launched in July 2024.
- Default blocking for new domains: Cloudflare set AI training crawlers to blocked by default on new domains, a change it calls Content Independence Day.
- Bot Preference Sync: On August 21, 2026, Cloudflare announced Bot Preference Sync, which writes your dashboard choices for search, agent and training bots into robots.txt. Publishers who say they earn from ads get training set to Disallow by default.
- Robots.txt is still rare at the top. Only about 37% of the top 10,000 domains had a robots.txt file in Cloudflare’s earlier check, and 7.8% of those files disallowed GPTBot.
How AI Crawler Blocking Affects AI Training Data
Blocking also shrinks the open data that AI models learn from. The Data Provenance Initiative audited 14,000 web domains used in popular training sets.
- 5% or more of all tokens in the C4 dataset became fully restricted in a single year, from 2023 to 2024.
- 28% or more of the most active, critical sources in C4 became fully restricted in the same year.
- 45% of C4 is restricted by crawling limits in website Terms of Service.
When the best-maintained sources opt out, AI companies have to license content instead. Cloudflare counted more than 50 publisher and AI licensing deals signed since 2023.
Does Blocking AI Crawlers Hurt AI Search Visibility?
Not always, and the data is mixed. HasData checked 52 sites cited in Google AI Mode answers. Of those, 51.9% blocked at least one AI crawler in robots.txt, compared with 15% across its full sample. So blocking AI training bots did not keep those sites out of AI Mode answers.
The safer approach for most businesses:
- Block training, allow search. You can disallow GPTBot and CCBot but allow OAI-SearchBot and PerplexityBot, so you can still appear in AI search answers.
- Treat Google separately. Blocking Google-Extended stops Gemini training use, but it does not remove you from Google Search or AI Overviews.
- Check your server, not just your file. Make sure your CDN and firewall match what your robots.txt says.
Our GEO statistics and Google AI Mode statistics cover how AI answers cite websites, and our GEO tools list helps you track it.
AI Crawler Blocking Statistics: Frequently Asked Questions
What percentage of websites block GPTBot?
About 5.89% of roughly 140 million websites block GPTBot in robots.txt. Only 0.50% name GPTBot in an explicit rule. The rest block all bots at once.
Which AI crawler is blocked the most?
GPTBot, at 5.89%, followed closely by CCBot at 5.85% and Amazonbot at 5.78%.
Do news websites block AI crawlers?
Yes, far more than other sites. About half of 1,148 news publishers blocked GPTBot in robots.txt in July 2026, against 7.9% of top websites. In our own check of 45 major news sites, 89% block at least one AI bot and 60% block GPTBot.
Does robots.txt stop AI crawlers?
Only the crawlers that choose to follow it. Robots.txt is a request. To enforce a block, you need server, CDN or firewall rules, and testing shows many sites have rules that do not match.
Should I block AI crawlers on my site?
It depends on your goals. Publishers and creators who sell their content often block training bots. Businesses that want AI search visibility usually allow search bots such as OAI-SearchBot and PerplexityBot while blocking training bots.
Can a Site Block AI Training but Stay in AI Search?
Yes. Block training bots such as GPTBot and ClaudeBot, and allow search bots such as OAI-SearchBot and Claude-SearchBot. In our check, 15 of the 40 sites that block GPTBot use this setup, including Reuters, Forbes and LinkedIn.
My Recommendation
Decide what each AI bot gets based on what it gives back. Training bots take content and send almost no visitors, so blocking them costs you little. Search and answer bots can send you traffic, so keep them open unless your content is your product. Then test your site with each bot’s user agent, because the numbers show that many robots.txt bans are not enforced the way site owners think. Review your rules every quarter, since new bots appear often and CDN defaults keep changing.
