Googlebot is on Cloudflare’s list of verified bots. Cloudflare checks it against Google’s published IP ranges, and verified bots have historically been excluded from Cloudflare’s default bot protections. So when Search Console shows Blocked due to access forbidden (403) or Server error (5xx), or the live test fails on a Cloudflare site, the natural reaction is that it can’t be Cloudflare.
It often is. Cloudflare doesn’t block Googlebot by default, but several settings will block it if you turn them on, and one change in September 2026 made a popular AI setting block Googlebot too. This guide shows how to find which Cloudflare feature is responsible, then fix it without letting fake Googlebots through.
The 2026 Change Most Site Owners Missed
In July 2026, Cloudflare introduced separate controls for Search, Agent and Training crawlers, available on every plan including Free. It also said crawlers that do both search and training, including Googlebot, Applebot and Bingbot, would be handled by the most restrictive matching rule from 15 September 2026. In plain terms: if you set AI Training to Block, you can block Googlebot too, unless you confirmed the opt-out that keeps Search plus Training crawlers allowed.
This isn’t theoretical. In August, Search Engine Journal reported a site owner whose sitemap returned 403s to both Googlebot and Bingbot after setting AI Training to Block. Switching the setting off restored access. Google’s John Mueller asked for details. Blocking Google-Extended in robots.txt doesn’t help here, because Google-Extended is only a robots.txt token, not a separate crawler Cloudflare can identify.
Step 1: Confirm Googlebot Is Being Blocked
- In Search Console, run URL Inspection, then Test Live URL on an affected page. A failure with a 403, 503 or “blocked” message points to the edge.
- Open Settings, then Crawl stats, and look at response codes and host status for a spike in 4xx or 5xx responses.
- Check the Page indexing report for growth in Blocked due to access forbidden (403) or Server error (5xx).
Step 2: Find the Cloudflare Feature Responsible
In the Cloudflare dashboard, open your zone’s Security events (under Security, Analytics) and filter by user agent containing Googlebot, or by the verified bot field. Each blocked or challenged request shows which Cloudflare service acted on it.
Table 1: What the Security Event Tells You
| Service or Setting Shown | What Happened | Where to Fix It |
|---|---|---|
| AI crawler controls (Training or Block AI bots) | Googlebot treated as a training crawler | Security settings: allow Search plus Training crawlers, or stop blocking Training |
| Bot Fight Mode | Free-plan bot protection challenged the request | Security, Bots: turn Bot Fight Mode off if it affects crawlers |
| Super Bot Fight Mode | A category action or the verified bots setting blocked it | Security, Bots: set Verified bots to Allow |
| Custom rules | Your own WAF rule matched (country, ASN, user agent, path) | Security, WAF: edit the rule or add a skip rule for verified bots |
| Rate limiting rules | Googlebot crawled too fast for a limit | Exclude verified bots or raise the threshold |
| IP Access rules or Zone Lockdown | An IP range or country is blocked | Remove ranges that overlap Google’s crawler IPs |
| Managed rules | A managed WAF rule flagged a request | Review the rule ID and add a targeted exception |
| No Cloudflare event at all | Cloudflare passed the request, so the origin blocked it | Check your host’s firewall, security plugins and server rules |
That last row matters. If Cloudflare shows nothing but Google still gets a 403, the block is behind Cloudflare, usually a host firewall or a WordPress security plugin.
Step 3: Apply the Right Fix
Fix AI Crawler Settings
- Go to your zone’s Security settings and review the Search, Agent and Training controls.
- If your site depends on Google traffic, don’t set Training to Block without the opt-out that keeps crawlers like Googlebot (Search plus Training) allowed.
- Remember that the older Block AI bots toggle now counts as blocking Training.
- To limit AI training on your content, use robots.txt tokens such as Google-Extended and block dedicated training crawlers individually, rather than a broad training block.
Fix Bot Fight Mode and Super Bot Fight Mode
Cloudflare’s Super Bot Fight Mode documentation says neither product can be reconfigured through WAF custom rules. With Super Bot Fight Mode (Pro and above), you can set Verified bots to Allow, and a custom rule with the Skip action can exclude requests from the Super Bot Fight Mode phase. Bot Fight Mode on the Free plan has no such controls, so if it interferes with crawling, switching it off is the practical fix.
Add a Skip Rule for Verified Search Crawlers
Custom rules run before Super Bot Fight Mode. A skip rule placed at the top lets verified search engine crawlers bypass rules that might catch them:
Rule name: Allow verified search engine crawlers
Expression: (cf.client.bot) or (cf.verified_bot_category eq "Search Engine Crawler")
Action: Skip
Skip: All remaining custom rules, Rate limiting rules,
Super Bot Fight Mode rules, Managed rules (as needed)
Position: FirstThese fields rely on Cloudflare’s own verification, not the user agent string, so a scraper calling itself Googlebot won’t match. Cloudflare’s verified bots page notes that legacy category values remain usable in custom rules.
Fix Overly Broad Custom Rules
Table 2: Custom Rules That Commonly Catch Googlebot
| Rule | Why It Blocks Google | Safer Version |
|---|---|---|
| Block all countries except your own | Googlebot crawls mostly from US IP addresses | Add an exception for verified bots first |
| Block cloud provider ASNs, including Google’s | Googlebot’s addresses belong to Google’s network | Exclude verified bots, or block specific abusive ranges |
| Challenge user agents containing “bot” | Googlebot’s user agent contains “bot” | Match known bad bots by name, not the word bot |
| Managed challenge on /wp-json/ or the whole site | Crawlers can’t solve challenges | Limit challenges to login and admin paths |
| Block empty or unusual referrers | Crawlers don’t send referrers | Remove the rule or exclude verified bots |
Check Site-Wide Modes
- I’m Under Attack mode puts an interstitial in front of visitors. Turn it off once an attack is over, and use targeted rules instead of leaving it on.
- Rate limiting should exclude verified bots or use thresholds well above a normal crawl rate.
- Maintenance or staging protections such as Cloudflare Access on the live hostname will lock out Google entirely.
Step 4: Don’t Let Fake Googlebots In
Allowing anything with Googlebot in its user agent is the wrong fix. Scrapers fake it constantly, as our bot traffic statistics show. Rely on Cloudflare’s verified bot fields, or verify manually with Google’s reverse DNS method:
host 66.249.66.1
# -> crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com
# -> 66.249.66.1 (must match the original IP)Step 5: Confirm Google Can Crawl Again
- Run the live test in URL Inspection again until it returns the page normally.
- Check Security events over the next day for any remaining Googlebot blocks.
- Click Validate fix in the Page indexing report and request indexing for your most important URLs.
- Watch Crawl stats for a return to normal crawl volume and response codes.
For other indexing problems, see our guide to fixing Google indexing issues, and these tips on getting Google to index your site faster once crawling is restored.
A Safe Cloudflare Setup for SEO
Table 3: Recommended Settings for Search-Dependent Sites
| Setting | Recommended |
|---|---|
| AI crawler controls | Search allowed; Training blocked only with Search plus Training crawlers allowed |
| Super Bot Fight Mode: Verified bots | Allow |
| Bot Fight Mode (Free plan) | Off if it challenges crawlers |
| First custom rule | Skip for verified search engine crawlers |
| Country and ASN blocks | Only after the verified bot skip rule |
| Rate limiting | Exclude verified bots |
| I’m Under Attack mode | Off except during active attacks |
| Review cadence | After every security change, run a URL Inspection live test |
FAQs
Does Cloudflare Block Googlebot by Default?
No. Googlebot is a verified bot and is normally allowed. It gets blocked when you enable settings such as a Training block in AI crawler controls, Super Bot Fight Mode with verified bots not allowed, or custom rules that match Google’s IPs or user agent.
Why Did Googlebot Start Getting 403s After I Blocked AI Bots?
Since 15 September 2026, Cloudflare treats Googlebot as a Search plus Training crawler. Blocking Training, or using the older Block AI bots toggle, can block it too unless you allow Search plus Training crawlers.
Can I Block AI Training but Still Allow Googlebot?
Yes. Keep Search plus Training crawlers allowed in Cloudflare’s settings, use the Google-Extended token in robots.txt to opt out of Gemini training, and block dedicated training crawlers individually.
Can a WAF Rule Bypass Bot Fight Mode?
No. Cloudflare says Bot Fight Mode can’t be adjusted with custom rules. Super Bot Fight Mode can be skipped with a custom rule’s Skip action, and its verified bots setting can be set to Allow.
How Do I Know If It’s Cloudflare or My Server?
Check Cloudflare’s Security events for blocked Googlebot requests. If none appear but Google still gets errors, the block is at your origin, such as a host firewall or security plugin.
