Cloudflare's AI Blocks Are Breaking Googlebot: Crawler Classification Is Now an IP Problem

On August 4, Search Engine Journal reported on an r/TechSEO thread in which a site owner had switched a single Cloudflare setting, AI Training, to Block, then noticed Googlebot and Bingbot collecting 403s whenever they came for the sitemap. Turning the setting off restored access right away. The thread drew enough attention that Google's John Mueller asked the poster to get in touch so the team could take a look.

Nothing was actually broken. Cloudflare classifies Googlebot and Bingbot as "Search + Training" crawlers, and its July 1 announcement spells out that mixed-purpose crawlers "will be allowed/blocked according to all of their behaviors." Block training and you block every crawler that also trains, even the one running the search index your business depends on. The setting did exactly what the documentation said it would; few people had read the documentation that carefully.

The stakes rise on September 15, 2026. From that date, new domains onboarding to Cloudflare will have Training and Agent crawlers blocked by default on pages that display ads, with only Search allowed through. Cloudflare fronts roughly a fifth of the web, so if you publish content and want to stay in Google while opting out of AI training, the next five weeks are a good time to understand what these categories actually match on. What they match on is crawler identity, and on the internet a crawler's identity ultimately resolves to IP addresses.

One crawler serving two masters

Googlebot stopped being just a search crawler some time ago, without much fanfare. The same fetches that feed the search index can also feed Gemini model training and grounding, unless you opt out. The opt-out Google offers is Google-Extended, announced on September 28, 2023, and it helps to be precise about what it is: a robots.txt token, nothing more. Google-Extended has no user agent of its own, sends no requests, and holds no IP addresses. The crawler is still Googlebot; the token only governs what Google does with the bytes after the fetch.

That design has a hard consequence. Any control enforced at the network layer, whether a WAF rule, a bot management category, or a rate limiter, sees exactly one Googlebot. It cannot admit the search half of a request and refuse the training half, because there is only one request.

Cloudflare knows this, and its response has been to push the problem back onto the crawler operators. In the company's own framing, mixing purposes in one crawler is not an accident but an advantage: the largest search engine gets access to roughly twice the content of leading AI companies precisely because publishers cannot refuse its training use without losing search discoverability. Per Cloudflare Radar's May 2026 measurements, 35.7 percent of AI crawler traffic came from these mixed-use crawlers.

Three categories, one deadline

The July 1 model sorts automated traffic into three purposes. Search collects and indexes content to answer questions about it later, Agent acts in real time on a person's behalf, and Training takes content to train or fine-tune a model. Every crawler Cloudflare recognizes carries one or more of these labels, a customer's allow and block decisions apply per label, and the strictest one wins for crawlers that carry several.

September 15 turns this model into the default, on the theory that an ad is a signal the page was meant for human eyes. Cloudflare has been explicit about what it wants from Google, Microsoft, Apple, and everyone else running a mixed crawler: split your automation into separate crawlers per purpose before the deadline, or inherit the strictest rule.

The question a user agent cannot answer

Underneath the news cycle sits a structural problem. What publishers actually want to know is intent: will these bytes end up in a search index or a training corpus? That fact does not exist at request time. It is a decision made later, in a datacenter you cannot see, by the crawler's operator. No HTTP header can carry it honestly, and the user agent header in particular is self-declared and freely forgeable (we covered what that forgeability does to robots.txt enforcement last week).

So the industry is converging on the only workable substitute: split intent across identities, and make each identity verifiable. OpenAI runs GPTBot for training, OAI-SearchBot for search, and ChatGPT-User for live fetches, each with its own published IP range file. Anthropic followed in May 2026 with bots.json, covering ClaudeBot, Claude-User, and Claude-SearchBot. Under this scheme a purpose is something you can enforce, because it maps to a set of prefixes you can match at the edge.

Google, for now, is the holdout: one crawler, one range file, two purposes, and a trust-based token to tell them apart.

Verifying Googlebot by IP, the reliable way

Whatever you decide to allow, the enforcement layer should key on verified identity rather than headers. For Googlebot there are two independent checks. Both have been around for two decades, and both are much harder to fake than anything in the request itself.

First, the published ranges. Google ships machine-readable prefix lists for its crawler infrastructure and documents them: googlebot.json (which now redirects to a consolidated common-crawlers.json) for Googlebot proper, special-crawlers.json for product-specific bots like AdsBot, and user-triggered-fetchers.json for on-demand fetches. The files change over time, so resolve against them at lookup time rather than baking a snapshot into a firewall rule from 2024.

Second, forward-confirmed reverse DNS. Resolve the PTR record of the connecting IP, check that the hostname ends in googlebot.com, google.com, or googleusercontent.com, then resolve that hostname forward and confirm it returns the original IP:

dig -x 66.249.66.1 +short
crawl-66-249-66-1.googlebot.com.
dig crawl-66-249-66-1.googlebot.com +short
66.249.66.1

Both lookups must agree; a PTR record alone says whatever the IP's owner wants it to say. A request claiming to be Googlebot from an IP that fails both checks is simply not Googlebot. The same procedure works for Bingbot (rDNS under search.msn.com, plus bingbot.json), and for operators that publish nothing at all, the autonomous system and hosting attribution of the source IP are the signals that remain.

A block policy that survives September 15

If we were running a publisher site today, this is roughly the policy we would deploy.

You want	Do this	Watch out for
Stay in Google Search, keep content out of Gemini training	robots.txt: User-agent: Google-Extended / Disallow: /	Trust-based. The token has no network identity, so compliance cannot be verified by IP.
Block dedicated training crawlers	Match GPTBot, CCBot, Meta-ExternalAgent and friends against their verified ranges at the edge	CCBot publishes no ranges; fall back to ASN and hosting attribution.
Keep AI search referrals	Allow OAI-SearchBot, Claude-SearchBot, PerplexityBot by their range files	Retrieval crawlers change ranges more often than search engines do.
Use Cloudflare's category toggles	Audit AI Crawl Control before September 15	AI Training: Block currently 403s Googlebot and Bingbot. Do not enable it on a site that lives on organic search.

The uncomfortable row is the first one. The only tool that blocks Google's training use without collateral damage to search is the one tool you cannot verify, while everything enforceable hits both purposes at once. That asymmetry is exactly the point Cloudflare is pressing on.

Where the IP space goes next

Our bet is that the September 15 standoff gets resolved the way these things usually do: in the address space. OpenAI, Anthropic, and Perplexity already run purpose-specific crawlers on purpose-specific published prefixes, which means a publisher's policy compiles down to a handful of range files. If Google splits a training crawler out of Googlebot, the interesting part will not be the announcement but the new JSON file, because from that day "search or training?" becomes a question your firewall can answer instead of your lawyer.

And if Google does not split, every Cloudflare-fronted site that blocks training keeps serving 403s to the crawler that built the modern web's front door. Either way, crawler IP ranges, who publishes them, what they contain, and which company and network each address actually belongs to, become the layer where this fight settles.

Sorting a crawl log into verified search traffic, training bots, and impostors is exactly the kind of attribution Ipregistry's data was built for: one lookup returns an IP's autonomous system along with its operating company and hosting classification, so every claimed crawler maps to the network that actually sent it. You can try it with 100,000 free lookups to get started.