On October 13, Fastly's security research team published a one-day sample from its Next-Gen WAF logs that is worth reading twice. During the 24 hours of September 23, 2026 (UTC), about 6.3 million requests arrived claiming to be an AI crawler or fetcher while probing for files that should never be reachable over HTTP. They came from nearly 10,200 distinct IP addresses. Roughly 1,600 of those addresses sent 94 percent of the traffic, nearly all of it from a single public cloud provider that Fastly does not name, and together they reached more than 100,000 sites.
The busiest path was /.env, probed by nearly 1,800 addresses. Three days later, on October 16, Cloudflare announced that even bots it has verified will be graded on conduct, with a trust band fed by robots.txt consistency and by how a bot reacts to an HTTP 429. Read the two posts together and the message from both ends of the industry is the same: the User-Agent header is a costume, and the only facts a server can check are facts about the connection.
Why the costume is an AI crawler and the target is .env
A .env file is where twelve-factor applications keep their secrets: database passwords, cloud keys, payment and API tokens, in plain text, one per line. It belongs outside the web root. A careless deploy, a container image built from the whole project directory, or a Vite dev server left exposed (Fastly saw the /@fs/.env route in the sample) is enough to put it one GET away. The campaign used a shared wordlist of 96 file names, which accounted for nearly all of its requests. Counting the long tail too, the day's probes reached about 3,000 distinct private files across more than 46,000 path variants, once every prefix, locale and traversal form such as /../../../../.env is counted.
The disguise is a rational choice. Many sites have spent three years writing allow rules for GPTBot, ClaudeBot, PerplexityBot and their search and user-triggered siblings, because the alternative is to vanish from AI answers. A WAF that waves through anything named like an AI crawler has a hole shaped like a User-Agent string, and the header costs nothing to set. Fastly's main campaign cycled through exactly a dozen identities and used around 60 distinct User-Agent strings, spreading its requests roughly evenly. Across the day, the ten most impersonated names, spread over OpenAI, Anthropic, Perplexity, DuckDuckGo and Common Crawl, each drew about 500,000 requests.
GreyNoise had described the same pattern on August 28: thirteen crawler names from eight companies, six of them arriving from 824 addresses spread across 795 separate /24 networks in nearly identical volume, hunting /.env, its backup and per-environment variants, and /.aws/credentials. One detail from that report stays with us. Anthropic's real crawler requested /robots.txt in 12 percent of its traffic. None of the six forged names in the cluster asked for it once.
Three ways to verify a crawler, in order of cost
Every serious crawler operator now gives you at least one way to check the claim against the source address. None of them involve the User-Agent string.
The first is a published prefix list. OpenAI serves one JSON file per identity: gptbot.json for training, searchbot.json for OAI-SearchBot and chatgpt-user.json for user-triggered fetches. Anthropic's bots.json listed 38 IPv4 prefixes when we pulled it on October 18: one /22, five /28s and 32 single hosts, with a creation time of October 7, 2026. Perplexity publishes perplexitybot.json and perplexity-user.json. DuckDuckGo's duckduckbot.json carried 486 prefixes for DuckDuckBot, last generated on September 1. Common Crawl publishes ccbot.json with IPv4 and IPv6 ranges. A match is a yes. A miss is a strong no, provided you refetch the files on a schedule, because they change. Anthropic's had been regenerated six days before Fastly's post.
The second is forward-confirmed reverse DNS, the check Google has documented since 2006, Microsoft has documented for Bingbot since 2012, and the one Common Crawl asks you to use. Resolve the PTR of the source address, confirm the hostname sits in the operator's domain, then resolve that hostname forward and insist on getting the original address back. Common Crawl's own example:
dig -x 18.97.14.84 +short
18-97-14-84.crawl.commoncrawl.org.
dig 18-97-14-84.crawl.commoncrawl.org +short
18.97.14.84A PTR alone proves nothing, since whoever controls the reverse zone for a block can write any name into it. The forward step is what turns it into an attestation by the domain owner. OpenAI, Anthropic and Perplexity document only their range files, not reverse DNS, so for those three the list is the check. (We wrote up how reverse DNS works and where it lies earlier this year if you want the long version.)
The third is a signature. Web Bot Auth, the scheme Cloudflare engineers brought to the IETF, now a working group document as draft-ietf-webbotauth-httpsig-protocol, has the bot sign each request using HTTP Message Signatures (RFC 9421), in practice with an Ed25519 key. The request carries Signature, Signature-Input and a Signature-Agent header that points at the operator's key directory at /.well-known/http-message-signatures-directory, and the signature carries an expiry, about a minute in Cloudflare's guidance, to limit replay. It is the only one of the three that survives a crawler moving to new address space, and the only one an impostor cannot satisfy by renting the right prefix. It is also the least deployed, which is why the other two still carry the load.
The cheap filter comes before any list
Range files and reverse DNS answer a narrow question: is this address the operator's? There is a broader question that a single lookup on the source address answers for every request, whether or not it claims to be a crawler: whose address is this, and what kind of network is it?
Fastly's main campaign was about 1,600 addresses on one public cloud. Group the 6.3 million requests by the company operating each source block and the campaign collapses into one row, no matter how many identities or User-Agent strings it rotates through. That is what hosting and company attribution gives you. The request comes from a tenant of a cloud provider, and tenants do not appear in OpenAI's, Anthropic's or Perplexity's range files. To be fair, the real crawlers run on cloud infrastructure too: Anthropic's list includes blocks inside Microsoft, Google and Amazon space, and Common Crawl's example address sits in AWS. The hosting flag on its own cannot tell a real ClaudeBot from a fake one. What it tells you is that an unverified claim from datacenter space is a bulk tool, while an unverified claim from a residential ISP is a different animal, often a home connection resold through a residential proxy network, and deserves a different rule.
In Ipregistry's response the answer lives in three fields. connection.type and company.type classify the announcing network and the organization using the block as business, education, government, hosting or isp, and security.is_cloud_provider is the blunt version of the same question. For an address rented from a public cloud, the expected answer is hosting on both type fields and true on the flag, with the cloud provider's name rather than an AI company's. A request in that state should never reach an allow rule written for a crawler.
Verified is becoming a grade, not a badge
Cloudflare's October 16 post starts from the opposite end. Its Verified bot program already requires operators to identify themselves accurately and securely, which Web Bot Auth and the BotBase directory cover. The new part is continuous. Cloudflare now observes whether a verified bot's traffic matches a site's robots.txt (RFC 9309) over time, and what the bot does after a 429 Too Many Requests (RFC 6585). Consistently slowing down or pausing raises trust; keeping the same pace lowers it. The output is a trust band, which the post describes as "a range showing how consistently a bot respects site preferences," explicitly not a permanent judgment. Cloudflare says it already runs the system internally, will publish the scores on Radar, and will let customers pick the threshold. No band boundaries, observation windows or violation counts were disclosed.
The precedent is Perplexity, de-listed from Verified bots in 2025 after fetching opted-out resources and working around blocks. That was a binary decision taken after the fact. The bands are meant to catch the same conduct earlier and on a gradient.
For a site operator the useful consequence is that a 429 stops being a polite suggestion and becomes an instrument. Send one to a bot that is pulling too fast, log what happens next, and you have your own behavioral signal, whether or not you sit behind Cloudflare. Fastly's impostors would fail that test in seconds. A wordlist scanner has no backoff logic because it has no intention of being invited back.
A rule you can copy today
Fastly's post ends with a single request rule for its own WAF, but the shape owes nothing to Fastly and works in front of any origin: block when the request claims a known AI crawler, the source address fails verification for that crawler, and the path is one no crawler has any business fetching. In pseudo-rule form:
claimed_bot in (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User,
Claude-SearchBot, PerplexityBot, Perplexity-User, DuckDuckBot, CCBot)
and verified_bot != true
and path matches (/.env, /.env.*, /*/.env, /@fs/*, /.git/config, /.aws/credentials)
=> block, log, alertNone of the three inputs is a vendor feature. The first is a pattern match on the User-Agent header, the third a path match, and the second is the lookup described above: does the source address fall inside the range file the claimed vendor publishes, plus forward-confirmed reverse DNS where the vendor documents it. Managed bot products expose that result as a ready-made rule field. On nginx, HAProxy, Caddy or in application middleware you compute it yourself from the published files, cache the answer per address, and refresh the files on a schedule, since Anthropic's changed six days before Fastly's post went out.
Key the first clause on the claimed name, not on the full User-Agent string. Fastly counted around 60 variants in one campaign, and an exact-string match loses every time the operator bumps a version number. Key the second on a verification result computed from the source address, which is why rotating addresses buys the attacker nothing: the new ones are not the vendor's either. Keep the third clause narrow. An unverified crawler fetching your home page may be someone's lab project. An unverified crawler fetching /.aws/credentials is not.
Two things the rule does not fix. A secret reachable from a web path is compromised the moment any scanner finds it, so the rotation and the move out of the web root come before the firewall work. And a verified crawler is still only verified: whether it reads robots.txt and backs off is a separate question, the one Cloudflare just started grading.
Telling a vendor's crawler from a cloud tenant wearing its name is the attribution Ipregistry's data is built for: one lookup returns the company and connection type, the hosting flag and the proxy and abuse signals for the source address, so a rule can check what the address is before it trusts what the request says. You can try it with 20,000 free lookups to get started.
Keep users in the loop Ship release notes that get read. Try Noticeable