Since August 2, 2026, an AI company that ignores your robots.txt risks a fine of up to €15 million or 3 percent of worldwide annual turnover, whichever is higher. But there is a catch nobody in the legal commentary dwells on: a robots.txt rule is addressed to a name, and the crawler picks its own name. A user agent is just a request header the client writes for itself. So both sides of any future enforcement case, the publisher claiming its opt-out was ignored and the provider claiming its crawlers behaved, end up arguing about the same thing: which requests in the server log actually belong to which operator. That is not a legal question. It is an IP attribution question.
The legal machinery behind that fine is new. On August 2, the European Commission's AI Office gained the power to actually enforce the AI Act's rules for general-purpose AI providers: request information, demand access to models, and fine violators. The obligations themselves have applied since August 2, 2025. For a year they sat on paper with no one entitled to police them. That year is over.
The obligation that matters to anyone running a web server is Article 53(1)(c) of the AI Act: every provider of a general-purpose model must maintain a copyright policy that identifies and complies with rights reservations expressed under Article 4(3) of the 2019 DSM Directive, including through machine-readable means. In practice, the machine-readable means everyone has converged on is robots.txt. A plain-text convention from 1994, standardized as RFC 9309 only in 2022, now carries regulatory weight backed by turnover-scaled fines.
What changed on August 2, and for whom
The Commission enforces the general-purpose AI chapter centrally through its AI Office; national authorities handle the rest of the Act. Models placed on the EU market on or after August 2, 2025 are in scope now. Models placed earlier get until August 2, 2027. The same date also switched on the Act's transparency rules: chatbots must disclose that they are machines, and synthetic media needs machine-readable marking.
Between those dates sits the General-Purpose AI Code of Practice, published July 10, 2025, the Commission's template for demonstrating compliance. Its copyright chapter is explicit in Measure 1.3: signatories commit to employ crawlers that read and follow robots.txt, and to identify and comply with other machine-readable rights reservations when crawling. OpenAI, Anthropic, Google, Microsoft, and Amazon signed. Meta publicly refused. Signing is voluntary, but Article 53 is not, and the Code is the yardstick the AI Office says it will measure against.
A user agent is a claim, not an identity
Every robots.txt opt-out in existence is addressed to a name: User-agent: GPTBot, User-agent: ClaudeBot, User-agent: Google-Extended. Nothing in HTTP binds that name to anything. This is one shell command:
curl -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.0; +https://openai.com/gptbot" https://example.com/Now OpenAI appears in example.com's logs, courtesy of anyone at all. The reverse move is the damaging one: a crawler that wants content from sites that opted out can simply stop announcing itself. On August 4, 2025, Cloudflare published evidence that Perplexity was doing exactly this: when its declared PerplexityBot hit a robots.txt block, requests resumed from undeclared IPs outside Perplexity's published ranges, wearing a generic Chrome-on-macOS user agent and rotating across autonomous systems, at a rate Cloudflare put at 3 to 6 million requests per day across tens of thousands of domains. Perplexity denied the characterization. Cloudflare removed it from its verified bots program anyway.
Whichever account you believe, the episode maps the failure mode precisely. A robots.txt keyed to user agent tokens is routed around by editing one header, and a log line that says "GPTBot" proves nothing in either direction. Under an enforcement regime with 3 percent fines, that ambiguity stops being an annoyance and becomes the whole case. A publisher complaining to the AI Office needs to show the offending requests actually came from the provider. A provider defending itself needs to show its fleet honored every reservation, and that the traffic in the complainant's logs was an impostor.
Verification runs on IPs
There are three checks, and they are all network-layer. First, published ranges: the major operators now ship machine-readable lists of their crawler egress prefixes, one file per bot in most cases. Second, forward-confirmed reverse DNS: resolve the PTR record of the connecting IP, check the hostname falls under the operator's designated domain, then resolve that hostname forward and confirm it returns the original IP. Google has recommended this procedure for Googlebot for two decades: 66.249.66.1 reverses to crawl-66-249-66-1.googlebot.com, which resolves forward to 66.249.66.1. Both lookups must agree, because a PTR record alone is whatever the IP holder wants it to say. Third, ASN and hosting attribution: when an operator publishes nothing, the announcing autonomous system and the nature of the network (datacenter, ISP, mobile) are what remain.
The direction of travel here is worth noticing. Anthropic's documentation used to state that it did not publish IP ranges because its crawlers used cloud provider IPs. In May 2026 that changed: claude.com/crawling/bots.json now lists the egress prefixes for ClaudeBot, Claude-User, and Claude-SearchBot (we pulled the file while writing this; its creation timestamp is May 1, 2026, and the first entry is 216.73.216.0/22). Google went the other way and consolidated: fetching its old googlebot.json now redirects to a combined common-crawlers.json. Publishing verifiable ranges has quietly become part of what operating a legitimate crawler means.
| Operator | Crawler tokens | How to verify |
|---|---|---|
| Anthropic | ClaudeBot, Claude-User, Claude-SearchBot | bots.json, published since May 2026 |
| Apple | Applebot, Applebot-Extended | rDNS under applebot.apple.com, per Apple's documentation |
| Googlebot, Google-Extended | Published JSON ranges plus rDNS under googlebot.com. Google-Extended is a robots.txt token only; it rides on Googlebot's infrastructure. | |
| Meta | Meta-ExternalAgent, facebookexternalhit | Ranges derived from AS32934 route objects, queried from RADb |
| Microsoft | Bingbot | bingbot.json plus rDNS under search.msn.com |
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User | Per-bot JSON: gptbot.json, searchbot.json, chatgpt-user.json |
| Perplexity | PerplexityBot, Perplexity-User | Per-bot JSON: perplexitybot.json and perplexity-user.json |
| Common Crawl | CCBot | Nothing published; crawls from AWS address space, so ASN and hosting detection is the only check |
Notice the two "Extended" entries. Google-Extended and Applebot-Extended, the tokens that specifically govern AI training use, have no network identity of their own. You cannot verify them directly; you verify the parent crawler and trust the operator to apply the right policy per token. The training opt-out, the thing the AI Act just made legally meaningful, is the least independently verifiable signal in the whole stack.
The traffic being regulated, by the numbers
Cloudflare Radar classifies AI crawler requests by purpose, and the May 2026 breakdown shows why the training opt-out is the fight that matters: 51.8 percent of AI crawler traffic was training-purpose crawling, another 35.7 percent came from mixed-use crawlers that blend training and retrieval, and only 9.3 percent was search. The asymmetry behind those numbers is stark. In first-quarter 2026 Radar measurements, ClaudeBot crawled roughly 24,000 pages for every visitor it referred back; Google's traditional ratio is about 5 to 1.
The infrastructure is moving to match the law. On July 1, 2026, Cloudflare gave every customer separate switches for Search, Agent, and Training traffic, and announced that from September 15, 2026, training and agent bots will be blocked by default on ad-monetized pages. The purpose split is why operators run separate user agents on separate published ranges: GPTBot is not ChatGPT-User, and a policy that treats a live fetch on behalf of a paying user the same as bulk training ingestion will make someone unhappy in both directions.
What a compliant crawl log looks like
If you operate websites, the practical move is to stop treating verification as a blocking decision and start treating it as evidence collection. For each request claiming a crawler identity, record the claim and the verdict together, resolved against the operator's ranges at the time of the request (the range files change, which is why both OpenAI's and Anthropic's carry a creation timestamp). A useful log entry looks like this:
{
"time": "2026-08-05T14:12:09Z",
"ip": "216.73.216.147",
"claimed_agent": "ClaudeBot/1.0",
"range_match": "216.73.216.0/22 (claude.com/crawling/bots.json)",
"asn": "AS16509 (Amazon.com, Inc.)",
"network_type": "hosting",
"operator": "Anthropic",
"verdict": "verified"
}Note the ASN line: that prefix is announced by Amazon, not Anthropic, because ClaudeBot rides on rented cloud capacity (we checked the routing tables while writing this). The BGP view alone would tell you "AWS" and stop there. Pinning the request to Anthropic takes the published range file, and when no range file exists, hosting and company attribution is the best remaining signal.
Three categories fall out of a log like that, each with its own policy. Verified crawlers get whatever your robots.txt and business decisions say, allow search, block or charge training, as you prefer. Unverifiable claimants, a "Googlebot" arriving from a residential ISP in the wrong country, are abuse and can be dropped without ceremony (we wrote about how residential proxy networks supply exactly those IPs). The interesting middle category is verified training crawlers hitting paths your robots.txt reserved: since August 2, that log entry is documentation of a possible Article 53 violation, timestamped and attributable to a named operator.
Providers face the mirror image. The AI Office can now ask a model provider to show its crawl infrastructure honored reservations, and the credible answer is the same artifact: egress logs that reconcile against published ranges, so that any request outside them is provably not yours. The Perplexity episode showed what the absence of that discipline looks like from the outside.
Attribution at this scale is the part Ipregistry is built for: a single lookup returns the IP's autonomous system, its operating company and hosting classification, so you can map every claimed crawler in your logs to the network that actually sent it. You can try it with 100,000 free lookups to get started.
Keep users in the loop Ship release notes that get read. Try Noticeable