Cloudflare will block mixed-use AI crawlers from ad-supported sites by default

This move forces a structural decoupling of search discovery from AI model training, a friction point that has historically benefited dominant engines like Google. By making the block a default setting for ad-supported sites, Cloudflare is protecting the referral-based economics of the open web while creating a standardized infrastructure for micro-monetization. For the streaming and media ecosystem, this professionalizes the 'agentic internet,' allowing smaller publishers to enforce intellectual property rights without individual legal negotiations. Watch for whether Google responds by further isolating its Googlebot functions to prevent mass exclusion from its search index.

The shift toward automated traffic management follows a June 2026 report from the World Federation of Advertisers, which found that 92% of major brands are currently overhauling their discoverability strategies for AI-mediated environments. This urgency is underscored by Cloudflare's own network data, which revealed that bots surpassed human traffic for the first time in early June 2026, accounting for 57.4% of all web requests. Industry pressure has been mounting for such a change; per Press Gazette, for instance, approximately 79% of the top 100 news sites in the US and UK had already manually blocked AI training bots via robots.txt by April 2026.

Cloudflare’s transition from 'Pay Per Crawl' to 'Pay Per Use' also mirrors a broader industry pivot toward revenue-sharing. In July 2024, AI search startup Perplexity launched a Publishers Program to share ad revenue with partners like Time and Fortune following plagiarism controversies. Similarly, Apple introduced 'Applebot-Extended' in June 2024, allowing publishers to opt out of foundation model training while remaining discoverable in Safari and Siri. By providing a technical layer for these transactions, Cloudflare aims to stabilize a marketplace that IAB Tech Lab CEO Anthony Katsur described in January 2026 as unsustainable due to 'the age of crawling' where value was extracted without compensation.