Cloudflare will block mixed-use AI crawlers from ad-supported sites by default
Cloudflare has announced a new default policy starting September 2026 that will block "mixed-use" crawlers from ad-supported sites. The initiative aims to protect publisher intellectual property from unauthorized AI training scraping while introducing "Pay-Per-Use" monetization partnerships with Ceramic.ai and You.com.
Key Takeaways
- Starting September 15, 2026, Cloudflare will block 'mixed-use' crawlers by default on pages featuring advertisements.
- The policy applies to all new customers, new sites for existing customers, and all existing users on the free tier.
- A new 'Pay Per Use' model replaces 'Pay Per Crawl,' allowing publishers to charge AI companies specifically when content appears in results.
- Initial partners Ceramic.ai and You.com will facilitate payments to publishers when content is used in AI search results or accessed by agents.
- Cloudflare data indicates over 50% of AI crawler traffic is currently wasted on re-fetching web pages that have not changed.
Why It Matters
This move forces a structural decoupling of search discovery from AI model training, a friction point that has historically benefited dominant engines like Google. By making the block a default setting for ad-supported sites, Cloudflare is protecting the referral-based economics of the open web while creating a standardized infrastructure for micro-monetization. For the streaming and media ecosystem, this professionalizes the 'agentic internet,' allowing smaller publishers to enforce intellectual property rights without individual legal negotiations. Watch for whether Google responds by further isolating its Googlebot functions to prevent mass exclusion from its search index.
Additional Context
The shift toward automated traffic management follows a June 2026 report from the World Federation of Advertisers, which found that 92% of major brands are currently overhauling their discoverability strategies for AI-mediated environments. This urgency is underscored by Cloudflare's own network data, which revealed that bots surpassed human traffic for the first time in early June 2026, accounting for 57.4% of all web requests. Industry pressure has been mounting for such a change; per Press Gazette, for instance, approximately 79% of the top 100 news sites in the US and UK had already manually blocked AI training bots via robots.txt by April 2026. Cloudflare’s transition from 'Pay Per Crawl' to 'Pay Per Use' also mirrors a broader industry pivot toward revenue-sharing. In July 2024, AI search startup Perplexity launched a Publishers Program to share ad revenue with partners like Time and Fortune following plagiarism controversies. Similarly, Apple introduced 'Applebot-Extended' in June 2024, allowing publishers to opt out of foundation model training while remaining discoverable in Safari and Siri. By providing a technical layer for these transactions, Cloudflare aims to stabilize a marketplace that IAB Tech Lab CEO Anthony Katsur described in January 2026 as unsustainable due to 'the age of crawling' where value was extracted without compensation.
Read full article at techcrunch.com
Get this in your inbox → Subscribe
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source