Cloudflare will block mixed-use AI crawlers from ad-supported sites by default
Cloudflare has announced a new default policy to block mixed-use web crawlers from ad-supported sites starting September 2026, aiming to force a distinction between search and AI training traffic. Concurrently, the company is introducing a 'Pay Per Use' monetization model, enabling publishers to charge AI companies when content generates value through specific AI search and retrieval tools.
Key Takeaways
- Beginning September 2026, Cloudflare will default to blocking crawlers that blend search indexing with AI training on all ad-supported sites.
- The new "Pay Per Use" monetization model allows publishers to charge AI companies when content is cited or used, rather than just for the initial crawl.
- Initial launch partners including Ceramic.ai and You.com will provide publishers with queries, citations, and rankings data in exchange for content access.
- Cloudflare data indicates that over 50% of current AI crawler traffic is wasted on re-fetching unchanged web pages.
Why It Matters
Cloudflare is leveraging its position as a gatekeeper for 20% of the web to force a structural shift in how AI companies access data. By defaulting to a block on mixed-use bots, the company effectively strips away the "implicit consent" model that allowed AI training to piggyback on search indexing. For the streaming and media ecosystem, this establishes a technical framework for micro-monetization that scales beyond the high-level licensing deals currently dominated by major studios and publishers. Watch for whether Google separates its primary Googlebot into distinct search and AI entities to maintain its 2x information advantage without triggering these default blocks.
Additional Context
The September 2026 deadline follows a year of preliminary technical maneuvers by Cloudflare to rebalance the web's economic model. In July 2025, per Nieman Lab, the company launched its "Pay Per Crawl" marketplace and shifted to an opt-in model for AI scraping on new domains. Major publishers including Condé Nast, Gannett, and Time endorsed the move, seeking to recoup value lost to AI answer engines. Cloudflare further expanded this infrastructure in January 2026 by acquiring Human Native Ltd., a startup focused on licensed-content tooling for rights holders, according to reports from SiliconANGLE. This shift coincides with a critical inflection point in global internet traffic. Per Cloudflare Radar data from June 2026, automated bot and AI agent requests reached 57.5% of all HTML web traffic, officially surpassing human activity earlier than industry forecasts. Competing analysis from Fastly in June 2026 noted that AI-driven requests grew at 6.5 times the pace of human traffic during the first half of the year. This volume surge has significant infrastructure implications; HUMAN Security’s 2026 report found that agentic AI traffic grew over 7,800% year-over-year, often visiting thousands of pages to complete a single human errand. Cloudflare’s specific critique of the "world's largest search engine" and its 2x information lead highlights the competitive friction between open infrastructure providers and integrated tech giants. While Google offers the "Google Extended" bot to let site owners opt out of Gemini training, critics argue that the core Googlebot remains a mixed-use crawler. Per Search Engine Land in June 2026, zero-click searches—where users receive AI-generated answers directly on the search results page—reached 68%, increasing the pressure on publishers to secure direct compensation for the data powering those summaries.
Read full article at techcrunch.com
Get this in your inbox → Subscribe
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source