Bot traffic hits machine majority as AI crawlers strain cloud budgets
Bad bot traffic, accounting for 37% of global internet traffic in 2024, is increasingly impacting infrastructure costs and ad fraud for streaming and media operators. The report argues that organizations must shift toward treating bot management as a critical FinOps and content-protection control rather than isolated security incidents.
Key Takeaways
- AI training now accounts for 52% of all crawler requests, up from 22% in early 2025
- Advanced bot traffic targeted APIs in 44% of cases during 2024, complicating legacy edge defenses
- Australian advertisers lose approximately $5 billion annually to bot-driven online ad fraud
- Automation is growing 8x faster than human traffic, particularly in streaming, media, and retail sectors
- Cloud providers like AWS are introducing flat-rate CDN plans to mitigate surprise bot-driven overages
Why It Matters
The transition to a machine-majority web transforms bot mitigation from a security task into a structural operating requirement. For streaming and media operators, the proliferation of AI crawlers creates an economic asymmetry where content is extracted for training without driving referral traffic or revenue. This pressure is forcing a move toward sophisticated behavioral scoring and API-specific controls to protect the application logic directly rather than just the perimeter. Market players must now balance blocking malicious automation against maintaining visibility in AI-driven search. Watch for more publishers to follow Nine Entertainment’s July 2026 deal with Microsoft to replace opportunistic scraping with contractually guaranteed licensing and attribution.
Additional Context
The escalation of machine-driven traffic has triggered a major defensive pivot among infrastructure providers. Per Cloudflare, July 2024 marked the first time automated requests surpassed 57% of HTML web traffic globally. In response, Cloudflare announced new default settings effective September 15, 2026, that will automatically block 'mixed-use' crawlers—those that simultaneously index for search and train AI models—on ad-supported pages. This move specifically targets the lack of granularity in current bot labels, allowing publishers to allow search indexing while blocking the high-volume 'agent' and 'training' fetches that consume resources without returning audience value. Cloud economics are similarly adapting to this permanent state of automation. Per AWS announcements in March 2026, the provider expanded its CloudFront flat-rate pricing tiers to include built-in WAF and bot management for a single monthly fee. This model reflects a shift away from traditional pay-as-you-go meters, where common bots could generate thousands in surprise egress and inspection charges during a single spike. By bundling security logging and serverless edge compute into predictable monthly plans, hyperscalers are essentially offering a 'bot insurance' model to stabilize digital budgets. For media organizations, the technical defense is increasingly paired with commercial licensing. In July 2026, Nine Entertainment secured a pilot agreement with Microsoft allowing its mastheads, including include The Sydney Morning Herald and The Australian Financial Review, to be used for Microsoft Copilot outputs. The deal ensures click-through links and verified snippets are provided in exchange for access to full-text content. This structure suggests the industry is moving toward a tiered internet: high-value content will be locked behind gated API access for licensed partners, while the open web faces increasingly aggressive filtration of non-referral automation.
Read full article at techbusinessnews.com.au
Get this in your inbox → Subscribe
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source