Akamai AI bot traffic controls now split into three granular categories
Akamai has updated its Bot Directory to categorize AI traffic into three distinct types: training crawlers, search crawlers, and fetchers/agents. This update allows customers to apply more granular security policies to manage how AI bots interact with their web content.
Key Takeaways
- New categories separate high-volume training crawlers like GPTBot and ClaudeBot from search-focused indexers like PerplexityBot.
- AI fetchers and agents, including ChatGPT-User and GoogleAgent-Mariner, are now identified as ad hoc requests triggered by specific user actions.
- The Akamai Bot Directory update enables businesses to block scrapers while remaining visible in AI-powered search citations.
- Granular controls are designed to support agentic commerce, where AI assistants navigate sites to complete transactions for consumers.
Why It Matters
The immediate implication is that content owners can now prevent LLM training scrapers from exhausting server resources without disappearing from AI-driven search results. In the broader streaming ecosystem, this provides a technical framework for defending proprietary video metadata and transcripts from unauthorized model training while maintaining brand discoverability. As AI agents begin to handle autonomous tasks like subscription management or content discovery, distinguishing between a helpful user agent and a background scraper becomes critical for security and revenue attribution. Watch for whether competitors like Cloudflare or Fastly introduce similar three-tier classification systems to match this level of visibility.
Additional Context
Akamai's three-tier AI bot categorization arrives amid intensifying competition among CDN and edge-security vendors seeking to monetize AI traffic governance. In June 2026, Cloudflare reported that AI crawler requests had grown to represent a significant share of total bot traffic on its network, prompting the company to expand its own bot management toolkit with AI-specific detection features. The broader market pressure is clear: content publishers and streaming platforms are demanding finer-grained controls as AI agents proliferate across the web, and Akamai's Bot Directory update positions it as the first major CDN to formally separate training, search, and fetcher traffic into distinct policy categories.
The business implications extend beyond Akamai's product roadmap. Nokia and AWS announced at DTW Ignite in June 2026 that they are building a unified data and cloud control layer for autonomous network operations, demonstrating how agentic AI frameworks are being embedded across infrastructure stacks that CDNs must interoperate with. For streaming platforms specifically, the distinction between a real-time fetcher agent (which may be performing legitimate content discovery or subscription management) and a background training scraper has direct revenue implications. Akamai's AI Brand Presence product, which tracks how content appears in AI-generated responses, signals a strategic bet that publishers will pay for visibility into how their intellectual property flows through LLM pipelines. This mirrors a broader trend where CDN vendors are repositioning from pure delivery infrastructure toward intelligence and governance layers.
On the technical side, the challenge of classifying AI agents at scale is non-trivial. Ericsson launched its AI in RAN commercial software subscription on June 11, 2026, claiming up to 20% higher downlink throughput across more than 15 live deployments, illustrating how agentic systems are being deployed in production environments where classification accuracy directly impacts service quality. For Akamai's streaming customers, the three-tier system must handle edge cases such as AI agents that both fetch content for real-time user queries and cache data for future model training. The company's approach of categorizing by intent rather than by user-agent string alone represents a meaningful technical advance, though independent benchmarking of classification accuracy across diverse AI agent populations has not yet been published. Competitors including Fastly and Cloudflare will likely need to demonstrate comparable precision to remain competitive in enterprise streaming accounts.
Read full article at akamai.com
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source