Half of enterprise AI deployments miss latency targets at peak load

The failure to meet enterprise AI latency targets signals a shift from model-centric to infrastructure-centric challenges in streaming and B2B applications. As streaming platforms integrate agentic AI for real-time personalization and metadata tagging, the reliance on centralized data centers creates a 'latency wall' that degrades user experience. This bottleneck suggests that simply scaling compute power is insufficient; instead, developers must optimize CPU-bound tasks and reduce network hops to maintain sub-500ms performance. The industry must now pivot toward edge-based inference and decentralized processing to support complex, multi-step AI agents. Watch for a rise in edge-native AI frameworks designed to minimize WAN transport times in production environments.

Akamai's findings on enterprise AI latency targets align with broader industry benchmarking efforts that are increasingly focused on inference performance under real-world conditions. In April 2025, MLCommons released MLPerf Inference v5.0 results showing that submissions to the Llama 2 70B benchmark increased 2.5x over the prior year, making it the highest-submission-rate test in the suite and surpassing ResNet50. The benchmark suite now covers both datacenter and edge systems, reflecting the industry's shift toward measuring inference latency across diverse deployment topologies rather than relying solely on training throughput metrics.

The latency thresholds that MLPerf has adopted for its interactive scenarios directly mirror the sub-500ms targets Akamai's report identifies as critical for enterprise deployments. MLPerf Inference v5.0 introduced a Llama 2 70B Interactive benchmark with a 99th percentile TTFT limit of 450ms and a TPOT threshold of 40ms, derived from analysis of production platforms including ChatGPT and Perplexity AI. The benchmark designers found that a 50th percentile token generation rate of 20 to 50 tokens per second is critical for user experience, and they set stricter 99th percentile targets to ensure consistent responsiveness during high-load scenarios. By September 2025, MLPerf Inference v5.1 expanded the interactive scenario specifically to address agentic AI applications, acknowledging that multi-step agent workflows impose distinct latency constraints beyond single-turn inference.

The agentic frameworks highlighted in Akamai's report, including Pydantic AI and LangChain, are themselves evolving to address production reliability challenges that compound latency issues. Pydantic AI partnered with Prefect to build an integration enabling automatic failure recovery and durable execution for agent workflows, where cached task results allow agents to resume from the point of failure rather than restarting entirely. The integration records call type and duration for each task, providing observability into where latency accumulates across multi-step agent pipelines. This approach directly addresses the network-hop and CPU-side processing delays that Akamai's report identifies as the primary causes of missed latency targets, since each failed and retried step in an agentic workflow adds cumulative delay that GPU scaling alone cannot resolve. Recent data shows that enterprise AI failures persist for nearly half of organizations, highlighting the need for such robust infrastructure.