Spheron launches three-pool disaggregated architecture for multimodal vLLM-Omni serving
Spheron, a GPU cloud provider, details a three-stage disaggregated architecture for vLLM-Omni, a multimodal model serving large language models with separate encoder, prefill, and decode GPU pools. This architecture significantly boosts throughput for image and audio-heavy workloads, especially at scale, by optimizing GPU types for each stage's bottleneck. The article includes a full deployment walkthrough on Spheron GPU Cloud with recommendations for GPU sizing and cost optimization.
Key Takeaways
- Three-pool topology uses specialized GPUs: L40S/A100 for encoding, H100/B200 for prefill, and H200 for memory-intensive decoding.
- Eliminates head-of-line blocking where image/audio encoding typically consumes prefill pool TFLOPS, causing stalls.
- NIXL transport layer maintains inter-pool latency between 4-16ms on RDMA, with break-even gains occurring above 64 concurrent requests.
- Deployment walkthrough recommends spot instances for the retriable encoder pool to reduce costs while keeping prefill and decode on-demand.
Why It Matters
Multimodal models have broken standard two-stage (prefill-decode) disaggregation because visual and audio encoders now create a third primary bottleneck. This transition to a three-pool model allows operators to right-size hardware for specific compute profiles, such as using high-bandwidth HBM3e for decoding while offloading encoding to cheaper PCIe cards. For the streaming industry, this represents a critical shift toward architecture that can handle mass-scale, any-to-any inference without the linear cost increases of homogeneous clusters. Watch for rival inference engines like SGLang to adopt similar three-stage connectors as multimodal request volumes cross the 64-request concurrency threshold.
Additional Context
The push toward three-stage disaggregation reflects a broader industry shift as multimodal 'omni' models like Qwen3-Omni and Cosmos3 enter production. Per vLLM project updates in June 2026, the ecosystem has moved to support 'any-to-any' pipelines where text, image, and video are processed in a single inference pass. This evolution has made traditional serving engines, which were optimized primarily for text-based autoregression, insufficient for high-concurrency visual workloads. Recent benchmarks from Nvidia and the vLLM-Omni team (May 2026) indicate that unmanaged encoder contention can degrade job completion times by over 90% in large-scale deployments. Simultaneously, the transport layer for these distributed architectures has matured. Tools like Mooncake and NVIDIA's NIXL are now standard for moving KV cache and feature tensors across heterogeneous GPU clusters. Per vLLM announcements in May 2026, Mooncake has been integrated as a distributed KV cache store specifically to manage the large memory footprints generated by long-context agentic and multimodal workflows. This infrastructure allows clusters to utilize under-exploited CPU and SSD resources for 'cold' cache storage while maintaining 'hot' data on GPUs, a technique that has reportedly boosted effective request capacity by up to 498% in tests on Kimi-class models. On the hardware side, the availability of specialized silicon like the H200 (4.8 TB/s HBM3e) and the B200 has pressured providers to offer more flexible procurement models. Spheron’s move to aggregate capacity from five separate providers in June 2026 aligns with a market-wide trend toward heterogeneous cloud marketplaces. According to industry analysis from April 2026, the 'buy-vs-rent' decision for H100 clusters has flipped, with competitive cloud pricing now beating the total cost of ownership for on-premise hardware even at 100% utilization, further driving the adoption of complex, multi-pool serving architectures.
Read full article at spheron.network