Splitting Vision-Language Model Inference Cuts GPU Costs 40%

Splitting VLM inference at the modality boundary rather than at the prefill/decode stage makes heterogeneous GPU deployment practical over commodity PCIe, avoiding the NVLink or InfiniBand requirement that has confined prior disaggregation systems to homogeneous datacenter hardware. Multiple serving frameworks are converging on this architecture — SGLang shipped EPD disaggregation in January 2026, vLLM-Omni extends the approach to any-to-any multimodal models, and IBM Research demonstrated llm-d running 3–5× faster across mixed-vendor GPU clusters in June 2026. The signal to track: whether a cross-vendor KV cache transfer library materializes — IBM identified this as the prerequisite for routing prefill and decode to different hardware vendors, and it does not yet exist.

The modality-boundary split described in the SudoAll explainer draws directly from research by Donglin Yu et al. (arXiv:2603.12707), who built HeteroServe — a phase-aware runtime that maps vision encoding to consumer GPUs (RTX 4090, ~$3k) and language generation to datacenter GPUs (A100, ~$16k). Their system achieved 54% higher throughput than vLLM v0.3.0 on identical 4×A100 hardware, and a $38k heterogeneous cluster outperformed a $64k homogeneous baseline by 37% in cost-efficiency, per their evaluation on LLaVA-1.5-7B and Qwen2.5-VL.

Multiple serving frameworks have moved toward encoder disaggregation in 2026. SGLang shipped Encoder-Prefill-Decode (EPD) disaggregation in January 2026 (per LMSYS Org blog), reporting 6–8× lower time-to-first-token for image-heavy workloads at 1 QPS by scaling vision encoders horizontally instead of increasing tensor parallelism. vLLM-Omni (arXiv:2602.02204), a 2026 extension of vLLM, generalizes the approach to any-to-any multimodal models handling text, images, video, and audio — reducing job completion time by up to 91.4% for Qwen3-Omni compared to baseline serving. EPD-Serve (arXiv:2601.11590), built for Huawei Ascend NPUs, improved throughput by 57–69% over PD-disaggregated deployment while meeting sub-50ms time-per-output-token SLOs.

IBM Research, working with Red Hat and NxtGen Cloud, demonstrated in June 2026 that the llm-d Kubernetes-native control plane ran IBM Granite and Sarvam AI models on mixed-vendor GPU clusters 3–5× faster than traditional Kubernetes round-robin scheduling, serving twice as many users. IBM noted that the next step is routing compute-heavy prefill to one vendor's GPUs and memory-intensive decode to another — but this requires a KV cache transfer library compatible across GPU backends, which does not yet exist. Practical deployment guidance has also emerged: DigitalOcean published a tutorial in June 2026 mapping the heterogeneous split onto GPU Droplets, with L40S or RTX 6000 Ada for the encoder pool and H100 or H200 for the decoder pool, connected over 25 Gbps private networking.