EncodingTechnical Development

Kafka share groups scale throughput beyond partition limits

A technical analysis demonstrates that Apache Kafka's new Queues feature, specifically share groups, can significantly increase consumer throughput by mitigating head-of-line blocking when message processing involves delays, showing linear scaling up to at least an 8x increase. This scaling comes with the trade-off of losing ordering guarantees for messages. The benchmark results indicate that share groups can be used to scale processing beyond the number of partitions.

Key Takeaways

The benchmark used the same setup as the prior post, but added configurable processing delays to model IO.
At 200ms delay, each instance processed 5 records per second; at 20ms, 50 rec/s; at 5ms, about 196 rec/s; and at 1ms, about 925 rec/s.
Standard and share-group consumers matched at 4 instances because the test topic had 4 partitions.
Share groups were scaled to 8, 16, and 32 instances, and total processing rate increased linearly in each case.
The main downside is that multiple instances can consume from the same partition, so message ordering is no longer guaranteed.

Why It Matters

For workloads with head-of-line blocking, Kafka’s share groups offer a way to push consumer throughput past the partition count. In these tests, delays from 1ms to 200ms made the benefit visible, with linear scaling out to 32 instances and at least an 8x throughput increase. That matters most for pipelines that wait on external IO, but the ordering trade-off is real: if message sequence matters, share groups may not be acceptable. The next signal to watch is where the scaling curve stops flattening as instance counts rise beyond 32.

Read full article at streamingdata.tech

Enjoy our coverage?

Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.

Add as preferred source

Beamr: Beamr ML-safe compression cuts bitrate 43% without degrading 3D detection

TVNewsCheck: CNBC UK deploys MediaKind Beam for cloud-connected live news operations

freenode: FFmpeg H.264 MVC decoding patch enables Blu-ray 3D multiview support

European Broadcasting Union: Media eXchange Layer open-source project named IBC2026 Innovation Awards finalist

Get this in your inbox → Subscribe

← Encoding & Software

EncodingTechnical Development

Kafka share groups scale throughput beyond partition limits

Streamingdata

Key Takeaways

The benchmark used the same setup as the prior post, but added configurable processing delays to model IO.
At 200ms delay, each instance processed 5 records per second; at 20ms, 50 rec/s; at 5ms, about 196 rec/s; and at 1ms, about 925 rec/s.
Standard and share-group consumers matched at 4 instances because the test topic had 4 partitions.
Share groups were scaled to 8, 16, and 32 instances, and total processing rate increased linearly in each case.
The main downside is that multiple instances can consume from the same partition, so message ordering is no longer guaranteed.