Netflix Conductor engine overhaul supports 30,000 tasks per workflow
Netflix has overhauled its internal Conductor orchestration engine, replacing legacy components like Dynomite with Cassandra and Timestone to support 30,000 tasks per workflow. The update includes a new type-safe Java SDK and managed serverless task integrations to improve scalability and developer experience for its media production and advertising pipelines.
Key Takeaways
- Workflow capacity increased 10x from 2,500 to 30,000 tasks per execution through a new partitioned data model.
- Asynchronous evaluation decoupled triggers from execution, reducing p99 latency by 40% and eliminating lock contention.
- Managed serverless task integrations for Titus and Stratum allow teams to run containerized compute without hosting dedicated worker fleets.
- A new type-safe Java SDK replaces JSON-based authoring to prevent runtime errors and improve developer experience.
Why It Matters
This infrastructure shift signals Netflix's transition toward a more unified, serverless-first orchestration model that reduces operational overhead for individual engineering teams. By moving away from the open-source version to a highly specialized internal fork, Netflix is prioritizing the extreme concurrency required for its expanding ads and games businesses. The move to a type-safe Java DSL reflects a broader industry trend toward 'infrastructure as code' to minimize costly runtime failures in complex media pipelines. Watch for whether Netflix introduces more complex 'Run Strategies' like idempotent de-duplication to further automate coordination across its global studio operations.
Additional Context
Netflix Conductor has become one of the most widely adopted open-source workflow orchestration engines since its 2016 release, and its evolution continues to influence how streaming and media companies architect their pipelines. Orkes, the commercial company founded by Conductor's original creators, raised $9 million in seed funding in early 2025 to offer a managed cloud version of the engine, signaling sustained enterprise demand for Conductor-based orchestration beyond Netflix's internal fork. The company reported that its platform processes workflows for customers in financial services, healthcare, and media, positioning itself as the neutral steward of the open-source project while Netflix pursues its own specialized trajectory.
The business case for orchestration engines at Netflix scale is closely tied to the company's expanding advertising and gaming operations. Netflix reported $1.9 billion in advertising revenue for fiscal year 2025, a figure that nearly doubled year over year and placed enormous pressure on the ad-serving and campaign-management pipelines that Conductor coordinates. Separately, Netflix acquired Spry Fox in late 2025 to bolster its cloud gaming infrastructure, adding another category of complex, multi-step workflows that the orchestration layer must handle. These revenue streams explain why the internal Conductor fork now supports 30,000 tasks per workflow, far beyond the open-source version's typical operational envelope.
On the technical side, the shift from Dynomite to Cassandra aligns Netflix Conductor with a broader industry preference for Cassandra in high-throughput orchestration backends. Apache Cassandra 5.0, released in September 2024, introduced storage-attached indexes and vector search capabilities that improve query performance for workloads requiring both transactional writes and analytical reads, characteristics relevant to Conductor's task-state tracking. Meanwhile, Temporal Technologies raised $146 million in a Series C round in 2025, reaching a $1.6 billion valuation as a competing orchestration platform that emphasizes durability guarantees over Conductor's throughput-first design. The competitive landscape now forces streaming infrastructure teams to weigh Conductor's proven scale against Temporal's formal execution semantics, particularly for pipelines where media orchestration platforms is non-negotiable.
Read full article at netflixtechblog.medium.com
Enjoy our coverage?
Add StreamingMeme as a preferred source on Google to see more of our streaming news at the top of your Search results.
Add as preferred source