NVIDIA Releases Dynamo v0.9.0: Major Infrastructure Fixes Including FlashIndexer, Multi-Condition Support, and Removed NATS and ETCD

NVIDIA recently released Dynamo v0.9.0. This is the most important infrastructure development for a distributed logic framework to date. This update simplifies how large models are used and managed. The release focuses on removing heavy dependencies and improving the way GPUs handle multi-type data.

Big Simplification: Removing NATS and more

The biggest change in v0.9.0 is the removal of the NATS again ETCD. In previous versions, these tools handled service discovery and messaging. However, they added ‘operational tax’ by requiring developers to manage more clusters.

NVIDIA replaced these with a new one Event Flight and a Discovery Flight. The system is now in use ZMQ (ZeroMQ) high-quality transportation and MessagePack for data processing. For teams using Kubernetes, Dynamo now supports it Kubernetes-native service discovery. This change makes infrastructure less complex and easier to maintain in production environments.

Multi-Modal and E/P/D Split Support

Dynamo v0.9.0 extends multi-mode support across 3 main areas: vLLM, SGlangagain TensorRT-LLM. This allows models to process text, images, and video more efficiently.

The key feature in this update is IE/P/D (Encode/Prefill/Decode) is separated. In a typical setup, a single GPU usually handles all 3 stages. This can cause bottlenecks during heavy video or image processing. v0.9.0 introduces Encoder Classification. Now you can use the Encoder on a different set of GPUs from Fill in advance again Delete the code employees. This allows you to scale your hardware based on the specific needs of your model.

Sneak preview: FlashIndexer

This release includes a private preview of the FlashIndexer. This component is designed to solve distribution delay problems KV storage managers.

When working with large context windows, moving Key Value (KV) data between GPUs is a slow process. FlashIndexer improves the way the system indexes and retrieves these cached tokens. This causes a drop Time To Start Token (TTFT). While it’s still a preview, it represents a big step toward making distributed computing feel as fast as local computing.

Smart Routing and Load Balancing

Managing traffic across 100 GPUs is difficult. Dynamo v0.9.0 introduces smart Editor using predictive load balancing.

The program uses a Kalman filter predicting future application load based on past performance. It also supports route plans from the Kubernetes Gateway API Inference Extension (GAIE). This allows the network layer to communicate directly with the indexing engine. If a particular GPU group is overloaded, the system can route new requests to idle workers with high accuracy.

Technology Stack at a Glance

The v0.9.0 release updates several key components from their latest stable versions. Here is a breakdown of the supported backends and libraries:

installation of dynamo tokens crate, written inside Rustensures that token management is always at a high speed. With data transfer between GPUs, Dynamo continues to be powerful NIXL (NVIDIA Inference Transfer Library) for It is based on RDMA communication.

Key Takeaways

1. Infrastructure Decentralization (Goodbye NATS and ETCD): The release completes the modernization of the communication architecture. By replacing NATS and ETCD with a new one Event Flight (using ZMQ again MessagePack) and Kubernetes-native service discoverythe system removes the ‘operational tax’ of managing external collections.
2. Multi-Modal Disaggregation (E/P/D Disaggregation): Dynamo now supports complete Enter code/Prefill/Generate code (E/P/D) split across 3 backends (vLLM, SGlang, and TRT-LLM). This allows you to run vision or video encoders on separate GPUs, preventing heavy encoding tasks from blocking the text generation process.
3. A preview of FlashIndexer for Lower Latency :’Preview’ of FlashIndexer introduces a special section to be developed distributed KV cache managers. It is designed to make the identification and retrieval of conversation ‘memory’ much faster, aiming to reduce the Time to First Token (TTFT).
4. Intelligent Programming with Kalman Filters: The system is now in use predictive load balancing powered by Kalman filters. This allows Scheduler to predict GPU load more accurately and handle traffic increases proactively, supported by route plans from the Kubernetes Gateway API Inference Extension (GAIE).

Check it out GitHub release here. Also, feel free to follow us Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to Our newspaper. Wait! are you on telegram? now you can join us on telegram too.