NVIDIA Dynamo Slashes LLM Crash Recovery to Seven Seconds

NVIDIA has unveiled a preview of Dynamo, an open-source tool designed to address the prolonged downtime associated with large language model (LLM) engine crashes. In a benchmark test using the GLM-5.2 model quantized to NVFP4 on NVIDIA B200 nodes, Dynamo's shadow engine recovery system restored the system in just 7.3 seconds. This represents a 39-fold speedup compared to the 283 seconds required for a traditional cold restart, which typically demands reloading massive model weights and rebuilding CUDA graphs from scratch.

The technology achieves this speed by keeping a pre-warmed backup engine idle on the same GPUs. At the heart of this architecture is the GPU Memory Service (GMS), a sidecar process that manages physical GPU memory independently of the active inference engine. Utilizing the CUDA Virtual Memory Management API, GMS ensures that model weights remain resident in high-bandwidth memory even if the primary engine process crashes. When a failure occurs, the parked shadow engine quickly maps the existing weights into its own virtual address space and takes over.

The rapid failover dramatically improves the user experience during a software fault. In NVIDIA's tests, the median time to first token (TTFT) after a crash dropped from 23,815 milliseconds under a cold restart to just 1,311 milliseconds with Dynamo. Additionally, the median decode rate rose from 12 to 46 tokens per second per user. While a cold restart left 201 out of 399 requests waiting longer than five seconds for their first token, Dynamo reduced that figure to just a single request.

However, Dynamo is currently a preview feature with specific infrastructure requirements. It requires Kubernetes version 1.34 or newer with Dynamic Resource Allocation (DRA) enabled, alongside the NVIDIA GPU DRA driver. While vLLM serves as the primary backend, integrations for SGLang and TensorRT-LLM are also supported. Practitioners should note that Dynamo only resolves software-level process crashes rather than physical hardware failures, and promoted shadow engines currently start with empty key-value caches.