Xiaomi optimized the operation of the MiMo-V2.6-Flash model on the DGX cluster

Developers presented the configuration of MiMo-V2.6-Flash-2x-DGX-Spark, allowing the launch of the 310-billion model Xiaomi on specialized hardware, the system demonstrates high performance when working with long contextual sequences. The basis of the solution is the use of 4 DGX Spark nodes running under the vLLM engine, the integration of the DFlash method for speculative decoding and the use of fp8 KV cache made it possible to achieve a generation speed of 155.8 tokens per second, the model successfully processes a contextual window of up to 300,000 tokens. The technical implementation of the project is focused on minimizing delays during inference of super-large language models, the use of 8-bit quantization of the key-value cache (KV cache) significantly reduces the requirements for video memory, while maintaining the accuracy of generation at the level of the original model weights. This approach allows deploying heavy LLMs in corporate environments with a limited number of computing nodes, optimizing the vLLM stack for the DGX Spark architecture provides stable operation under high loads, which is critical for systems that require processing large amounts of data in real-time.