A patch has been released for running the 125B Qwen3.8 model on consumer hardware

Developers presented Qwen3.8-Flash-Next-INT4-Mixed-AutoRound — a specialized patch for the vLLM engine, allowing the launch of heavy MoE models on limited hardware resources. The solution uses a mixed quantization scheme INT4/INT6, implemented through the AutoRound method. Technical implementation allows deploying a model with 125 billion parameters on a configuration of 3–4 RTX 3090 video cards. The system supports work with a contextual window of up to 262,000 tokens. During tests, the performance during decoding reached 60 tokens per second, and the prefill speed was 3700 tokens per second. To achieve such indicators, weight optimization is used, which minimizes the loss of accuracy when switching to a low-bit representation. The use of AutoRound ensures a balance between data compression and preserving the model's ability to reason, which is critical for MoE (Mixture of Experts) architectures. This tool lowers the entry threshold for local launch of large language models that previously required server power with a large amount of VRAM. The patch is aimed at researchers and developers using consumer equipment to work with long context and high-parameter neural networks.