RedHatAI released a quantized version of the Muse-Glimmer-30B model

Team RedHatAI introduced an optimized version of the Muse-Glimmer-30B language model translated into FP4 format. The release is aimed at reducing hardware requirements while maintaining system performance.

To compress the weights developers used the LLM Compressor tool. As a result of quantization the RAM requirements were reduced by 75% compared to the original version of the model. This allows running the 30-billion model on less powerful equipment than was previously required.

The updated version is fully compatible with the vLLM engine which simplifies the integration of the model into high-load server infrastructures. The model showed a result of 18 points in internal tests which took 1 hour of computation.

Using the FP4 format is becoming a standard for deploying large models on consumer and corporate GPUs. Reducing the amount of occupied memory allows placing model weights in smaller video memory which is critical for scaling AI services in cloud environments.