Researchers presented the BITCOS method for compressing ternary LLMs

Researchers presented BITCOS — a new method of adaptive data placement that allows overcoming the standard compression barrier of 1.58 bits for ternary language models. Unlike the traditional method of packing five trits into a byte, which requires 1.625 bits per weight, the new technology takes into account the actual distribution of characters in neural network weights. Analysis of 29 ternary models showed that the share of zero weights reaches 51.5%. The BITCOS method uses a dense bit map of the presence of non-zero values in combination with a compact vector of signs. This allows reducing the memory volume to 1.485 bits per weight for the most sparse models, which is more efficient than standard approaches in 26 out of 29 tested architectures. Developers implemented optimized unpacking sequences for AVX-512, AVX2 instructions and Intel Xe2 graphics processors. The use of this format provides a performance boost of matrix-vector multiplication of up to 1.28x compared to current industrial solutions. Testing on five different hardware platforms, including client and server processors, showed an increase in decoding throughput of up to 1.18x on CPU and up to 1.27x on GPU. The use of BITCOS allows for more efficient use of memory resources when deploying models with limited weight precision. This development optimizes inference for systems where parameter storage density is critical. The transition to adaptive compression reduces memory bandwidth requirements, which accelerates the operation of LLM on a wide range of equipment, from integrated graphics cores to high-performance server accelerators.