Researchers have compiled an atlas of inference optimization for LLM

A team of researchers published The Inference Engineering Pareto Atlas, in which they analyzed the impact of various optimization methods on the cost, quality, and latency of language models. For testing, the Qwen2.5-7B-Instruct model was used, running on the vLLM 0.12 engine. During the work, 54 configurations were tested on L4, A100, and H100 graphics processors. Researchers used a simulator calibrated to real measurements, with a drift error of less than 1.5%. Quality was evaluated on 200 questions from the GSM8K set, including checking FP16, AWQ 4bit, and FP8 quantization methods. The results showed that FP8 weights retain 99.4% accuracy while reducing latency to 0.61–0.65 of the base level. At the same time, the AWQ 4bit method reduces token latency to 0.34 of the base, but leads to a loss of 5.9% accuracy. Using FP8 KV cache without additional settings was ineffective, as the model stopped giving correct answers, despite maintaining throughput. The choice of optimal configuration depends on project constraints. For tasks with strict latency requirements, H100 was the most effective, while A100 showed better results in terms of cost, providing inference for $0.106 per million tokens. Combined optimization methods more often fall on the Pareto boundary than single settings. This atlas allows engineers to select optimal deployment parameters for models under specific hardware resources. The work is published under the CC BY 4.0 license.