Researchers proposed a method of calibrated routing of requests for LLM

A group of researchers presented a calibrated routing method for systems with separated computational powers for prefill and decoding. The new algorithm dynamically distributes the load between GPU nodes, considering the length of the prompt, the predicted length of the response, the pressure on the KV cache, and the requirements for SLO. During testing on eight NVIDIA A40 GPUs running the vLLM engine, the system demonstrated an average goodput of 0.864. This exceeds the results of standard strategies, such as round robin (0.835) and least loaded. Using a calibrated router made it possible to achieve similar performance using six GPUs instead of seven, which reduces the requirements for infrastructure. The authors emphasize the importance of hardware calibration: using constants obtained only through a simulator reduces the effectiveness of goodput by 4.5 points and reduces the advantage in tail latency by 40%. In conditions of extreme resource scarcity, greedy cost minimization becomes less effective than simple load distribution. The advantages of the method are most noticeable when working with heterogeneous traffic and scaling decoding pools. With a small number of nodes (three or fewer), the efficiency of the algorithm is neutralized, as a simple queue length count is sufficient for balancing. The work is published under the CC BY 4.0 license. The research results allow optimizing the work of disaggregated systems, such as DistServe or Mooncake, by more accurately predicting the completion time of requests on each node.