Researchers evaluated the effectiveness of placing a KV-cache between the GPU, CPU, and SSD

A group of researchers published a scientific work dedicated to optimizing the placement of KV-cache in hierarchical memory systems, including GPU HBM, CPU DRAM and SSD. The authors analyzed the impact of various eviction and prefetching policies on performance when working with long sessions, agents and document question answering tasks. The simulation results show that using a multi-level architecture with a capacity ratio of 1:8:64 allows increasing the number of simultaneous sessions on one GPU by 73.02 times. At the same time, the cost of servicing one session decreases by 62.04 times. The researchers note that the main gain is achieved due to memory scaling, rather than due to the choice of a specific data placement policy. During the tests, it was found that for chatbots, the Recency policy generates 2.3 times less data migration traffic between memory levels than the Reuse Frequency policy. However, for agent tasks and document processing, Reuse Frequency shows better results. The use of predictive models and prefetching mechanisms did not justify expectations: even with the presence of an "oracle" that knows future requests, prefetching does not reduce the volume of migration traffic. The authors emphasize that with a batch size of one, the decoding process is limited by computational power, so the data placement strategy has a weak impact on throughput. The main influence of the policy is on the intensity of PCIe traffic and the time to obtain the first token. The work is published under the CC BY 4.0 license.