Developers have accelerated work with long context on RTX 5090

Enthusiasts have presented the project fast-long-context, which allows expanding the working context window to 256K tokens on RTX 5090 graphics cards. The solution is implemented as a patch for the SGLang framework, optimizing memory usage when performing inference.

The system uses weights and KV-cache in NVFP4 format, providing high data density while maintaining performance. During testing on the specified hardware, the generation speed reaches 240–300 tokens per second. The tool requires a Linux operating system and an NVIDIA architecture graphics accelerator to work.

The technology is aimed at developers who need to process large amounts of data without using server clusters. Using NVFP4 significantly reduces the requirements for video memory while maintaining the ability to work with long sequences, which was previously limited by the hardware resources of consumer GPUs.

The project is available in an open repository and requires preliminary setup of the SGLang environment. This approach allows effectively utilizing the potential of new graphics cards for tasks that require deep context analysis in real-time.