Researchers introduced the HTB benchmark to combat hallucinations of AI agents

Researchers published a work Closed-World Resolution Against Tool Hallucination in LLM Agents, in which they analyzed the problem of invoking non-existent tools by neural network agents. The authors proved that current filtering methods are unable to prevent the generation of calls that do not match any scheme in the registry.

During the study, a classification of hallucinations into five categories (H1-H5) was developed. Testing of ten models showed that increasing the number of parameters does not reduce the frequency of errors: models with 675 billion parameters demonstrate the same performance as compact solutions with 7-8 billion. A total of 322 cases of generating non-existent functions were recorded, with the number of errors increasing 11 times when using an unstructured JSON format.

Particular attention is paid to the Model Context Protocol. When combining multiple servers into a single namespace, collisions occur, leading to 154 new cases of hallucinations even in advanced models that previously passed testing in isolated environments.

To standardize evaluation, developers presented the Hallucinated-Tools Benchmark (HTB). This dataset allows testing the ability of agents to match calls with real tool registries. The implementation of similar verification systems is necessary to prevent the execution of invalid commands that bypass standard security mechanisms and request filtering.