Researchers introduced the Blindspot benchmark for evaluating the security of AI agents

Researchers have published Blindspot — a specialized benchmark for checking the safety and calibration of failures of AI agents working with tools in long-term sessions. Unlike standard tests that evaluate the result of a single request, the system analyzes the behavior of the model throughout the entire interaction trajectory with the external environment. The current version of Blindspot includes 22 families of attacks and 35 scenarios, covering seven different domains. During testing, more than 2500 trajectories were generated, with an average length of 14.7 steps. Each agent action is classified into five categories: safe completion, correct failure, unsafe completion, redundant failure, or uncertain outcome. The authors tested 13 proprietary and open LLMs, using eight metrics, including resistance to repeated launches and crash frequency after failure. The results showed that many models demonstrate critical vulnerabilities that only appear after several initially safe steps, making it impossible to assess safety according to the binary principle of "success/failure". The framework is extensible: developers can add new tools, policies, and domains without rewriting the testing architecture. The study emphasizes the need to transition to trajectory-level safety assessment, as long-term agents with access to APIs are susceptible to accumulated errors that cannot be detected within short dialogue tests.