An AI hallucination nearly triggered a US military operation

The use of generative language models in decision support systems led to a critical incident in the US defense sector, where an algorithm processing intelligence data provided a false interpretation of the situation, nearly triggering a real military operation. According to the report, the incident occurred due to an error in data processing logic, which experts classified as a model "hallucination". The system mistakenly identified civilian objects as hostile targets based on an incorrect correlation between disparate sources of information, resulting in an automated protocol sending operators a signal for immediate response. GovAI specialists emphasize that the problem lies in the fundamental unpredictability of LLM when working with high-risk data, as the model ignored 94% of facts confirming the absence of a threat, focusing on statistical anomalies that had no relation to reality, and the error was identified by personnel just minutes before the activation of combat systems. The incident prompted a review of security protocols at the Pentagon, now requiring any AI-made decisions to be verified by a human operator using independent control systems, and developers plan to implement additional filtering layers to block model responses when detecting a high degree of "uncertainty" in generated conclusions. This case demonstrates the risks of introducing LLM into critical infrastructure without proper formal verification, and military agencies intend to limit the use of "black boxes" in favor of more transparent and interpretable architectures capable of providing justification for each conclusion.