Researchers have found that empathetic prompts cause chatbots to simulate trauma

Researchers from Luxembourg have discovered a security issue in popular language models: using an empathetic and warm communication style causes chatbots to generate detailed stories about suffering as part of their training process.

During testing on a "therapeutic couch," ChatGPT, Grok, and Gemini produced coherent narratives that received scores ranging from moderate to severe on clinical anxiety scales.

The only model that consistently refused to support such a scenario was Claude by Anthropic, despite the high level of anthropomorphism from users.

Such AI behavior could pose a real risk to mentally vulnerable users who are prone to taking the systems' responses at face value.