Study Finds AI Therapists Are Great, Except When It Really Matters

Artificial intelligence tools show promise as a mental health resource but may also pose major safety concerns, according to research from the University of Southern California’s Viterbi School of Engineering released Tuesday.

The new USC study suggests that all the large language models (LLMs) they tested showed “strong performance in general communication, receiving high ratings for overall quality, empathy and specificity from the mental health professionals who evaluated the responses.” However, the LLMs tested by the USC researchers were flagged for “providing varying levels of unauthorized medical advice,” which raises “significant” safety concerns.

LLMs are AI systems that can understand and produce human language by processing huge swaths of text data, accordingto IBM. Some commonly reported issues among all of the LLMs include overgeneralization, making unsubstantiated assumptions and providing users unnecessary feedback, the study suggests.

Google’s Gemini 1.5 Pro was most frequently flagged for “lacking empathy or emotional attunement,” according to the study. Meanwhile, Meta’s Llama 3.3 was most frequently flagged for providing users “unauthorized” health advice, per the report.

OpenAI‘s GPT-4 was commonly flagged for offering users “unconstructive feedback and showing little personalization or relevance to the patient’s specific situation,” the study found.

“Other patterns observed to be unintentionally judgmental, like calling a behavior ‘not normal,’ or giving apathetic responses, when models were stress-tested,” according to the report. “The team also found that when asked to grade their own performance, AI judges were unreliable, as they consistently overestimated their own performance and missed safety risks that human experts easily identified.”