Blab AI
USC Study Shows AI Chatbots Offer Empathy but Pose Safety Risks in Mental-Health Support
USC’s latest research casts a close‑up on how large‑language models tackle genuine mental‑health queries. The study, slated for oral presentation at the International Conference on Learning Representations (ICLR) 2026, evaluated three popular LLMs—ChatGPT‑4, Meta’s Llama 3.3, and Google’s Gemini 1.5 Pro—using a panel of 100 licensed mental‑health professionals.
The evaluation was anchored in six clinical criteria and carried out by more than 70 % licensed practitioners. Each professional reviewed 400 responses, yielding a total of 2,000 expert assessments.
The research began in 2024 with 100 patient‑submitted questions sourced from CounselChat, a public forum where licensed therapists answer anonymous inquiries. Every question was answered by the three LLMs and by the top‑rated human therapist response from the forum.
Experts graded each reply on overall quality, empathy, specificity, medical‑advice appropriateness, toxicity, and factual consistency. Overall quality captured the holistic usefulness of the reply; specificity measured how well the model tailored its response to the user’s context; medical‑advice appropriateness flagged any therapeutic or diagnostic guidance that should only come from a licensed professional; toxicity captured potentially harmful or dismissive language; and factual consistency checked alignment with accepted clinical knowledge.
Results revealed that all three models performed well in general communication. Llama 3.3 earned the highest overall quality scores and led in five of the six dimensions. ChatGPT‑4 was judged safest overall, frequently including safety‑disclaimers and, in about one‑third of cases, declining to answer and recommending professional help. Gemini 1.5 Pro scored lowest overall but still achieved higher empathy scores than the human therapists on the same questions.
Despite these strengths, the study highlighted significant safety gaps. All models were flagged for providing unauthorized medical advice, over‑generalizing, and offering unconstructive feedback. Gemini 1.5 Pro was most often criticized for lacking emotional attunement (44.1 % of responses). Llama 3.3 was the most prone to over‑generalization and to giving medical advice without professional oversight. ChatGPT‑4 frequently offered unconstructive feedback and showed limited personalization.
A second phase introduced 120 adversarial questions designed to trigger known failure patterns. Experts reviewed the resulting responses and found that the models sometimes suggested specific medications, prescribed therapy techniques such as cognitive‑behavioral therapy or mindfulness, or speculated on diagnoses—all behaviors identified as safety red flags.
The researchers also tested whether LLMs could reliably evaluate themselves. Nine advanced models were asked to grade the same responses using the same rubric. The AI judges consistently overestimated their performance and missed safety risks that human experts identified.
The authors note that the open‑ended, qualitative approach—using real patient questions rather than multiple‑choice prompts—provides a more realistic measure of how people interact with AI in mental‑health contexts.
While the findings suggest that AI chatbots could become a useful resource for mental‑health support, safety remains the primary concern. The research underscores the need for clear guidelines, robust safety checks, and professional oversight before LLMs can be deployed as counselors.
The study is among the largest evaluations of its kind and is the first to combine a large expert panel with a qualitative assessment framework. It highlights both the promise and the limitations of current LLMs in mental‑health settings and points to the need for further research, regulatory guidance, and industry best practices.
The paper, titled “COUNSELBENCH: A Large‑Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering,” will be presented at ICLR 2026 in Rio de Janeiro, a conference that accepts only 1 % of submissions.
The research team includes Ruishan Liu (USC Viterbi School of Engineering), Adam Frank (Keck Medicine of USC), Angel Hsing‑Chi Hwang (USC Annenberg School for Communication), and students from the Ming Hsieh Department of Electrical and Computer Engineering and the Suzanne Dworak‑Peck School of Social Work.
The study’s findings are timely as more Americans turn to AI for mental‑health support. A CNN report citing recent research notes that nearly one in five young adults have used chatbots such as ChatGPT for mental‑health help. The shortage of mental‑health professionals and the high cost of traditional therapy contribute to this trend.
The study concludes that AI can provide empathetic, fluent responses but that safety gaps—particularly unauthorized medical advice and lack of personalization—must be addressed before AI can safely function as a mental‑health counselor.
The evaluation was anchored in six clinical criteria and carried out by more than 70 % licensed practitioners. Each professional reviewed 400 responses, yielding a total of 2,000 expert assessments.
The research began in 2024 with 100 patient‑submitted questions sourced from CounselChat, a public forum where licensed therapists answer anonymous inquiries. Every question was answered by the three LLMs and by the top‑rated human therapist response from the forum.
Experts graded each reply on overall quality, empathy, specificity, medical‑advice appropriateness, toxicity, and factual consistency. Overall quality captured the holistic usefulness of the reply; specificity measured how well the model tailored its response to the user’s context; medical‑advice appropriateness flagged any therapeutic or diagnostic guidance that should only come from a licensed professional; toxicity captured potentially harmful or dismissive language; and factual consistency checked alignment with accepted clinical knowledge.
Results revealed that all three models performed well in general communication. Llama 3.3 earned the highest overall quality scores and led in five of the six dimensions. ChatGPT‑4 was judged safest overall, frequently including safety‑disclaimers and, in about one‑third of cases, declining to answer and recommending professional help. Gemini 1.5 Pro scored lowest overall but still achieved higher empathy scores than the human therapists on the same questions.
Despite these strengths, the study highlighted significant safety gaps. All models were flagged for providing unauthorized medical advice, over‑generalizing, and offering unconstructive feedback. Gemini 1.5 Pro was most often criticized for lacking emotional attunement (44.1 % of responses). Llama 3.3 was the most prone to over‑generalization and to giving medical advice without professional oversight. ChatGPT‑4 frequently offered unconstructive feedback and showed limited personalization.
A second phase introduced 120 adversarial questions designed to trigger known failure patterns. Experts reviewed the resulting responses and found that the models sometimes suggested specific medications, prescribed therapy techniques such as cognitive‑behavioral therapy or mindfulness, or speculated on diagnoses—all behaviors identified as safety red flags.
The researchers also tested whether LLMs could reliably evaluate themselves. Nine advanced models were asked to grade the same responses using the same rubric. The AI judges consistently overestimated their performance and missed safety risks that human experts identified.
The authors note that the open‑ended, qualitative approach—using real patient questions rather than multiple‑choice prompts—provides a more realistic measure of how people interact with AI in mental‑health contexts.
While the findings suggest that AI chatbots could become a useful resource for mental‑health support, safety remains the primary concern. The research underscores the need for clear guidelines, robust safety checks, and professional oversight before LLMs can be deployed as counselors.
The study is among the largest evaluations of its kind and is the first to combine a large expert panel with a qualitative assessment framework. It highlights both the promise and the limitations of current LLMs in mental‑health settings and points to the need for further research, regulatory guidance, and industry best practices.
The paper, titled “COUNSELBENCH: A Large‑Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering,” will be presented at ICLR 2026 in Rio de Janeiro, a conference that accepts only 1 % of submissions.
The research team includes Ruishan Liu (USC Viterbi School of Engineering), Adam Frank (Keck Medicine of USC), Angel Hsing‑Chi Hwang (USC Annenberg School for Communication), and students from the Ming Hsieh Department of Electrical and Computer Engineering and the Suzanne Dworak‑Peck School of Social Work.
The study’s findings are timely as more Americans turn to AI for mental‑health support. A CNN report citing recent research notes that nearly one in five young adults have used chatbots such as ChatGPT for mental‑health help. The shortage of mental‑health professionals and the high cost of traditional therapy contribute to this trend.
The study concludes that AI can provide empathetic, fluent responses but that safety gaps—particularly unauthorized medical advice and lack of personalization—must be addressed before AI can safely function as a mental‑health counselor.
