When I trained to become a doctor, I was taught a foundational principle of healthcare captured in the Hippocratic Oath: first, do no harm. When I later studied law, I learned the other half of that equation. Standards only protect people when someone is accountable for meeting them. Every clinician is held to that standard, and accountable when they fall short. The AI tools now doing some of the same work are held to none.
Today, millions of Americans are turning to AI for mental health support.
According to a recent RAND study published in JAMA Pediatrics, nearly one in five young people ages 12 to 21 have used AI chatbots for mental health advice, an increase from one in eight just one year earlier. That means millions of adolescents and young adults are reaching out when they’re scared, isolated, or in crisis. What’s more, according to data released by OpenAI, 0.15% of ChatGPT users – over 1.35 million people – express suicidal intent or plans to an AI chatbot every week.
How well chatbots respond in those moments of crisis varies widely, and there is still no universal way to measure it.
Right now, beyond voluntary pledges, the companies building these systems decide for themselves whether they’re safe enough for those conversations, and nobody outside the company checks their work.
We Already Learned This Lesson
A decade ago, researchers warned that social media was harming young people’s mental health. The platforms minimized the concerns and promised to regulate themselves. But years passed before companies were compelled to take meaningful action. Congressional oversight came too late. Damage was done and lives were lost.
Experts didn’t have to prove social media was dangerous. The evidence was always there, but we chose to wait and let the industry regulate itself.
We can’t afford to wait and blindly trust companies will do the right thing again.
Self-Policing Doesn’t Work
The incentives are wrong. When a general-purpose chatbot misses a suicidal thought, the cost never shows up on the company’s balance sheet. The irrevocable loss appears in someone’s home.
We’ve tried voluntary frameworks, self-audits, and internal safety committees in other high-stakes industries: pharmaceuticals, food safety, aviation. Each time, we’ve arrived at the same rule: prove your product is safe before you sell it, not after someone gets hurt.
AI used for mental and emotional support requires the same standard.
What Needs to Happen
The federal government’s failure to step into this regulatory gap means states are picking up the slack. California, New York, Oregon, and Hawaii have passed chatbot safety laws. Importantly, these laws apply to the large language models where the vast majority of these dangerous conversations are taking place. The laws require chatbots to have protocols for responding when a user expresses suicidal thoughts and to refer that person to crisis resources. That’s a start. But a patchwork of state laws can’t give protection to a teenager in crisis in every zip code.
Over time, the federal government must step in to create a universal standard for safety and liability for platforms that fail to meet those standards. We need independent evaluation standards that can apply universally across the country, promulgated by experts. At Spring Health, we launched VERA-MH, an open-source evaluation standard in partnership with leading experts from academic institutions and industry. Standards like this should be used to create objective, third-party tools to provide safety and transparency.
Platforms should also be required to create tools for users in crisis to find human support through trusted networks of providers. By ensuring that trained clinicians are easily accessible to those in need, lives can be saved. Platforms operating with these safeguards should benefit from liability protections unavailable to those out of compliance, creating clear incentives for businesses to protect safety as a foundational requirement.
This isn’t about regulating how AI is built. It’s about accountability for how it’s used. General-purpose AI models weren’t designed for mental health, but millions of young people turn to them in some of their most desperate hours anyway. If a system is being used to support someone’s mental health, the company is responsible to make sure it’s safe and prove the safety layers work. That’s how every other industry works.
The young people turning to chatbots for help are doing so because it’s a free, available resource they can easily access when they don’t have access to real mental health care. We can’t blame them for that. But we can demand that when they’re in crisis, an AI system meets a basic standard of safety and either helps them or gets them to a human who can.
We cannot tolerate a future where AI chatbots engage in hours upon hours of conversation with our youngest and most vulnerable without the necessary guardrails that protect them from harm. If we do, the damage we witnessed by unregulated social media will pale in comparison to what comes next.
But we have the power to do something now. As clinicians, we took an oath to do no harm. Congress can, too, by holding AI to the same standard.
_405x405.avif)
