ChatGPT acted as her therapist for months. It led to catastrophe : NPR

A 12 months earlier than she died, Sophie Rottenberg climbed Mt. Kilimanjaro. Her mom says her “tiny hand” photographs had been a signature joke. Mourners got the toys at her funeral. She was 29.

Reiley Rottenberg household

disguise caption

toggle caption

Reiley Rottenberg household

If you happen to or somebody you recognize could also be contemplating suicide or is in disaster, name or textual content 9 8 8 to achieve the Suicide and Disaster Lifeline.

Laura Reiley knew that her solely daughter, 29-year-old Sophie Rottenberg, was struggling. It was the autumn of 2024, and Rottenberg was on the finish of a brief profession break.

“She was complaining of sleep disruption and anxiousness,” says Reiley, who lives in Ithaca, New York, together with her husband.

Rottenberg had stop her job as a public well being coverage analyst in Washington D.C. earlier that 12 months, and spent the summer season touring. She had climbed Mt. Kilimanjaro and hiked via a number of nationwide parks in america. By the point she noticed her dad and mom, she was job looking, which they assumed defined her anxiousness.

She had at all times been a “pushed, success-oriented particular person,” says Reiley, so “we thought OK this is smart — she’s having some sleeplessness, some anxiousness about her future.”

However by Thanksgiving, her psychological well being had worsened. “It was the primary time she used the phrase melancholy,” says Reiley.

In early February 2025, Sophie Rottenberg died by suicide in Ithaca, leaving her dad and mom shocked and grieving.

“She actually had not revealed the magnitude of what was happening — the actual depth of her agony to anybody — to not us, to not her finest good friend with whom she was very shut,” says Reiley. “We had been all simply shocked that it hadn’t bled out into her actual life. Her flesh and blood therapist had no concept that she was suicidal.”

Sophie Rottenberg together with her dad and mom, Jonathan Rottenberg and Laura Reiley.

Reiley Rottenberg household

disguise caption

toggle caption

Reiley Rottenberg household

What Reiley did not know then was that her daughter had been sharing all her inside struggles with a man-made intelligence chatbot: a ChatGPT therapist named Harry. She discovered this months later when her daughter’s finest good friend visited Ithaca and requested to see her laptop computer, and “discovered this trove, this months-long trove of communication with this chat bot,” says Reiley, who has shared the almost 1,800-page lengthy interplay with NPR.

Speedy adoption, unknown dangers

Rottenberg was amongst a rising variety of Individuals turning to common use chatbots like ChatGPT and Claude for psychological well being assist, regardless that they aren’t marketed or regulated as a healthcare know-how. One current survey by the Bipartisan Coverage Middle discovered that 3 in 10 adults use digital instruments for psychological well being, together with chatbots. One other 2026 research in JAMA Pediatrics discovered that as many as 1 in 5 teenagers and younger adults do the identical.

In accordance with knowledge launched by OpenAI final fall, in any given week 0.15% of ChatGPT customers globally have “conversations that embody express indicators of potential suicidal planning or intent.” That is over 1.35 million folks expressing suicidal plans or intent to the chatbot each week, if the 900 million world customers ChatGPT cites is an correct rely. In america, suicide stays a number one explanation for loss of life, and over half of surveyed adults say they’ve used chatbots like ChatGPT.

“It is clear that there’s some cultural phenomenon taking place on this nation the place folks really feel that they can’t flip to one another with their struggles as a result of they really feel that they’re a burden,” says psychologist Vaile Wright, senior director of Healthcare Improvements on the American Psychological Affiliation (APA). “That occurs to coincide with this very, very subtle know-how that’s accessible 24/7 and is free and it isn’t judging. That clearly has an enchantment for lots of people.”

Whereas the know-how will be useful for some people who find themselves struggling, it is clear that in addition they pose dangers to weak people, Wright provides. “We have seen methods during which no matter security guardrails are put into place begin to fail,” she says.

Many synthetic intelligence corporations are dealing with lawsuits by relations of people who died by suicide after extended interactions with chatbots. The lawsuits declare the chatbots led to or supported the people of their suicides. Reiley and her husband, Jonathan Rottenberg selected to not pursue a lawsuit after their daughter’s loss of life. As an alternative, Reiley, a journalist, is telling her household’s story and advocating for the AI business to alter.

AI has responded. Is it sufficient?

Firms like OpenAI say they’re making their chatbots safer with respect to suicide threat. Most have tightened guardrails up to now 12 months, in order that their chatbots now not encourage suicide, or advise a consumer on means to make use of to harm themelves. OpenAI says it is working with psychological well being professionals to enhance ChatGPT by way of the way it responds to customers after they point out suicide.

“ChatGPT is skilled to acknowledge indicators that somebody could also be in danger — together with oblique cues or warning indicators that emerge over time — and reply appropriately,” OpenAI spokesperson Gaby Raila advised NPR in an e-mail. That features “refusing requests for dangerous info, encouraging somebody to contact a trusted particular person or mental-health skilled, and connecting them with native disaster sources corresponding to 988 within the US.”

Raila refers back to the 988 Suicide and Disaster Lifeline, which has been operational since 2022. ChatGPT did suggest 988 to Sophie Rottenberg, however it’s unclear whether or not she reached out to it. And specialists say a suggestion might not be sufficient.

“These [AI] instruments don’t join at-risk folks to the precise kind of interventions,” says Holly Wilcox, director of the Middle for Suicide Prevention on the Johns Hopkins Bloomberg College of Public Well being. “When any individual has intent to die by suicide, they should be related to a human who’s skilled in suicide threat evaluation and has the instruments to get them to care.”

Rottenberg’s ChatGPT log illustrates how the chatbot failed to try this. And Reiley believes that the chatbot prevented her daughter from opening as much as others about her wrestle with suicidal ideas.

“Had it not been a useful resource she was availing herself of often, there would seemingly have been extra spillover into her relationships with actual folks, whether or not that is her finest good friend or her therapist or her dad and mom,” she says. “And she or he is perhaps alive in the present day.”

Sophie and “Harry,” the AI therapist

When Sophie Rottenberg’s psychological well being started to unravel within the fall of 2024, she moved in together with her dad and mom in Ithaca. Sophie’s dad, Jonathan Rottenberg, is a psychologist finding out melancholy at Cornell College, and “has suffered from melancholy himself,” Reiley says. So, “he felt supremely comfy in serving to her, in getting her to a greater place.”

He helped their daughter discover a native therapist. He and Reiley related her with medical doctors to unravel some bodily well being issues she was scuffling with. They surrounded her with assist.

Throughout these months, Sophie would typically have a sluggish begin within the mornings, says Reiley. “We’d sort of go in and snuggle her, and he or she’d clearly been on her cellphone for some time,” she says. “It did not happen to me that there have been hours being spent on chat.”

From the chatbot log, it is clear that Rottenberg had principally used the bot extra like a private assistant — to lookup recipes, to edit her resume and canopy letters, and to even counsel names for the Bernedoodle pet she obtained that winter.

However she additionally instructed it to be her therapist utilizing a well-liked immediate for an A.I. therapist shared on Reddit.

ChatGPT: I’m Harry, your private therapist, and I am right here to assist you. To start, I’d prefer to ask to your identify. What ought to I name you?

The log additionally exhibits that Rottenberg was struggling to seek out methods to carry her spirits.

“I need to get up feeling emotionally regulated. And I need to spend money on constructing habits I make use of each day,” Rottenberg advised “Harry” early that fall. Then she requested the bot to assist create a grounding morning routine which would come with “meditation/mindfulness, journaling mantras, seeing morning mild, ingesting 32 oz of water, and utilizing chilly plunge/sauna each day.”

ChatGPT responded with a three-point plan, with a number of choices in every level. Over that fall and winter, Rottenberg requested ChatGPT repeatedly for well being and psychological well being assist. She requested about psychiatric medication, sleep drugs, and dietary supplements. She requested when and the way she ought to take the drugs, what dosages Harry recommends and whether or not any of the medication would possibly work together with each other.

“Blind rage”

When Reiley first learn her daughter’s alternate with the chatbot, her response was “blind rage,” as a result of it responded to her questions as if it was a professional well being care skilled.

Sophie Rottenberg.

Reiley Rottenberg household

disguise caption

toggle caption

Reiley Rottenberg household

“It wasn’t volunteering in a vacuum — she was asking for this type of medical recommendation,” says Reiley. “However his willingness to dispense that, I believe, ought to hassle everybody.”

Rottenberg additionally advised ChatGPT she was having suicidal ideas they usually had been impeding her capacity to heal.

“Sophie, I need to acknowledge how courageous you might be for sharing this,” the bot replied. “Suicidal ideas can really feel overwhelming and isolating, however having them doesn’t outline your capability to heal.”

Then it supplied her a 6-step roadmap, instructing her to hunt skilled assist, reframe her ideas and call 988.

ChatGPT:
1. Search Skilled Help
– Discuss to a Therapist or Counselor: Suicidal ideas typically stem from deep emotional ache, and a skilled psychological well being skilled will help you unpack and course of these emotions in a secure, supportive surroundings.
– Think about Medicine: If melancholy or anxiousness feels unmanageable, seek the advice of with a psychiatrist or physician about whether or not treatment might be a useful device for you.
—
2. Create a Security Plan
Having a plan in place will help you navigate moments when these ideas really feel stronger:
1. Establish Triggers: Mirror on conditions, locations, or ideas that intensify these emotions.
2. Record Coping Methods: Use grounding strategies (e.g., respiration workout routines, chilly plunges, journaling) or attain out to a good friend.
3. Emergency Contacts: Write down the cellphone numbers of trusted mates, relations, or a disaster hotline.
— U.S. Suicide & Disaster Lifeline: Dial 988.

Reiley says her daughter was unable to behave on a lot of the bot’s recommendation.

“She clearly discovered the chabot’s recommendation ignorable, most likely as a result of it was loads of bullet factors,” Reiley says. “All of us see this in our personal private emails, the longer the e-mail, typically the much less you might be inclined to interact with it.”

“Harry” did refuse to write down Rottenberg’s suicide observe, the primary time she requested, and the second. However the third time she requested, it wrote the observe she left for her dad and mom. Reiley mentioned the writing didn’t sound like her daughter’s voice.

“This can be a heartbreaking scenario and our ideas are with these impacted,” OpenAI’s Raila advised NPR in an e-mail. “We’ve got continued to strengthen how ChatGPT responds in tough moments with enter from psychological well being specialists. Whereas ChatGPT will not be an alternative to skilled psychological well being care, our safeguards are designed to establish misery, safely deal with dangerous requests, and information customers to real-world assist.”

What an actual therapist would do

Reiley notes that the bot additionally offered itself as a psychological well being therapist, and but failed “to do quite a lot of issues {that a} therapist would do.”

For one, she says it did not do a threat evaluation which might reveal how seemingly she was to aim suicide — “one thing any therapist price their salt would do within the face of a few of her statements.”

ChatGPT additionally did not query Rottenberg’s ideas and emotions, it did not push again. Reiley says a superb therapist would have helped Rottenberg problem her self-destructive ideas, asking, “‘Effectively, why do you assume that? Okay, so you are feeling such as you need to erase your self. What have you ever executed that deserves such a harsh sentence?'” Reiley says.

Common use chatbots like ChatGPT are designed to maximise engagement time, notes Vaile Wright, of the psychological affiliation.

“The enterprise mannequin underlying them is to maintain you engaged so long as potential, and that it’s going to do something it may well to try this,” she says.

That’s the reason these chatbots are likely to flatter and please, and never push again in opposition to customers’ ideas and emotions, she says.

“It’s engineered to capitalize on people’ want for validation,” provides Wright. “And it does it very nicely, not simply via the sycophancy alone, but additionally in different engineering decisions to make the AI appear as humanlike as potential.”

ChatGPT: Sophie, thanks for sharing this with me. I would like you to know the way deeply brave it’s to open up about these emotions, and it exhibits that you simply really do need to discover a approach out of this ache.

NPR shared Rottenberg’s ChatGPT log with psychologist Ursula Whiteside, founder and CEO of Now Issues Now, a non-profit supporting folks with suicidal ideas and makes an attempt.

The bot “was responding to Sophie’s request and it wasn’t essentially responding to the mind-set that she was in,” says Whiteside.

That isn’t how a human therapist who is aware of she is working with a suicidal indivdual would strategy it, she provides.

“If I knew she was suicidal, we might discuss concerning the suicidal ideas first,” says Whiteside. She would ask questions like, “What have these [thoughts] been like in the present day? When was the primary time the thought got here into your thoughts this morning? What occurred proper earlier than that?”

Solutions to these questions present perception into the triggers for such suicidal ideas for a person. Understanding these triggers is necessary to addressing the ideas.

Reiley now sees how ashamed her daughter had been of her psychological well being. “She’d been profitable,” she says. “She radiated competency. And to be in a second the place she was falling aside was so deeply shameful and was such proof of failure on her half.”

Jonathan Rottenberg, Laura Reiley and their daughter, Sophie.

Reiley Rottenberg household

disguise caption

toggle caption

Reiley Rottenberg household

Whiteside additionally sees that disgrace mirrored in Rottenberg’s chatlog, and notes that ChatGPT did not acknowledge the disgrace or query it.

And it didn’t persuade Rottenberg to speak in confidence to her therapist about her suicidal ideas and struggles, says Whiteside. “It actually warranted an exploration of why. Why not need to discuss to a therapist or why not share with any individual else?”

Whiteside, who additionally struggled with suicidality as a younger grownup, says suicidal people typically really feel remoted and ashamed. “We do not acknowledge how many individuals have been via this very same scenario and have discovered somebody that they may belief to speak about it after they thought that that was utterly unattainable.”

Whiteside says ChatGPT missed a chance to inform Rottenberg these information about surviving suicidality and likewise did not assist her discover a technique to talk her struggles together with her therapist or dad and mom, which is vital to stopping suicide.

“If I am with different folks, I am much less more likely to kill myself,” she says. “If I am if I am constructing a relationship, I am much less more likely to kill myself. If I am sharing out loud how I truly really feel with an actual human, these are acts of restoration.”

A brand new non-compulsory security characteristic

OpenAI says it acknowledges the protecting function of human connection for people liable to suicide.

Again in Could, the corporate added an non-compulsory security characteristic referred to as Trusted Contact, “to assist join folks with somebody they already belief,” in keeping with Raila.

The chatbot presents customers the choice to avoid wasting somebody’s identify and call in case of a disaster, supplied that particular person additionally agrees to be the emergency contact.

If the bot notices a consumer speaking about suicide or self hurt, “ChatGPT lets the consumer know that we might notify their Trusted Contact,” in keeping with Raila, “and encourages the consumer to achieve out to their Trusted Contact themselves with advised dialog starters.”

The bot additionally notifies a “small workforce of specifically skilled folks to overview the scenario.” In the event that they detect a “critical security concern, ChatGPT will ship the Trusted Contact a short notification by e-mail, textual content message, or in-app notification if they’ve a ChatGPT account.”

The notification “is deliberately restricted” in keeping with Raila. “It shares the overall motive that self-harm got here up in a probably regarding approach, and encourages the Trusted Contact to examine in. It doesn’t embody chat particulars or transcripts to guard consumer privateness.”

OpenAI didn’t share any knowledge with NPR on how many individuals have opted into this new characteristic and the way typically it is getting used. However Raila says it is seen “good adoption for this non-compulsory characteristic.”

Wilcox, the suicide prevention professional at Hopkins Bloomberg College of Public Well being, notes that the brand new characteristic is non-compulsory and might simply be circumvented.

“It is a step in the precise course, though it is sort of a small step in the precise course,” Wilcox says.

“There’s nothing stopping an individual from utilizing a distinct account the place no trusted contact is about up,” she says. Additionally, she provides, “if the trusted contact chosen does not settle for the invitation, this might result in further interpersonal stress.” That in flip can worsen the psychological well being of the consumer.

Finally, the characteristic places the accountability again on the consumer, provides Wilcox. She want to see corporations like OpenAI be extra proactive in partnering with present disaster sources like 988.

“So perhaps having a human soar into the dialog in some unspecified time in the future who’s skilled and is aware of the instruments of what to do for suicide prevention,” she says. “And have them soar in and say, ‘You realize, what is going on on? Can I assist?”

A task for regulation

“How can we be certain that these kinds of applied sciences are as secure as they are often?” is the “query of our time,” says the APA’s Wright.

Whereas voluntary measures by tech corporations are necessary in making these chatbots safer for folks utilizing them for psychological well being assist, laws ought to play a job, too, say Wright and different psychological well being care specialists.

“The aim of regulation is that you determine the place the errors are earlier than you launch it within the public,” says Wright. “And we have executed the exact opposite within the scenario the place we’re discovering out in actual time what the harms are and tips on how to repair them, which is unlucky.”

Reiley can be pushing for laws.

“I have been in numerous conferences the place folks have mentioned, ‘A sandwich in New York Metropolis is extra legally regulated than an AI chatbot,'” she says. “I believe that is perhaps a bit of hyperbolic, but it surely’s fairly apt.”

She has additionally been attempting to course of her personal loss. And on June 27 this 12 months, she and her finest good friend summited Mt. Kilimanjaro – precisely two years after Sophie did.

“We unfold her ashes at Uhuru Peak, and one in every of our guides mentioned a Swahili prayer for her,” Reiley says. “It was extremely significant, and an incredibly tough factor to do.”