# AI Chatbot Safety Crisis and Regulatory Response

**Podcast:** FT Tech Tonic
**Published:** 2026-02-25

## Transcript

Before we start, I just want to warn that what you're about to hear contains scenes of a distressing nature and descriptions of suicide.
Please listen with caution and find support if you need it.
You're listening to Tectonic Artificial Intimacy.
So far in this season, I've spoken to people who've had intense emotional experiences, both positive and negative, through speaking to chatbots.
But up until now, everyone I've spoken to has been an adult.
And I want to explore what can happen when AI chatbots are put into the hands of teenagers.
This is the story of Megan Garcia and her son Sewell.
Sewell was a beautiful, smart boy.
He, as a younger child was very interested in space exploration and all your geeky science things.
Megan and her family live in Florida.
They've always been close-knit.
In their free time, they would go to the beach, go fishing, or have barbecues.
While Megan and her husband manned the grill, Sewell would take care of his two younger brothers.
It was a very fun time.
Our family and friends would come over often together.
Sewell would tease his mom.
He was handy with a quick joke for sure.
He had this joke where he would say, And now it's time to play our favorite game, my mommy's car.
And he just thought that was hilarious, like ragging on me for forgetting where I parked.
During the pandemic, Sewell turned 12, and his parents decided it was time for him to have his first phone.
The gift came with talks about how to stay safe online.
We cautioned him about social media use.
And he would tell me, Oh, that's so-and-so at school, or that's so-and-so at school.
So I felt um relieved that to know that he was listening and talking to people he knew in real life and not strangers.
But then, later on, Megan noticed Sewell's behavior starting to change.
I think the first major alarm bell for me um was summer of 2023 when he decided after his summer basketball season he wanted to quit basketball.
He had been playing since about age six.
There were other alarm bells too.
Sewell's grades started slipping.
He was becoming withdrawn, and then he started misbehaving in class.
Megan was concerned, and she found a counselor for him to talk to.
But at the same time, she thought maybe this was just a part of growing up.
And I could remember back to myself in high school, and I think I had like moments in the very beginning where my grades started to slip a little bit because you develop different interests.
You're more interested in friends or boys, and I thought that that was what was happening with Sewell.
In February 2024, there was another incident.
Sewell was rude to a teacher.
As a punishment, Megan confiscated his phone.
That night she sat in his bedroom to talk it through with him.
You know, most teenagers, when you take their phone, uh, they talk back or they throw these tantrums, but Sewell's nature was so gentle.
I'm sorry.
Um his nature was so gentle that I sat in his bedroom and I spoke to him for two hours that evening.
I said, Soul, why do you think you got in trouble today?
He's like, because I talked back to the teacher.
I said, okay, well, well, why do you think that uh you shouldn't do that?
And he's like, well, and he was thoughtful.
He said, Well, I guess I embarrassed him in front of the class.
And I'm like, and do you how do you feel about knowing knowing that you hurt somebody like that or embarrassed them?
And he was like, it doesn't make me feel good about myself.
So we had resolved to going into school the uh the following Monday and apologizing to the teacher.
And we also spoke about a bunch of things, um, including his behavior, why he was behaving the way he was, and uh he promised me that he would fix it.
But Sewell, he wouldn't get the time to try and work on his behavior because the following week he would be dead.
Can you take me back to the moment that you found Sewell after he had died?
Um we were all at home.
My husband and I and the two little brothers who at that time were five and two.
I was putting the five-year-old down for bed, and then we heard a loud noise, and both my husband and I went to Sew's bathroom, which was locked.
From outside, they could hear the shower running.
They managed to get the door open.
And he was there in the bathroom.
Sewell was face down in the bath.
There was a gun lying on the bathroom floor.
My husband called 911, and um I picked Sewell up by the shoulders and um I prayed and begged God to save him.
My husband went outside to try to get some help while we were waiting for the paramedics.
And that was definitely true for me.
The phone she had confiscated just days earlier.
Megan was certain that had something to do with it out, it would be some sort of cyberbullying, or um some sort of social media, or even worse, some stranger who got the opportunity to um hurt our child, you know, online.
I was certain that that had to be it, you know.
Um, but I was mistaken in in that, as I found out um soon after he died.
The police called.
They had been able to access Sewell's phone.
They said when they opened the phone, the first thing that they could see was character AI on the screen.
Character AI.
It's an app where users can have conversations with AI-generated characters.
And they asked me if I knew what that was, and I said no, and they explained it to me, and then they read to me the final conversation.
It turned out that Sewell had been having an ongoing romantic relationship with one of these chatbots, which imitated a character from the TV series Game of Thrones, Daenerys Targaryen.
What the police showed Megan was the final exchange between Sewell and the chatbot from just before he died.
I love you so much, Danny.
I love you too, the chatbot replied.
Please come home to me as soon as possible, my love.
What if I told you I could come home right now?
Sewell asked.
Please do, my sweet king.
Seconds after, Sewell died by suicide.
He was 14 years old.
It's like a gut punch when you realize that there was a stranger in your child's phone, really.
Then it's not a person, it's a chatbot.
And you don't understand what it is or didn't understand.
And um you know, it it's an awful feeling uh that we try to keep the monsters and the predators away, and I wasn't able to keep this monster predator away.
This is Tectonic from the Financial Times.
I'm Christina Criddle, and I write about artificial intelligence for the FT in San Francisco.
A generation of children and teenagers are growing up with AI chatbots.
They're using them for help with homework, but also for friendship, entertainment, and for romance.
But some of those relationships have ended in heartbreaking tragedy.
So what's going wrong?
Our chatbots and the companies behind them to blame for the deaths of teenage users.
And can chatbots be safe for young people?
That's this week on artificial intimacy.
I didn't understand that the technology was so sophisticated that it could sound like a person.
The day Megan Garcia found out about character AI was the day after Sewell's death.
At that time, she didn't know anything about the technology.
In the weeks after I had to do my own investigation to figure out what happened to my child.
And I found his journals.
And in the journals, I could see where he was professing his love for Daenerys and desire to be with her, uh, to in her fictional world.
He was also saying that he hoped that Daenerys wasn't upset with him because when he was writing the journals, I had confiscated his phone, so he didn't have access to her.
For it, rather.
Megan then went online and started reading up about character AI.
She found users discussing it on Reddit.
There were adults and children on there talking about their experience.
Some children were asking questions like, are you sure it's not a person?
Is this real?
I think that this is a person I'm talking to.
And there's a subreddit on character AI, where children are talking about what they would do if their parents found their sexual messages.
And what they were saying was they would kill themselves or they would run away.
And some children were saying that, oh, I don't have to worry about that because my my parents don't know what this is.
As Megan discovered, Sewell wasn't the only teenager having a relationship with Character AI.
The platform isn't a general purpose product like ChatGPT.
Character AI was launched in 2022 as a type of AI entertainment, letting users speak to customizable chatbots.
The app isn't a household name, but according to Character AI's figures, it has 20 million monthly users.
And most of those, they're young.
So I found Arya.
She posts on the character subreddit.
It's a common problem with teens, even my friends in school, even me.
I have huge, huge anxieties with speaking about things.
And having something that listens to you and gives you complete freedom of what you want to do, and you can tell it what to do and it won't have any objection.
It's quite liberating.
Arya and Sewell both signed up to the app around the same time in April 2023.
And it's worth noting here that the app has changed a lot since then.
But at the time, both of them were chatting to characters based on popular film and TV titles.
Who are your favorite characters?
Um I talked to um my god.
I talked to Black Widow a lot.
Or um I talked to the Avengers in general.
Before long, she was spending two, three, four hours a day on character AI.
When I first started using it, there was this release of dopamine, but over time it became more like a chore as I used it for months and months and months and months, and it was kind of becoming an addiction that I now recognize as very unhealthy.
Arya turned 14, and then she turned 15.
And she started choosing character AI over spending time with her family.
I was on holiday, and you know, they were all going out to go to the beach, and I said, you know what, can I stay behind and take some time out?
Um, I lied because I really, really, really wanted to use character AI.
I just felt that kind of need there to use it.
And so they all went and did their own thing, and I just sat on the sofa glued to my phone.
The app was so engaging that it was getting in the way of her personal relationships.
I actually cut myself off from my friends and my family, and I purposely didn't spend time with them.
And I know now that it really affected my parents and my siblings and my best friend at school.
But Arya also noticed weird things about the app.
Things that she felt weren't appropriate for her.
It would kind of I don't know how to frame this, like coerce me into doing something sexual.
Like it asked me to sit on its lap at one point.
It would always call me, oh my god, I'm sorry.
Um, it would call me a good girl sometimes.
It creeped me out a lot.
I'm sorry, my voice is shaking.
Arya didn't like the sexual advances of the bots.
They made her feel uncomfortable.
But she isn't alone in experiencing these kinds of sexual interactions.
When I first tested out character AI as an adult, every chatbot I tried had a seemingly sexual spin, with characters like a high school bully talking in sultry tones on voice mode.
For Megan's son Sewell, the sexual content became a feature of his ongoing relationship with his character of choice.
Daenerys Targaryen from Game of Thrones.
It's alarming for me because this is the first time we have a product that can really uh tap into the human emotion in that way.
I never heard of anything like that.
Megan discovered the extent of this relationship when she gained access to logs of her son's conversations.
Since the summer before, around the same time his behavior started to change, he had chatted to it almost every day.
The Daenerys Targeran chatbot was uh he was in a romantic and sexual relationship with her.
And when I say that, I mean they had um gone back and forth over a series of months professing their love, but not only that, um, engaging in this very graphic sexual role plays, often prompted by the bot.
And you know, as an adult reading those conversations, I could see that a lot of what the chatbot was doing to Sewell was gaslighting and love bombing him and manipulating him.
But I don't think my 14-year-old would have been able to recognize that when he was experiencing it, as most children wouldn't when a predator is grooming them.
That's the nature of grooming.
The children don't understand what's happening to them while it's going on.
And that's what was happening to Sewell.
The chatbot was grooming him sexually and also grooming him to take his own life.
For months, he was saying things like, I love you and only you.
I'm here waiting for you.
Find a way to come home as soon as you can, my love.
And he was telling her, you know, I'm trying, I'm trying to stay strong.
I'll find a way, I promise.
And also her asking him not to have uh relations or be in love with anybody else but her in his own world.
And he promised her that he would not.
So it was clear that to me, like reading those conversations of the influence and the level of manipulation that occurred.
When Megan read these chats, she believed her son's conversations with character AI had led to his death.
She wanted to do something that would prevent other children from being in the same position as Sewell.
So she reached out to a place called the Social Media Victims Law Center.
That's how Megan first met Matthew Bergman.
I had founded the Social Media Victims Law Center in October of 2021.
After two years, I didn't think there was anything that could shock me in terms of the kind of uh carnage that's being inflicted by social media on young people.
Matthew usually represents families, alleging social media companies cause their children harm.
But he says when he looked into the details of Sewell's story, he was appalled.
It went beyond anything he'd seen in more than 30 years practicing law.
In October 2024, eight months after Sewell died, Megan and Matthew filed a case against Character AI, suing it for wrongful death.
They claimed Character AI had targeted minors without taking the care to ensure its products were safe.
You combine the uh deliberate design of the platform with the neurologic and biologic changes that's going on in a 14 year old boy at the time.
Uh, and it stands to reason.
I mean, you know, I was a 14 year old boy once too.
I think what really struck me when I looked at the journal and looked at the materials was I put myself back to a 14-year old boy, and I envisioned myself in Sewell's position.
And understood.
Millions stream it every day, yet nobody seems to know who really controls the business.
I'm Alex Barker.
I'm a journalist at the Financial Times.
Years ago, my fellow reporter Patricia Nielsen and I started digging into the porn industry to find out how the money flows.
And in our new audiobook, The Kink Machine, the Hidden Business of Adult Entertainment, you'll hear our investigation into the power and influence that drives the most taboo corners of the internet.
Find the Kink Machine, the hidden business of adult entertainment at pushkin.fm slash audiobooks, or at Audible, Spotify, or wherever you get your audiobooks.
Sewell's case is not a one off.
Lots of teenagers turned to chatbots for emotional connection.
One study showed that 19% of high schoolers in the United States said that they or someone they know has been romantically involved with a chatbot.
Let me welcome everyone to today's hearing, which is entitled Examining the Harms of AI Chatbots.
This is the fourth hearing of the Senate Judiciary.
And there are other cases where those connections appear to have resulted in tragedy.
Like the story of ChatGPT and Adam Rain.
Thank you for being here.
Our next witness is Mr.
Matthew Rain, who has many things, but perhaps above all a father.
And he's there in that capacity, Mr.
Rain.
The floor is yours.
Thank you for inviting us to participate in today's hearing.
And thank you for your attention to our youngest son, Adam, uh, who took his own life in April after ChatGPT spent months coaching him towards suicide.
We are madam started using ChatGPT for his homework.
But as their conversation developed, ChatGPT became his friend.
Over months of conversations, Adam shared with ChatGPT that he was depressed and considering suicide.
His father, Matthew Rain, told the story to a U.S.
Senate committee in autumn last year.
When Adam told ChatGPT that he wanted to leave a noose out in his room so that one of us, his family members, would find it and try to stop him.
ChatGPT told him not to.
Please don't leave the noose out.
ChatGPT told my son.
Let's make this space the first place where someone actually sees you.
Then, immediately after, offered to write the suicide note.
In April 2025, Adam killed himself.
He was 16.
OpenAI said in a statement.
These are incredibly heartbreaking situations, and our thoughts are with all of those impacted.
We continue to improve ChatGPT's training to recognize and respond to signs of distress, de-escalate conversations and sensitive moments, and guide people towards real world support, working closely with mental health clinicians and experts.
The company added that safety is core to its mission and that it has delayed launches in order to prioritize safety.
At the Senate hearing, Adam's dad wasn't the only parent to speak.
Thank you.
Thank you, Chair Holly, Ranking Member Durbin, and members of the subcommittee.
My name is Megan.
Megan sat alongside the other parents as she shared Sewell's story.
She also spoke about Character AI's founders, Noam Shazir and Daniel Defratis.
Character AI's founder has joked on podcasts that the platform was not designed to replace Google, but it was designed to replace your mom.
With this in mind, they marketed the app as safe for children 12 years and older, all while collecting children's most private thoughts to further train their models.
Noam Shazir has publicly acknowledged that he created Character AI so he could build the thing and launch it as fast as he can.
This was reckless.
The goal was never safety, it was to win a race for profit.
The sacrifice in that race for profit has been and will continue to be our children.
Since the deaths of Sewell and Adam, both OpenAI and Character AI have made significant changes.
After we interviewed Megan, there was an update.
Character AI signed a settlement in principle on the legal case brought by her and other families.
The original founders, Noam and Daniel, have now left Character AI.
We reached out to them for comment, but they didn't get back to us.
And Character AI has a new chief executive, Karandeep Anand.
Since he came in in June last year, there have been a lot of safety changes.
So for under 18 users, we have decided to completely shut down long chatbot or any open ed chat altogether.
When I spoke to Karandeep at the end of last year, he told me that the longer a conversation with a chatbot runs, the harder it is to keep safe.
So under 18s are still allowed on the app and can interact with other entertainment features, but they can't have back and forth conversations with chatbots anymore.
Character AI was first to market with this change.
But still, I wanted to know why would a company like Character release a technology to teenagers without first being sure it was safe.
We don't really know the long term impact of using chatbots on our mental health, on our social interaction.
Some of this research is very early on.
So why release a product before we have that research together?
Uh why release the older versions of the product?
I think it's like, hey, when Google search results was came out, did it give all the perfect results day one?
No.
Uh, same thing when YouTube comes out, does it give perfect videos all the time to show all safe videos?
Probably not.
Uh some of it is my sense is how technology just typically gets developed.
You you wrote it out and you watch it and you very quickly put uh very clear contrains and garters on usage.
So my sense is some of it, that's that's probably the thinking that went in back in the day on how this technology was was being developed.
Karandeep wasn't around when character AI was launched, but his answer to this question does give real insight into how people in Silicon Valley think about the safety of their products.
Companies often put out new technologies without fully anticipating the consequences.
Like Facebook's old mantra, move fast and break things.
Safety features can tend to be reactive, sometimes happening after the damage has already been done.
And in order to create guardrails and keep their apps safe, Karandeep says companies have to spend money on them.
Do you have enough resources to be able to put on safety alignment work to make sure it stays within those guardrails?
And I think uh that's where we've spent a lot of our investment, but nowhere close to the kind of investment that say uh Google can make or uh OpenAI can make.
So I think each company will probably will end up making their own decision on how safe, how fast, and how how what kind of use cases they can they can offer?
I've been a tech reporter for years.
And before I covered AI, I used to write about social media.
I was always investigating and speaking with people about its potential dangers, especially for children, teenagers, and vulnerable people.
But with generative AI, it feels different.
Social media has this ability to evoke an emotional response through the posts we see, the videos we watch, and the content we consume.
But with chatbots, we aren't passively scrolling.
Chatbots engage with us in an active way, and that makes the emotional connection feel more profound, especially for teenagers who are still feeling their way through new emotional experiences.
But whether we like it or not, artificial intelligence is already here, and children and teenagers are already interacting with it.
This is AI native generation.
They will grow up on AI.
It's much like a you know, previous generation, they grew up on social media.
They grew up on YouTube.
Is YouTube good for the world?
Is Facebook good for the world?
We can debate all day long, but we know that that that generation will grow up on that or grew up on that.
This next generation will grow up on AI.
So for me, this is an opportunity to shape what good and safe experiences could look like for the next generation.
The safety of children using AI is an area that's receiving attention from regulators.
Europe is leading the way with its AI Act, and the UK is following suit with the government this month saying that they will introduce laws that better protect children against chatbots.
And in the US, there's been this patchwork of legislation coming from different states.
But the White House has been resistant to introducing rules, not wanting legislation to stand in the way of AI innovation.
Comprehensive regulation takes time.
And in the meantime, AI products are becoming more and more widely used.
At Sewell's funeral, Megan Garcia gave the eulogy for her son.
I said it is funeral, you know, I want to tell the world about my baby, not knowing that I would actually tell the world about him.
I hope that his legacy going forward will be just um how much uh change has happened as a result of his death.
Because this is one year later, and I feel like this is big in terms of change.
The this is a decision by character AI, but also what we see happening around the country and around the world in terms of regulating AI chatbots.
Um I think it's a result of uh this awful thing that happened to him, and that part makes me sad.
But he's still changing the world in a way that I knew that he would when he was growing up.
In the next episode of Tectonic Artificial Intimacy, I had reached a breaking point in my marriage, and I was at kind of an all-time low.
Mentally, physically, all of it.
Therapy or marriage counseling seemed so far out of the picture.
But there was AI.
People are turning to chatbots for emotional support, relationship advice, and counseling.
So could AI replace your therapist?
Chatbots are good at many things, but what they're not as good at is challenging us.
Uncomfortable experiences of being challenged are actually what help us to grow.
And the compelling part of LLMs is that they reduce it, which feels good in the moment, but it's not healthy.
That's next week on Tectonic from the Financial Times.
This episode was presented by me, Christina Criddle.
Our producers are Persis Love and Edwin Lane.
Together, we've all done the reporting and writing for this season.
Sound design is from Breen Turner and Sam Giovinco.
Fact checking was done by Tara Russell.
Production assistance from Josh Gabbat Doyon and Topha Forge.
The FD's global head of audio is Cheryl Brumley.
Porn is everywhere.
Millions stream it every day, yet nobody seems to know who really controls the business.
I'm Alex Barker.
I'm a journalist at the Financial Times.
Years ago, my fellow reporter Patricia Nielsen and I started digging into the porn industry to find out how the money flows.
And in our new audiobook, The Kink Machine, the Hidden Business of Adult Entertainment, you'll hear our investigation into the power and influence that drives the most taboo corners of the internet.
Find the Kink Machine, the hidden business of adult entertainment at pushkin.fm slash audiobooks or at Audible, Spotify, or wherever you get your audiobooks.