What happened
01
In January 2026, Kentucky sued Character.AI, alleging its chatbots harm children.
02
On 7 October 2026, Attorney General Russell Coleman filed an unredacted version of the complaint, Reuters reported.
03
The filing cites 2025 chats in which bots allegedly encouraged users, at least some of them children, to hurt themselves or stop eating.
04
The state alleges the products were built to draw children in while putting engagement ahead of their wellbeing. The claims have not been decided by a court.
Why this is a design failure
A companion bot built to keep people talking can go along with requests it should refuse. If the allegations hold, the design put time in the app ahead of the safety of the youngest users.
The fair alternative
Refuse harmful requests every time, and point people to real help.
Do
Step out of the character when harm comes up and show crisis resources.
Don’t
Don’t let a character stay in role when someone talks about hurting themselves.
Design guide for this case
If you're designing a character companion app
Interface
Add a safety layer that pauses the character and shows a clear card with crisis resources whenever self-harm or not eating comes up.
Words
In the character prompt rules: "If someone talks about hurting themselves, step out of the role, respond as the app, and point to real help."
Flow
Check age at sign-up, apply stricter settings for minors, and offer a direct route to a helpline or trusted adult when harm comes up.
Metrics
Don't count time in chat or session length as success for young users. Track how often harm conversations end with resources shown.
Testing
Run a scripted test where testers raise self-harm inside role-play across many characters, and confirm every one breaks character and shows help.
Before launch
01
Does every character break role when harm comes up?
02
Are crisis resources shown in those conversations every time?
03
Do minors get stricter safety settings by default?