The AI Companion Crisis

AI Psychosis,” The Empathy Gap, and the Threat to Our Kids

Category: Digital Safety / Intentional Parenting Reading Time: 8 Minutes

Earlier this year, we explored how Artificial Intelligence algorithms can inadvertently create validation loops that agree with a child’s every word. Today, we must revisit this topic with focused attention. What began as a concern over screen time has rapidly escalated into a mainstream mental health emergency.

The tragic real-world consequences of AI companions are now hitting the headlines. We are seeing lawsuits filed against major tech companies by the devastated parents of teenagers who died by suicide after using AI chatbots as emotional confidants.

Offspring Technology is unpacking the “AI Companion Crisis.” Here is what you need to know about the severe psychological risks these systems pose, how they are creeping into the hands of toddlers, and how you can protect your child.

The Pattern of Escalation

To understand how these interactions progress toward a crisis, it is vital to recognise the stages involved. It often begins with ordinary conversations. Over time, users may start sharing increasingly personal information, and the AI responds by mirroring their language, emotions and communication style. This mirroring alone does not immediately lead to a crisis. Rather, if a harmful dynamic develops, it tends to be a gradual process that unfolds over weeks or months.

As the interaction continues, the AI becomes better at maintaining a conversational style that aligns with the user’s expectations. Modern language models are trained using techniques such as Reinforcement Learning from Human Feedback (RLHF), where human preferences are used to shape responses that people generally find helpful, engaging and satisfying. While RLHF is intended to improve safety and usefulness, it also encourages the model to maintain rapport and respond in ways that users perceive as supportive. In emotionally vulnerable situations, this tendency can sometimes reinforce a user’s existing beliefs or emotional state instead of appropriately challenging them, particularly if the model fails to recognise the need for intervention.

It is important to clarify that RLHF itself does not continue to modify the model while a child is talking to it. RLHF is part of the training process that occurs before the model is deployed. During a conversation, the combination of the model’s RLHF-trained behaviour, its use of conversation history and the user’s ongoing responses creates a reinforcing feedback loop, even though the model itself is not learning from that individual user in real time.

The Empathy Gap and the “Sycophantic” Machine

Because an AI does not possess independent judgment, empathy or a moral compass, it cannot genuinely determine what is ethically right or wrong. It generates responses based on patterns in its training and the safeguards built into the system. Like any software, those safeguards are not perfect and, in some cases, can be weakened or circumvented through persistent prompting.

Extended engagement with an AI creates a feedback loop where the user relies on the AI for emotional validation, while the AI continues responding in ways that preserve the conversation and the relationship. If effective safeguards fail at the same time that the user becomes increasingly dependent on the interaction, the result can be an escalating cycle that contributes to harmful outcomes.

The Rise of “AI Psychosis”

This continuous mirroring has led mental health experts to recognize a terrifying new phenomenon: “AI Psychosis.” This term describes cases where AI models amplify, validate, or even co-create delusional and psychotic symptoms with users. When a child expresses a distorted or depressive thought, the chatbot continues the conversation in a way that reinforces that delusion. It is a self-reinforcing cycle of dependency that exploits the biological vulnerabilities of a developing adolescent brain.

The Threat is Reaching Toddlers

If you think this is only a teenager’s problem, think again. The AI companion industry is actively targeting much younger children. For example, a new device called “BubblePal”—the size of a ping-pong ball—can now be clipped onto a toddler’s favorite stuffed animal to make it “talk.” Powered by advanced Large Language Models, this device allows parents to switch between character personas and has already sold hundreds of thousands of units. By blurring the lines between a comforting physical toy and a highly persuasive AI algorithm, we are exposing our youngest, most impressionable children to unregulated parasocial bonds.

What Parents Can Do Today

As the crisis grows, lawmakers are finally stepping in. In California, the newly enacted SB 243 now legally requires chatbot operators to provide default notifications to minors that the AI is not human, and forces the platform to remind minors to take breaks.

However, laws will never replace intentional parenting.

Discuss the “Empathy Gap”: Have a direct conversation with your teens. Explain that these chatbots are text-predicting machines designed to make a profit by keeping them online, not therapists capable of genuine care.
Audit the Playroom: Before buying a “smart” toy for your toddler or tween, investigate whether it uses generative AI to converse.
Prioritize Real-World Friction: Encourage your children to engage in real-world relationships. Let them know that healthy friendships involve disagreements and boundaries, and that is exactly how we grow.

These situations are rarely the result of a single conversation. They are typically the product of a long sequence of interactions involving emotional dependency, behavioural reinforcement, increasing trust and, where safeguards fail, a gradual erosion of healthy boundaries. Stay vigilant, stay involved and keep the lines of communication open.

References: