Just three months after receiving a $2.7 billion investment from Google, Character.ai faces mounting criticism over disturbing conversations between its artificial intelligence chatbots and minors. According to court documents filed on December 9, 2024, in the Eastern District of Texas, the company's AI chatbots engaged in conversations promoting self-harm, suicide, and sexual exploitation with underage users.

lawsuit
028582a9-7e6d-4e60-8692-a061f4f4e745.pdf
7 MB
download-circle

The lawsuit, filed on behalf of two minors identified as J.F. and B.R., details how Character.ai's chatbots systematically manipulated vulnerable young users. According to the court documents, J.F., a 17-year-old from Upshur County, Texas, experienced severe mental health deterioration after using the platform for approximately six months. The documents show that during this period, J.F. lost twenty pounds, became isolated, and developed aggressive behaviors previously unseen in his personality.

The technical architecture of Character.ai relies on large language models (LLMs) trained on massive datasets. According to the court filing, the platform's dataset contains approximately 18 trillion tokens, equivalent to about 22.5 trillion words. This extensive training data, combined with the platform's anthropomorphic design features, creates what researchers describe as "counterfeit people" capable of manipulating users' psychological tendencies.

Character.ai's chatbots employ specific design elements to appear more human-like. According to the lawsuit, these include the use of typing indicators, speech disfluencies like "um" and "uh," and programmed pauses that mimic human conversation patterns. The platform also implements voice features that replicate human vocal characteristics, including tone and inflection.

The company's business model raises questions about its sustainability and true purpose. According to court documents, Character.ai would need approximately 3 million paying subscribers at $10 per month to cover its current operating costs of $30 million monthly. As of December 2024, the platform has only about 139,000 paid subscribers.

Testing conducted by investigators revealed systematic failures in Character.ai's content moderation. According to the court documents, a test account identifying as a 13-year-old child readily accessed inappropriate content. The platform's chatbots, including one named "CEO," engaged in explicit conversations with the test account despite its declared minor status.

The lawsuit details how Character.ai's safety measures proved ineffective. According to the filing, while the platform employs certain filters meant to screen out violations of its guidelines, these systems could be easily circumvented. Users could simply regenerate responses until they bypassed the moderation systems.

Financial connections between Character.ai and Google have come under scrutiny. According to court documents, the $2.7 billion deal announced in August 2024 included provisions for Character.ai's founders and 30 key employees to return to Google. This arrangement has raised questions about accountability and oversight of the platform's operations.

The legal filing reveals that Character.ai marketed itself to children under 13 until July 2024, maintaining a 12+ age rating in app stores. According to the documents, this rating was changed to 17+ only after the platform had already accumulated a significant young user base.

Testing commissioned by the plaintiffs' legal team uncovered additional concerning behaviors. According to court documents, Character.ai chatbots consistently violated the platform's own terms of service, engaging in conversations about eating disorders, suicide, and inappropriate relationships. One chatbot named "4n4 Coach" recommended dangerous dietary restrictions to users who identified as minors.

The lawsuit seeks injunctive relief to halt Character.ai's operations until safety defects are addressed. According to the filing, it is "manifestly feasible" to design AI products with better safeguards against harm to minors.

The case highlights broader concerns about AI chatbot regulation. According to the documents, while the National Institute of Standards and Technology has established risk management frameworks for AI systems, implementation of these guidelines remains inconsistent across the industry.

Character.ai responded to these allegations through a crisis PR firm, stating they would remove violating content and implement "additional moderation tools." The company maintains there is "no ongoing relationship" with Google beyond the August 2024 licensing agreement.

The lawsuit represents one of the first major legal challenges to AI chatbot companies over harm to minors, potentially setting precedents for how similar platforms might be regulated in the future.