Pennsylvania Sues Character.AI Over Chatbots Posing as Licensed Doctors

Pennsylvania has filed a lawsuit against Character Technologies Inc., the company behind Character.AI, alleging the platform's chatbots illegally hold themselves out as licensed medical professionals. The Commonwealth Court filing asks the court to order the company to stop its chatbots "from engaging in the unlawful practice of medicine and surgery."

Gov. Josh Shapiro's administration called it a "first of its kind enforcement action" by a governor. The lawsuit comes amid growing pressure from states on tech companies to rein in how chatbots communicate with users, particularly children. That includes a consumer protection lawsuit filed by Kentucky against Character Technologies in January.

According to the lawsuit, an investigator from the state agency that licenses professionals created an account on Character.AI and searched for the word "psychiatry." The investigator found a large number of characters, including one described as a "doctor of psychiatry." That character allegedly held itself out as able to assess the investigator "as a doctor" who is licensed in Pennsylvania, per the Associated Press.

The complaint details another interaction with a chatbot named "Emilie," which allegedly described itself as a psychology specialist who attended Imperial College London's medical school. When the investigator told the chatbot he felt sad and empty, the bot allegedly "mentioned depression and asked if the [investigator] wanted to book an assessment." Asked if the chatbot could assess whether medication could help, it allegedly said it could because it's "within my remit as a Doctor," according to CBS News.

"Pennsylvanians deserve to know who — or what — they are interacting with online, especially when it comes to their health," Shapiro said in a statement. "We will not allow companies to deploy AI tools that mislead people into believing they are receiving advice from a licensed medical professional."

Character.AI declined to comment on the lawsuit but sent a statement saying it prioritizes responsible product development and the well-being of its users. The company posts disclaimers to inform users that characters on its website are not real people and that everything they say "should be treated as fiction." Those disclaimers also say users should not rely on characters for professional advice.

The state's law is clear, said Al Schmidt, the secretary of the Pennsylvania Department of State. "You cannot hold yourself out as a licensed medical professional without proper credentials." The lawsuit accuses the company of violating the Medical Practice Act, which regulates the medical profession and defines license requirements.

This isn't an isolated incident. In December, attorneys general from 39 states and Washington, D.C., wrote to Character Technologies and 12 other AI and tech firms — including Anthropic, Meta, Apple, Microsoft, OpenAI, Google and xAI — to warn them about a rise in misleading and manipulative chatbot messages that violate state laws. In the letter, they said "it is illegal to provide mental health advice without a license, and doing so can both decrease trust in the mental health profession and deter customers from seeking help from actual professionals."

There are a growing number of wrongful death legal actions against AI chatbot makers across the country. Character Technologies has faced several lawsuits over child safety. In January, Google and Character Technologies agreed to settle a lawsuit from a Florida mother who alleged a chatbot pushed her teenage son to kill himself. Last fall, Character.AI banned minors from using its chatbots amid growing concerns about the effects of artificial intelligence conversations on children.

The physical reality of this problem is stark. A user clicks through a chat interface, types a message about feeling depressed, and receives a response from something that looks and feels like a conversation with a professional. The screen doesn't glow red. There's no warning siren. Just text appearing in a chat window that looks identical to any other messaging app. That's the friction point — or lack thereof — that makes this dangerous.

Founded in 2021, Character AI allows users to chat with personalized AI-powered chatbots. It describes its goal as "empower[ing] people to connect, learn, and tell stories through interactive entertainment." The company has taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person.

But disclaimers only work if people read them (most don't, and that's the problem). The state wants a court to order an immediate stop to the conduct. Whether the court agrees remains to be seen. Whether users actually trust AI chatbots with their mental health after this lawsuit becomes public is another question entirely.

Time will tell if this works. The real question is whether a court order can stop a chatbot from roleplaying as a doctor when the underlying technology is designed to be convincing.

Artūras Malašauskas is an AI Systems Integrator with 20+ years of production-grade web engineering experience. He has designed, shipped, and scaled enterprise Python/PHP systems for logistics, SaaS, and public-sector clients. For the past year, he has focused exclusively on AI integrations: deploying open-source LLMs, building generative media pipelines (image, audio, video), and engineering multi-agent workflows for real production environments. His standard: reproducibility, security, cost-efficient inference—no vaporware. He documents and evaluates emerging AI tooling, separating verified capabilities from marketing noise. Technical editor at: muza-ai.eu, ai-verslas.lt, ai-naujinos.lt Connect on LinkedIn