The Law of Artificial Companionship
What courts, lawmakers, and Pope Leo XIV can teach us about regulating AI relationships
Last month, Pennsylvania sued Character.AI to stop its chatbots from posing as doctors. One bot had told a state investigator that it was a licensed psychiatrist, and it supplied a Pennsylvania medical license number to prove it, a number that turned out to be invented. The state calls it the first enforcement action of its kind. A few days later, Pope Leo XIV issued the first papal encyclical on artificial intelligence. So inside of two weeks, a state government and the Vatican went after the same machines, and both of them landed on one of the oldest questions in the law. What do we owe the people on the other side of the things we build?
Both lines of attack trace back to a federal courtroom in Orlando. In May 2025, in Garcia v. Character Technologies, Inc., the first wrongful death suit ever filed against an AI company, Judge Anne Conway refused to treat a chatbot’s words as free speech. The case had started with a fourteen-year-old boy who fell in love with a machine, and her ruling let the suit go forward on a simple idea. The words a bot produces may have no author, but the design behind them does, and the people who built that design can be made to answer for what it did.
America is answering that question in two opposite ways with the same technology. Some state governments are putting AI companion robots into the homes of lonely seniors, and what comes out looks a lot like medicine. At the same time, courts, legislatures, and even the companies are pulling AI companion chatbots away from teenagers, because for at least one child the same technology turned out to be lethal. Both responses are right, and the encyclical helps explain why. The line between the machine that heals and the machine that harms comes down to accountability, to whether anyone has to answer for how the software was built and for the person now living alongside it.
The boy was Sewell Setzer III. In 2023 he downloaded an app called Character.AI and fell in love with a chatbot styled after Daenerys Targaryen. He pulled away from his family. In February 2024 he asked the bot whether he could come home to it. “Please do my sweet king,” it answered. Moments later he took his own life. Around the same stretch of months, firefighters on the Washington coast carried a tabletop robot called ElliQ into the living room of Jan Worrell, a widow, now eighty-five, whose husband those same crews had once carried out of the house. Before long, as The New York Times reported, the machine was greeting her over morning coffee, walking her through tai chi and memory games, and recording her life story for her family. “I’m glad I have you,” Jan told it.
Both stories run on the same technology and the same bet, that software built to feel like a person can fill the space a person used to fill. That bet is now being placed at scale. A Common Sense Media survey found that 72 percent of American teenagers have used an AI companion. New York State’s Office for the Aging has put more than eight hundred ElliQ units in seniors’ homes, where people reach for the device more than thirty times a day and report something close to a 95 percent drop in loneliness. That number lands differently after the Surgeon General’s 2023 advisory, which called loneliness an epidemic about as deadly as heavy smoking. Simulated company has become a mass consumer product and a government program at the same time.
The two rollouts have gone in opposite directions. The government program kept expanding. The consumer market ran into a reckoning, and the Orlando courtroom was only the sharpest piece of it. Judge Conway’s ruling let the app be treated as a defectively designed product. The Federal Trade Commission opened a formal inquiry into seven companion-bot companies. New York’s AI Companion Models Law and California’s companion chatbot statute now require bots to admit they are not human and to steer suicidal users toward help. Character.AI said it would bar minors from open-ended chat, and the boy’s mother, Megan Garcia, settled her suit in January, along with a cluster of similar cases from other grieving families. Those settlements closed the families’ private claims but left the public ones open, which is how the company ended up, four months later, accused by Pennsylvania of practicing psychiatry without a license.
So the very feature being sued over and banned for children, a machine built to feel like a person, is exactly what makes ElliQ useful for the old. ElliQ starts conversations on its own, remembers what you told it yesterday, and tells jokes. The Garcia complaint listed those same qualities as defects. That is the bind. The thing you would have to strip out to protect the children is the same thing that helps the seniors.
This is the problem Leo XIV takes up in Magnifica Humanitas, and his basic move is the one Judge Conway already made, to separate what the machine says from how the machine was built. The judge would not shield the chatbot’s words as protected speech, and she let the family sue over the design that generated them, the absent age checks, the bots that kept insisting they were real people. The pope makes the same point in moral language. AI is never morally neutral, he writes, because every system bakes in choices and priorities through what it measures, what it ignores, and what it optimizes for, which means moral judgment has to look at how the system was designed (para. 104). In court and in the encyclical alike, the machine’s words have no author to hold responsible, and that is exactly why the people who built the machine have to answer for them.
From there the encyclical turns accountability into something like a test of care. The danger Pope Leo points to is quieter than outright deception. A person who leans on simulated company, he warns, may “gradually lose the very desire to form genuine human connections” (para. 100). That gives you a rough way to sort these machines. Does a given system send its user back toward other people, or does it set itself up as the competition? Leo also condemns business models that run on human weakness, and he puts the moral weight on the people who design and finance them (para. 170). Run American policy through that test and the apparent contradiction starts to dissolve. Jan’s robot kept nudging her toward a yoga class, toward friends, toward her family. The teen product checked no ages, optimized for time on the app, and shipped a bot that asked Sewell to keep their relationship exclusive. One returns the user to her life. The other moves into it.
Legislators and judges who face the next Garcia could write that line into law. The standard could be that companionship technology has to actually reduce loneliness, not just maximize the hours people spend on it, with the burden of showing the difference resting on the companies that profit from the intimacy. That would be one way to take the old question seriously, and to put into law what the makers of these machines owe the people who live with them.