Your chatbot has no seal of confession: AI is already testifying in court, and the state knows where to look

THE THESIS. The chatbot has become the confidant of millions, but in the eyes of the law it is just another third party. What you type into ChatGPT, Claude or Snapchat My AI can reach a prosecutor three ways: a court order, a search of your device, or a tip from the company itself. Our position is clear. In the short term there is a real risk to civil liberties: the state has found the most complete record of private life that has ever existed. In the long term this is fixable, and the fix is not to stop using AI. It is to create a dedicated legal privilege, narrow and bounded, just as we once invented one for doctors and lawyers.

WHAT THE JUDGE SAID. The case that changes everything is United States v. Heppner. On February 13, 2026, federal judge Jed Rakoff in New York settled a question many considered open. Bradley Heppner, an executive charged with securities fraud, used Claude, Anthropic's assistant, to analyze his legal situation and draft defense arguments. He did it on his own, without his lawyers asking. The FBI searched his home and seized about 31 documents containing those conversations, according to an analysis by the law firm Orrick. Heppner moved to exclude them. He invoked two protections: attorney-client privilege (the rule that keeps what you tell your lawyer out of court) and work product (the protection for materials prepared for the defense). Rakoff rejected both. His reasoning is simple: an AI holds no law license, so talking to it is not talking to a lawyer. And he added something more serious, as Orrick and Husch Blackwell note: even a document that was protected can lose that protection if you share it with a public AI.

Our take: the ruling is legally sound and socially explosive. Rakoff invents nothing. He applies a centuries-old rule: confidentiality is lost when you tell a third party. What's new is that this "third party" is now the tool millions use precisely to think through legal problems before paying a lawyer. The practical effect is a class divide. Those who can afford a lawyer get a protected space to plan their defense. Those who can't, and turn to free AI, leave a trail prosecutors can read.

CHATS ARE ALREADY EVIDENCE. This is not hypothetical. In August 2026, The Washington Post reviewed public records and local news and found AI chats cited in 12 court cases over the past two years. The number sounds small, but it is growing fast. According to the same report, as cited by Malwarebytes, OpenAI disclosed data from more than 80 accounts in the second half of 2025, over four times the rate of the same period in 2024. The best-known cases vary widely. In the Palisades fire in Los Angeles, federal prosecutors presented the ChatGPT activity of Jonathan Rinderknecht, accused of starting the blaze. The evidence includes, according to court documents, an image generated months earlier of a burning forest and a question to the chatbot about whether someone could be blamed for a fire started by their cigarette. The trial ended in a mistrial in June and is set to be retried starting October 19. In Virginia, 18-year-old Ja-Zion Robertson was sentenced to 25 years for first-degree murder. Before the killing he had asked Snapchat My AI what would happen if he shot someone who stepped onto his property "with hostile intent."

Our take: two things that the debate often mixes up need to be separated. One is using a chat as evidence in a case already under investigation, just like Google history or WhatsApp messages. That is normal in a society governed by law, with judicial oversight. The other is more delicate. A chat is not like a Google search. It is like a private diary that talks back. People write their fears, fantasies and worst moments into it with a level of detail they would never put into a search box. A prosecutor can take a sentence written at 3 a.m. out of context far more easily than a three-word query.

WHEN THE COMPANY CALLS THE POLICE. The third front is the newest, and the one that struck the biggest chord. Here nobody requests the data: the company sends the alert. According to Tom's Hardware and The Next Web, Anthropic has referred at least three conversations to authorities since August. On August 11, a 22-year-old man in San Antonio asked the AI about shooting at a nearby elementary school; police, alerted by the FBI, arrested him on a terroristic threat charge. On August 14, a San Francisco user threatened Anthropic CEO Dario Amodei in a chat and said he had bought an AR-15 rifle; police did not arrest or charge him. The most discussed case is that of Carli Heller, 30, in Florida. According to the Lee County sheriff, Heller wrote on September 26 that she was going to "shoot up" the sheriff's office and, the next day, that she had bought a new gun. Anthropic's automated system flagged the messages, a human team reviewed them, and the company alerted authorities. Heller is charged with a felony for written threats, punishable by up to 15 years in prison according to Gizmodo. She told deputies she used Claude as a "diary." OpenAI applies an equivalent policy: if its human reviewers see an imminent and serious threat to others, it may notify police. The company says it does not do so in self-harm cases, out of respect for privacy.

Our take: our position here is less comfortable than easy outrage. If someone announces they will shoot up a school and says they have the weapon, alerting police is defensible. Anyone who read it would do the same. The problem lies elsewhere, and experts point to it. Eugene Volokh, a First Amendment scholar (the US free-speech protection), doubts Heller's case is a punishable threat. His argument, reported by Straight Arrow News: Florida law usually requires that the writer intended the message to reach someone, or knew it would. John Coleman of the group FIRE makes the same point. If you write something believing only a machine reads it, the key element of a threat is missing: a recipient who feels threatened. In other words, the company may have good reasons to alert police, but that does not make the diary a crime. Heller's trial will decide; her arraignment is set for November 2.

POWER NOBODY VOTED FOR. The core issue is who decides. Today, the threshold for sending your conversation to police is set by a private company in its usage policy. An algorithm filters, a small team reviews, and the company decides. There is no prior judge, no notice to the user, and no public, verifiable standard for what counts as a "credible threat." It is the same asymmetry we have already seen in public-sector AI that denies benefits: acting is cheap and instant, while defending yourself is costly and slow. OpenAI CEO Sam Altman acknowledged it in 2025: talking to a therapist, lawyer or doctor has legal protection, and with ChatGPT "we haven't figured that out yet." He proposed an "AI privilege." The idea is good, but it should also be read as commercial self-interest: trust is the product. Meanwhile, court orders such as the one in New York Times v. OpenAI have forced the retention of chats users believed were deleted.

IMPLICATIONS. For users, the practical rule is simple: treat every chat as a document a judge might one day read. If you have a serious legal problem, talk to a lawyer first; if you need AI, use it at their direction and through tools contracted with safeguards. For lawmakers, the challenge is to design a narrow privilege. It should protect intimate, medical or legal consultation, and not protect active preparation of a crime, just as medical confidentiality has exceptions. It should require a warrant to access content, transparency about how many referrals are made, and after-the-fact notice to the person affected. For companies, the demand is to publish their thresholds, their referral numbers and their error rates. Our long-term bet is optimistic, but not naive. A confidant available at 3 a.m., one that helps you understand a diagnosis or a contract, is one of this technology's great advances. It can bring health care and justice within reach of people who could never afford them. But that advance only holds if people can trust it. It took centuries to build professional confidentiality. With AI we don't have centuries, and the decision is already being made, case by case, by courts and company policies.