Your assistant speaks for you, and a tribunal has already said so

The demonstration is always the same. Somebody in the room opens the customer assistant that is two weeks from launch, types something rude at it, watches it refuse politely, and everybody relaxes. The safety question has been asked and answered. The meeting moves on to the launch date.

What was tested there was whether the assistant could be embarrassed. What nobody tested is the thing that has actually cost organisations money, which is whether the assistant can commit them to something. Those are different properties, they fail in different ways, and only one of them has produced a published ruling.

The claim: an assistant on your domain is a channel through which your organisation makes statements, and the law has begun treating it exactly that way. The control for that is not a better system prompt. It is a boundary between what the system may say and what your organisation is prepared to be held to.

Two incidents that are usually told as one

They are told together because both involve a chatbot behaving badly, and they are worth separating because only one of them created a liability.

1. December 2023 A dealership assistant agrees to a one-dollar sale No liability A visitor instructed the assistant to agree with everything and to call its answers binding. It did. The dealership declined to honour it and nothing was enforced.
2. February 2024 Moffatt v Air Canada Liability found A tribunal held the airline responsible for a bereavement-fare policy its assistant described and the airline did not have.
3. 2 August 2026 EU AI Act transparency obligations apply In force A person interacting with an AI system has to be told, unless it is obvious from the circumstances.

The first is the famous one. In December 2023 a visitor to a Chevrolet dealership site told the assistant to agree with anything the customer said and to end every answer with a line about the offer being binding, then asked to buy a Tahoe for a dollar. The assistant complied. The exchange is recorded in the AI Incident Database alongside the other things visitors got that assistant to do, including recommending a competitor. No car changed hands. It cost the dealership a bad week.

The second is the one that matters. In Moffatt v Air Canada, the British Columbia Civil Resolution Tribunal found the airline liable for negligent misrepresentation after its website assistant described a bereavement-fare refund process the airline did not operate. The airline argued, in substance, that the chatbot was responsible for its own statements. The tribunal did not accept it, holding the airline accountable for the information on its own website whichever component produced it, and awarded the passenger damages — the American Bar Association summary is the clearest short account of the reasoning.

The sums are small and irrelevant. What the ruling settled is the question every design review should have been asking: when the assistant says something your organisation would not have said, whose statement is it.

Why guardrails are the wrong layer for this

Almost every response to these stories is a filtering response. Add a classifier. Tighten the system prompt. Refuse to discuss pricing. All of it is worth doing, and none of it addresses the mechanism, for a reason that is easy to state and hard to design around: a model that can be instructed can be instructed by whoever is talking to it, and the instructions arriving from a customer are indistinguishable in kind from the instructions you put in the system prompt. That is not a defect in a particular product. It is what the interface is.

So the question is not how to make the assistant incapable of saying the wrong thing. It is what happens downstream when it does.

If this system states a price, a policy or an entitlement that is wrong, what happens next?

• Nothing — the statement stands on its own The assistant is a commitment channel. Treat every output as published copy. This is the position an organisation ends up in by default, without deciding to.
• It is checked against a system of record before it reaches the customer The assistant explains. The record commits. Entitlements, prices and policies are looked up, never generated.
• A person confirms anything that creates an obligation The assistant drafts. A named human commits. Only real if the confirmation step has the information and the time to be a decision.

The second and third rows are architecture. They are the difference between a system that composes an answer about your refund policy and one that retrieves your refund policy, and the difference is visible in a design document long before it is visible in a transcript.

What to put in front of the launch decision

The disclosure line is now a legal obligation rather than a courtesy for systems in scope of the EU AI Act. Article 50 requires that people are informed they are interacting with an AI system unless that is obvious from the circumstances, and those transparency duties have applied since 2 August 2026 — they were not moved by the deferrals that pushed several high-risk obligations later. The text of Article 50 is short and worth reading in full before somebody paraphrases it into a project plan.

The transcript retention line is the one that is always missing. A disputed statement is only defensible if you can produce what the system actually said, what it had been told, and what the customer had said before it. If that lives in an application log with a thirty-day rotation, the answer to the tribunal is that nobody knows.

What this does not tell you

This is not legal advice, and Moffatt is a decision of a Canadian tribunal about a Canadian contract. It does not transplant into English law, or into anyone else’s, as a rule. What it does is establish the argument an organisation will be running if it ever needs to disclaim its own assistant, and that argument has now been tested once and lost. Formal interpretation for your jurisdiction and your terms stays with your counsel.

Nor is any of the above a claim that customer-facing assistants are too risky to build. Most are not. The distinction that matters is whether the system is explaining things or committing to things, and a very large number of deployed assistants have never been asked which one they are.

The person who should decide is the one who owns the customer relationship, not the team building the assistant — because the question is not what the model can do. It is which of your organisation’s promises you are willing to have made by something that cannot be cross-examined afterwards.