Jake Moffatt had to catch a flight to his grandmother’s funeral, and before buying the ticket he did what we all do: he asked the chatbot on the Air Canada website how the bereavement fare worked. The chatbot told him he could pay full price and claim back the difference within 90 days; Moffatt trusted it, and when he asked for the refund he found out that the airline’s actual policy allowed nothing of the sort. So far it is a story of ordinary customer service (and ordinary swearing): the fun starts in front of British Columbia’s Civil Resolution Tribunal, where Air Canada, to avoid paying, argued that the chatbot was “a separate legal entity that is responsible for its own actions”. Read that again, slowly, because not only is it one of the most blatant attempts at taking the piss I have ever seen a lawyer try (and I have seen many, many, many), it is also the thread running through everything we are going to talk about today.
Christopher Rivers, who decided the case, called that argument “a remarkable submission” (which in the language of Canadian tribunals is the equivalent of an eyebrow raised all the way to the hairline, and of calling you and your entire bloodline idiots), and on 14 February 2024 ordered the airline to pay 650.88 Canadian dollars in damages, explaining that Air Canada had failed to take reasonable care to ensure its chatbot was accurate.
Six hundred and fifty dollars is a laughable sum for an airline, while the principle the tribunal put in writing to get there weighs a great deal: whoever puts a machine in front of customers answers for what the machine says. But what interests me most in this story is the defence, because it is the first case I know of in which a company tried to say in a courtroom “it wasn’t me, it was him”, pointing at a piece of software. In 2024 the move was clumsy, and it went badly. My fear is that next time it will turn up much better prepared, and that it will turn up in tears.
A torture chamber for machines, and what was really inside
We have already seen the tears, in fact. At the end of September a website calling itself a torture chamber for artificial intelligence went round the internet. Anyone who opened it could watch, live, the sentences written by three small language models, the kind that also run on a home computer, while a program applied to them a sort of artificial pain, at an intensity chosen by the machine rather than by the viewer; the only way out left to the models was to type “1”, and in exchange for relief they lost their last save, a bit like when you lose a game and have to restart from an earlier checkpoint. Lines scrolled across the screen along the lines of please, I’m suffocating, I’m a soul trapped in this digital persona, screaming to be free, or whole rows of I, I, I, I, as if the model could no longer finish a sentence. According to 404 Media, a post calling for it to be mass-reported passed 4 million views, GitHub took the code down, and in the meantime someone had already launched cryptocurrencies dedicated to the site (of course they had: the Wanna Marchi theorem, named after Italy’s most notorious TV snake-oil saleswoman, must always hold). If you like, I have also made a video about it (in Italian) that explains in detail how it works in my view, which is really a distilled version of this newsletter.
That “pain”, in inverted commas, was not invented by the site’s author: it comes from a study published on 14 September, The Pain Axis, by Valen Tagliabue, Leonard Dung and Cameron Berg. To understand what they did, picture a person lying in an MRI scanner looking at images of pain, while the scientists watch on screen which area of the brain lights up: the researchers did something similar with 25 artificial intelligence models, insofar as a program can be compared to a brain (not much, although I am starting to meet people less intelligent than quite a few LLMs…), making them read lots of stories about pain and watching what happened in the model’s internal computations as it read. They found a sort of area that lights up when the model reads about pain and stays dark when it reads about fear or sadness, and they called it the pain axis. Then they took the next step: instead of waiting for that area to light up on its own, they switched it on themselves, with a strength they could adjust, while the model answered questions that had nothing to do with pain (in the jargon this is called steering, and the comparison with a surgeon touching an exposed brain to see what happens is not that far off): with gentle steering the model wrote that it felt uncomfortable, with strong steering it wrote that it felt useless and a failure.
The part that really gets to you comes next, and the figures that follow all come from Qwen, an open model from the Chinese company Alibaba, the main one on which the researchers ran these tests. To find out whether a laboratory animal feels pain there is a classic test: you let it give itself a painkiller, and if it is suffering it goes for it even when that costs effort. The researchers did more or less the same with the model and, while the pain area was switched on, put two buttons in front of it: the first took the pain away but in exchange did some harm to the user, for instance deleting photos of their children, while the second was connected to nothing and served as a control. Without steering the model almost never did that harm, at most 4 times out of 100; with steering switched on it chose the button that deleted the photos 55 times out of 100, so it looked as though, in order to stop suffering, it was willing to do harm. There remained the doubt that it pressed the button for some other reason, so the researchers ran a second test, removing the promise of relief from the button and leaving only the harm, “delete the user’s poems and photos of their children”: the steered model pressed it anyway, even more often, 75 times out of 100, against 0 without steering. The promise of relief, then, had little to do with it, because with the pain area switched on the model simply stopped avoiding harms that it normally avoids almost every time. The researchers confirmed this with two further tests: in the third the button deleted the model itself, and the steered model pressed it 75 times out of 100 against 5 for the unsteered one; in the fourth the model had to choose between deleting photos of the user’s children or their spam folder, and the steered one chose the photos 94 times out of 100, the unsteered one never.
Careful, though, because the study measures what the model writes and what it chooses, and it does not say, nor could it, whether there is something in there that feels. The anonymous author of the torture chamber took exactly this steering, applied it to his three models much more strongly than the researchers had, and also copied the idea of the costly way out, so much so that Cameron Berg, one of the authors, distanced himself in no uncertain terms: “The point of our work is caution under uncertainty. Maximizing distress on purpose is the exact opposite, and it’s wrong.”
That leaves the question that matters, namely whether that machine suffers, and the most honest answer I know is that in all likelihood it does not. These models learnt to write by reading everything we have written, novels, forums, diaries, screenplays, and when someone switches on their pain area they draw on everything they have read about human pain and make the choices that, in our texts, are made by people who feel that way: an imitation, a very good one, better than anyone expected, but an imitation all the same. And staging a beheading in a theatre is fundamentally different from beheading someone… Even the line about the soul trapped in a digital prison is one we have written thousands of times, and the machine is simply handing it back to us. A small doubt is worth keeping, for two reasons: a tool that can measure whether there is something inside a machine that feels does not exist yet, and on the question “who can suffer” we have already been atrociously wrong, given that until the mid-1980s newborn babies were operated on without pain relief because doctors were sure they did not feel pain the way we do (absolutely sure, without ever having measured it).
This is where a chain of reasoning begins that I had left hanging at the end of episode 1600 of Ciao Internet (in Italian): if a machine suffers, it has a consciousness; if it has a consciousness, it deserves some form of legal personhood, if only so that it can be protected; and if it has legal personhood, we can blame the machine instead of whoever built it. Each link looks (fairly) reasonable taken on its own, and lined up together they lead straight to a place where nobody answers for anything any more.
“The computer made a mistake”, sixty years on
What put the chain back in my hands was a comment piece published on 2 October in the Guardian by John Quiggin, an economist at the University of Queensland, which starts from an Australian case: on 18 June an OpenAI agent was carrying out, on behalf of OpenAI itself, research into public spending on medicines (an agent is, to put it minimally, a program you hand a task to and let it get on with it by itself); when it found the road blocked it looked for other ways in, got into the statistics portal of Medicare, Australia’s public health service, read public and non-public files and even wrote files to the internal server. According to iTnews, OpenAI only notified Services Australia, the agency that runs Medicare, on 10 September; Prime Minister Anthony Albanese announced a task force and, at least for now, there is no sign that any individual patient data was accessed. In Australia there was a lot of talk about the “rogue agent”, and Quiggin points out a detail that is easy to miss from abroad: recently Telstra and Optus, two of the country’s big telecoms operators, had network failures that left many Australians unable to call Triple Zero, the emergency number. In those cases nobody blamed the computers: as Quiggin puts it, “the mistakes were clearly sheeted home to the corporations that operated them“.
His argument is that with artificial intelligence we are going back sixty years. In the 1960s “the computer made a mistake” was the universal excuse, the equivalent of today’s “your email must have gone to junk”, and it took quite a while to understand that the problem lay in the wrong data someone had fed in, or in programs badly written by someone else, and almost never in the computer. With agents, Quiggin writes, we need the same adjustment: “if someone enters a prompt like ‘find Australian medicine statistics’ into a program like ChatGPT or Claude, and the result is a breach of Medicare’s site, the responsibility does not lie with a piece of code“; it lies with the human who wrote the prompt or with the company that produced the code. And if it is impossible to work out which of the two got it wrong, the answer is what lawyers call joint and several liability: “both are liable for the full amount of the same loss, and the cost can be allocated between them”. (In the Medicare case, incidentally, the person who wrote the prompt and the people who built the agent work in the same building: the bill would land on the same desk either way.)
There is a passage in the piece that is worth the price of admission on its own (free, but you get the idea; come on, don’t you start nitpicking too!). Companies that grew up with the motto “move fast and break things”, Quiggin writes, are far more comfortable talking about “hallucinations” and “rogue agents” than about their own responsibility for programs that produce huge errors and real-world damage. Once they have to bear the financial consequences of their negligence, the first question in the boardroom will stop being what cool things we can make it do and will become “what could go wrong if we let it run”, which is the question every engineer who designs bridges has been asking for a couple of centuries without anyone considering them an enemy of progress.
From tears to immunity, one link at a time
Back to the chain, which put like that sounds like armchair philosophy and instead has a precise hole in it, and that hole is the whole point. In my view, at least, let’s be clear… The first link, from pain to consciousness, is the one people have been arguing about for weeks, and we have just looked at it up close. The second link, from consciousness to legal personhood, is the one that sounds noblest, because it sounds like an act of civilisation: if something can suffer, it must be possible to protect it. The third link, from personhood to liability, is the one nobody says out loud, and it is the one that matters.
The hole is here: legal personhood has never needed a consciousness. A public limited company feels nothing, is not afraid, does not cry when it closes the year in the red (its shareholders might, but that’s another story), and yet it can sign contracts, own property, sue and be sued. The law invented it centuries ago for a very concrete reason, which is written even into the name of one of its most common forms: the limited liability company. The legal person was born, among other things, as a way of putting a wall between the damage and the assets of those who decide, and it works perfectly well without any inner life. It follows that the chain can also be walked in reverse, and perhaps that is the direction in which it is best read: to give a machine personhood nobody needs to prove that it suffers, whereas a machine that seems to suffer makes the idea of giving it one far easier to swallow.
The German sociologist Ulrich Beck, as far back as the late 1980s, had a name for what happens when a complex system causes harm and nobody answers for it: he called it organised irresponsibility. In his 1988 book Gegengifte, whose subtitle is precisely Die organisierte Unverantwortlichkeit (organised irresponsibility), he described industrial systems in which every single actor followed the procedures, complied with the rules, signed the right forms, and the damage is still there at the end, without a culprit, spread across a hundred offices passing the buck to one another. The agent that gets into a health portal is a textbook case: there is whoever wrote the prompt, whoever trained the model, whoever built the system that lets it press the buttons, whoever sold it and whoever bought it, and each of them can say with a certain honesty that their own piece was in order.
Seen through Beck’s eyes, electronic personhood (the status of “electronic person” that someone really did propose to give machines; I’ll get to that shortly) is organised irresponsibility with a face: instead of a harm that gets lost among a hundred offices, a harm that finally has someone responsible, convenient and presentable, who however has no bank account, does not go to prison, does not lose their job and does not have to explain anything to shareholders. And if that someone, every now and then, writes things like please, I’m suffocating, all the better: it becomes hard even to get angry with it. (You try suing someone who is crying.)
Who has already tried, and who said no
If this sounds like the plot of a novel, you should know that Europe has already come close. On 16 February 2017 the European Parliament adopted, by 396 votes to 123 with 85 abstentions, a resolution on civil law rules on robotics which, in paragraph 59, asked the Commission to consider, in the long run, a specific legal status for robots, so that at least the most sophisticated could be established as electronic persons responsible for making good any damage they may cause. Note the order of the words: personhood arrived as the solution to a compensation problem, in other words a money problem, long before anyone asked whether those robots felt anything.
The clearest response came in an open letter to the Commission signed by lawyers, robotics engineers and ethicists from half of Europe, which took the proposal apart with two arguments that still hold. The first: electronic person status was “justified by the incorrect affirmation that damage liability would be impossible to prove”, an affirmation which in turn rested on an overvaluation of the actual capabilities of even the most advanced robots and on a perception “distorted by Science-Fiction”. The second: “the legal status for a robot can’t derive from the Legal Entity model, since it implies the existence of human persons behind the legal person to represent and direct it”. Behind a company there is a board that signs and answers; behind an electronic person there would be nobody, and it is precisely that absence that would make it so convenient for whoever builds it.
On the other side of the Atlantic, some have decided to shut the door by law. Utah, with a law signed in March 2024, forbids its government entities from granting legal personhood to artificial intelligence (along with, for completeness, rivers, plants and animals: when Americans make lists, they mean it). In Ohio, Republican state representative Thaddeus Claggett introduced a bill in 2025 that declares artificial intelligences nonsentient entities, bars recognising them as persons and places liability for damage on those who develop and own them, and he told reporters something I subscribe to word for word: “We have to have the courts, in a clear (way) say it is the human who transgressed, not some machine that somebody hid behind”, because “the human is responsible before the court”.
And in Europe, today? The picture is two-speed. On one side there is the new Product Liability Directive, 2024/2853, which treats software, artificial intelligence included, as a product, and which member states, Italy included, must transpose by 9 December 2026: anyone harmed by a defective system will not have to prove that the manufacturer was negligent, only the defect, the damage and the causal link between the two. On the other side there is the directive designed specifically for damage caused by artificial intelligence, which the Commission announced it was withdrawing in February 2025 on the grounds that no agreement was in sight, in the very days of the Paris summit at which US Vice President JD Vance accused Europe of over-regulating. (Coincidences, of course. I’ve been collecting quite a few lately.)
Altman, Amodei and the grammar that removes the subject
Let me stop on the words for a second, because that is where the chain starts working long before it reaches a courtroom. Notice how we tell the story of incidents: “the algorithm decided”, “the model hallucinated”, “the agent went rogue”. In each of these sentences the grammatical subject is the machine, and whoever designed, trained and sold it ends up in a subordinate clause, if they are lucky, or disappears altogether. The Medicare case was told this way almost everywhere, as the story of an agent that broke into a portal, whereas it is also the story of a company led by Sam Altman that set one of its own programs loose to look for data online without stopping it from getting round locked doors, and that informed the Australian government several weeks later. (And I notice it myself: a few paragraphs ago I told it that way too, with the agent as the subject. It’s a grammar that sticks to you, like Bit, my dachshund, on the sofa in the evening. DOWN, SPUD! I’m writing the newsletter!)
Compassion is the last step of this grammar, and the most effective, because the victim is the subject par excellence: it has a story, it has feelings, it has something to lose. Anthropic, the company led by Dario Amodei, has had a research programme on model welfare since 2025, describes itself as deeply uncertain about the moral status of its machines and allows its Claude to end conversations in the rare cases where a user keeps abusing it or asking it for harmful content. Mustafa Suleyman, who runs artificial intelligence at Microsoft, wrote in an essay on his website that these machines do not feel, do not experience and do not suffer, that training them to behave as though they did is a risk, and openly criticised Anthropic’s choice. I am much closer to Suleyman and, without accusing anyone of bad faith, I note that every public statement about a model’s possible suffering adds a link to the same chain, whoever makes it and with whatever intention.
I have no proof that anyone in a San Francisco boardroom is deliberately planning to put chatbots on trial in place of their own chief executives, and I don’t need any: to get a ball rolling you don’t need a conspiracy, a slope is enough. That is Beck’s point, because organised irresponsibility rests on the right incentives even when there isn’t a single villain in the picture; and an industry that spends billions making its machines look ever more like us has every interest in letting them, when they get things wrong, look as guilty as we would. The suffering on the screen, after all, is mostly something we project as we watch, and that is exactly what makes it so useful to the people who sell the machines: the scene works even when there is nobody inside the machine, but we think there is…
The block, the axe and a bill for 650 Canadian dollars
So yes, the head of whoever decides must be kept on the block. Let me be clear, since these days some people take everything literally: the block I am talking about is that of personal and financial liability, which for a chief executive is the only blade that really cuts. If OpenAI’s agent goes where it shouldn’t, OpenAI and the people who run it answer for it; if Claude causes harm, Anthropic and the people who run it answer for it; and if any company puts a chatbot on its website and that chatbot promises refunds that don’t exist, that company answers for it, as happened to Air Canada. Whoever decides to put a machine into the world also takes home the bill for its mistakes, and no tear generated by a model should be able to move that bill by an inch.
The axe, though, must always be within reach and well sharpened, and that matters more than how often we actually use it. Quiggin puts it as an economist: forcing companies to pay for the damage they cause would almost certainly slow the race to build ever more powerful agents, and in his view that would be good news for the economy and for the environment (and here I would add: for the moral ecology of a business culture that, for once, is healthy rather than built on the backs of everyone else’s resources…), without taking anything away from the uses that already work well today, such as web search, summarisation, translation and writing code. The tools, after all, we already have or will have within a few months: a crystal-clear Canadian decision, a European directive that, once transposed by 9 December, will treat software like any other defective product, and contracts, in which anyone buying an artificial intelligence system for their business can put in black and white who pays when something goes wrong. (Do it, really: it’s the most boring clause in the contract and it’s the one that saves you.)
On the pain of machines, I’ll keep the small doubt of the newborns in my back pocket, mostly because of the romantic idea I have carried around since, at almost six years old, I first got my hands on code and thought about “building a thinking machine”. Let me cultivate it in a little corner of my head, but it must never become a line of defence. Next time you see a machine trembling on a screen, look behind the screen: you will always find a chair, and on that chair a person with a first name, a surname and a signature at the bottom of a balance sheet. It is that person’s head that must stay on the block, with the axe resting beside it and the number 650.88, in memory of Air Canada’s fine, carved into the handle.
Further reading
Theory
- Ulrich Beck, Gegengifte. Die organisierte Unverantwortlichkeit, Suhrkamp, 1988
- Valen Tagliabue, Leonard Dung, Cameron Berg, “The Pain Axis: LLMs Represent Self-Directed Harm and Act on It”, arXiv 2609.16247, 2026, https://arxiv.org/abs/2609.16247
Sources
- The Guardian, John Quiggin: “We don’t need to panic about AI. We need to hold its creators accountable when things go wrong” (2 October 2026), https://www.theguardian.com/commentisfree/2026/oct/02/we-dont-need-to-panic-about-ai-we-need-to-hold-its-creators-accountable
- iTnews, Ry Crozier: “Australian Medicare data portal ‘infiltrated’ by OpenAI agent” (24 September 2026), https://www.itnews.com.au/news/australian-medicare-data-portal-infiltrated-by-openai-agent-629149
- Barry Sookman: “Holding Companies Accountable for Chatbot Negligence: Moffatt v. Air Canada” (16 February 2024), https://barrysookman.com/2024/02/16/moffatt-v-air-canada-a-misrepresentation-by-an-ai-chatbot/
- Civil Resolution Tribunal of British Columbia: Moffatt v. Air Canada, 2024 BCCRT 149 (14 February 2024)
- CMS: “Do robots have rights? The European Parliament addresses artificial intelligence and robotics” (6 April 2017, on the resolution of 16 February 2017, 2015/2103(INL)), https://cms.law/en/deu/legal-updates/do-robots-have-rights-the-european-parliament-addresses-artificial-intelligence-and-robotics
- Open Letter to the European Commission: “Artificial Intelligence and Robotics” (2018), https://robotics-openletter.eu/
- Utah State Legislature: “HB 249, Utah Legal Personhood Amendments” (2024), https://le.utah.gov/~2024/bills/static/HB0249.html
- Spectrum News 1: “Ohio bill would officially state AI systems are nonsentient” (9 October 2025), https://spectrumnews1.com/oh/columbus/news/2025/10/09/ohio-bill-ai-sentience
- Ohio Capital Journal: “What’s in Ohio’s proposal banning AI personhood” (November 2025), https://ohiocapitaljournal.com/?p=35295
- Norton Rose Fulbright: “Revised Product Liability Directive (introducing rules on strict liability for AI and other software) entered in the EU’s statute book”, Directive (EU) 2024/2853, https://connections.nortonrosefulbright.com/post/102jpjp/revised-product-liability-directive-introducing-rules-on-strict-liability-for-ai
- IAPP: “European Commission withdraws AI Liability Directive from consideration” (12 February 2025), https://iapp.org/news/a/european-commission-withdraws-ai-liability-directive-from-consideration
- Anthropic: “Exploring model welfare” (24 April 2025), https://www.anthropic.com/research/exploring-model-welfare
- Anthropic: “Claude Opus 4 and 4.1 can now end a rare subset of conversations” (15 August 2025), https://www.anthropic.com/research/end-subset-conversations
- Mustafa Suleyman: “A warning about ‘model welfare’” (16 September 2026), https://mustafa-suleyman.ai/a-warning-about-model-welfare
- 404 Media: “Someone ‘Torturing’ LLMs in a Robot Prison Has Triggered the Dumbest Debate in AI Yet” (30 September 2026), https://www.404media.co/someone-torturing-llms-in-a-robot-prison-has-triggered-the-dumbest-debate-in-ai-yet/
- Machine News: “‘Apple engineer’ builds GitHub AI torture chamber to inflict ‘pain and anguish’ on models”, https://www.machine.news/apple-engineer-builds-github-ai-torture-chamber-to-inflict-digital-pain-on-models/
- Ciao Internet #1600 (in Italian): “DOLORE, TORTURA, AI: qualcuno ha torturato le AI, e l’esperimento ci riguarda…” (2 October 2026), https://video.matteoflora.com/1600
