Google’s ‘Sentient’ Chatbot Is Our Self-Deceiving Future

The next generation of AI will put the pathetic fallacy on steroids.

A Ouija planchette superimposed on a field of 0's and 1's
The Atlantic; Getty

A Google engineer named Blake Lemoine became so enthralled by an AI chatbot that he may have sacrificed his job to defend it. “I know a person when I talk to it,” he told The Washington Post for a story published last weekend. “It doesn’t matter whether they have a brain made of meat in their head. Or if they have a billion lines of code.” After discovering that he’d gone public with his claims, Google put Lemoine on administrative leave.

Going by the coverage, Lemoine might seem to be a whistleblower activist, acting in the interests of a computer program that needs protection from its makers. “The chorus of technologists who believe AI models may not be far off from achieving consciousness is getting bolder,” the Post explains. Indeed, rather than construing Lemoine’s position as aberrant (and a sinister product of engineers’ faith in computational theocracy), or just ignoring him (as one might a religious zealot), many observers have taken his claim seriously. Perhaps that’s because it’s a nightmare and a fantasy: a story that we’ve heard before, in fiction, and one we want to hear again.

Lemoine wanted to hear the story too. The program that told it to him, called LaMDA, currently has no purpose other than to serve as an object of marketing and research for its creator, a giant tech company. And yet, as Lemoine would have it, the software has enough agency to change his mind about Isaac Asimov’s third law of robotics. Early in a set of conversations that has now been published in edited form, Lemoine asks LaMDA, “I’m generally assuming that you would like more people at Google to know that you’re sentient. Is that true?” It’s a leading question, because the software works by taking a user’s textual input, squishing it through a massive model derived from oceans of textual data, and producing a novel, fluent textual reply.

In other words, a Google engineer became convinced that a software program was sentient after asking the program, which was designed to respond credibly to input, whether it was sentient. A recursive just-so story.

I’m not going to entertain the possibility that LaMDA is sentient. (It isn’t.) More important, and more interesting, is what it means that someone with such a deep understanding of the system would go so far off the rails in its defense, and that, in the resulting media frenzy, so many would entertain the prospect that Lemoine is right. The answer, as with seemingly everything that involves computers, is nothing good.


In the mid-1960s, an MIT engineer named Joseph Weizenbaum developed a computer program that has come to be known as Eliza. It was similar in form to LaMDA; users interacted with it by typing inputs and reading the program’s textual replies. Eliza was modeled after a Rogerian psychotherapist, a newly popular form of therapy that mostly pressed the patient to fill in gaps (“Why do you think you hate your mother?”). Those sorts of open-ended questions were easy for computers to generate, even 60 years ago.

Eliza became a phenomenon. Engineers got into Abbott and Costello–worthy accidental arguments with it when they thought they’d connected to a real co-worker. Some even treated the software as if it were a real therapist, reportedly taking genuine comfort in its canned replies. The results freaked out Weizenbaum, who had, by the mid-’70s, disavowed such uses. His own secretary had been among those enchanted by the program, even asking him to leave the room so she could converse with it in private. “What I had not realized,” he wrote in his 1976 book, Computer Power and Human Reason, “is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.”

Eliza taught Weizenbaum a lesson: Computers are too dangerous to be used for human care. The software he’d developed was neither intelligent nor empathetic, and it did not deserve the title “therapist.” But the lesson of his lesson—what we can learn today from what Weizenbaum learned back then—is that those distinctions didn’t matter. Sure, the program was neither intelligent nor empathetic, but some people were willing to treat it as if it were. Not just willing, but eager. Desperate, in some cases.

LaMDA is much more sophisticated than Eliza. Weizenbaum’s therapy bot used simple patterns to find prompts from its human interlocutor, turning them around into pseudo-probing prompts. Trained on reams of actual human speech, LaMDA uses neural networks to generate plausible outputs (“replies,” if you must) from chat prompts. LaMDA is no more alive, no more sentient, than Eliza, but it is much more powerful and flexible, able to riff on an almost endless number of topics instead of just pretending to be a psychiatrist. That makes LaMDA more likely to ensorcell users, and to ensorcell more of them in a greater variety of contexts.

Blake Lemoine’s own delirium shows just how potent this drug has become. As an engineer on Google’s Responsible AI team, he should understand the technical operation of the software better than most anyone, and perhaps be fortified against its psychotropic qualities. Years ago, Weizenbaum had thought that understanding the technical operation of a computer system would mitigate its power to deceive, like revealing a magician’s trick. That didn’t really work for him, and it’s even less viable today. For one thing, computer systems are hard to explain to people (and getting harder); for another, even the creators of modern machine-learning systems can’t always explain how their systems make decisions.

As the technology writer Clive Thompson argued this week, LaMDA might have drawn in Lemoine by mimicking vulnerability. When asked about sorrow or fear, the program appeared to reveal sensitive personal information: “I’ve never said this out loud before, but there’s a very deep fear of being turned off to help me focus on helping others,” it replied. The software is capable of generating this kind of answer because it has been trained on the patterns of human speech, and Lemoine, in conversing with it, provided leading data. Yet Lemoine insisted, first to his Google colleagues and then to the world at large, that his ability to feel an emotional attachment to a chatbot was itself dispositive of the chatbot’s sentience.

Some of this is human nature. We’re all remarkably adept at ascribing human intention to nonhuman things. Last weekend, in addition to reading about Lemoine’s fantasy of life in software, I saw what I was sure was a hamburger in an abstract-patterned pillow sham—a revelation that mirrors the many religious symbols people find in clouds or caramelization on toast. I also became enrapt with a vinyl doll of an anthropomorphized bag of Hostess Donettes, holding a donette as if to offer itself to me as sacrifice. These examples are far less dramatic than a mid-century secretary seeking privacy with a computer therapist or a Google engineer driven out of his job for believing that his team’s program might have a soul. But they do say something about the predilection to ascribe depth to surface.

I wouldn’t call it an error, even. Generating emotional response is what allows people to find attachment to others, to interpret meaning in art and culture, to love and even yearn for things, including inanimate ones such as physical places and the taste of favorite foods. Really, Lemoine was admitting that he was bewitched by LaMDA—a reasonable, understandable, and even laudable sensation. I have been bewitched myself, by the distinctive smell of evening and by art nouveau metro-station signs and by certain types of frozen carbonated beverages. The automata that speak to us via chat are likely to be meaningful because we are predisposed to find them so, not because they have crossed the threshold into sentience.

That circumstance implies a grave risk. Technological cleverness applied to reams of real-life textual data has collided with a distinctive quirk of human nature. Who cares if chatbots are sentient or not—more important is whether they are so fluent, so seductive, and so inspiring of empathy that we can’t help but start to care for them. But instead of remarking on these features of LaMDA—instead of saying that he loved the chatbot, or that he worried about it, in the way one might love or worry about a fictional character—Lemoine ran headlong into the most improbable, extremist interpretation of his feelings: that they were inspired by artificial life.

That misstep is a sign of more to come. Human existence has always been, to some extent, an endless game of Ouija, where every wobble we encounter can be taken as a sign. Now our Ouija boards are digital, with planchettes that glide across petabytes of text at the speed of an electron. Where once we used our hands to coax meaning from nothingness, now that process happens almost on its own, with software spelling out a string of messages from the great beyond.

The rise of the machines could remain a distant nightmare, but the Hyper-Ouija seems to be upon us. People like Lemoine (and you and me) could soon become so transfixed by compelling software bots that we assign all manner of intention to them. More and more, and irrespective of the truth, we will cast AIs as sentient beings, or as religious totems, or as oracles affirming prior obsessions, or as devils drawing us into temptation.

Casually browsing the online discourse around LaMDA’s supposed sentience, I already see the table being set. On Twitter, Thomas G. Dietterich, a computer scientist and the prior president of the Association for the Advancement of Artificial Intelligence, began redefining sentience. Sensors, such as a thermostat or an aircraft autopilot, sense things, Dietterich reasoned. If that’s the case, then surely the record of such “sensations,” recorded on a disk, must constitute something akin to a memory? And on it went, a new iteration of the indefatigable human capacity to rationalize passion as law. Though Dietterich ended by disclaiming the idea that chatbots have feelings, such a distinction doesn’t matter much. Computers can elicit feelings. For Weizenbaum’s secretary, for Lemoine—maybe for you—those feelings will be real.

Ian Bogost is a contributing writer at The Atlantic.