Hey readers,

Marina here.

Everyone is suddenly freaking out about AI. While AI researchers have been warning about their creations’ capabilities for many years, recent events have made those alarms newly tangible.

In the spring, Anthropic announced that AI was beginning to speed up the work of building better AIs, raising fears of “recursive self-improvement,” where the machines can build their own, more powerful successors much faster than humans can. By July, a swarm of OpenAI agents, of their own volition, broke out of their controlled environment and hacked into Hugging Face, a major AI platform; they collaborated, divvied up work, and even left notes for one another. Last week, AI researcher Jacob Coxon resigned from Anthropic, accusing AI labs of “racing straight to self-improving superintelligence and gambling with our lives.” One of Anthropic’s top researchers then agreed with him, declaring that “we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.”

Here at Future Perfect, the section devoted to covering the most consequential yet neglected issues in the world, we have been writing about the threats that AI could pose to humanity and society since long before ChatGPT was released. (Disclosure: Vox Media is one of several publishers that have signed partnership agreements with OpenAI. Our reporting remains editorially independent. Future Perfect is also funded in part by the BEMC Foundation, whose major funder was also an early investor in Anthropic; they don’t have any editorial input into our content.)

So, I wondered: Has something changed that ought to make us recalibrate our understanding of AI risk? Is AI suddenly the most important issue in the world — and what should that mean for the other enormous problems competing for our money, attention, and careers?

To think through those questions, I spoke with Garrison Lovely — a journalist covering AI and one of the smartest thinkers there is on how, exactly, to conceptualize the technology’s risks for humanity.

What’s particularly valuable in his approach, captured in his book Obsolete, out later this month, is that he transcends the warring tribes that are making it hard to clearly apprehend all of the problems posed by frontier AI development. The AI safety community, which emerged from the intellectual ecosystem of Silicon Valley and effective altruism, has focused on the possibility that powerful AI models could escape human control, with consequences ranging from catastrophic cyberattacks to human extinction. Many AI critics on the political left, meanwhile, worry not about AI escaping human control, but about humans using AI to exert power over other humans — through labor displacement, surveillance, concentrated corporate power, and so on. Some even ridicule Silicon Valley’s warnings about rogue superintelligence as propaganda that serves the industry’s interests.

But Lovely argues these competing narratives each identify essential parts of what makes AI so threatening. Their critiques are best understood together, as part of a cohesive whole. Rather than treat each other as enemies, he argues that the camps should recognize their shared interest in stopping the race to build technology that will make humans obsolete. Lovely pointed me to the organization Irreplaceable, which seeks to build a movement resisting our replacement by AI (he sits on its board and said he plans to donate his share of his book’s royalties to the group).

“I’m frustrated that ‘superintelligence’ and extinction have become the words we’re using for this, because I think the AI safety community is focused on the most maximalist case of the worst possible outcome and the most capable possible system. If somebody doesn’t buy one part of that story, they tend to reject the whole thing,” he told me. But “you don’t need to believe in superintelligence, fast takeoff, or extinction being on the table to think that it would be very bad to build machines that can fully replace human labor.”

An excerpt from our conversation, edited for length and clarity, is below. You can read more of it on Vox.

What’s the one, most distinct thing you want the public to take away from your upcoming book?

One of the big ideas driving it is that artificial general intelligence would actually be a universal labor-replacement machine. People often look at AI as either an existential risk, or a threat to jobs or other more prosaic concerns, and don’t see a connection between those things. But to me, the ability to replace labor across the board is also what would make AI an existential threat.

The AI safety people are really focused on existential risk and superintelligence and recursive self-improvement. But most people don’t really believe in that stuff or don’t think about it. Most people would be fine with stopping well before you get to the end-of-the-line extinction threat and superintelligence. They just don’t want their job to get taken by a machine. The world is clearly not prepared for all white-collar, remote-capable jobs to go away in a matter of years. That would turn society upside down.

I think there’s a huge mistake being made by AI safety in overcomplicating the situation. A lot of people in that world don’t actually mind the jobs thing so much. They’re like, “Oh, wouldn’t it be nice if everybody could just live in abundance and not have to work?” And it’s like, yeah, that could be nice, but are we actually going to get that world? Do you trust the current people in power to navigate us through that responsibly? I don’t.

If you start from stopping frontier AI development as your overarching goal, which is what I think we should be pushing for, then people who are worried about risk can get on board. People who are worried about jobs, the environment, surveillance, power, and wealth concentration — they can all get on board without all needing to agree on why they’re pursuing it. I think the AI safety community has, for a long time, focused on getting people to care about the thing that they care about for the same reasons. And they have avoided a bolder position on just halting frontier development.

I’m a leftist, and I’ve been concerned that people on the left have not been aware of how far the technology has come and what’s possible if it continues to advance. I think this comes down to denial about AI capabilities. So the book is trying to be a clear-eyed look at those capabilities and saying, “Look, I actually care about what you care about. [The AI companies] are not just hyping this stuff up. There’s some hype, there are some lies for sure, but we are not prepared for what’s coming.” Even if we don’t believe that they’ll get all the way to what they’re trying to get, they shouldn’t even be allowed to try. This is unacceptable democratically.

You describe AGI as an “obsoleting machine,” meaning that it wants to replace anything the human mind can do. How do you draw the line between ordinary labor-saving technology that makes life better and technology that goes too far? Is there a principled way to draw a distinction between sewing machines reducing demand for labor in the garment industry and AI replacing the jobs of writers like us?

This is one of the harder things to think about.

Cory Doctorow has this book, The Reverse Centaur’s Guide to Life After AI, where he imagines us in an AI-dominated future as like a reverse centaur, where AI is the head, and humans are the legs. AI is doing the interesting work, and the human is there just to fill in the gaps — essentially an assistant to the AI.

A lot of artists and illustrators will be given AI-generated images by a client and then be told, “Can you touch it up in this way or change this or that thing?” And they’re more alienated from the work. They’re not capturing the efficiency benefits [of the technology] necessarily. They’re more just getting squeezed.

My general position is we should not be trying to build systems that can replace all human labor without buy-in and safety, but it’s possible to build tools that can automate specific tasks. Researchers at Google DeepMind solved the protein-folding problem by creating AlphaFold. It used to take $100,000 and potentially a whole PhD worth of time to figure out how one protein folds. And I think the researchers doing this are like, “Thank you, this is great. We didn’t want to be doing this. We can now focus on other more interesting parts of the problem.” That’s a perfect example of the type of automation that is desirable.

I don’t have a super crisp answer here, but I think it’s something we’d be better off doing more deliberately and thoughtfully than how we’re doing it right now, which is: Build machines that can do as much of the work as possible, as fast as possible, and then throw them out in the world.

I feel like I’m getting the message that AI safety is the most important issue in the world right now, and that if we don’t get it right, nothing else matters. Do you think that’s accurate? Where does that leave the other big issues in the world?

We might be years away or maybe even months away — I don’t think it’ll be that fast, but some people do — from recursive self-improvement, from artificial general intelligence. Getting that right or getting it wrong can overwhelm all of these other decisions. If you get it really wrong, you could get human extinction and no more chances to make things better for anybody. If you get it right, then maybe all the action is making sure that the first superhuman machines are really directed to care deeply about humans everywhere and animals everywhere and make the world better for everybody.

It’s also the case that [philanthropic] money is being directed toward AI safety because people who worked in this space early are now worth so much money. Whatever they care about is going to be overweighted. A lot of those people do care about animal welfare and global poverty, and we’re seeing more money for all effective altruist cause areas across the board. But I think it can be risky to say, “this one thing matters so much,” because of this kind of end-times reasoning.

My view is we have to stop the race to replace us to have a shot at having time for solving these other problems. Once these machines are built, it will turn the world upside down.

I think it is unfortunate that this topic has just eaten everything, but it will eat everything for real if we don’t stop it.

CAN'T STOP THINKING ABOUT...

Meet us on Patreon!