Hey readers,
On Tuesday, OpenAI announced the release of GPT-4, their latest, biggest language model, only a few months after the splashy release of ChatGPT. GPT-4 was already in action; Microsoft has been using it to power Bing’s new assistant function. OpenAI has written that they think the best way to handle powerful AI systems is to develop and release them as quickly as possible, and that’s certainly what they’re doing.
Also on Tuesday, I sat down with Holden Karnofsky, the co-founder and co-CEO of Open Philanthropy, to talk about AI and where it’s taking us.
Karnofsky, in my view, should get a lot of credit for his prescient views on AI. Since 2008, he’s been engaging with what was then a small minority of researchers who were saying that powerful AI systems were one of the most important social problems of our age — a view that I think has aged remarkably well.
Some of his early published work on the question, from 2011 and 2012, raises questions about what shape those models will take and how hard it would be to make developing them go well — all of which will only look more important with a decade of hindsight.
In the last few years, he has started to write about the case that AI may be an unfathomably big deal and about what we can and can’t learn from the behavior of today’s models. Over that same time, Open Philanthropy has been investing more in making AI go well. And recently, Karnofsky announced a leave of absence from his work at OpenPhil to explore working directly on AI risk reduction.
A much longer transcript of our conversation will be on the site, but here are some excerpts for you:
How AI could get crazy
Kelsey: You’ve written about how AI could mean that things get really crazy in the near future.
Holden: The basic idea would be to imagine what the world would look like in the far future after a lot of scientific and technological development. Generally, I think most people would agree the world could look really, really strange and unfamiliar. There's a lot of science fiction about this.
What is most high stakes about AI, in my opinion, although there's many things about it that could be strange and could transform society a lot, is the idea that AI could potentially serve as a way of automating all the things that humans do to advance science and technology, and so we could get to that wild future a lot faster than people tend to imagine.
Today, we have a certain number of human scientists who try to push forward science and technology, so the day that we're able to automate everything they do, that could be a massive increase in the amount of scientific and technological advancement that's getting done. And furthermore, it can create a kind of feedback loop that we don't have today where basically as you improve your science and technology, that leads to a greater supply of hardware and more efficient software that runs a greater number of AIs.
And because those are the ones doing the science and technology research and advancement, that goes in a loop. If you get that loop, you get very explosive progress. The upshot of all this is that the world most people imagine thousands of years from now in some wild sci-fi future could be more like 10 years out or one year out or months out from the point where AI systems are doing all the things that humans typically do to advance science and technology.
Kelsey: That sounds great, right? Star Trek future overnight? What’s the catch?
Holden: I think there are big risks. I mean, it could be great. But as you know, I think that if all we do is we kind of sit back and relax and let scientists move as fast as they can, we'll get some chance of things going great and some chance of some things going terribly.
I am most focused on standing up where normal market forces will not and trying to push against the probability of things going terribly. In terms of how things could go terribly, maybe I'll start with the broad intuition: When we talk about scientific progress and economic growth, we're talking about the few percent per year range. That's what we've seen in the last couple hundred years, that's all any of us know. That's all anyone alive today has ever experienced.
But how you would feel about an economic growth rate of, let's say, 100 percent per year, 1000 percent per year. Some of how I feel is that we just are not ready for what's coming. I think society has not really shown any ability to adapt to a rate of change that fast. The appropriate attitude toward the next sort of industrial revolution-sized transition is caution.
Like, gosh, it might be better if we could do this a bit more gradually and thoughtfully and not just have everything kick off with a handful of companies racing forward as fast as they can to do whatever they can.
Another broad intuition is that these AI systems we’re building, they might do all the things humans do to automate scientific and technological advancement, but they're not humans. If we get there, that would be the first time in all of history that we had anything capable of autonomously developing its own new technologies, autonomously advancing science and technology that is not humans. No one has any idea what that's going to look like, and I think we shouldn't assume that the result is going to be good for humans. I think it really depends on how the AI are designed.
And that's where I think if you look at this current state of machine learning, it's just very clear that we have no idea what we're building. To a first approximation, the way these systems are designed is that someone takes a relatively simple learning algorithm and they pour in an enormous amount of data. They put in the whole internet and it sort of tries to predict one word at a time from the internet and learn from that. That's an oversimplification, but it's like they do that and out of that process pops some kind of thing that can talk to you and make jokes and write poetry, but no one really knows why.
You can think of it as analogous to human evolution, where there were lots of organisms and some survived and some didn’t and at some point there were humans who have all kinds of things going on in their brains that we still don’t really understand. Evolution is a simple process that resulted in complex beings that we still don’t understand.
When Bing Chat came out and it started threatening users and, you know, trying to seduce them and God knows what, people asked, "Why is it doing that?" And I would say not only do I not know, but no one knows because the people who designed it don't know, the people who trained it don't know.
The benefits of going slow on AI
Kelsey: Some people — OpenAI for example — have argued that yes, you’re right, AI is going to be a huge deal, dramatically transform our world overnight, and that’s why we should be racing forward as much as possible because by releasing technology sooner, we’ll give society more time to adjust.
Holden: I think there's some pace at which that would make sense and I think the pace we’ve already seen may already be too fast for that. I think society just takes a while to adjust to anything.
And this is a lot of my point: Historically, economic growth of a few percent seems to have gone pretty well, but a few percent a year means that no one year is too crazily radically different from the previous year, at least technologically or at least in terms of the size of the world economy, so people are able to adjust.
Most technologies that come out, it takes a long time for them to be appropriately regulated, for them to be appropriately used in government. People who are not early adopters or tech lovers learn how to use them, integrate them into their lives, learn how to avoid the pitfalls, learn how to deal with the downsides.
So I think that if we may be on the cusp of a radical explosion in growth or in technological progress, I don’t really see how rushing forward is supposed to help here. I don't see how it's supposed to get us to a rate of change that is slow enough for society to adapt if we’re pushing forward as fast as we can.
I think the better plan is to actually have a societal conversation about what pace we do want to move at and whether we want to slow things down on purpose and whether we want to move a bit more deliberately, and if not, how we can have this go in a way that avoids some of the key risks or that reduces some of the key risks.
I guess I don't really see how just making things happen as soon as possible is going to somehow cause society to adapt if we're looking at the possibility of a truly explosive rate of change.