This is a clip from a conversation with Stuart Russell from Dec 2018. Check out Stuart's new book on this topic "Human Compatible": https://amzn.to/2pdXg8G New full episodes every Mon & Thu and 1-2 new clips or a new non-podcast video on all other days. You can watch the full conversation here: https://www.youtube.com/watch?v=KsZI5oXBC0k
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on iTunes:
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Note: I select clips with insights from these much longer conversation with the hope of helping make these ideas more accessible and discoverable. Ultimately, this podcast is a small side hobby for me with the goal of sharing and discussing ideas. For now, I post a few clips every Tue & Fri. I did a poll and 92% of people either liked or loved the posting of daily clips, 2% were indifferent, and 6% hated it, some suggesting that I post them on a separate YouTube channel. I hear the 6% and partially agree, so am torn about the whole thing. I tried creating a separate clips channel but the YouTube algorithm makes it very difficult for that channel to grow unless the main channel is already very popular. So for a little while, I'll keep posting clips on the main channel. I ask for your patience and to see these clips as supporting the dissemination of knowledge contained in nuanced discussion. If you enjoy it, consider subscribing, sharing, and commenting.

Stuart Russell is a professor of computer science at UC Berkeley and a co-author of the book that introduced me and millions of other people to AI, called Artificial Intelligence: A Modern Approach.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

Let's just talk about maybe the control problem.

So this idea of losing ability to control the behavior in our AI system.

What do you think we can do to manage it?

Well, so it doesn't take a genius to realize that if you make something that's smarter than you, you might have a problem.

You know, and Turing, Alan Turing, you know, wrote about this and gave lectures about this in

He did a lecture on the radio and he basically says, you know, once the machine thinking method starts, very quickly they'll outstrip humanity.

And if we're lucky we might be able to, I think he says, we may be able to turn off the power at strategic moments, but even so our species would be humbled.

And actually he was wrong about that, right?

Because if it's a sufficiently intelligent machine, it's not going to let you switch it off.

So What do you think is meant, just for a quick tangent, if we shut off this superintelligent machine, that our species would be humbled?

I think he means that we would realize that we are inferior, right?

That we only survive by the skin of our teeth because we happen to get to the off switch.

we hadn't, then we would have lost control over the earth.

So are you more worried when you think about this stuff about super intelligent AI or are you more worried about super powerful AI that's not aligned with our values?

I think, So the main problem I'm working on is the control problem, the problem of machines pursuing objectives that are, as you say, not aligned with human objectives.

And this has been the way we've thought about AI since the beginning.

You build a machine for optimizing and then you put in some objective and it optimizes.

And we can think of this as the King Midas problem.

Because if King Midas put in this objective, everything I touch should turn to gold, and the gods, that's like the machine, they said, okay, done.

You now have this power, And of course his food and his drink and his family all turned to gold and then he dies of misery and starvation.

And this is, you know, it's a warning, It's a failure mode that pretty much every culture in history has had some story along the same lines.

You know, there's the genie that gives you 3 wishes, and the third wish is always you know, please undo the first 2 wishes because I messed up.

And you know, when Arthur Samuel wrote his chess, his checker playing program, which learned to play chequers considerably better than Arthur Samuel could play, and actually reached a pretty decent standard, Norbert Wiener, who was 1 of the major mathematicians of the 20th century, sort of the father of modern automation control systems, he saw this and he basically extrapolated, as Turing did, and said, okay, this is how we could lose control and Specifically that We have to be certain that the purpose we put into the machine is the purpose which we really desire.

You mean it's very difficult to encode, to put our values on paper is really difficult, or you're just saying it's impossible?

So theoretically it's possible, but in practice it's extremely unlikely that we could specify correctly in advance the full range of concerns of humanity.

You talked about cultural transmission of values, I think is how humans to human transmission of values happens, right?

I mean, as we grow up, we learn about the values that matter, how things should go, what is reasonable to pursue and what isn't reasonable to pursue.

I think machines can learn in the same kind of way.

Yeah, so I think that what we need to do is to get away from this idea that you build an optimizing machine and then you put the objective into it.

Because if it's possible that you might put in a wrong objective, and we already know this is possible because it's happened lots of times, right?

That means that the machine should never take an objective that's given as gospel truth.

Because once it takes the objective as gospel truth, then it believes that whatever actions it's taking in pursuit of that objective are the correct things to do.

So you could be jumping up and down and saying, you know, no, no, no, you're going to destroy the world, But the machine knows what the true objective is and is pursuing it.

This is, I think, many of the 20th century technologies, right?

So in statistics, you minimize a loss function.

The loss function is exogenously specified.

In control theory, you minimize a cost function.

In operations research, you maximize a reward function, and so on.

So in all these disciplines, this is how we conceive of the problem.

And it's the wrong problem, because we cannot specify with certainty the correct objective, right?

We need uncertainty, we need the machine to be uncertain about what it is that it's supposed to be maximizing.

Favorite idea of yours, I've heard you say somewhere, well, I shouldn't pick favorites, but it just sounds beautiful of, we need to teach machines humility.

Yeah, I mean, that's a beautiful way to put it.

That they're humble In that they know that they don't know what it is they're supposed to be doing.

And that those objectives, I mean, they exist.

They're within us, but We may not be able to explicate them.

We may not even know how we want our future to go.

And a machine that's uncertain is going to be deferential to us.

So if we say don't do that, well now the machine's learned something a bit more about our true objectives because something that it thought was reasonable in pursuit of our objective turns out not to be, so now it's learned something.

So it's going to defer because it wants to be doing what we really want.

And that point, I think, is absolutely central to solving the control problem.

And it's a different kind of AI when you take away this idea that the objective is known, then in fact a lot of the theoretical frameworks that we're so familiar with, You know, Markov decision processes, goal-based planning, you know, standard game tree search, all of these techniques actually become inapplicable.

And you get a more complicated problem because, because now, the interaction with the human becomes part of the problem.

Because the human, by making choices, is giving you more information about the true objective and that information helps you achieve the objective better.

And so that really means that you're mostly dealing with game theoretic problems where you've got the machine and the human and they're coupled together, rather than a machine going off by itself with a fixed objective.

Which is fascinating on the machine and the human level that when you don't have an objective, means you're together coming up with an objective.

I mean, there's a lot of philosophy that, you know, you could argue that life doesn't really have meaning.

We together agree on what gives it meaning and we kind of culturally create things that give why the heck we are on this earth anyway.

We together as a society create that meaning and you have to learn that objective.

And 1 of the biggest, I thought that's where you were gonna go for a second.

1 of the biggest troubles we run into outside of statistics and machine learning and AI and just human civilization is when you look at, I came from, I was born in the Soviet Union and the history of the 20th century, we ran into the most trouble, us humans, when there was a certainty about the objective and you do whatever it takes to achieve that objective, whether you're talking about Germany or communist Russia.

And I would say with corporations, In fact, some people argue that we don't have to look forward to a time when AI systems take over the world.

They already have, and they're called corporations.

Corporations happen to be using people as components right now, but they are effectively algorithmic machines and they're optimizing an objective which is quarterly profit that isn't aligned with overall well-being of the human race, and they are destroying the world.

They are primarily responsible for our inability to tackle climate change.

So I think that's 1 way of thinking about what's going on with corporations.

But I think the point you're making is valid, that there are many systems in the real world where we've sort of prematurely fixed on the objective and then decoupled the machine from those that it's supposed to be serving.

And I think you see this with government.

Government is supposed to be a machine that serves people, but instead it tends to be taken over by people who have their own objective and use government to optimize that objective regardless of what people want.


Speaker 0: Let's just talk about maybe the control problem. So this idea of losing ability to control the behavior in our AI system. So how do you see that? How do you see that coming about? What do you think we can do to manage it?

Speaker 1: Well, so it doesn't take a genius to realize that if you make something that's smarter than you, you might have a problem. You know, and Turing, Alan Turing, you know, wrote about this and gave lectures about this in

Speaker 0: 1951.

Speaker 1: He did a lecture on the radio and he basically says, you know, once the machine thinking method starts, very quickly they'll outstrip humanity. And if we're lucky we might be able to, I think he says, we may be able to turn off the power at strategic moments, but even so our species would be humbled. And actually he was wrong about that, right? Because if it's a sufficiently intelligent machine, it's not going to let you switch it off. It's actually in competition with you.

Speaker 0: So What do you think is meant, just for a quick tangent, if we shut off this superintelligent machine, that our species would be humbled?

Speaker 1: I think he means that we would realize that we are inferior, right? That we only survive by the skin of our teeth because we happen to get to the off switch. You know, just in time. You know, and if

Speaker 0: we hadn't, then we would have lost control over the earth. So are you more worried when you think about this stuff about super intelligent AI or are you more worried about super powerful AI that's not aligned with our values? So the paperclip

Speaker 2: scenarios kind of...

Speaker 1: I think, So the main problem I'm working on is the control problem, the problem of machines pursuing objectives that are, as you say, not aligned with human objectives. And this has been the way we've thought about AI since the beginning. You build a machine for optimizing and then you put in some objective and it optimizes. And we can think of this as the King Midas problem. Because if King Midas put in this objective, everything I touch should turn to gold, and the gods, that's like the machine, they said, okay, done.

You now have this power, And of course his food and his drink and his family all turned to gold and then he dies of misery and starvation. And this is, you know, it's a warning, It's a failure mode that pretty much every culture in history has had some story along the same lines. You know, there's the genie that gives you 3 wishes, and the third wish is always you know, please undo the first 2 wishes because I messed up. And you know, when Arthur Samuel wrote his chess, his checker playing program, which learned to play chequers considerably better than Arthur Samuel could play, and actually reached a pretty decent standard, Norbert Wiener, who was 1 of the major mathematicians of the 20th century, sort of the father of modern automation control systems, he saw this and he basically extrapolated, as Turing did, and said, okay, this is how we could lose control and Specifically that We have to be certain that the purpose we put into the machine is the purpose which we really desire. And the problem is, we can't do that.

Speaker 0: You mean it's very difficult to encode, to put our values on paper is really difficult, or you're just saying it's impossible?

Speaker 2: The line is gray between

Speaker 1: the 2. So theoretically it's possible, but in practice it's extremely unlikely that we could specify correctly in advance the full range of concerns of humanity.

Speaker 0: You talked about cultural transmission of values, I think is how humans to human transmission of values happens, right?

Speaker 1: Well, we learn, yeah. I mean, as we grow up, we learn about the values that matter, how things should go, what is reasonable to pursue and what isn't reasonable to pursue.

Speaker 0: I think machines can learn in the same kind of way.

Speaker 1: Yeah, so I think that what we need to do is to get away from this idea that you build an optimizing machine and then you put the objective into it. Because if it's possible that you might put in a wrong objective, and we already know this is possible because it's happened lots of times, right? That means that the machine should never take an objective that's given as gospel truth. Because once it takes the objective as gospel truth, then it believes that whatever actions it's taking in pursuit of that objective are the correct things to do. So you could be jumping up and down and saying, you know, no, no, no, you're going to destroy the world, But the machine knows what the true objective is and is pursuing it.

And tough luck to you. And this is not restricted to AI, right? This is, I think, many of the 20th century technologies, right? So in statistics, you minimize a loss function. The loss function is exogenously specified.

In control theory, you minimize a cost function. In operations research, you maximize a reward function, and so on. So in all these disciplines, this is how we conceive of the problem. And it's the wrong problem, because we cannot specify with certainty the correct objective, right? We need uncertainty, we need the machine to be uncertain about what it is that it's supposed to be maximizing.

Speaker 2: Favorite idea of yours, I've heard you say somewhere, well, I shouldn't pick favorites, but it just sounds beautiful of, we need to teach machines humility. Yeah, I mean, that's a beautiful way to put it.

Speaker 1: I love it. That they're humble In that they know that they don't know what it is they're supposed to be doing. And that those objectives, I mean, they exist. They're within us, but We may not be able to explicate them. We may not even know how we want our future to go.

And a machine that's uncertain is going to be deferential to us. So if we say don't do that, well now the machine's learned something a bit more about our true objectives because something that it thought was reasonable in pursuit of our objective turns out not to be, so now it's learned something. So it's going to defer because it wants to be doing what we really want. And that point, I think, is absolutely central to solving the control problem. And it's a different kind of AI when you take away this idea that the objective is known, then in fact a lot of the theoretical frameworks that we're so familiar with, You know, Markov decision processes, goal-based planning, you know, standard game tree search, all of these techniques actually become inapplicable.

And you get a more complicated problem because, because now, the interaction with the human becomes part of the problem. Because the human, by making choices, is giving you more information about the true objective and that information helps you achieve the objective better. And so that really means that you're mostly dealing with game theoretic problems where you've got the machine and the human and they're coupled together, rather than a machine going off by itself with a fixed objective.

Speaker 2: Which is fascinating on the machine and the human level that when you don't have an objective, means you're together coming up with an objective. I mean, there's a lot of philosophy that, you know, you could argue that life doesn't really have meaning. We together agree on what gives it meaning and we kind of culturally create things that give why the heck we are on this earth anyway. We together as a society create that meaning and you have to learn that objective. And 1 of the biggest, I thought that's where you were gonna go for a second.

1 of the biggest troubles we run into outside of statistics and machine learning and AI and just human civilization is when you look at, I came from, I was born in the Soviet Union and the history of the 20th century, we ran into the most trouble, us humans, when there was a certainty about the objective and you do whatever it takes to achieve that objective, whether you're talking about Germany or communist Russia.

Speaker 1: You get into trouble with humans. And I would say with corporations, In fact, some people argue that we don't have to look forward to a time when AI systems take over the world. They already have, and they're called corporations. Corporations happen to be using people as components right now, but they are effectively algorithmic machines and they're optimizing an objective which is quarterly profit that isn't aligned with overall well-being of the human race, and they are destroying the world. They are primarily responsible for our inability to tackle climate change.

So I think that's 1 way of thinking about what's going on with corporations. But I think the point you're making is valid, that there are many systems in the real world where we've sort of prematurely fixed on the objective and then decoupled the machine from those that it's supposed to be serving. And I think you see this with government. Right? Government is supposed to be a machine that serves people, but instead it tends to be taken over by people who have their own objective and use government to optimize that objective regardless of what people want.

See all Lex Fridman transcripts on Youtube

Stuart Russell: The Control Problem of Super-Intelligent AI | AI Podcast Clips