The following is a conversation with Stuart Russell.

He's a professor of computer science at UC Berkeley and a co-author of a book that introduced me and millions of other people to the amazing world of AI called Artificial Intelligence, The Modern Approach.

So it was an honor for me to have this conversation as part of MIT course in artificial general intelligence and the Artificial Intelligence Podcast.

If you enjoy it, please subscribe on YouTube, iTunes, or your podcast provider of choice, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Stuart Russell.

So you've mentioned in 1975 in high school, you've created 1 of your first AI programs that played chess.

Were you ever able to build a program that beat you at chess or another board game?

I actually wrote the program at Imperial College.

So I used to take the bus every Wednesday with a box of cards this big and shove them into the card reader.

It took about 5 seconds to read the cards in and compile the code, so we had 3 seconds of CPU time, which was enough to make 1 move with a not very deep search.

And then we would print that move out and then we'd have to go to the back of the queue and wait to feed the cards in again.

No, I think we got an 8 move, you know, depth 8 with Alpha Beta and we had some tricks of our own about move ordering and some pruning of the tree.

But you were still able to beat that program?

I was a reasonable chess player in my youth.

I did an Othello program and a backgammon program.

So when I got to Berkeley I worked a lot on what we call meta-reasoning, which really means reasoning about reasoning.

And in the case of a game-playing program, you need to reason about what parts of the search tree you're actually going to explore, because the search tree is enormous, or bigger than the number of atoms in the universe.

The way programs succeed, and the way humans succeed, is by only looking at a small fraction of the search tree.

And if you look at the right fraction, you play really well.

If you look at the wrong fraction, if you waste your time thinking about things that are never going to happen, moves that no one's ever going to make, then you're going to lose, because you won't be able to figure out the right decision.

So that question of how machines can manage their own computation, how they decide what to think about, is the meta-reasoning question.

We developed some methods for doing that, and very simply, a machine should think about whatever thoughts are going to improve its decision quality.

We were able to show that both for Othello, which is a standard two-player game, and for Backgammon, which includes dice rolls, so it's a two-player game with uncertainty.

For both of those cases, we could come up with algorithms that were actually much more efficient than the standard alpha-beta search which chess programs at the time we're using.

And I think you can see the same basic ideas in AlphaGo and AlphaZero today.

The way they explore the tree is using a form of meta-reasoning to select what to think about based on how useful it is to think about it.

Is there any insights you can describe with our Greek symbols of how do we select which paths to go

There's really 2 kinds of learning going on.

So as you say AlphaGo learns to evaluate board position.

So it can look at a Go board And it actually has probably a superhuman ability to instantly tell how promising that situation is.

To me, the amazing thing about AlphaGo is not that it can beat the world champion with its hands tied behind his back, but the fact that if you stop it from searching altogether, so you say okay you're not allowed to do any thinking ahead, right, you can just consider each of your legal moves and then look at the resulting situation and evaluate it.

So just the immediate outcome of your moves and decide if that's good or bad.

That version of AlphaGo can still play at a professional level, right?

And human professionals are sitting there for 5, 10 minutes deciding what to do, and AlphaGo in less than a second can instantly intuit what is the right move to make based on its ability to evaluate positions.

And that is remarkable, because we don't have that level of intuition about Go.

We actually have to think about the situation.

So anyway, that capability that AlphaGo has is 1 big part of why it beats humans.

The other big part is that it's able to look ahead 40, 50, 60 moves into the future.

And if it was considering all possibilities 40 or 50 or 60 moves into the future, that would be, you know, 10 to the 200 possibilities.

So way more than, you know, atoms in the universe and so on.

So it's very, very selective about what it looks at.

So let me try to give you an intuition about how you decide what to think about.

Right, so if you're already convinced that a move is terrible, there's no point spending a lot more time convincing yourself that it's terrible because it's probably not gonna change your mind.

So the real reason you think is because there's some possibility of changing your mind about what to do.

And it's that changing of mind that would result then in a better final action in the real world.

So That's the purpose of thinking, is to improve the final action in the real world.

And so if you think about a move that is guaranteed to be terrible, you can convince yourself it's terrible, you're still not going to change your mind, right?

But on the other hand, suppose you had a choice between 2 moves.

1 of them you've already figured out is guaranteed to be a draw, let's say.

And then the other 1 looks a little bit worse.

It looks fairly likely that if you make that move you're going to lose.

But there's still some uncertainty about the value of that move.

There's still some possibility that it will turn out to be a win.

So even though it's less promising on average than the other move, which is guaranteed to be a draw, there's still some purpose in thinking about it because there's a chance that you will change your mind and discover that in fact, it's a better move.

So it's a combination of how good the move appears to be and how much uncertainty there is about its value.

The more uncertainty, the more it's worth thinking about, because there's a higher upside if

And of course, in the beginning, especially in the AlphaGo 0 formulation, it's everything is shrouded in uncertainty.

So you're really swimming in a sea of uncertainty so it benefits you to...

I mean actually following the same process as you described but because you're so uncertain about everything you basically have to try a lot of different directions.

Yeah so the early parts of the search tree are fairly bushy, that it will look at a lot of different possibilities, but fairly quickly, the degree of certainty about some of the moves, I mean if a move is really terrible, you'll pretty quickly find out, right?

You'll lose half your pieces or half your territory and then you'll say, okay, this is not worth thinking about anymore.

And then, so further down, the tree becomes very long and narrow and you're following various lines of play, you know, 10, 20, 30, 40, 50 moves into the future.

And, you know, that again is something that human beings have a very hard time doing, mainly because they just lack the short-term memory.

You just can't remember a sequence of moves that's 50 moves long.

And you can't imagine the board correctly for that many moves into the future.

Of course the top players, I'm much more familiar with chess, but the top players probably have, they have echoes of the same kind of intuition, instinct that in a moment's time, AlphaGo applies when they see a board.

I mean, they've seen those patterns, human beings have seen those patterns before at the top, at the grandmaster level.

It seems that there is some similarities, or maybe it's our imagination creates a vision of those similarities, but it feels like this kind of pattern recognition that the AlphaGo approaches are using is similar to what human beings at the top level are using.

Yeah, I mean, I think the extent to which a human grandmaster can reliably, instantly recognize the right move and instantly recognize the value of a position, I think that's a little bit overrated.

But if you sacrifice a queen, for example, I mean, there's these beautiful games of chess with Bobby Fischer, somebody where it's seeming to make a bad move.

And I'm not sure there's a perfect degree of calculation involved where they've calculated all the possible things that happen, but there's an instinct there, right?

Yeah, so I think what happens is you get a sense that there's some possibility in the position, even if you make a weird looking move, that it opens up some lines of calculation that otherwise would be definitely bad.

And it's that intuition that there's something here in this position that might yield a win.

And in some sense, when a chess player is following a line in his or her mind, they're mentally simulating what the other person is going to do, what the opponent is going to do.

And they can do that as long as the moves are kind of forced, right?

As long as there's a, you know, there's a, we call a forcing variation where the opponent doesn't really have much choice how to respond, and then you see if you can force them into a situation where you win.

You know, we see plenty of mistakes, even in Grandmaster games, where they just miss some simple 3, 4, 5 move combination that wasn't particularly apparent in the position but was still there.

So when you mentioned that in Othello, those games were after some meta reasoning improvements and research was able to beat you.

Part of the meta reasoning capability that it had was based on learning.

And you could sit down the next day and you could just feel that it had got a lot smarter.

And all of a sudden you really felt like you're sort of pressed against the wall because it was much more aggressive and was totally unforgiving of any minor mistake that you might make.

And actually it seemed understood the game better than I did.

And Garry Kasparov has this quote where during his match against Deep Blue, he said he suddenly felt that there was a new kind of intelligence across the board.

think that's a scary or an exciting possibility for Kasparov and for yourself?

In the context of chess, purely sort of in this like, that feeling, whatever that is.

I think it's definitely an exciting feeling.

You know, this is what made me work on AI in the first place, was as soon as I really understood what a computer was, I wanted to make it smart.

You know, I started out with the first program I wrote was for the Sinclair programmable calculator.

And I think you could write a 21-step algorithm that was the biggest program you could write, something like that, and do little arithmetic calculations.

So I think I implemented Newton's method for square roots and a few other things like that.

But then, you know, I thought, okay, if I just had more space, I could make this thing intelligent.

Started thinking about AI and I think the thing that's scary is not the chess program because you know chess programs they're not in the taking over the world business.

But if you extrapolate, you know, there are things about chess that don't resemble the real world, Right?

The chessboard is completely visible to the program, where of course the real world is not.

Most of the real world is not visible from wherever you're sitting, so to speak.

And to overcome those kinds of problems, you need qualitatively different algorithms.

Another thing about the real world is that we regularly plan ahead on the timescales involving billions or trillions of steps.

Now we don't plan those in detail, but when you choose to do a PhD at Berkeley, that's a 5 year commitment and that amounts to about a trillion motor control steps that you will eventually be committed to.

Including going up the stairs, opening doors, drinking water.

Every finger movement while you're typing, every character of every paper and the thesis and everything.

So you're not committing in advance to the specific motor control steps, but you're still reasoning on a time scale that will eventually reduce to trillions of motor control actions.

And so for all these reasons, AlphaGo and Deep Blue and so on don't represent any kind of threat to humanity, but they are a step towards it, right?

And progress in AI occurs by essentially removing 1 by 1 these assumptions that make problems easy, like the assumption of complete observability of the situation, right?

We remove that assumption, you need a much more complicated kind of computing design.

You need something that actually keeps track of all the things you can't see and tries to estimate what's going on, and there's inevitable uncertainty in that.

So it becomes a much more complicated problem.

We are starting to have algorithms that can cope with much longer time scales, that can cope with uncertainty, that can cope with partial observability.

And so each of those steps sort of magnifies by a thousand the range of things that we can do with AI systems.

So the way I started in AI, I wanted to be a psychiatrist for a long time, I wanted to understand the mind in high school and of course program and so on.

And I showed up, University of Illinois, to an AI lab, and they said, okay, I don't have time for you, but here's a book, AI, a Modern Approach.

I think it was the first edition at the time.

And I remember the lay of the land was, well, it's incredible that we solved chess, but we'll never solve Go.

I mean, it was pretty certain that Go, in the way we thought about systems that reason, was impossible to solve.

Well, I think I would have said that it's unlikely we could take the kind of algorithm that was used for chess and just get it to scale up and work well for Go.

And at the time, what we thought was that in order to solve Go, we would have to do something similar to the way humans manage the complexity of Go, which is to break it down into kind of sub-games.

So when a human thinks about a Go board, they think about different parts of the board as sort of weakly connected to each other.

And they think about, okay, within this part of the board, here's how things could go.

In that part of the board, here's how things could go.

And then you try to sort of couple those 2 analyses together and deal with the interactions and maybe revise your views of how things are going to go in each part and then you've got maybe 5 6 07:10 parts of the board And that actually resembles the real world much more than chess does.

Because in the real world, we have work, we have home life, we have sport, whatever, different kinds of activities, shopping.

These all are connected to each other, but they're weakly connected.

So when I'm typing a paper, I don't simultaneously have to decide which order I'm going to get the milk and the butter.

But I do need to realize, okay, I better finish this before the shops close because I don't have any food at home.

So there's some weak connection, but not in the way that chess works, where everything is tied into a single stream of thought.

So the thought was that to solve Go, we'd have to make progress on stuff that would be useful for the real world.

And in a way, AlphaGo is a little bit disappointing, because the program design for AlphaGo is actually not that different from Deep Blue or even from Arthur Samuel's checker playing program from the 1950s.

And in fact, the 2 things that make AlphaGo work is 1 is this amazing ability to evaluate the positions and the other is the meta-reasoning capability, which allows it to explore some paths in the tree very deeply and to abandon other paths very quickly.

So this word meta-reasoning, while technically correct, inspires perhaps the wrong degree of power that AlphaGo has, for example.

So let me ask you, sort of, so you were part of the symbolic AI world for a while, like where AI was, There's a lot of excellent, interesting ideas there that unfortunately met a winter.

Oh, so I would say, yeah, it's not quite as simple as that.

So The AI winter, the first winter that was actually named as such was the 1 in the late 80s.

And that came about because in the mid-80s there was really a concerted attempt to push AI out into the real world using what was called expert system technology.

And for the most part, that technology was just not ready for prime time.

They were trying in many cases to do a form of uncertain reasoning, judgment, combinations of evidence, diagnosis, those kinds of things, which was simply invalid.

And when you try to apply invalid reasoning methods to real problems You can fudge it for small versions of the problem, but when it starts to get larger the thing just falls apart so many companies found that the stuff just didn't work, and they were spending tons of money on consultants to try to make it work.

There were other practical reasons, like they were asking the companies to buy incredibly expensive Lisp machine workstations, which were literally between 50 and $100,000 in, you know, in 1980s money, which was, would be like between 150 and $300,000 per workstation in current prices.

And the bottom line, they weren't seeing a profit from it.

Every major company was starting an AI department, just like now.

And I worry a bit that we might see similar disappointments.

Not because the current technology is invalid, but it's limited in its scope.

And it's almost the dual of the scope problems that expert systems had.

So- what have you learned from that hype cycle and what can we do to prevent another winter, for example?

Yeah, so when I'm giving talks these days, that's 1 of the warnings that I give.

1 is that rather than data being the new oil, data is the new snake oil.

And then the other is that we might see a kind of very visible failure in some of the major application areas, and I think self-driving cars would be the flagship.

And I think when you look at the history, So the first self-driving car was on the freeway, driving itself, changing lanes, overtaking in 1987.

And that kind of looks like where we are today, right?

You know, prototypes on the freeway, changing lanes and overtaking.

Now I think significant progress has been made, particularly on the perception side.

So we worked a lot on autonomous vehicles in the early mid-90s at Berkeley.

We put congressmen into self-driving cars and had them zooming along the freeway.

So in simulation, with perfect perception, you could actually show that you can drive safely for a long time, even if the other cars are misbehaving and so on.

But simultaneously we worked on machine vision for detecting cars and tracking pedestrians and so on, and we couldn't get the reliability of detection and tracking up to a high enough level, particularly in bad weather conditions, nighttime rain fog.

Good enough for demos, but perhaps not good enough to cover the general operation.

Yeah, see the thing about driving is, so suppose you're a taxi driver, and you drive every day, 8 hours a day for 10 years, that's a hundred million seconds of driving.

And any 1 of those seconds, you can make a fatal mistake.

So you're talking about 8 nines of reliability, right?

Now if your vision system only detects 98.3% of the vehicles, right?

Then that's sort of, you know, 1 and a bit nines of reliability.

So you have another 7 orders of magnitude to go.

And this is what people don't understand.

They think, oh, because I had a successful demo, I'm pretty much done.

But you're not even within 7 orders of magnitude of being done.

And it's not the, can I follow a white line?

We follow a white line all the way across the country.

The edge cases, other drivers doing weird things.

So if you talk to Google, right, so they had actually a very classical architecture where you had machine vision which would detect all the other cars and pedestrians and the white lines and the road signs, and then basically that was fed into a logical database.

And then you had a classical 1970s rule-based expert system telling you, okay, if you're in the middle lane and there's a bicyclist in the right lane, who is signaling this, then, then, then, then do that.

And what they found was that every day they'd go out and there'd be another situation that the rules didn't cover.

You know, so they'd come to a traffic circle and there's a little girl riding her bicycle the wrong way around the traffic circle.

And then you, you know, they'd come back and add more rules and they just found that this was not really converging.

And if you think about it, how do you deal with an unexpected situation, meaning 1 that you've never previously encountered, and the reasoning required to figure out the solution for that situation has never been done.

It doesn't match any previous situation in terms of the kind of reasoning you have to do.

Well, you know, in chess programs this happens all the time.

You're constantly coming up with situations you haven't seen before, and you have to reason about them, and you have to think about, okay, here are the possible things I could do, here are the outcomes, here's how desirable the outcomes are, and then pick the right 1.

You know, in the 90s we were saying, okay, this is how you're going to have to do automated vehicles.

They're going to have to have look-ahead capability.

But the look-ahead for driving is more difficult than it is for chess

the other right There's humans and they're less predictable than chess pieces.

Then it well then when you have an opponent in chess Who's also somewhat unpredictable?

But for example in chess, you always know the opponent's intention.

They're trying to beat you Right, whereas in driving you don't know is this guy trying to turn left, or has he just forgotten to turn off his turn signal, or is he drunk, or is he changing the channel on his radio, or whatever it might be, you've got to try and figure out the mental state, the intent of the other drivers to forecast the possible evolutions of their trajectories.

And then you've got to figure out, okay, which is the trajectory for me that's going to be safest?

And those all interact with each other because the other driver is going to react to your trajectory and so on.

So you've got the classic merging onto the freeway problem where you're kind of racing a vehicle that's already on the freeway and you either pull ahead of them or you're going to let them go first and pull in behind and you get this sort of uncertainty about who's going first.

So all those kinds of things mean that you need a decision-making architecture that's very different from either a rule-based system or it seems to me a kind of an end-to-end neural network system.

You know, so just as AlphaGo is pretty good when it doesn't do any look-ahead but it's way way way way better when it does, I think the same is going to be true for driving.

You can have a driving system that's pretty good when it doesn't do any look ahead, but that's not good enough.

You know, and we've already seen multiple deaths caused by poorly designed machine learning algorithms that don't really understand what they're doing.

the perception side, there's mistakes being made by those algorithms where the perception is very shallow.

On the planning side, the look ahead, like you said.

And the thing that we come up against that's really interesting when you try to deploy systems in the real world is you can't think of an artificial intelligence system as a thing that responds to the world always.

You have to realize that it's an agent that others will respond to as well.

So in order to drive successfully, you can't just try to do obstacle avoidance.

Right, you can't pretend that you're invisible.

I mean, but you have to assert, yet others have to be scared of you.

Just, we're all, there's this tension, there's this game.

So we study a lot of work with pedestrians.

If you approach pedestrians as purely an obstacle avoidance, so you're doing look ahead as in modeling the intent, they're not going to, they're going to take advantage of you, They're not going to respect you at all.

There has to be a tension, a fear, some amount of uncertainty.

Yeah, or at least just a kind of a resoluteness.

See all Lex Fridman transcripts on Youtube

Stuart Russell: Long-Term Future of Artificial Intelligence | Lex Fridman Podcast #9