David Silver leads the reinforcement learning research group at DeepMind and was lead researcher on AlphaGo, AlphaZero and co-lead on AlphaStar, and MuZero and lot of important work in reinforcement learning.

Support this podcast by signing up with these sponsors:
- MasterClass: https://masterclass.com/lex
- Cash App - use code "LexPodcast" and download:
- Cash App (App Store): https://apple.co/2sPrUHe
- Cash App (Google Play): https://bit.ly/2MlvP5w

EPISODE LINKS:
Reinforcement learning (book): https://amzn.to/2Jwp5zG

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
4:09 - First program
11:11 - AlphaGo
21:42 - Rule of the game of Go
25:37 - Reinforcement learning: personal journey
30:15 - What is reinforcement learning?
43:51 - AlphaGo (continued)
53:40 - Supervised learning and self play in AlphaGo
1:06:12 - Lee Sedol retirement from Go play
1:08:57 - Garry Kasparov
1:14:10 - Alpha Zero and self play
1:31:29 - Creativity in AlphaZero
1:35:21 - AlphaZero applications
1:37:59 - Reward functions
1:40:51 - Meaning of life

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with David Silver, who leads the reinforcement learning research group at DeepMind and was the lead researcher on AlphaGo, AlphaZero, and co-led the AlphaStar and MuZero efforts and a lot of important work in reinforcement learning in general.

I believe AlphaZero is 1 of the most important accomplishments in the history of artificial intelligence.

And David is 1 of the key humans who brought AlphaZero to life together with a lot of other great researchers at DeepMind.

We were both jet lagged, but didn't care and made it happen.

It was a pleasure and truly an honor to talk with David.

This conversation was recorded before the outbreak of the pandemic.

For everyone feeling the medical, psychological, and financial burden of this crisis, I'm sending love your way.

Stay strong, we're in this together, we'll beat this thing.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, review it with 5 stars on Apple Podcasts, support on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D-M-A-N.

As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation.

I hope that works for you and doesn't hurt the listening experience.

Please consider supporting the podcast by signing up to Masterclass at masterclass.com slash Lex, and downloading Cash App and using code LexPodcast.

This show is presented by Cash App, the number 1 finance app in the App Store.

Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1.

Since Cash App allows you to buy Bitcoin, let me mention that cryptocurrency in the context of the history of money is fascinating.

I recommend Ascent of Money as a great book on this history.

Debits and credits on ledgers started around 30,000 years ago.

The US dollar created over 200 years ago.

And Bitcoin, the first decentralized cryptocurrency, released just over 10 years ago.

So given that history, cryptocurrency is still very much in its early days of development, but it's still aiming to, and just might, redefine the nature of money.

So again, If you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10, and Cash App will also donate $10 to THIRST, an organization that is helping to advance robotics and STEM education for young people around the world.

Sign up at masterclass.com slash Lex to get a discount and to support this podcast.

In fact, for a limited time now, if you sign up for an all access pass for a year, you get to get another all access pass to share with a friend.

When I first heard about Masterclass, I thought it was too good to be true.

For $180 a year, you get an all access pass to watch courses from, to list some of my favorites.

Chris Hadfield on space exploration, Neil deGrasse Tyson on scientific thinking and communication, Will Wright, the creator of SimCity and Sims on game design, Jane Goodall on conservation, Carlos Santana on guitar, his song Europa could be the most beautiful guitar song ever written.

Garry Gasparo on chess, Daniel Negrano on poker, and many, many more.

Chris Hadfield explaining how rockets work and the experience of being launched into space alone is worth the money.

For me, the key is to not be overwhelmed by the abundance of choice.

Pick 3 courses you want to complete, watch each of them all the way through.

It's not that long, but it's an experience that will stick with you for a long time, I promise.

You can watch it on basically any device.

Once again, Sign up on masterclass.com slash Lex to get a discount and to support this podcast.

And now here's my conversation with David Silver.

What was the first program you ever written and what programming language?

Yeah, my parents brought home this BBC Model B microcomputer.

It was just this fascinating thing to me.

I was about 7 years old and couldn't resist just playing around with it.

So I think first program ever was writing my name out in different colors and getting it to loop and repeat that.

And there was something magical about that, which just led to more and more.

How did you think about computers back then?

Like the magical aspect of it, that you can write a program and there's this thing that you just gave birth to that's able to create sort of visual elements and live on its own.

Or did you not think of it in those romantic notions?

It was something where, you know, there was this limitless possibilities.

Once you have a computer in front of you, you can do anything with it.

I used to play with Lego with the same feeling.

You can make anything you want out of Lego, but even more so with a computer.

You're not constrained by the amount of kit you've got.

And so I was fascinated by it and started pulling out the user guide and the advanced user guide and then learning.

So I started in basic and then later 6502.

My father also became interested in this machine and gave up his career to go back to school and study for a master's degree in artificial intelligence, funnily enough, at Essex University when I was 7.

So I was exposed to those things at an early age.

He showed me how to program in Prolog and do things like querying your family tree and those are some of my earliest memories of trying to figure things out on a computer.

Those are the early steps in computer science programming, but when did you first fall in love with artificial intelligence, or with the ideas, the dreams of AI?

I think it was really when I went to study at university, so I was an undergrad at Cambridge and studying computer science.

And I really started to question, you know, what really are the goals?

Where do we want to go with computer science?

And it seemed to me that the only step of major significance to take was to try and recreate something akin to human intelligence.

If we could do that, that would be a major leap forward.

And that idea, I certainly wasn't the first to have it, but it nestled within me somewhere and became like a bug, you know, I really wanted to crack that problem.

So you thought it was, like you had a notion that this is something that human beings can do, that it is possible to create an intelligent machine?

Well, I mean, unless you believe in something metaphysical, then what are our brains doing?

Well, at some level, they're information processing systems, which are able to take whatever information is in there, transform it through some form of program and produce some kind of output, which enables that human being to do all the amazing things that they can do in this incredible world.

So then do you remember the first time you've written a program that, because you also had an interest in games, do you remember the first time you were in a program that beat you in a game?

Sort of achieved super David Silver level performance?

So for 5 years, I programmed games for my first job.

So it was an amazing opportunity to get involved in a startup company.

And so I was involved in building AI at that time.

And so for sure, there was a sense of building handcrafted what people used to call AI in the games industry, which I think is not really what we might think of as AI in its fullest sense, but something which is able to take actions in a way which makes things interesting and challenging for the human player.

And at that time, I was able to build these handcrafted agents, which in certain limited cases could do things which were able to do better than me, but mostly in these kind of twitch-like scenarios where they were able to do things faster or because they had some pattern which was able to exploit repeatedly.

I think if we're talking about real AI, the first experience for me came after that when I realized that this path I was on wasn't taking me towards, it wasn't dealing with that bug which I still had inside me to really understand intelligence and try and solve it.

Everything people were doing in games was short-term fixes rather than long-term vision.

So I went back to study for my PhD, which was, funnily enough, trying to apply reinforcement learning to the game of Go.

I built my first Go program using reinforcement learning, a system which would by trial and error play against itself and was able to learn which patterns were actually helpful to predict whether it was going to win or lose the game.

And then choose the moves that led to the combination of patterns that would mean that you're more likely to win.

I mean, was there sort of the, yeah, is it's a mix of a sort of excitement and Was there a tinge of sort of like, almost like a fearful awe?

You know, it's like in space, 2001, Space Odyssey, kind of realizing that you've created something that, that is, you know, that's achieved human level intelligence in this 1 particular little task.

And in that case, I suppose neural networks weren't involved.

There were no neural networks in those days.

But it was a principled self-learning system based on a lot of the principles which people are still using in deep reinforcement learning.

I think I found it immensely satisfying that a system which was able to learn from first principles for itself was able to reach the point that it was understanding this domain better than I could and able to outwit me.

I don't think it was a sense of awe, it was a sense that satisfaction, that something I felt should work, had worked.

So to me, AlphaGo, and I don't know how else to put it, but to me, AlphaGo and AlphaGo 0, mastering the game of Go is, again, to me, the most profound and inspiring moment in the history of artificial intelligence.

So you're 1 of the key people behind this achievement, and I'm Russian, so I really felt the first sort of seminal achievement when Deep Blue beat Garry Kasparov in 1987.

So as far as I know, the AI community at that point largely saw the game of Go as unbeatable in AI, using the sort of the state of the art brute force methods, search methods.

Even if you consider, at least the way I saw it, even if you consider arbitrary exponential scaling of compute Go would still not be solvable, hence why it was thought to be impossible.

So, given that the game of Go was impossible to master, what was the dream for you?

You just mentioned your PhD thesis of building the system that plays Go.

What was the dream for you that you could actually build a computer program that achieves world-class, not necessarily beats the world champion, but achieves that kind of level of play and go.

And funnily enough, I just came from a panel where I was actually in a conversation with Garry Kasparov and Murray Campbell, who was the author of Deep Blue.

And it was their first meeting together since the match.

So I'm literally fresh from that experience.

So these are amazing moments when they happen, but where did it all start?

Well for me it started when I became fascinated in the game of Go.

So Go for me, I've grown up playing games, I've always had a fascination in board games.

I played chess as a kid, I played Scrabble as a kid.

When I was at university I discovered the game of Go and to me it just blew all of those other games out of the water.

It was just so deep and profound in its complexity with endless levels to it.

What I discovered was that I could devote endless hours to this game.

And I knew in my heart of hearts that no matter how many hours I would devote to it, I would never become a, you know, AAA grandmaster, or there was another path and the other path was to try and understand how you could get some other intelligence to play this game better than I would be able to.

And so even in those days, I had this idea that, what if, what if it was possible to build a program that could crack this?

And as I started to explore the domain, I discovered that this was really the domain where people felt deeply that if progress could be made in Go, it would really mean a giant leap forward for AI.

It was the challenge where all other approaches had failed.

This is coming out of the era you mentioned, which was in some sense the golden era for the classical methods of AI like heuristic search in the 90s.

They all fell 1 after another, not just chess with deep blue, but checkers, backgammon, Othello.

There were numerous cases where systems built on top of heuristic search methods with these high performance systems had been able to defeat the human world champion in each of those domains.

And yet in that same time period, there was a million dollar prize available for the game of Go for the first system to be a human professional player.

And at the end of that time period, in year 2000 when the prize expired, the strongest Go program in the world was defeated by a 9 year old child when that 9 year old child was giving 9 free moves to the computer at the start of the game to try and even things up.

And computer go expert beat that strongest, same strongest program with 29 handicapped stones, 29 free moves.

So that's what the state of affairs was when I became interested in this problem in around 2003 when I started working on Computer Go.

There was just, there was very, very little in the way of progress towards meaningful performance again, at anything approaching human level.

And so people, they, it wasn't through lack of effort.

And so there was a strong sense that something different would be required for Go than had been needed for all of these other domains where AI had been successful.

And maybe the single clearest example is that Go, unlike those other domains, had this kind of intuitive property that a Go player would look at a position and say, Hey, you know, here's this mess of black and white stones.

But from this mess, oh, I can, I can predict that, that this part of the board has become my territory?

This part of the board has become your territory.

And I've got this overall sense that I'm going to win and that this is about the right move to play.

And that intuitive sense of judgment, of being able to evaluate what's going on in a position, it was pivotal to humans being able to play this game and something that people had no idea how to put into computers.

So this question of how to evaluate a position, how to come up with these intuitive judgments, was the key reason why Go was so hard, in addition to its enormous search base, and the reason why methods which had succeeded so well elsewhere failed in Go.

And so people really felt deep down that in order to crack Go, we would need to get something akin to human intuition.

And if we got something akin to human intuition, we'd be able to solve many, many more problems in AI.

So to me, that was the moment where it's like, okay, this is not just about playing the game of Go, this is about something profound.

And it was back to that bug, which had been itching me all those years, now this is the opportunity to do something meaningful and transformative and I guess a dream was born.

That's a really interesting way to put it.

So almost this realization that you need to find, formulate goals, a kind of a prediction problem versus a search problem was the intuition.

I mean, maybe that's the wrong crude term, but to give it the ability to kind of intuit things about positional structure of the board.

Now, okay, but what about the learning part of it?

Did you have a sense that you have to, that learning has to be part of the system?

Again, something that hasn't really, as far as I think, except with TD Gammon and the 90s with RL a little bit, hasn't been part of those state-of-the-art game playing systems.

So I strongly felt that learning would be necessary.

And that's why my PhD topic back then was trying to apply reinforcement learning to the game of Go.

And not just learning of any type, but I felt that the only way to really have a system to progress beyond human levels of performance wouldn't just be to mimic how humans do it, but to understand for themselves.

And how else can a machine hope to understand what's going on except through learning?

If you're not learning, what else are you doing?

Well, you're putting all the knowledge into the system.

And that just feels like something which decades of AI have told us is maybe not a dead end, but certainly has a ceiling to the capabilities.

It's known as the knowledge acquisition bottleneck, that the more you try to put into something, the more brittle the system becomes.

That's the only way you're going to be able to get a system which has sufficient knowledge in it, millions and millions of pieces of knowledge, billions, trillions, of a form that it can actually apply for itself and understand how those billions and trillions of pieces of knowledge can be leveraged in a way which will actually lead it towards its goal without conflict or other issues.

Yeah, I mean, if I put myself back in that time, I just wouldn't think like that.

Without a good demonstration of RL I would think more in the symbolic AI, like not learning but sort of a simulation of knowledge base, like a growing knowledge base, but it would still be sort of pattern-based, like basically have little rules that you kind of assemble together into a large knowledge base.

Well, in a sense, that was the state of the art back then.

So if you look at the Go programs, which had been competing for this prize I mentioned, they were an assembly of different specialized systems, some of which used huge amounts of human knowledge to describe how you should play the opening, how you should, all the different patterns that were required to play well in the game of Go, end game theory, combinatorial game theory, and combined with more principled search-based methods which were trying to solve for particular sub-parts of the game, like life and death, connecting groups together, all these amazing sub-problems that just emerge in the game of Go.

There were different pieces all put together into this collage, which together would try and play against a human.

Although not all of the pieces were handcrafted, the overall effect was nevertheless still brittle and it was hard to make all these pieces work well together.

So really, what I was pressing for and the main innovation of the approach I took was to go back to first principles and say, well, let's back off that and try and find a principled approach where the system can learn for itself.

If you try something, did that help or did it not help?

And only through that procedure can you arrive at knowledge which is verified.

The system has to verify it for itself, not relying on any other third party to say this is right or this is wrong.

And so that principle was already very important in those days, that unfortunately we were missing some important pieces back then.

So before we dive into maybe discussing the beauty of reinforcement learning.

Let's take a step back, we kind of skipped it

The elements of it perhaps contrasting to chess that sort of you really enjoy as a human being and also that make it really difficult as a AI machine learning problem.

So the game of Go has remarkably simple rules.

In fact, so simple that people have speculated that if we were to meet alien life at some point that we wouldn't be able to communicate with them, but we would be able to play a game of Go with them.

So they probably have discovered the same rule set.

So the game is played on a 19 by 19 grid, and you play on the intersections of the grid and the players take turns.

It's to surround as much territory as you can, as many of these intersections with your stones and to surround more than your opponent does.

And the only nuance to the game is that if you fully surround your opponent's piece then you get to capture it and remove it from the board and it counts as your own territory.

Now from those very simple rules immense complexity arises.

There's kind of profound strategies in how to surround territory, how to kind of trade off between making solid territory yourself now compared to building up influence that will help you acquire territory later in the game, how to connect groups together, how to keep your own groups alive, which patterns of stones are most useful compared to others.

There's just immense knowledge and human Go players have played this game for, it was discovered thousands of years ago and human Go players have built up this immense knowledge base over the years.

It's studied very deeply and played by something like 50 million players across the world, mostly in China, Japan, and Korea, where it's an important part of the culture, so much so that it's considered 1 of the 4 ancient arts that was required by Chinese scholars.

So if I was to compare to chess, chess is in the same way as it is in Chinese culture for Go and chess in Russia is also considered 1 of the sacred arts.

So if we contrast Go with chess, there's interesting qualities about Go.

Maybe you can correct me if I'm wrong, but the evaluation of a particular static board is not as reliable.

In chess, you can kind of assign points to the different units.

And it's kind of a pretty good measure of who's winning, who's losing.

Yeah, So in the game of Go, you find yourself in a situation where both players have played the same number of stones.

Actually captures a strong level of play happen very rarely, which means that at any moment in the game, you've got the same number of white stones and black stones.

And the only thing which differentiates how well you're doing is this intuitive sense of where are the territories ultimately going to form on this board.

And if you look at the complexity of a real go position, it's mind boggling that kind of question of what will happen in 300 moves from now when you see just a scattering of 20 white and black stones intermingled.

And so that challenge is the reason why position evaluation is so hard in Go compared to other games.

In addition to that, it has an enormous search space.

So there's around 10 to the 170 positions in the game of Go.

And that search space is so great that traditional heuristic search methods that were so successful in things like Deep Blue and chess programs just kind of fall over and go.

So at which point did reinforcement learning enter your life, your research life, your way of thinking?

We just talked about learning, but reinforcement learning is

a very particular kind of learning, 1 that's both philosophically sort of profound, but also 1 that's pretty difficult to get to work, as

So when did that enter your life And how did that work progress?

So I had just finished working in the games industry at this startup company.

And I took a year out to discover for myself exactly which path I wanted to take.

I knew I wanted to study intelligence, but I wasn't sure what that meant at that stage.

I really didn't feel I had the tools to decide on exactly which path I wanted to follow.

And 1 of the things I read was Sutton and Bartow, the sort of seminal textbook on an introduction to reinforcement learning.

And when I read that textbook, I just had this resonating feeling that this is what I understood intelligence to be.

And this was the path that I felt would be necessary to go down to make progress in AI.

So I got in touch with Rich Sutton and asked him if he would be interested in supervising me on a PhD thesis in computer go.

And he basically said that if he's still alive, he'd be happy to.

But unfortunately, he'd been struggling with very serious cancer for some years, and he really wasn't confident at that stage that he'd even be around to see the end of it.

But fortunately, that part of the story worked out very happily.

They've got a great games group out there with a history of fantastic work in board games as well, as Rich Sutton, the father of RL.

So it was the natural place for me to go, in some sense, to study this question.

And the more I looked into it, the more strongly I felt that this wasn't just the path to progress in computer go, but really, this was the thing I'd been looking for.

This was really an opportunity to frame what intelligence means.

Like what are the goals of AI in a single clear problem definition such that if we're able to solve that clear single problem definition, in some sense we've cracked the problem of AI.

So to you, reinforcement learning ideas, at least sort of echoes of it, would be at the core of intelligence.

It is at the core of intelligence, And if we ever create a human level intelligence system, it would be at the core of that kind of system.

Let me say it this way, that I think it's helpful to separate out the problem from the solution.

So I see the problem of intelligence, I would say it can be formalized as the reinforcement learning problem, and that that formalization is enough to capture most, if not all of the things that we mean by intelligence.

See all Lex Fridman transcripts on Youtube

David Silver: AlphaGo, AlphaZero, and Deep Reinforcement Learning | Lex Fridman Podcast #86