The following is a conversation with Ariel Vinales.

He's a senior research scientist at Google DeepMind, and before that, he was at Google Brain and Berkeley.

His research has been cited over 39,000 times.

He's truly 1 of the most brilliant and impactful minds in the field of deep learning.

the biggest papers and ideas in AI, including sequence to sequence learning, audio generation, image captioning, neural machine translation, and of course, reinforcement learning.

He's a lead researcher of the AlphaStar Project, creating an agent that defeated a top professional at the game of StarCraft.

This conversation is part of the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, iTunes,

or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Ariel Vinales.

You spearheaded the DeepMind team behind AlphaStar that recently beat a top professional player at StarCraft.

So you have an incredible wealth of work and deep learning in a bunch of fields, but let's talk about StarCraft first.

Let's go back to the very beginning, even before AlphaStar, before DeepMind, before deep learning first, what came first for you, A love for programming or a love for video games?

I think for me, it definitely came first, the drive to play video games.

I didn't really code much, but what I would do is I would just mess with the computer, break it and fix it.

That was the level of skills I guess that I gained in my very early days.

And then I really got into video games, especially Starcraft, actually, the first version.

I spent most of my time just playing kind of pseudo professionally, as professionally as you could play back in 98 in Europe, which was not a very mainstream like what's called nowadays eSports.

So as a player, I tended to try to play not many games, not to kind of disclose the strategies that I kind of developed.

And I like to play random, actually, not in competitions, but just to, I think in StarCraft, there's well, there's 3 main races and I found it very useful to play with all of them.

And so I would choose random many times, even sometimes in tournaments, to gain skill on the 3 races, because it's not how you play against someone, but also if you understand the race because you played, you also understand what's annoying, what then when you're on the other side, what to do to annoy that person, to try to gain advantages here and there and so on.

So I actually played random, although I must say in terms of favorite race, I really like Zerk.

I was probably best at Zerk, and that's probably what I tend to use towards the end of my career before starting university.

Could you try to describe StarCraft to people that may never have played video games, especially the massively online variety

So StarCraft is a real-time strategy game.

And the way to think about StarCraft, perhaps if you understand a bit chess, is that there's a board which is called map or like, yeah, like the map where people play against each other.

There's obviously many ways you can play, but the most interesting 1 is the 1 versus 1 setup where you just play against someone else or even the built-in AI, right?

Blizzard put a system that can play the game reasonably well, if you don't know how to play.

And then in this board, you have, again, pieces like in chess, but these pieces are not there initially like they are in chess.

You actually need to decide to gather resources, to decide which pieces to build.

So in a way you're starting almost with no pieces.

You start gathering resources in StarCraft, there's minerals and gas that you can gather.

And then you must decide how much do you want to focus, for instance, on gathering more resources or starting to build units or pieces.

And then once you have enough pieces or maybe like attack, you know, a good attack composition, then you go and attack the other side of the map.

And now the other main difference with chess is that you don't see the other side of the map.

So you're not seeing the moves of the enemy.

So as a result, you must not only decide trading off economy versus building your own units, but you also must decide whether you want to scout to gather information, but also by scouting you might be giving away some information that you might be hiding from the enemy.

So there's a lot of complex decision-making, all in real time.

There's also, unlike chess, this is not a turn-based game.

You play basically all the time, continuously, and thus some skill in terms of speed and accuracy of clicking is also very important.

And people that train for this really play this game at an amazing skill level.

I've seen many times this, and if you can witness this live, it's really, really impressive.

So In a way, it's kind of a chess where you don't see the other side of the board.

You're building your own pieces and you also need to gather resources to basically get some money to build other buildings, pieces, technology and so on.

From the perspective of the human player, the difference between that and chess, or maybe that and a game like turn-based strategy like Heroes of Might and Magic, is that there's an anxiety because you have to make these decisions really quickly.

And if you are not actually aware of what decisions work, it's a very stressful balance.

Everything you describe is actually quite stressful, difficult to balance for an amateur human player.

I don't know if it gets easier at the professional level, like if they're fully aware of what they have to do, but at the amateur level there's this anxiety, oh crap I'm being attacked, oh crap I have to build up resources, oh I have to probably expand, and all these of the time, the real-time strategy aspect is really stressful and computation I'm sure difficult.

We'll get into it, But for me, Battle.net, so StarCraft was released in

And Blizzard Battle.net with Diablo in 96 came out.

And to me, it might be a narrow perspective, but it changed online gaming and perhaps society forever.

But I may have made way too narrow a viewpoint, but from your perspective, can you talk about the history of gaming over the past 20 years?

Is this, how transformational, how important is this line of games?

So I think I kind of was an active gamer whilst this was developing, the internet, online gaming.

So for me, the way it came was I played other games, strategy related.

I played a bit of Common and Conquer, and then I played Warcraft 2, which is from Blizzard.

I didn't understand about what Blizzard was or anything.

Warcraft 2 was just a game, which was actually very similar to Starcraft in many ways.

It's also a real-time strategy game where there's orcs and humans, so there's only 2 races.

So I remember a friend of mine came to school saying, oh, there's this new cool game called StarCraft.

And I just said, oh, this sounds like just a copy of Warcraft 2 until I kind of installed it.

And at the time I am from Spain, so we didn't have very good internet.

So there was, for us, StarCraft became first kind of an offline experience where you kind of start to play these missions, right?

You play against some sort of scripted things to develop the story of the characters in the game.

And then later on, I start playing against the built-in AI, and I thought it was impossible to defeat it.

Then eventually you defeat 1 and you can actually play against 7 built-in AIs at the same time, which also felt impossible.

But actually, it's not that hard to beat 7 built-in AIs at once.

So once we achieved that, also, we discovered that we could play, as I said, internet wasn't that great, but we could play with LAN, right?

On like basically against each other if we were in the same place, because you could just connect machines with cables.

So we started playing in LAN mode as a group of friends, and it was really, really much more entertaining than playing against AIs.

And later on, as the internet was starting to develop and being a bit faster and more reliable, then it's when I started experiencing Battle.net, which is this amazing universe, not only because of the fact that you can play the game against anyone in the world, but you can also get to know more people.

You just get exposed to now like this vast variety of it's kind of a bit when the chats came about, right?

There was a chat system, you could play against people, but you could also chat with people, not only about Stalker, but about anything.

And that became a way of life for kind of 2 years.

And obviously then it became like kind of it exploded in me that I started to play more seriously, going to tournaments and so on, so forth.

have a sense on a societal, sociological level, what's this whole part of society that many of us are not aware of, and it's a huge part of society, which is gamers?

I mean, every time I come across that in YouTube or streaming sites, I mean, this is a huge, the huge number of people play games religiously.

Do you have a sense of those folks, especially now that you've returned to that realm a little bit on the AI side?

So in fact, I, even after StarCraft, I actually played World of Warcraft, which is maybe the main sort of online world and presence that you get to interact with lots of people.

It was, to me, it was a bit less stressful than StarCraft because winning was kind of a given.

You just put in this world and you can always complete missions.

But I think it was actually the social aspect of especially Starcraft first and then games like World of Warcraft really shaped me in a very interesting way, because what you get to experience is just people you wouldn't usually interact with.

So even nowadays, I still have many Facebook friends from the era where I played online, and their ways of thinking, even political, we don't live in it, we don't interact in the real world, but we were connected by basically fiber.

And that way I actually get to understand a bit better that we live in a diverse world.

And these were just connections that were made by because I happened to go in a city, in a virtual city as a priest, and I met this warrior and we became friends and then we started playing together.

So I think it's transformative and more and more and more people are more aware of it.

But back in the day, as you were saying, in

even, it was very, still very strange thing to do, especially in Europe.

I think there were exceptions, like Korea, for instance, it was amazing that everything happened so early in terms of cyber cafes.

Like if you go to Seoul, it's a city that back in the day, Starcraft was kind of, you could be a celebrity by playing Starcraft.

So yeah, it's quite interesting to look back.

The same way, of course, like technology and social networks and so on are also transforming things.

And a quick tangent, let me ask, you're also 1 of the most productive people in your particular chosen passion and path in life.

And yet you're also appreciate and enjoy video games.

Do you think it's possible to do, to enjoy video games in moderation?

Someone told me that you could choose 2 out of 3.

When I was playing video games, you could choose having a girlfriend, playing video games or studying.

And I think for the most part, it was relatively true.

Games like StarCraft, if you take the game pretty seriously and you want to study it, then you obviously will dedicate more time to it.

And I definitely took gaming and obviously studying very seriously.

So to me, especially when I started university undergrad, I kind of stepped off StarCraft.

And then World of Warcraft was a bit more casual.

But as I said, that was not as much time investment as it was for me in StarCraft.

What are the, you're behind the team, so DeepMind's been working on StarCraft and released a bunch of cool open source agents and so on in the past few years, but AlphaStar really is the moment where the first time you beat a world class player.

So what are the parameters of the challenge in the way that AlphaStar took it on?

And how did you and David and the rest of the DeepMind team get into it?

Consider that you can even beat the best in the world or top players.

I think it was 2014 when DeepMind was acquired by Google.

And I, at the time, was at Google Brain, which was in California, is still in California.

We had this summit where we got together, the 2 groups, so Google Brain and Google DeepMind got together and we gave a series of talks.

And given that they were doing deep reinforcement learning for games, I decided to bring up part of my past, which I had developed at Berkeley, like this thing which we call Berkeley Overmind, which is really just a Starcraft 1 bot, right?

And I remember Demis just came to me and said, well, Maybe not now, it's perhaps a bit too early, but you should just come to DeepMind and do this again with deep reinforcement learning, right?

And at the time, it sounded very science fiction for several reasons.

But then in 2016, when I actually moved to London and joined DeepMind, transferring from Brain, it became apparent that because of the AlphaGo moment and kind of Blizzard reaching out to us to say, wait, like, do you want the next challenge?

And also me being full time at DeepMind, So sort of kind of all these came together.

And then I was I went to to Irvine in California, to the Blizzard headquarters, to just chat with them and try to explain how would it all work before you do anything.

And the approach has always been about the learning perspective.

So in Berkeley, we did a lot of rule-based, you know, conditioning and if you have more than 3 units, then go attack.

And if the other has more units than me, I retreat and so on and so forth.

And of course, the point of deep reinforcement learning, deep learning, machine learning in general is that all these should be learned behavior.

So that kind of was the DNA of the project since its inception in 2016, where we just didn't even have an environment to work with.

So if you go back to that conversation with Demis or even in your own head, how far away did you, because we're talking about Atari games,

we're talking about Go, which is kind of, if you're honest about it, really far away from Starcraft.

Well, now that you've beaten it, maybe you could say it's close, but it seems like Starcraft is way harder than Go, philosophically and mathematically speaking.

Yeah, when I kind of thought about, okay, I'm gonna dedicate a lot of my time and focus on this.

And obviously I do a lot of different research in deep learning.

So spending time on it, I mean, I really had to kind of think there's going to be something good happening out of this.

So really, I thought, well, this sounds impossible.

And it probably is impossible to do the full thing, like the full game, where you play 1 versus 1, and it's only a neural network playing and so on.

So it really felt like, I just didn't even think it was possible.

But on the other hand, I could see some stepping stones, like towards that goal.

Clearly you could define sub problems in StarCraft and sort of dissect it a bit and say, okay, here is a part of the game, here is another part.

So this was really also critical to me, the fact that we could access human replays, right?

And in fact, they open sourced this for the whole community where you can just go.

And it's not every single StarCraft game ever played, but it's a lot of them.

And every day they will you can just query a data set and say, well, give me all the games that were played today.

And given my kind of experience with language and sequences and supervised learning, I thought, well, that's definitely going to be very helpful and something quite unique now, because ever before we had such a large data set of replays of people playing the game at this scale of such a complex video game, right?

And as soon as I knew that Blizzard was able to kind of give this to the community, I started to feel positive about something non-trivial happening.

But I also thought the full thing, like really no rules, no single line of code that tries to say, well, I mean, if you see this, you need to build a detector.

All these, not having any of these specializations seemed really, really, really difficult to me.

I do also like that Blizzard was teasing or even trolling you.

Sort of almost, yeah, pulling you in into this really difficult challenge.

What's the interest from the perspective of Blizzard, except just curiosity?

Yeah, I think Blizzard has really understood and really bring forward this competitiveness of esports in games.

Starcraft really sparked a lot of something that almost was never seen, especially as I was saying back in Korea.

So they just probably thought, well, this is such a pure 1 versus 1 setup that it would be great to see if something that can play Atari or Go and then later on chess could even tackle this kind of complex real-time strategy game.

So for them, they wanted to see first, obviously, whether it was possible, if the game they created was in a way solvable to some extent.

And I think on the other hand, they also are a pretty modern company that innovates a lot.

So just starting to understand AI for them, how to bring AI into games is not AI for games, but games for AI.

And we obviously at DeepMind use games for AI to drive AI progress, but Blizzard might actually be able to do, and many other companies, to start to understand and do the opposite.

So I think that is also something they can get out of this.

And they definitely, we have brainstormed a lot about this, right?

But 1 of the interesting things to me about Starcraft and Diablo and these games that Blizzard has created is the task of balancing classes, for example, sort of making the game fair from the starting point and then let skill determine the outcome.

Is there, I mean, can you first comment, there's 3 races, Zerg, Protoss, and Terran.

I don't know if I've ever said that out loud.

Yeah, I don't think I've ever in person interacted with anybody about Starcraft, That's funny.

I wonder if the AI, the work that you're doing with AlphaStar would help balance them even further.

Is that something that Blizzard is thinking about?

So balancing when you add a new unit or a new spell type is obviously possible, given that you can always train or pre-train at scale some agent that might start using that in unintended ways.

But I think actually, if you understand how Starcraft has kind of co-evolved with players, in a way, I think it's actually very cool the ways that many of the things and strategies that people came up with, right?

So I think it's we've seen it over and over in StarCraft that Blizzard comes up with maybe a new unit, and then some players get creative and do something kind of unintentional or something that Blizzard designers just simply didn't test or think about.

And then after that becomes kind of mainstream in the community, Blizzard patches the game, and then they maybe weaken that strategy or make it actually more interesting, but a bit more balanced.

So this kind of continual talk between players and Blizzard is kind of what has defined them actually in actually most games.

In StarCraft, but also in World of Warcraft, they would do that.

There are several classes and it would be not good that everyone plays absolutely the same race or and so on.

So I think They do care about balancing, of course, and they do a fair amount of testing.

But it's also beautiful to also see how players get creative anyways.

And I mean, whether AI can be more creative at this point, I don't think so.

I mean, it's just sometimes something so amazing happens.

Like I remember back in the days, you have these dropships that could drop the rivers, and that was actually not thought about, that you could drop this unit that has this what's called splash damage that would basically eliminate all the enemy's workers at once.

No 1 thought that you could actually put them in really early game, do that kind of damage, and then, you know, things change in the game.

don't know, I think it's quite an amazing exploration process from both sides, players and Blizzard alike.

Well, it's almost like a reinforcement learning exploration, but the scale of humans that play Blizzard games is almost on the scale of a large-scale, deep-mind RL experiment.

I mean, if you look at the numbers, it's, I mean, you're talking about, I don't know how many games, but hundreds of thousands of games probably a month.

I mean, so you could, it's almost the same as running RL agents.

What aspect of the problem of StarCraft do you think is the hardest?

Is it the, like you said, the imperfect information?

Is it the fact they have to do long-term planning?

Is it the real-time aspects, so you have to do stuff really quickly?

Is it the fact that large action space, so you can do so many possible things?

Or is it, you know, in the game theoretic sense, there is no Nash equilibrium, at least you don't know what the optimal strategy is because there's way too many options.

Is there something that stands out as just like the hardest, the most annoying thing?

So when we sort of looked at the problem and start to define the parameters of it, what are the observations, what are the actions, it became very apparent that the very first barrier that 1 would hit in StarCraft would be because of the action space being so large and us not being able to search like you could in Chess or Go, even though the search space is vast.

The main problem that we identified was that of exploration, right?

So without any sort of human knowledge or human prior, if you think about StarCraft and you know how deep reinforcement learning algorithm works, which is essentially by issuing random actions and hoping that they will get some wins sometimes so they could learn.

So if you think of the action space in StarCraft, almost anything you can do in the early game is bad, because any action involves taking workers, which are mining minerals for free.

That's something that the game does automatically sends them to mine.

And you would immediately just take them out of mining and send them around.

So just thinking how is it going to be possible to get to understand these concepts, but even more like expanding, right?

There's these buildings you can place in other locations in the map to gather more resources, but the location of the building is important.

And you have to select a worker, send it walking to that location, build the building, wait for the building to be built, and then put extra workers there.

That just feels like impossible if you just randomly click to produce that state, desirable state, that then you could hope to learn from, because eventually that may yield to an extra win.

So for me, the exploration problem and due to the action space and the fact that there's not really turns, there's so many turns because the game essentially ticks at 22 times per second.

I mean, that's how they can discretize sort of time.

Obviously, you always have to discretize time.

But it's really a lot of time steps of things that could go wrong.

And that definitely felt a priori like the hardest.

You mentioned many good ones, I think partial observability, the fact that there is no perfect strategy because of the partial observability.

We start seeing more and more now in terms of as we solve the previous ones, but the core problem to me was exploration and solving it has been basically kind of the focus on how we saw the first breakthroughs.

So exploration in a multi-hierarchical way.

So like 22 times a second exploration has a very different meaning than it does in terms of should I gather resources early or should I wait or so on.

See all Lex Fridman transcripts on Youtube

Oriol Vinyals: DeepMind AlphaStar, StarCraft, and Language | Lex Fridman Podcast #20