- Hardware for running neural nets is expected to get faster in the upcoming years, allowing for scaled-up models and qualitatively different behaviors.
- There is little research happening on understanding the limits of existing AI methods.
- The shift towards specialized hardware that is more brain-like and runs in parallel will greatly improve the speed of running models.
- Becoming a good engineer is more valuable than spending time implementing exotic models in TensorFlow.
- A CS major with no AI experience can become productive at OpenAI from day one, with a focus on engineering and infrastructure.
- The Dota 2 project at OpenAI involved more engineering work than machine learning science.
- The selection of Dota 2 as a testbed for AI algorithms was based on its compatibility with Linux, availability of game API, and support from the developer.
- Valve's release of a bot-focused API coincided perfectly with the project.

- The team had to become familiar with the game API and ensure it could run on a large scale.
- One person worked on building a scripted bot using Lua, while another worked on turning the game into a Docker container.
- The scripted bot was then ported into Python by renaming files and uncommenting code.
- A file with dummy calls was created and implemented using gRPC and protobuf-based protocol.
- The infrastructure was stabilized, allowing time for focus on machine learning, specifically behavioral cloning.
- Reinforcement learning with self-play was used to train the bot, which played against itself to improve.
- The bot was able to beat the scripted bot after a week or two of engineering effort.
- The team then decided to compete in the tournament.

- The speaker spent a month learning the workflow and created a bot that could imitate actions but not understand intent.
- They used reinforcement learning to fine-tune the bot and improve its performance.
- The team had a scoreboard to measure the bot's win rate and made continuous improvements to its strength over time.
- The project management approach shifted to focusing on trying out new ideas and setting weekly goals.
- The bot's performance improved gradually, with the ability to beat semi-pro players a month before the competition.
- The team was unsure of the bot's true skill and had varying estimates of its chances at the competition.
- During the competition, the team was holed up in a locker room near the stadium with laptops and a partition to play against pro players.
- The bot beat one pro player 3-0, lost to another due to an item build it hadn't encountered before, and beat a third pro player 3-0.
- The pros had mixed reactions to the bot, but it sparked discussions and emotional responses.
- After the event, a LAN party was held where players found exploits in the bot's gameplay.

- The bot's learning is limited by the environment it plays in, leading to exploitable weaknesses.
- The team is excited for the next step of 5v5 gameplay, as it presents a new challenge of exploiting the other team.
- The team identifies and fixes exploits by making small tweaks to the bot's training.
- The process of tweaking the bot is compared to digitizing a human, where missing features are added to improve performance.
- The team focuses on engineering the observation and action spaces to optimize the bot's learning capacity.
- The team works tirelessly to make improvements to the bot, even performing surgery on the running experiment.
- The bot's performance is tested against professional players, and it learns strategies such as baiting opponents.
- The bot's baiting strategy has a psychological effect on humans, making them fall for its seemingly dumb moves.

- The team faced a dilemma when their bot broke just hours before the best players were set to arrive.
- They stitched together two bots to create a functioning one just in time for the event.
- They debated whether to abandon the bot or let it continue training to fix it.
- The bot was able to beat Arteezy, a professional player, and received positive feedback.
- They continued to make small changes and improvements to the bot before playing against Sumail.
- The team had a break before the final game and then let the bot train for over 24 hours.
- The bot went undefeated against Sumail and has since played against other pros.
- Some humans have been able to consistently beat the bot after playing it for a long time.

- A top player who played hundreds of games with a bot became as good as the bot at the game.
- The bot has helped the player improve his focus and play style.
- OpenAI values knowledge of distributed systems and bug-free code in their team.
- Good engineering skills and basic knowledge of linear algebra and statistics are important.
- Non-technical people can be helpful to AI startups by educating themselves about AI and being involved in ethical discussions.
- Working long hours is not recommended for productivity, but doing what you love is important.
- The term AI is overused by startups for publicity.
- The last job that will remain as AI advances is difficult to predict, but AI researchers will be important.
- Playing video games is relevant to building AGI because it provides complex environments for algorithms to learn skills.
- OpenAI has job postings on their website for those interested in getting involved.

- Tips for getting a job in AI are specific to the job posting for a large-scale reinforcement learning engineer.
- The company looks for people who are highly skilled in their specific technical area.
- AI PhD is not a requirement for the job.
- The speaker thanks the audience.

- Hardware for running neural nets is expected to improve in speed and capabilities.
- There is limited research on the limits of existing AI methods.
- Specialized hardware that is more brain-like and runs in parallel will greatly enhance model speed.
- OpenAI values engineering skills over implementing exotic models.
- The Dota 2 project at OpenAI required significant engineering work.
- The selection of Dota 2 as a testbed was based on compatibility and support.
- The team had to become familiar with the game API and ensure scalability.
- The bot was trained using reinforcement learning with self-play.
- The bot's performance gradually improved and it was able to beat semi-pro players.
- During the competition, the bot had mixed results against pro players.
- The bot's learning is limited by its environment and has exploitable weaknesses.
- The team is excited for the challenge of 5v5 gameplay.
- The team focuses on engineering the bot's observation and action spaces.
- The bot's performance is tested against professional players and it learns strategies.
- The team faced a dilemma when the bot broke just before a major event.
- The bot was able to beat professional players and received positive feedback.
- OpenAI values engineering skills, knowledge of distributed systems, and bug-free code.
- Non-technical people can contribute to AI startups through education and ethical discussions.
- OpenAI has job postings for those interested in AI.
- Tips for getting a job in AI vary based on the specific job posting.

Greg Brockman is the CTO and cofounder of OpenAI - https://twitter.com/gdb

Szymon Sidor is a Research Scientist at OpenAI - https://twitter.com/sidorszymon

Sam Altman is the President of Y Combinator and Co-Chairman of OpenAI - http://twitter.com/sama

Watch their bot compete at The International - https://openai.com/the-international/

YC's podcast is hosted by Craig Cannon - http://twitter.com/craigcannon

Now if you look forward to what's going to happen over upcoming years is the hardware for these applications for running neural nets really, really quickly are going to get fast, faster than people expect.

And I think that what that's going to unlock is you're going to be able to scale up these models and you're going to see qualitatively different behaviors from what you've seen so far.

For example, we had a paper on this unsupervised learning where you train a language model to—you train a model to predict the next character in Amazon reviews.

And just by learning to predict the next character in Amazon reviews.

And just by learning to predict the next character in Amazon reviews, somehow it learned a state-of-the-art sentiment analysis classifier.

And so it's kind of crazy if you think about it, right?

You just were told, hey, predict the next character.

And if you were told to do this, Well, the first thing you do is you'd learn the spelling of words and you learn punctuation The next thing you do you start to learn semantics, right?

If you have extra capacity there and that this effect goes away If you use a slightly smaller model and what happens if you have a slightly larger model Well, we don't know because we can't run those models yet.

What do you guys think are the most promising underexplored areas in AI?

If we're trying to make it come faster, what should people be working on that they're not?

Yeah, so there are many areas of AI that we already developed by quite a bit.

There is some basic research in just classification, deep learning, reinforcement learning.

And what people do is people kind of try to invent problems and such as solving some complicated games of hierarchical structure and they try to add kind of extra features to their models to combat those problems.

But I think there's very little research happening on actually understanding the existing methods and their limits.

So for example, it was a long-held belief in deep learning that kind of to parallelize your computation, you need to cram as small batches as possible on every on every device.

And in fact, Baidu did this impressive engineering feat where they took recurrent neural networks and they implemented the kind of GPU assembly code to make sure that you can fit like bat size 1 RNNs on every GPU and you know despite like all those smart people working on those problems like only very recently that Facebook kind of took a cold quiet look at just like very basic problem of classification.

And in their great paper called Image that in 1 hour, they showed that if you actually take a code that does image classification and if you fix all the bugs you actually can get away with much larger batch size and therefore finish the classification problem much faster.

And you know it's not the kind of sexy research that people want to see where you have like some hierarchy, big RNN but it actually this kind of research I think at this point will advance field the most.

So Greg you mentioned hardware and your initial answer.

In the near term, what are the actual innovations that you foresee happening?

So the big change is that the kinds of computers that we've been trying to make really fast are general purpose computers that are built on the Von Neumann architecture.

You basically have a processor, you have a big memory, and you have some bottleneck between the 2.

With the applications that we're starting to do now, suddenly you can start making use of massively parallel compute.

That the architectures that these models can run on, sort of the fastest, are going to look kind of like the brain, where the brain is basically you have a bunch of neurons that all have their own memory right near to them, and that they all talk to their neighbors, and maybe there's some kind of longer range skip connections.

And that just no one's really had incentive to develop hardware like this.

And so what we've seen is that, well, you move your neural networks from running on a CPU to a GPU, and now suddenly you have 1,000 CUDA cores running in parallel, and that you can get massive performance boost there.

Now if you move to specialized hardware that is much more brain-like and that runs in parallel with a bunch of tiny little cores that you're going to be able to run these models sort of insanely faster.

I think 1 of the most common questions or threads of question that were asked on Twitter and Facebook were generally how to get into AI.

Could you guys give us just a primer of where someone should start if they're just a CS major in college?

So it really depends on the nature of the project that you would like to do.

I can tell you a bit about our project, which is essentially developing large-scale reinforcement learning for Dota 2.

And their majority of the work is actually engineering.

And you know, like, essentially taking, you know, taking, taking the algorithms that we have already implemented and trying to scale them up is usually the fastest way to get improvement in our experiments.

So essentially, becoming a good engineer for our team is much more valuable than, for example, people spending months upon months implementing exotic models in TensorFlow.

So just to echo this, because I hear this come up all the time people say it's like my dream to work at open AI But I got to go get an AI PhD So I'll see you in like 5 or 7 years if people are just a really solid engineer But no experience at all with AI How long does it take someone like that to become productive for the kind of work at open AI that we that we're looking for?

So someone like that can actually become productive from day 1.

And with different engineers who show up at OpenAI, there's a spectrum of where they end up specializing.

There are some people who focus on building out infrastructure, and that actually this infrastructure can range from, well, we have a big Kubernetes deployment that we run on top of a cloud platform, and building tooling and monitoring, and managing this underlying layer.

It actually looks quite a bit like running a startup.

And a lot of people who are most successful at that have quite a bit of running that large scale in a startup or production environment.

There's kind of a next level of getting closer to the actual machine learning.

Where if you think of how machine learning systems look, that they tend to be this magical black box of machine learning, this core.

And you actually try to make that core be as small as possible because machine learning is really hard, needs a lot of compute, it's really hard to tell what's going on there.

And so you want it to be as simple as possible, but then you surround it by as much engineering as you possibly can.

So what percent of the work on the Dota 2 project would you guys say was what people would really think of as like machine learning science versus engineering?

Essentially like as far as day to day work goes, this kind of work was almost non-existent.

There was like maybe a few person weeks spent on that compared to like person hours spent on engineering.

And I think maybe placing some good bets was 1 of it.

And they're often more about what not to do rather than what to do.

So at the very beginning of the project, we knew we wanted to solve a game, a hard game.

We didn't know exactly which 1 we wanted to do, because these are great testbeds for pushing the limits of our algorithms.

And just to be clear, you guys are 2 of the key people, but the entire team was like 10 people?

And these things, good testbeds for our algorithms to see what the limits are to really push the limit of what's possible.

And you know for sure that when you've done it, that you've done it.

And so actually the way that we selected the game was we went on Twitch and just looked down the list of most popular games in the world and starting, you know, number 1 is League of Legends.

The thing about League of Legends is it doesn't run on Linux and it doesn't have a game API.

And little things like that actually are the biggest barrier to making AI progress in a lot of ways.

And so looking down the list, Dota actually was the first 1 that kind of had all the right properties.

Runs on Linux, that it has a big community around replay parsing, that there's a built-in Lua API.

This API was meant for building mods rather than for building bots, but we could probably use it to build bots.

1 of the great things about Valve as a company is that they're very into having these open hackable games where people can go and do a bunch of custom things.

And so, kind of philosophically, it was very much the right kind of company to be working with.

So, we actually did this initial selection back in November, and we were working on some other projects at the time, and so it didn't really get started until late in December.

And 1 of the funny things is that, by total coincidence, in mid-December, Valve released a new bot-focused API, and that they were saying, hey, our bots are famously bad, maybe the community can solve this problem, so we'll actually build an API specific for it for people to do this.

And that was just 1 of those coincidences of the universe that just worked out extremely well.

So we were kind of in close contact with the developer of this API all throughout.

So at the very end of the project, well, what are you going to do?

And so the first thing was We had to become very familiar with this game API to make sure we understood all the little semantics and all of the different corner cases, and also to make sure that we could run this thing on large scale and to turn it into a pleasant development environment.

1 person was working with the bot API, building a scripted bot.

And so basically this is the learn all the game rules, think really hard about how it works.

The particular person who wrote it, Raffal, has played about 3 or 4 games of Dota in his life, but he's watched over a thousand hours of Dota gameplay and has now written the best Dota scripted bot in the world.

And so that, you know, sort of a lot of just writing this thing in Lua, getting very intimately familiar with all those details.

In the meanwhile, what I was working on was trying to figure out how do you turn this thing into a Docker container.

And so they had this whole build process.

It turns out that Steam can only be in offline mode for 2 weeks at a time, that they pushed new patches all the time, so you needed to go from this like, you know, sort of manually download the game and whatever to actually have an automated repeatable process.

It turns out that the full game files are about 17 gigabytes and that our docker registry can only support 5 gigabyte layers, and so I had to write a thing to chunk up things into 5 gigabyte tar balls and put those in S3 and slip them back down.

So a bunch of things there where it's really just about figuring out what the right workflow is, what the right abstractions are.

And then the next step was, well, we know we want to be writing our bots in TensorFlow and Python.

Machine learning, it's actually quite interesting that a lot of the highest order bit on progress, just like having the game API as a higher order bit, it's also can you use tools that are familiar and easy to iterate with?

Before the world of modern machine learning frameworks, everyone would write their code in Matlab.

If you had a new idea, it would take you 2 months to do it.

And so it was really all about iteration speed.

And so if you can get into the Python world, well, we have these large code bases that we built up of high-quality algorithms, that there's just so much tooling built around it that that's the optimal experience.

The next step was to port the scripted bot into Python.

The way I did that was I literally just renamed all of the .lua files to .py, commented out the code, and then started uncommenting function by function.

And then you run the function, you get an exception, you then go and uncomment whatever code it depends on, and as mechanically as possible, I tried to be like a human transpiler, and that Lua is 1 index, Python 0 index, so you have to do that.

That Lua doesn't distinguish between an array type and a dictionary type, and so you kind of have to disambiguate those 2.

But for the most part, I did something that could have been totally mechanically done.

And it's great because I didn't have to understand any of the game logic.

I didn't have to understand anything that was going on under the hood.

I could basically just port over, and it just kind of came together.

But then you end up with a small set of functions that you do not have implementations of, which are all of the actual API calls.

And so I ended up with a file with a bunch of dummy calls, and I knew exactly which calls I needed, and then implemented on top of gRPC, a protobuf-based protocol where on every tick, the game would dump the full game state, send the diff over the wire, reassemble that into an in-memory state object, Python, and then all of these API methods would be implemented in Python.

And so at the end of this, you know, it sounds like a bit of a Frankenstein process, but it actually worked really well.

And In the end, we had something that looked just like a typical OpenAI gym environment.

And so all you have to do is you say, gym.make this Dota environment ID, and suddenly you're playing Dota.

And your Python code just has to call into some object that implements the Lua API, and suddenly these characters are running around the screen doing what you want.

And so this was like a lot of the kind of thing that I was working on in the pure engineering side.

And actually, as time went on, so Shimon and Jakob and Jay and others joined the project.

And most people were building on top of this API and really didn't have to dig into any of the underlying implementation details.

So personally, my 1 machine learning contribution to the project, I'll tell you about that.

Because my background is primarily startup engineering, building large infrastructure, not machine learning, definitely not a machine learning PhD.

So I reached a point where I'd gotten the infrastructure into a pretty stable point that I felt like, all right, I don't have to be fighting the fires there very constantly.

I have some time to actually focus on digging into some of the machine learning.

1 particular piece that we were interested in doing was behavioral cloning.

So we had 1 of the systems that we had built was to go and download all of the replays that are published each day.

And so the way this game works is that there's about 1.5 million replays that are available for public download.

Valve clears them out after 2 weeks, and so you have to have some discovery process.

Originally, we were downloading all of them every day and realized that was about 2 terabytes worth of data a day.

So we ended up filtering down to the most expert players.

But we wanted to actually take this data, parse it, and use it to clone the behavior for a bot.

And so I spend a lot of time with like sort of, you know, it's basically you need this whole pipeline to download the replays, to parse them, to kind of iterate on that, to then take it, train a model, and try to predict what the behavior would be.

And at first, it's just like, 1 thing I find very interesting is the sort of different workflow that you end up with when doing machine learning.

Like there are a bunch of things where when software engineers join open AI that are just very surprising.

For example, if you look at a typical research workflow, you'll see a lot of files named like, you know, experiment, whatever the name of the experiment is, 1234, and you look at them and they're just like slight forks of the same thing.

And you're like, is this what version controls for?

And after doing this cloning project, I learned exactly why.

Because the thing is, if you have a new idea for, okay, well, I've kind of got this thing working and now I'm going to try something slightly different.

As you're doing the new thing, well, machine learning is to some extent very binary.

At the start, it just doesn't work at all, and you don't know why.

Or it kind of works, but it has some weird performance, and you're not sure exactly, is it a bug?

And so if you've gotten it working at all, and then you make a change, You're always going to want to go back and kind of compare to the previous thing you had running and so you actually do Want the new thing running side-by-side with the old thing and if you're constantly stashing and unstashing and checking out and whatever then you're just going to be sad and and There are a lot of like kind of workflow issues like that Then you just you got to bang your head against the wall and then you see like, ah, I've been enlightened.

So, but before we progress further on the story, can you just explain the basics of training a bot in a game?

Like how are you actually giving it the feedback?

So on a high level we are using reinforcement learning with self-play.

So what that means, I mean, it's no rocket science even though reinforcement learning sounds all fancy.

Essentially what's happening is we have a bot which observes some state in the environment and performs some actions based on that state and based on those actions that it executes then it continues playing and eventually you know either does well or poorly so that can that's something that we can quantify in a number and that's more of an engineering problem than a research problem, how to quantify how good the bot is doing.

And then the bot gets feedback on whether it's doing good or not, and then tries to select the actions that yield to high, that positive feedback to high reward.

And to give us a sense for how well that works, so the bot plays against itself to get better, once you had everything working, how good would a bot from day n do against a bot from day n-1?

So I guess we have a story that kind of illustrates what to expect from those techniques.

So when we started this project, our goal wasn't to really do research.

At some high level it was, but we were very goal-oriented.

We wanted to solve Dota 5.5 and we had our MySlove.onv1.

The way it started was early days when there was just Greg and Rafał and Rafał was implementing scripted bots.

When he sees a creep, he needs to attack it, yada yada.

And Rafał is actually a really good engineer, so we had a really good script at work.

So what happened then, kind of like he got to the point of which he didn't think he could improve it much more, so we tried, okay, let's try some reinforcement learning.

And I was actually on vacation at the time, but there was another engineer, Jakub, who throughout my vacation, which I found super surprising, said, I leave, there is nothing, I come back, there is this reinforcement learning bot.

And actually it's beating our scripted bot after like week worth of engineering effort.

Possibly those 2 weeks, but it was something very miniature compared to the development of scripted bot.

So actually our bot, which didn't have any assumptions baked about the game, figured out the underlying game structure well enough to beat anything that we could code by hand, which was pretty amazing to see.

And at what point do you decide to compete in the tournament?

Sorry if it's running a bit long, but it'll get good shortly.

So, just to finish up my machine learning contribution.

So I basically spent about a month really learning the workflow, got something that like, you know, was able to do some signs of life where it would like run to the middle and you're

And it's very clear when you're just doing cloning that these algorithms learn to imitate what it sees rather than the actual intent.

And so it'd get kind of confused, even run, try to do some creep blocking or something, but the creeps wouldn't be around, and so it would just be zigzagging back and forth.

And anyway, I got this to the point where it was actually creep blocking pretty reliably, pretty well.

And then at that point, I turned it over to Jay, who's also working on the project, and he used reinforcement learning to fine tune that.

And so suddenly, it went from only understanding the actions rather than the intent to suddenly it really knew what it was doing and has the best creep block that anyone has seen.

That was my 1 machine learning contribution to the project.

So time went on and 1 of the most important parts of the project was having a scoreboard.

So we had a metric on the wall, which was the true skill of our best bot.

So true skill is basically like an evil rating that measures the win rate of your bot versus others.

And you put that on the wall and each week people just try all the ideas and some of them work.

And that we actually ended up with this very smooth, almost linear curve.

And that really means exponential increase in the strength of this bot over time.

And that part of that is sometimes these data points where you just train the same experiment for longer.

Typically, our experiments would last for maybe up to 2 weeks.

But also a lot of those were, well, we had a new idea.

We removed this other component that wasn't necessary.

And so we knew that, so we chose the goal of 1v1.

I don't recall exactly when, but it must have been in the spring or maybe even early summer.

But we really didn't know, are we actually going to be able to make it?

And unlike normally when you're building an engineering system, you think really hard about all the components.

It's like, well, you decompose it into this subsystem, that subsystem, that subsystem, and you can measure your progress as what percent of the components are built.

Here, you really have ideas that you need to try out and that it's sort of unpredictable in some sense.

And actually 1 of the most important changes to the project in terms of making progress was initially the way that the project management was happening was that each week, well, so we'd written down our milestones of let's beat this person by this date, let's beat this other person by this date, let's be able to do these outcome-based milestones on a weekly or biweekly basis.

Those things would come and go, and you wouldn't have them.

It's like, it's not like there was anything else you could have done.

It's just you have more ideas you need to try.

instead of shifting it to what are all the things we're going to try by next week.

And then, yeah, if you didn't actually do everything you said you were going to do, then you should feel bad and then you should do more of it.

And if you did all those and it didn't work, then fair enough.

But you achieved what you wanted to achieve.

And so even going into the international, so 2 weeks before the international was kind of our cutoff.

For at this point, there's not much more we can do.

We're going to do our biggest experiment ever, put all of our compute into 1 basket and see where it goes.

And at that point, like at 2 weeks out, how good was the bot?

Oh, it was barely sometimes winning with professionals that we had testing them.

So to be specific, I'm just pulling this back in.

So July 8th is when we had our first win against our semi-protester.

And then we were kind of more consistent with it.

And so he was on some laptop somewhere that was not very good.

And then we were consistently beating him.

This was the week before the international.

And so we didn't really know how good we were getting.

We knew that the true skill was going up.

When was the last time that an open AI employee beat the bot?

Okay, but so like a month or 2 out it could beat all the open AI people.

So 4 weeks was the first time that it beat the semi-pro.

See all Y Combinator transcripts on Youtube

Building Dota Bots That Beat Pros - OpenAI's Greg Brockman, Szymon Sidor, and Sam Altman