First lecture of MIT course 6.S091: Deep Reinforcement Learning, introducing the fascinating field of Deep RL. For more lecture videos on deep learning, reinforcement learning (RL), artificial intelligence (AI & AGI), and podcast conversations, visit our website or follow TensorFlow code tutorials on our GitHub repo.

INFO:
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Slides: http://bit.ly/2HtcoHV
Playlist: http://bit.ly/deep-learning-playlist

OUTLINE:
0:00 - Introduction
2:14 - Types of learning
6:35 - Reinforcement learning in humans
8:22 - What can be learned from data?
12:15 - Reinforcement learning framework
14:06 - Challenge for RL in real-world applications
15:40 - Component of an RL agent
17:42 - Example: robot in a room
23:05 - AI safety and unintended consequences
26:21 - Examples of RL systems
29:52 - Takeaways for real-world impact
31:25 - 3 types of RL: model-based, value-based, policy-based
35:28 - Q-learning
38:40 - Deep Q-Networks (DQN)
48:00 - Policy Gradient (PG)
50:36 - Advantage Actor-Critic (A2C & A3C)
52:52 - Deep Deterministic Policy Gradient (DDPG)
54:12 - Policy Optimization (TRPO and PPO)
56:03 - AlphaZero
1:00:50 - Deep RL in real-world applications 
1:03:09 - Closing the RL simulation gap
1:04:44 - Next step in Deep RL

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman

Today I'd like to overview the exciting field of deep reinforcement learning Introduce, overview and provide you some of the basics I think it's 1 of the most exciting fields in artificial intelligence It's marrying the power and the ability of deep neural networks to represent and comprehend the world with the ability to act on that understanding, on that representation.

Taking as a whole, that's really what the creation of intelligent beings is.

And the exciting breakthroughs that recently have happened captivate our imagination about what's possible.

And that's why this is my favorite area of deep learning and artificial intelligence in general and I hope you feel the same.

We've talked about deep learning, which is taking samples of data, being able to in a supervised way compress and code the representation of that data in a way that you can reason about it.

I would take that power and apply it to the world where sequential decisions are to be made.

So it's looking at problems and formulations of tasks where an agent, an intelligent system has to make a sequence of decisions.

And the decisions that are made have an effect on the world around the agent.

How do all of us, any intelligent being that is tasked with operating in the world, how do they learn anything?

Especially when you know very little in the beginning.

It's trial and error, is the fundamental process by which reinforcement learning agents learn.

And the deep part of deep reinforcement learning is neural networks.

Is using the frameworks and reinforcement learning where the neural network is doing the representation of the world based on which the actions are made.

And we have to take a step back when we look at the types of learning, sometimes the terminology itself can confuse us to the fundamentals.

There is supervised learning, there's semi-supervised learning, there's unsupervised learning, there's reinforcement learning, And there's this feeling that supervised learning is really the only 1 where you have to perform the manual annotation, where you have to do the large-scale supervision.

Every type of machine learning is supervised learning.

It's supervised by a loss function or a function that tells you what's good and what's bad.

You know even looking at our own existence is how we humans figure out what's good and bad.

There's all kinds of sources direct and indirect by which our morals and ethics we figure out what's good and bad.

The difference between supervised and unsupervised and reinforcement learning is the source of that supervision.

What's implied when you say unsupervised is that the cost of human labor required to attain the supervision is low.

It's turtles and then there's a human at the bottom.

At some point there needs to be human intervention, human input to provide what's good and what's bad.

And this will arise in reinforcement learning as well.

We have to remember that because the challenges and the exciting opportunities of reinforcement learning lie in the fact of how do we get that supervision in the most efficient way possible but supervision nevertheless is required for any system that has an input and output that's trying to learn like a neural network does to provide an output that's good.

It needs somebody to say what's good and what's bad.

For you curious about that, there's been a few books, a couple written throughout the last few centuries from Socrates to Nietzsche, I recommend the latter especially.

So let's look at supervised learning and reinforcement learning.

I'd like to propose a way to think about the difference that is illustrative and useful when we start talking about the techniques.

So supervised learning is taking a bunch of examples of data and learning from those examples where ground truth provides you the compressed semantic meaning of what's in that data and from those examples 1 by 1 whether it's sequences or single samples we learn how to then take future such samples and interpret them Reinforcement learning is teaching we teach an agent through experience Not by showing a singular sample of a data set, but by putting them out into the world.

The distinction there, the essential element of reinforcement learning then for us, now we'll talk about a bunch of algorithms, but the essential design step is to provide the world in which to experience.

From the world, it gets the dynamics of that world, the physics of the world.

From that world, it gets the rewards, what's good and bad.

And us as designers of that agent do not just have to do the algorithm, we have to do design the world in which that agent is trying to solve a task.

The design of the world is the process of reinforcement learning, the design of examples, the annotation of examples is the world of supervised learning.

And the essential, perhaps the most difficult element of reinforcement learning is the reward, the good versus bad.

Here a baby starts walking across the room.

We want to define success as a baby walking across the room and reaching the destination.

And failure is the inability to reach that destination.

And reinforcement learning in humans, the way we learn from these very few examples, appear to learn from very few examples through trial and error, is a mystery, a beautiful mystery, full of open questions.

It could be from the huge amount of data, 230 million years worth of bipedal data that we've been walking, mammals walking, ability to walk, or 500 million years the ability to see, having eyes.

So that's the the hardware side, somehow genetically encoded in us is the ability to comprehend this world extremely efficiently.

It could be through not the hardware, not the 500 million years, but the few minutes, hours, days, months, maybe even years in the very beginning when we're born.

The ability to learn really quickly through observation to aggregate that information, filter all the junk that you don't need and be able to learn really quickly through imitation learning, through observation.

The way for walking that might mean observing others to walk.

The idea there is if there was no others around we would never be able to learn this the fundamentals of this walking or as efficiently.

And then it could be the algorithm totally not understood is the algorithm that our brain uses to learn.

The back propagation, that's an artificial neural networks, the same kind of processes not understood in the brain, that could be the key.

So I want you to think about that as we talk about the very trivial by comparison accomplishments and reinforcement learning and how do we take the next steps.

But it nevertheless is exciting to have machines that learn how to act in the world.

The process of learning for those who have fallen in love with artificial intelligence, the process of learning is thought of as intelligence.

It's the ability to know very little and through experience, examples, interaction with the world in whatever medium, whether it's data or simulation, so on.

Be able to form much richer and interesting representations of that world, be able to act in that world.

So let's look at this stack of what it means to be an agent in this world.

From top, the input to the bottom, the output, it's, there's an environment, we have to sense that environment.

We have just a few tools, us humans have several sensory systems.

On cars, you can have LiDAR camera, stereo vision, audio microphone networking, GPS, IMU sensors, so on.

Whatever robot you can think about, there's a way to sense that world.

And you have this raw sensory data, and then once you have the raw sensory data, you're tasked with representing that data in such a way that you can make sense of it, as opposed to all the raw sensors in the eye, the cones and so on, that take in just giant stream of high bandwidth information.

We have to be able to form higher abstractions of features based on which we can reason from edges to corners to faces and so on.

That's exactly what deep learning neural networks have stepped in to be able to in an automated fashion with as little human input as possible be able to form higher order representations of that information.

Then there's the learning aspect, building on top of the greater abstractions formed through representations be able to accomplish something useful whether it's discriminative task, generative task and so on.

Based on the representation be able to make sense of the data, be able to generate new data and so on.

From sequence to sequence, sequence to sample, from sample to sequence and so on and so forth to actions as we'll talk about.

And then there is the ability to aggregate all the information that's been received in the past to the useful information that's pertinent to the task at hand.

It's the thing, the old, it looks like a duck, quacks like a duck, swims like a duck.

I'm sure there's state of the art algorithms for the 3, image classification, audio recognition, video classification, activity recognition, so on.

Aggregating those 3 together is still an open problem.

Again, I want you to think about as we think about reinforcement learning agents.

How do we play, how do we transfer from the game of Atari to the game of Go to the game of Dota to the game of a robot navigating an uncertain environment in the real world.

And once you have that, once you sense the raw world, once you have a representation of that world, then we need to act, which is provide actions within the constraints of the world in such a way that we believe can get us towards success.

The promise excitement of deep learning is the part of the stack that converts raw data into meaningful representations.

The promise, the dream of deeper reinforcement learning is going beyond and building an agent that uses that representation and acts, achieves success in the world.

The framework and the formulation of reinforcement learning at its simplest is that there's an environment and there's an agent that acts in that environment.

The agent senses the environment by some observation, whether it's partial or complete observation of the environment.

It acts in that environment and through the action, the environment changes in some way and then a new observation occurs.

And then also as you provide the action, make the observations, you receive a reward.

In most formulations of this framework, this entire system has no memory that the only thing you need to be concerned about is the state you came from, the state you arrived in, and the reward received.

The open question here is what can't be modeled in this kind of way?

Can we model all of it from human life to the game of Go?

And is this a good way to formulate the learning problem of robotic systems in the real world and simulated world?

The environment could be fully observable or partially observable like in poker.

It could be single agent or multi-agent, Atari versus driving like deep traffic, deterministic or stochastic, static versus dynamic, static as in chess, dynamic again in driving in most real-world applications, discrete versus continuous like games, chess or continuous in carpool, balancing a pole on a cart.

The challenge for RL in real world applications is that as a reminder, Supervised learning is teaching by example, learning by example, teaching from our perspective.

Reinforcement learning is teaching by experience.

And the way we provide experience to reinforcement learning agents currently for the most part is through simulation or through highly constrained real-world scenarios.

So the challenge is in the fact that most of the successes is with systems environments that are simulatable.

So there's 2 ways to then close this gap, 2 directions of research and work.

1 is to improve the algorithms, improve the ability of the algorithms to then to form policies that are transferable across all kinds of domains, including the real world, including especially the real world.

So train and simulation transfer to the real world.

Or as we improve the simulation in such a way that the fidelity of the simulation increases to the point where the gap between reality and simulation is minimal to a degree that things learned in simulation are directly trivially transferable to the real world.

Okay, the major components of an RL agent.

An agent operates based on a strategy called a policy.

It sees the world, it makes a decision, that's a policy.

Sees the reward, sees a new state, acts, sees a reward, sees a new state and acts.

And this repeats forever until a terminal state.

The value function is the estimate of how good a state is or how good a state action pair is, meaning taking an action in a particular state.

And then the model, different from the environment, from the perspective of the agent.

So the environment has a model based on which it operates and then the agent has a representation, best understanding of that model.

So the purpose for an RL agent in this simply formulated framework is to maximize reward.

The way that the reward mathematically and practically is talked about is with a discounted framework.

So we discount further and further future reward.

So the reward that's farther into the future is means less to us in terms of maximization than reward that's in the near term.

So first, a lot of it is a math trick to be able to prove certain aspects, analyze certain aspects of conversions.

And in general, on a more philosophical sense, because environments either are or can be thought of a stochastic random, it's very difficult to, there's a degree of uncertainty, which makes it difficult to really estimate the reward that will be in the future because of the ripple effect of the uncertainty.

Let's look at an example, a simple 1, helps us understand policies, rewards, actions.

It tries to get rewards on the top right.

It's a really good thing at the top right, it wants to get there by walking around.

There's a negative 1, which is really bad, it wants to avoid that square.

And the choice of actions is up, down, left, right, 4 actions.

So you could think of there being a negative reward of 0.04 for each step, so there's a cost to each step, and there's a stochastic nature to this world, potentially, we'll talk about both determinacy and stochastic.

So in the stochastic case, when you choose the action up, with an 80% probability, with an 80% chance you move up, but with 10% chance you move left, another 10 move right.

So that's stochastic nature, even though you try to go up, you might end up in a block to the left and to the right.

So for a deterministic world, the optimal policy here, given that we always start in the bottom left, is really shortest path.

Is, you know, you can't ever, because there's no stochasticity, you're never going to screw up and just fall into the hole, negative 1 hole, that you just compute the shortest path and walk along that shortest path.

Because every single step hurts, there's a negative reward to it, 0.04.

So shortest path is the thing that minimizes the reward, shortest path to the plus 1 block.

Like I mentioned the 80% up and then split 10% to left and right.

Well first of all we need to have a plan for every single block in the area because you might end up there due to the stochasticity of the world.

Okay, the basic addition there is that we're trying to go avoid up the closer you get to the negative 1 hole.

So just try to avoid up because up, the stochastic nature of up means that you might fall into the hole with a 10% chance.

And given the .04 step reward, you're willing to take the long way home in some cases in order to avoid that possibility, the negative 1 possibility.

Now let's look at a reward for each step if it decreases to negative 2.

Then again, we go to the shortest path despite the fact that there's a stochastic nature.

In fact, you don't really care that you step into the negative 1 hole because every step really hurts.

And then you can play with this reward structure, right?

Yes, instead of a negative 2 or negative 0.04, you can look at a negative 0.1.

And you can see immediately that the structure of the policy, it changes.

So with a higher value, the higher negative reward for each step, immediately the urgency of the agent increases versus the less urgency, the lower the negative reward.

And when the reward flips, so it's positive, the every step is a positive.

So the entire system, which is actually quite common in reinforcement learning, the entire system is full of positive rewards.

And so then the optimum policy becomes the longest path.

Is grad school taking as long as possible and never reaching the destination.

So what lessons do we draw from robot in the room?

2 things, the environment model, the dynamics, just there in the trivial example, the stochastic nature, the difference between 80%, 100% and 50%, the model of the world, the environment has a big impact on what the optimal policy is.

And the reward structure, most importantly, The thing we can often control more in our constructs of the task we try to solve in reinforcement learning is the what is good and what is bad and how bad is it and how good is it?

And that has a complete change, like Robert Frost said, a complete change on the policy, the choices the agent makes.

So when you formulate a reinforcement learning framework as researchers, as students, what you often do is you design the environment, you design the world in which the system learns, even when your ultimate goal is the physical robot, there's a lot of work still done in simulation.

So you design the world, the parameters of that world, and you also design the reward structure.

And it can have transformative results, slight variations in those parameters can be huge results on huge differences on the policy that's arrived.

And of course the example I've shown before, I really love, is the impact of the changing reward structure might have unintended consequences and those consequences for real world system can have obviously highly detrimental costs that are more than just a failed game of Atari.

So here's a human performing the task, playing the game of Coast Runners racing around the track and so it's when you finish first and you finish fast you get a lot of points and so it's natural to then okay let's do an RL agent and then optimize this for those points and what you find out in the game is that you also get points by picking up the little green turbo things and what the agent figures out is that you can actually get a lot more points even by simply focusing on the green turbos focusing on the green turbos just rotating over and over slamming into the wall, fire and everything just picking it up especially because ability to pick up those turbos can avoid the terminal state at the end of finishing the race.

In fact, finishing the race means you stop collecting positive reward.

So you never want to finish, collect the turbos.

It's not actually easy to find such examples but they're out there of unintended consequences that can have highly negative detrimental effects when put in the real world.

We'll talk about a little bit of robotics.

When you put robots, 4 wheeled ones, like autonomous vehicles into the real world, and you have objective functions that have to navigate difficult intersections full of pedestrians, so you have to form intent models of those pedestrians.

Here you see cars asserting themselves through dense intersections, taking risks.

And within those risks that are taken by us humans when we drive vehicles, we have to then encode that ability to take subtle risk into AI-based control algorithms, perception then you have to think about at the end of the day there's an objective function And if that objective function does not anticipate the green turbos that are to be collected and then result in some unintended consequences could have very negative effects especially in situations that involve human life.

And some of the folks who talk about deep mind and open AI, they're doing incredible work in RL, also have groups that are working on AI safety for a very good reason.

This is a problem that I believe that artificial intelligence will define some of the most impactful positive things in the 21st century.

But I also believe we are nowhere close to solving some of the fundamental problems of AI safety that we also need to address as we develop those algorithms.

So, okay, examples of reinforcement learning systems.

All of it has to do with formulation of rewards, formulation of state and actions.

You have the traditional, the often used benchmark of a cart balancing a pole, continuous.

So the action is the horizontal force of the cart.

The goal is to balance the pole so it stays top in the moving cart.

And the reward is 1 at each time step if the pole is upright.

And the state measured by the cart by the agent is the pole angle angular speed and of course self-sensing of the cart position and the horizontal velocity.

Another example here, didn't want to include the video because it's really disturbing, but I do want to include this slide because it's really important to think about is by sensing the raw pixels, learning and teaching an agent to play a game of Doom.

So the goal there is to eliminate all opponents.

The state is the raw game pixels, the actions up, down, shoot, reload, and so on.

And the positive reward is when an opponent is eliminated and negative when the agent is eliminated, simple.

I added it here because again, on the topic of AI safety, we have to think about objective functions and how that translate into the world of not just autonomous vehicles, but things that even more directly have harm, like autonomous weapon systems.

We have a lecture on this in the AGI series.

And on the robotics platform, the object manipulation and grasping objects, there's a few benchmarks, there's a few interesting applications, learning the problem of grabbing objects, moving objects, manipulating objects, rotating and so on, especially when those objects don't have complicated shapes.

And so the goal is to pick up an object in the purely in the grasping object challenge.

The state is the visual information, so it's visual based, the raw pixels of the objects.

The action is to move the arm, grasp the object, pick it up, and obviously it's positive when the pickup is successful.

The reason I'm personally excited by this is because it will finally allow us to solve the problem of the claw, which has been torturing me for many years.

I don't know, that's Not at all why I'm excited by it.

Okay, and then we have to think about as we get greater and greater degree of application in the real world with the robotics, like cars, the main focus of my passion in terms of robotics is how do we encode some of the things that us humans encode?

How do we, you know, we have to think about our own objective function, our own reward structure, our own model of the environment about which we perceive and reason about in order to then encode machines that are doing the same.

And I believe autonomous driving is in that category.

We have to ask questions of ethics, we have to ask questions of risk, value of human life, value of efficiency, money and so on.

All these are fundamental questions that an autonomous vehicle unfortunately has to solve before it becomes fully autonomous.

So here are the key takeaways of the real world impact of reinforcement learning agents.

On the deep learning side, okay, these neural networks that form higher representation, the fun part is the algorithms, all the different architectures, the different encoder-decoder structures, all the attention, self-attention, recurrence, LSTMs, GRUs, all the fun architectures and the ability to leverage different data sets in order to perform discriminatory tasks better than, MIT does better than Stanford, that kind of thing.

The hard part is asking good questions and collecting huge amounts of data that's representative of the task.

That's for real world impact, not CVPR publication.

Real world impact, a huge amount of data.

On the deep reinforcement learning side, the key challenge, the fun part again is the algorithms.

The hard part is defining the environment, defining the access space and the reward structure.

As I mentioned, this is the big challenge.

And the hardest part is how to crack the gap between simulation and the real world.

We don't even know how to solve that transfer learning problem yet for the real world impact.

There's countless algorithms and there's a lot of ways to taxonomize them but at the highest level, there's model-based and there's model-free.

Model-based algorithms learn the model of the world.

So as you interact with the world, you construct your estimate of how you believe the dynamics of that world operates.

The nice thing about doing that is once you have a model or an estimate of a model, you're able to anticipate, you're able to plan into the future.

You're able to use the model to, in a branching way, predict how your actions will change the world, and so you can plan far into the future.

This is the mechanism by which you can do chess in the simplest form, because in chess, you don't even need to learn the model, the model is given to you, chess, go and so on.

The most important way in which they're different, I think, is the sample efficiency, is how many examples of data are needed to be able to successfully operate in the world.

And so model-based methods, because they're constructing a model, if they can, are extremely simple efficient.

Because once you have a model, you can do all kinds of reasoning that doesn't require experiencing every possibility of that model.

You can unroll the model to see how the world changes based on your actions.

Value-based methods are ones that look to estimate the quality of states, the quality of taking a certain action in a certain state.

So they're called off policy versus the last category that's on policy.

It means that they constantly, value-based agents, constantly update how good it is to take an action in a state, and they have this model of that goodness of taking an action in a state, and They use that to pick the optimal action.

They don't directly learn a policy, a strategy of how to act.

See all Lex Fridman transcripts on Youtube

MIT 6.S091: Introduction to Deep Reinforcement Learning (Deep RL)