This is lecture 2 of course 6.S094: Deep Learning for Self-Driving Cars taught in Winter 2017. This lecture introduces types of machine learning, the neuron as a computational building block for neural nets, q-learning, deep reinforcement learning, and the DeepTraffic simulation that utilizes deep reinforcement learning for the motion planning task.

INFO:
Slides: http://bit.ly/2H8Fs7g
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Playlist: https://goo.gl/SLCb1y

Links to individual lecture videos for the course:

Lecture 1: Introduction to Deep Learning and Self-Driving Cars
https://youtu.be/1L0TKZQcUtA

Lecture 2: Deep Reinforcement Learning for Motion Planning
https://youtu.be/QDzM8r3WgBw

Lecture 3: Convolutional Neural Networks for End-to-End Learning of the Driving Task 
https://youtu.be/U1toUkZw6VI

Lecture 4: Recurrent Neural Networks for Steering through Time 
https://youtu.be/nFTQ7kHQWtc

Lecture 5: Deep Learning for Human-Centered Semi-Autonomous Vehicles 
https://youtu.be/ByZF8_-OJNI

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

we will unveil the first tutorial, the first project.

This deep traffic, code named Deep Traffic, where your task is to solve the

traffic problem using deep reinforcement learning.

And I'll talk about what's involved in designing a network there, how you submit your own network, how you participate in the competition.

As I said, the winner gets a very special prize to be announced later.

As I mentioned yesterday, that's what's meant usually when you discuss about, you talk about machine learning and talk about its successes.

Supervised learning requires a data set where you know the ground truth.

And you provide that to a machine learning algorithm in order to learn the mapping between the inputs and the outputs in such a way that you can generalize to further examples in the future.

Unsupervised learning is the other side when you know absolutely nothing about the outputs, about the truth of the data that you're working with.

All you get is data and you have to find underlying structure, underlying representation of the data that's meaningful for you to accomplish a certain task, whatever that is.

There's semi-supervised data where only parts, usually a very small amount, is labeled.

There's ground truth available for just a small fraction of it.

of images that are out there on the internet and then you think about ImageNet, a data set where every image is labeled, the size of that ImageNet data set is a tiny subset of all the images

But that's the task we're dealing with as human beings, as people interested in doing machine learning is how to expand the size of that,

of the part of our data that we know something confidently about.

And reinforcement learning sits somewhere in between.

It's semi-supervised learning where there's an agent that has to exist in the world and that agent knows the inputs that the world provides but knows very little about that world except through occasional time-delayed rewards.

to just live it and every once in a while you find out that all that stuff you did last week was pretty bad idea.

in the sense that only a small subset of the data comes with some ground truth, some certainty that you have to then extract knowledge from.

So first, at the core of anything that works currently

in a practical sense, there has to be some ground truth.

There has to be some truth that we can hold on to as we try to generalize and that's supervised learning.

Even as in reinforcement learning, the only thing we can count on is that truth that comes in a form of a reward.

So the standard supervised learning pipeline is you have some raw data, the inputs, you have ground truth, the labels, the outputs that matches to the inputs.

Then you run a certain, any kind of algorithm, whether that's a neural network or another pre-processing algorithm that extracts the features from that data set.

extract the nose, the eyes, the corners of the eyes, the pupil or even lower level features in that image.

After that, we insert those features into a model, a machine learning model.

Then we, whatever that algorithm is, as we pass it through that training process, we then

After we've seen this 1 particular example, how much better are we at other tasks?

And as we repeat this loop, the model learns to perform better and better at generalizing from the raw data to the labels that we have.

And finally, you get to release that model into the wild to actually do prediction on data it has never seen before, that you don't know about.

And the task there is to predict the labels.

Okay, so neural networks is what this class is about.

It's 1 of the machine learning algorithms that has proven to be very successful.

And the building block, the computational building block of a neural network is a neuron.

It's the original old-school neuron where the output is binary, a 0 or 1.

And the process that a Perceptron goes through is it has multiple inputs and a single output.

The inputs, each of the inputs have weights on them, shown here on the left as 0.7, 0.6, 1.4.

And a perceptron, the inputs are ones or zeros, binary.

A bias on each neuron is then added on top

There's a test whether that summed value plus the bias is below or above a threshold.

If it's below a threshold, it produces a 0.

It's 1 of the only things we understand about neural networks, confidently.

We can prove a lot of things about this neuron.

For example, what we know is that a neuron can approximate a NAND gate.

A NAND gate is a logical operation, logical function that takes as input.

It has 2 inputs, A and B, here on the diagram on the left.

And the table shows what that function is.

When the inputs are zeros, 0, 1 in any order, the output is a 1.

The cool thing about a NAND gate is that it's a universal gate that

Your phone in your pocket today can be built out of just NAND gates.

You can build any logical function out of them if you stack them together in arbitrary ways.

The problem with NAND gates and computers is they're built from the bottom up.

You have to design these circuits of NAND gates.

So the cool thing here is the Perceptron, we can learn this magical NAND gate, we can learn this function.

So let's go through how we can do that, how a perceptron can perform the NAND operation.

If we put the weights of negative 2 on each of the inputs and a bias of 3 on the neuron, then if we perform that same operation of summing the weights times the inputs plus the bias, in the top left, we get,

when the inputs are zeros and there's sum to the bias, we get a

That's a positive number, which means the output of a perceptron will be a

a 1, that sum is still a positive number, again produces a 1 and so on.

When the inputs are both ones, then the output is a negative 1, less than 0.

So while this is simple, it's really important to think about.

It's a sort of the 1 basic computational truth you can hold on to as we talk about some

of the magical things neural network can do.

Because if you compare a circuit of NAND gates and a circuit of neurons,

The difference while a circuit of neurons, which is what we think of as a neural network, can perform the same thing as the circuit of NAND gates.

It can learn the arbitrary logical functions that an arbitrary circuit of NAND gates can represent but it doesn't require the human designer.

So 1 of the key aspects here, 1 of the key drawbacks of Perceptron is it's not very smooth in its output.

As we change the weights on the inputs and we change the bias and we tweak it a little bit, it's very likely that when you get, it's very easy to make the neuron output a 0 instead of a 1, or 1 instead of a 0.

So when we start stacking many of these together, it's hard to control the output of the thing as a whole.

Now, the essential step that makes a neural network work, that a circuit of perceptrons doesn't, is if the output is made smooth, is made continuous with an activation function.

And so instead of using a step function like a perceptron does,

shown there on the left, we use any kind of smooth function, a sigmoid,

where the output can change gradually as you change the weights and the bias.

And so learning is generally the process of adjusting those weights gradually and seeing how it has an effect on the rest of the network.

You just keep tweaking weights here and there and seeing how much closer you get to the ground truth.

And if you get farther away, you just adjust the weights in the opposite direction.

There is what we'll mostly talk about today is feed-forward neural networks.

On the left, going from inputs to outputs with no loops.

these amazing things called Recurrent Neural Networks.

They're amazing because they have memory.

But the painful thing is that they're really hard to train.

Today we'll talk about feed-forward neural networks.

An example of stacking a few of these neurons together.

Let's think of the task, the basic task now famous using the classification of numbers.

You have an image of a number, handwritten number, and your task is given that image to say what number is in that image.

And on the left of the network, the size of that input, despite the diagram, is

It's called the hidden layer because it has no interaction with the input or the output.

at the core of the computational power of neural networks, is the hidden layer.

It's tasked with forming a representation

of the data in such a way that it maps from the inputs to the outputs.

In this case, there is 15 neurons in a hidden layer.

There is 10 values on the output corresponding to each of the numbers.

There's several ways you can build this kind of network and this is what the magic of neural networks is you can do it a lot of ways.

You only really need 4 outputs to represent values 0 through 9.

But in practice, it seems that having 10 outputs works better.

Whenever the input is a 5, the output neuron in charge of the 5 gets really excited and outputs a value that's close to 1.

And then the other ones get an output of value, hopefully, that is close to 0.

And when they don't, we adjust the weights in such a way that they get closer to 0 and closer to 1 depending on whether it's the correct neuron associated with the picture.

We'll talk about the details of this training process more tomorrow when it's more relevant.

what we've discussed just now is the forward pass through the network.

It's the pass when you take the inputs, apply the weights, sum them together, add the bias, produce the output and check which of the outputs produces the highest confidence of the number.

Then once those probabilities for each of the numbers is provided, we determine the gradient that's used to punish or reward the weights that resulted in either the correct or the incorrect decisions and that's called back propagation.

We step backwards through the network applying those punishments or rewards.

Because of the smoothness of the activation functions, that is a mathematically efficient operation.

So for example of numbers, the ground truth for number 6 looks like the following in the slides.

Y of X equals to a 10-dimensional vector, where only 1 of them,

The loss function here, the basic loss function is the squared error.

Y of X is the ground truth and A is the output of the neural network resulting from the forward pass.

So when you input that number of a 6 and it outputs, whatever it outputs, that's A, a 10-dimensional vector.

And it's summed over the inputs to produce the squared error.

The loss function, the objective function, that's what's used to determine

how much to reward or punish the back propagated weights throughout the network.

And the basic operation of optimizing that loss function, of minimizing that loss function, is done with various variants of gradient

It's hopefully a somewhat smooth function, but it's a highly nonlinear function.

This is why we can't prove much about neural networks.

It's a highly high-dimensional, highly nonlinear function that's hopefully smooth enough where the gradient descent can find its way to at least a good solution.

And there has to be some stochastic element there

that jumps around to ensure that it doesn't get stuck in a local minima of this very complex function.

There's inputs, there's outputs, ground truth.

We're pretty confident, we know what's going on.

you have this data set, you train and You train a network on that data set and you can evaluate it, you can write a paper and try to beat a previous paper.

The problem is when you then use that neural network to create an intelligent system that you put out there in the world.

And now that system no longer is working with your data set.

It has to exist in this world that's maybe very different

So the takeaway from supervised learning is that neural networks are great memorization

but in a sort of philosophical way they might not be great at generalizing, at reasoning beyond the specific flavor of data set that they were trained on.

The hope for reinforcement learning is that we can extend the knowledge we gain in a supervised way to the huge world outside where we don't have the ground truth of how to act, of what does, how good a certain state is or how bad a certain state is.

This is a kind of brute force reasoning and I'll talk about kind of what I mean there.

But it feels like it's closer to reasoning as opposed to memorization.

That's a good way to think of supervised learning is memorization.

You're just studying for an exam and as many of you know, that doesn't mean you're going to be successful in life just because you get an A.

Just any agent, a human being or any machine existing in this world

can operate in the following way from the perspective of the agent.

It can execute an action, It can receive an observation

resulting from that action in a form of a new state and it can receive a reward or a punishment.

You can break down our existence in this way.

Simplistic view but it's a convenient 1 on the computational side.

And from the environment side, the environment receives the action,

emits the observation, so your action changes the world, therefore that world has to change

and then tell you about it and give you a reward or punishment for it.

I'll try to convey why this is fascinating a little bit later on.

and there's a bouncing ball and you're trying to move your actions to the right, move right, move left.

You're trying to move in such a way that the ball doesn't get past you.

And so here is a human level performance of that agent.

Each action changes the state of the world.

This may seem obvious but moving right changes visually the state of the world.

In fact, what we're watching now on the slides is the world changing before your eyes for this little guy.

They're racking up points in the top left of the video And then when the ball gets past the paddle, it gets punished by dying, quote unquote.

And that's the number of lives that's left, going from 5 to 4 to 3, down to 0.

And so the goal is to select at any 1 moment the action that maximizes future reward.

Without any knowledge of what a reward is, in a greater sense of the word, all you have is an instantaneous reward or punishment, instantaneous response of the world to your actions.

And this can be modeled as a Markov decision process.

Markov decision process is a mathematically convenient construct.

All you get is you have a state that you're currently in, you perform an action, you get a reward and you find yourself in a new state and that repeats over and over.

You start from state 0, you go to state 1, you once again repeat an action, get a reward, go to the next state.

Okay, that's the formulation that we're operating in.

When you're in a certain state, you have no memory of what happened 2 states ago.

Everything is operating on the instantaneous,

And so what are the major components of a reinforcement learning agent.

That's the agent, the function broadly defined of an agent's behavior.

That means, That includes the knowledge of how for any given state, what is an action that I will take with some probability.

Value function is how good each state and action are in any particular state.

a little, a subtle thing that is actually the biggest problem with everything you'll see today.

Is the model is how we represent the environment.

And what you'll see today is some amazing things that neural networks can achieve on

a relatively simplistic model of the world.

And the question whether that model can extend to the real world where human lives are

So let's look at the simplistic world, a robot in a room.

You start at the bottom left, your goal is to get to the top right.

Your possible actions are going up, down, left and right.

See all Lex Fridman transcripts on Youtube

MIT 6.S094: Deep Reinforcement Learning for Motion Planning