This is lecture 3 of course 6.S094: Deep Learning for Self-Driving Cars taught in Winter 2017. This lecture introduces computer vision, convolutional neural networks, and end-to-end learning of the driving task.

INFO:
Slides: http://bit.ly/2HdXYvf
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Playlist: https://goo.gl/SLCb1y

Links to individual lecture videos for the course:

Lecture 1: Introduction to Deep Learning and Self-Driving Cars
https://youtu.be/1L0TKZQcUtA

Lecture 2: Deep Reinforcement Learning for Motion Planning
https://youtu.be/QDzM8r3WgBw

Lecture 3: Convolutional Neural Networks for End-to-End Learning of the Driving Task 
https://youtu.be/U1toUkZw6VI

Lecture 4: Recurrent Neural Networks for Steering through Time 
https://youtu.be/nFTQ7kHQWtc

Lecture 5: Deep Learning for Human-Centered Semi-Autonomous Vehicles 
https://youtu.be/ByZF8_-OJNI

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

So today we will, we talked a little bit about neural networks, started to talk about neural networks yesterday.

Today we'll continue to talk about neural networks that work with images, convolutional neural networks, and see how those types of networks can help us drive a car.

If we have time, we'll cover a simple illustrative case study of detecting traffic lights.

The problem of detecting green, yellow, red.

If we can't teach our neural networks to do that, we're in trouble.

But it's a good, clear, illustrative case study of a three-class classification problem.

Here, looped over and over in a very short GIF.

This is actually running live on a website right now.

We'll show it towards the end of the lecture.

This once again, just like deep traffic, is a neural network that learns to steer a vehicle based on the video of the forward roadway.

And once again, doing all of that in the browser using JavaScript.

So you'll be able to train your own very network to drive using real world data.

We will also have a tutorial and code briefly described today at the end of lecture, if there's time, how to do the same thing in TensorFlow.

So if you want to build a network that's bigger, deeper, and you want to utilize GPUs to train that network, you want to not do it in your browser.

You want to do it offline using TensorFlow and having a powerful GPU on your computer.

So we talked about vanilla machine learning where there's no, where the size, yesterday, where the size of the input is small for the most part.

The number of neurons, in the case of neural networks, is on the order of 10, 100, 1000.

When you think of images, images are a collection of pixels.

1 of the most iconic images from computer vision in the bottom left there is Lana.

I encourage you to Google it and figure out the story behind that image.

It was quite shocking when I found out recently.

So once again, computer vision is these days dominated a data-driven approaches, by machine learning where all of the same methods that are used on other types of data are used on images where the input is just a collection of pixels and pixels are numbers from 0 to 255, discrete values.

So we can think exactly what we've talked about previously.

You can think of images in the same exact way.

We can do supervised learning where you have an input image, an output label.

The input image here is a picture of a woman, the label might be woman.

Again, semi-supervised and reinforcement learning.

In fact, the Atari games we talked about yesterday do some pre-processing on the images.

They're using convolutional neural networks as we'll discuss today.

And the pipeline for supervised learning is again the same.

There's raw data in the form of images, there's labels on those images.

We perform a machine learning algorithm, performs feature extraction, it trains given the inputs and outputs on the images and the labels of those images, constructs a model and then test that model.

Accuracy is the term that's used to often describe how well a model performs.

I apologize for the constant presence of cats throughout this course.

I assure you this course is about driving, not cats.

We're really good at looking at and converting visual perception as human beings, converting visual perception into semantics.

We see this image and we know it's a cat.

But a computer only sees numbers, RGB values for a colored image.

There's 3 values for every single pixel from 0 to 255.

And so given that image, We can think of 2 problems.

1 is regression and the other is classification.

Regression is when given an image, we want to produce a real value to output back.

So if we have an image of the forward roadway, we want to produce a value for the steering wheel angle.

And if you have an algorithm that's really smart, it can take any image of the forward roadway and produce the perfectly correct steering angle that drives the car safely across the United States.

We'll talk about how to do that and where that fails.

Classification is when the input again is an image and the output is a class label, a discrete class label.

Underneath it though, often is still a regression problem and what's produced is a probability that this particular image belongs to a particular category.

And we use a threshold to chop off the outputs associated with low probabilities and take the labels associated with the high probabilities and convert it into a discrete classification.

I mentioned this yesterday but it bears saying again, computer vision is hard.

As human beings, we're really good at dealing with all these problems.

The object looks totally different in terms of the numbers behind the images, in terms of the pixels when viewed from a different angle, viewpoint variation.

Objects, when you're standing far away from them or up close, are totally different size.

We're good at detecting that they're different size, it's still the same object as human beings but that's still a really hard problem because those sizes can vary drastically.

We talked about occlusions and deformations with cats.

You have to separate the object of interest from the background.

And given the three-dimensional structure of our world, there's a lot of stuff often going on in the background, the clutter.

There are inter-class variation that's often greater than inter-class variation, Meaning objects of the same type often have more variation than the objects that you're trying to separate them from.

There's the hard 1 for driving, Illumination.

Light is the way we perceive things, the reflection of light off the surface.

And the source of that light changes the way that object appears.

So the image classification pipeline is the same as I mentioned.

There's categories, it's a classification problem, so there's categories of cat, dog, mug, hat.

You have a bunch of examples, image examples of each of those categories.

And so the input is just those images paired with the category.

And you train to map, to estimate a function that maps from the images to the categories.

For all of that you need data, a lot of it.

There is unfortunately, There's a growing number of datasets but they're still relatively small.

We get excited, there are millions of images but they're not billions, they're trillions of images.

And these are the datasets that you will see if you read academic literature most often.

MNIST, the 1 that's been beaten to death and then we use as well in this course, is a data set of handwritten digits where the categories are 0 to 9.

ImageNet, 1 of the largest image datasets, fully labeled image datasets in the world, has images with a hierarchy of categories from WordNet.

And what you see there is a labeling of what images associated with which words are present in the dataset.

CIFAR-10 and CIFAR-100 are tiny images that are used to prove in a very efficient and quick way, offhand that your algorithm that you're trying to publish on or trying to impress the world with works well.

And places is a data set of natural scenes, woods, nature, city, so on.

It's a data set of 10 categories, airplane, automobile, bird, cat, so on.

They're shown there with sample images as the rows.

So let's build a classifier that's able to take images from 1 of these 10 categories and tell us what is shown in the image.

Once again, all the algorithm sees is numbers.

So we have to try to have, at the very core, we have to have an operator for comparing 2 images.

So given an image and I want to say if it's a cat or a dog, I want to compare it to images of cats and compare it to images of dogs and see which 1 matches better.

So there has to be a comparative operator.

Okay, so 1 way to do that is take the absolute difference between the 2 images, pixel by pixel.

Take the difference between each individual pixel shown on the bottom of the slide for a 4x4 image and then we sum that pixel-wise absolute difference into a single number.

So if the image is totally different pixel-wise, that'll be a high number.

If it's the same image, the number will be 0.

0, it's the absolute value too of the difference.

When we speak of distance, we usually mean L2 distance.

And so if we try to, so we can build a classifier that just uses this operator to compare to every single image in the dataset and say, I'm going to pick the category that's the closest using this comparative operator.

I'm going to find, I have a picture of a cat and I'm going to look through the dataset and find the image that's the closest to this picture and say that is the category that this picture belongs to.

So if we just flip the coin and randomly pick which category an image belongs to, we get that accuracy would be on average 10%.

The accuracy we achieve with our brilliant image difference algorithm that just goes to the data set and finds the closest 1 is 38%.

So you can think about this operation of looking through the dataset and finding the closest image as what's called K-nearest neighbors.

Where K in that case is 1, meaning you find the 1 closest neighbor to this image that you're asking a question about and accept the label from that image.

You could do the same thing increasing K.

Increasing K to 2 means you take the 2 nearest neighbors.

You find the 2 closest in terms of pixel-wise image difference, images to this particular query image and find which category do those belong to.

What's shown up top on the left is the data set we're working with, red, green, blue.

What's shown in the middle is the 1 nearest neighbor classifier, meaning this is how you segment the entire space of different things that you can compare.

And if a point falls into any of these regions, it will be immediately associated with the nearest neighbor algorithm to belong to that region.

With 5 nearest neighbors, there's immediately an issue.

The issue is that there's white regions, there's type breakers, where your 5 closest neighbors are from various categories.

So if we, this is a good example of parameter tuning.

You have 1 parameter, k, and your task as a teacher of machine learning, you have to teach this algorithm how to do your learning for you, is to figure out that parameter.

That's called parameter tuning or hyperparameter tuning as it's called in neural networks.

And So on the bottom right of the slide is, on the x-axis is K as we increase it from 0 to 100 and on the y-axis is classification accuracy.

It turns out that the best K for this data set is 7, 7 years neighbors.

Human level performance, and I should say that The way that number is, we get that number as we do with a lot of the machine learning pipeline process is you separate the data into the parts of the dataset you use for training and another part they use for testing.

You're not allowed to touch the testing part.

You construct your model of the world on the training data set and you use what's called cross validation where you take a small part of the training data, shown fold 5 there in yellow, to leave that part out from the training and then use it as part of the hyperparameter tuning.

As you train, figure out with that yellow part, fold 5, how well you're doing and then you choose a different fold and see how well you're doing and keep playing with parameters, never touching the test part.

And when you're ready, you run the algorithm on the test data to see how well you really do, how well it really generalizes.

Is there any way to determine an intuition of what a good K may be?

Or do you just have to run through all the data?

So the question was, Is there a good way to, is there any good intuition behind what a good K is?

There's general rules for different datasets, but usually you just have to run through it, grid search, brute force.

Yes, the question was, is each pixel 1 number or 3 numbers?

For majority of computer vision throughout its history, you use grayscale images, so it's 1 number.

But RGB is 3 numbers and there's sometimes a depth value too, so it's 4 numbers.

So it's, if you have a stereo vision camera that gives you the depth information of the pixels, that's a fourth.

And then if you stack 2 images together, that could be 6.

In general, everything we work with will be 3 numbers for a pixel.

So you're using the absolute value for 1 number?

So the question was for the absolute value is just 1 number exactly right.

So in that case, those gray scale images, so it's not RGB images.

So that's, you know, this algorithm is pretty good.

If we use the best, We optimize the hyperparameters of this algorithm, choose K of 7, seems to work well for this particular CIFAR-10 dataset.

Human beings perform at about 94, slightly above 94% accuracy for CIFAR-10.

So given an image, it's a tiny image, I should clarify, it's like a little icon.

Given that image, human beings are able to determine accurately 1 of the 10 categories with 94% accuracy.

And the currently state-of-the-art convolutional neural networks is 95.4% accuracy.

And believe it or not, it's a heated battle.

But the most important, the critical fact here is, it's recently surpassed humans and certainly surpassed the K-Nearest Neighbors algorithm.

It all still boils down to this little guy, the neuron, that sums the weights of its inputs, adds a bias, produces an output based on an activation, a smooth activation function.

The question was, you take a picture of a cat so you know it's a cat?

Like you have to write that down somewhere.

So you have to write as a caption, this is my cat.

And then the unfortunate thing, given the Internet and how witty it is, you can't trust the captions on images because maybe you're just being clever and it's not a cat at all.

Are CNNs in general better than 8-year standards?

Are, do, commonly streamed neural networks in general do better than 8-year standards?

Yeah, so the question was, do convolutional neural networks generally do better than nearest neighbors?

There's very few problems on which neural networks don't do better.

Yes, they almost always do better except when you have almost no data.

And convolutional neural networks isn't some special magical thing.

It's just neural networks with some cheating up front that I'll explain.

Some tricks to try to reduce the size and make it capable to deal with images.

So again, yeah, the input is, in this case that we looked at, classifying an image of a number as opposed to doing some fancy convolutional tricks.

We just take the entire 28x28 pixel image, That's 784 pixels as the input.

That's 784 neurons in the input, 15 neurons on the hidden layer and 10 neurons in the output.

Now everything we'll talk about has the same exact structure, nothing fancy.

There is a forward pass through the network where you take an input image and produce an output classification and there's a backward pass through the network, through back propagation, where you adjust the weights when your prediction doesn't match the ground truth output.

And learning just boils down to optimization.

It's just optimizing a smooth function, differentiable function that's defined as the loss function.

That's usually as simple as a squared error between the true output and the 1 you actually got.

Convolutional neural networks take inputs that have some spatial consistency, have some spatial meaning in them, like images.

You can think of the dimension of time and you can input audio signal into a convolutional neural network.

And so the input is usually, for every single layer that's a convolutional layer, the input is a 3D volume and the output is a 3D volume.

I'm simplifying because you can call it 4D too but it's 3D.

The height and the width is the width and the height of the image and then the depth for grayscale image is 1, for an RGB image is 3, for a 10 frame video of grayscale images the depth is 10.

It's just a volume, a 3D matrix of numbers.

And the only thing that a convolutional layer does is take a 3D volume as input, produce a 3D volume as output, and has some smooth function operating on the inputs, on the sum of the inputs that may or may not be a parameter that you tune, that you try to optimize.

It's a Lego pieces that you stack together in the same way as we talked about before.

So what are the types of layers that a convolutional neural network have?

So for example, a color image of 32 by 32 will be a volume of 32 by 32 by 3.

A convolutional layer takes advantage of the spatial relationships of the input neurons.

And a convolutional layer, it's the same exact neuron as for a fully connected network.

The regular network we talked about before.

But it just has a narrower receptive field, It's more focused.

The inputs to a neuron on the convolutional layer come from a specific region from the previous layer.

And the parameters on each filter, you can think of this as a filter because you slide it across the entire image.

So as opposed to taking the, if you think about 2 layers, as opposed to connecting every single pixel in the first layer to every single neuron in the following layer.

You only connect the neurons in the input layer that are close to each other, to the output layer.

And then you enforce the weights to be tied together spatially.

And what that results in is a filter, every single layer on the output, you could think of as a filter, that gets excited, for example, for an edge.

And when it sees this particular kind of edge in the image, it'll get excited.

It'll get excited in the top left of the image, top right, bottom left, bottom right.

The assumption there is that a powerful feature for detecting a cat is just as important no matter where in the image it is.

And this allows you to cut away a huge number of connections between neurons.

But it still boils down on the right as a neuron that sums a collection of inputs and applies weights to them.

The spatial arrangement of the output volume relative to the input volume is controlled by 3 things.

So for every single quote-unquote filter, you'll get an extra layer on the output.

So if the input, let's talk about the very first layer, the input is 32 by 32 by 3.

If the number of filters is 10, then the resulting depth, the resulting number of depth, the resulting number of stacked channels in the output will be 10.

Stride is the step size of the filter that you slide along the image.

Oftentimes that's just 1 or 3 and that directly reduces the spatial size, the width and the height of the output image.

And then there is a convenient thing that's often done is padding the image on the outside with zeros so that the input and the output have the same height and width.

So this is a visualization of convolution.

I encourage you to kind of maybe offline think about what's happening.

It's similar to the way human vision works, crudely so, if there's any experts in the audience.

So the input here on the left is a collection of numbers, 0, 1, 2, and a filter.

Well, there is 2 filters shown as W0 and W1.

Those Filters shown in red are the different weights applied on those filters.

And each of the filters have a depth just like the input, a depth of 3.

And so you slide that filter along the image, keeping the weights the same.

And so your first filter, you pick the weights.

You pick the weights in such a way that it fires, gets excited at useful features and doesn't fire for not useful features.

And then this second filter that fires for useful features and not.

And produces a signal on the output depending on a positive number, meaning there's a strong feature in that region and a negative number if there isn't.

This allows for drastic reduction in the parameters and so you can deal with inputs that are a thousand by a thousand pixel image for example or video.

There's a really powerful concept there, the spatial sharing of weights.

That means there's a spatial invariance to the features you're detecting.

See all Lex Fridman transcripts on Youtube

MIT 6.S094: Convolutional Neural Networks for End-to-End Learning of the Driving Task