This is lecture 4 of course 6.S094: Deep Learning for Self-Driving Cars (2018 version). This class is free and open to everyone. It is an introduction to the practice of deep learning through the applied theme of building a self-driving car.

OUTLINE:
0:00 - Computer Vision and Convolutional Neural Networks
22:15 - Network Architectures for Image Classification
34:39 - Fully Convolutional Neural Networks
44:35 - Optical Flow
50:07 - SegFuse Dynamic Scene Segmentation Competition

INFO:
Slides: http://bit.ly/2HdjksA
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Playlist: https://goo.gl/SLCb1y

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- Show your support: https://www.patreon.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

LINKS:
Playlist: https://goo.gl/SLCb1y
Lecture 1: Deep Learning - https://youtu.be/-6INDaLcuJY
Lecture 2: Self-Driving Cars - https://youtu.be/_OCjqIgxwHw
Lecture 3: Deep Reinforcement Learning - https://youtu.be/MQ6pP65o7OM
Lecture 4: Computer Vision - https://youtu.be/CLOAswsxudo
Lecture 5: Deep Learning for Human Sensing - https://youtu.be/Z2GfE8pLyxc

Guest talk: Sacha Arnoud, Waymo - https://youtu.be/LSX3qdy0dFg
Guest talk: Emilio Frazolli, nuTonomy - https://youtu.be/dWSbItd0HEA
Guest talk: Sterling Anderson, Aurora - https://youtu.be/HKBhP9JISF0

2017:
Guest talk: Sertac Karaman, MIT - https://youtu.be/0fLSf3NO0-s
Guest talk: Chris Gerdes, Stanford - https://youtu.be/LDprUza7yT4

Today we'll talk about how to make machines see Computer vision And we'll present Thank you for said yes And today we will present a competition That unlike deep traffic

which is designed to explore ideas, teach you about concepts of deep reinforcement learning.

SegFuse, the deep dynamic driving scene segmentation competition that I'll present today is at the very cutting edge.

Whoever does well in this competition is likely to produce a publication or ideas that would lead the world in the area of perception.

Perhaps together with the people running this class, perhaps on your own.

Computer vision, today as it stands, is deep learning.

Majority of the successes in how we interpret, form representations Understand images and videos Utilize to a significant degree neural networks

That applies for supervised, unsupervised, and reinforcement learning.

And for the supervised case, which is the focus of today, the process is the same.

There's annotated data where the human provides the labels That serves as the ground truth in the training process Then the neural network goes

through that data learning to map From the raw sensory input to the ground truth labels And then generalize over the testing data set

And the kind of raw sensors we're dealing with are numbers I'll say this again and again that for human vision, for us here, we take for granted this particular aspect of our ability.

It's to take in raw sensory information through our eyes and interpret it.

That's something, whether you're an expert computer vision person Or new to the field, you have to always go back to meditate on Is what kind of things the machine is given What

is the data that It's tasked to work with in order to Perform the task you're asking it to do Perhaps the data it's given is highly Insufficient to do what you want it to do That's the question that'll come up again and again Are images Enough to understand the world around you And given these numbers This set of numbers sometimes with 1 channel Sometimes with 3 RGB Where every single pixel have 3 different colors The task is to classify or regress.

Producing continuous variable or 1 of a set of class labels.

As Before, we must be careful about our intuition Of what is hard and what is easy in computer vision Let's take a step back To the inspiration for neural networks Our own biological neural networks

Because the human vision system And the computer vision system Is a little bit more similar in these regards

The structure of the human visual cortex is in layers And as information passes from the eyes to the To

Of the raw sensor information Higher and higher order representations are formed This

is the inspiration, the idea behind Using deep neural networks for images Higher and higher order representations are formed through the layers

The early layers Taking in the very raw sensory information Then extracting edges Connecting those edges, forming those edges

to form more complex features and finally into the higher order semantic meaning that we hope to get from these images.

In computer vision, deep learning is hard.

The illumination variability is the biggest challenge Or at least 1 of the biggest challenges in driving For visible light cameras Pose variability The objects As I'll also discuss about some of the advances from Jeff Hinton and the capsule networks The idea with the neural networks as they are currently used for computer vision are not good with representing variable pose These objects in images in this 2D plane of color and texture look very different numerically when the object is rotated and the object is mangled and shaped in different ways the deformable truncated cat Interclass variability the...

For the classification task which would be an example today throughout to introduce some of the networks over the past decade that have received success and some of the intuition and insight that made those networks work.

Classification, there is a lot of variability inside the classes and very little variability between the classes.

All of these are cats at top, all of those are dogs at bottom.

And the other, I would say the second biggest problem in driving perception, visible light camera perception, is occlusion.

When part of the object is occluded due to the three-dimensional nature of our world, Some objects in front of others And they occlude the background object And yet we're still tasked with identifying the object When only part of it is visible And sometimes that part Told you there's cats Is very hardly visible Here, we're tasked with classifying a cat with just an ear is visible, just the leg.

And then the philosophical level, As we'll talk about the motivation for our competition here.

Here's a cat dressed as a monkey eating a banana.

level, Most of us Understand what's going on in the scene In fact, a neural network Today, successfully classified this Image, this video as a cat but the context, the humor of the situation, and

the fact that you could argue it's a monkey,

is missing, and what else is missing is the dynamic information, the temporal dynamics of the scene.

That's what's missing in a lot of the perception work that has been done to date in the autonomous vehicle space in terms of visible light cameras.

Image classification pipeline, there's a bin with different categories inside each class.

Those bins, there's a lot of examples of each.

And you're tasked with when a new example comes along you've never seen before to put that image in a bin It's the same as the machine learning task before And everything relies on the data That's been ground truth That's been labeled by human beings MNIST is a toy data set of handwritten digits often used as examples And COCO, CIFAR, ImageNet, Places and a lot of other incredible data sets rich data sets of a hundred thousands, millions of images out there represent scenes, people's faces and different objects.

Those are all ground-truth data for testing algorithms and for competing architectures to be evaluated against each other.

CIFAR-10, 1 of the simplest, almost toy data sets of tiny icons with 10 categories of airplane, automobile, bird, cat, deer, dog, frog, horse, ship and truck is commonly used to explore some

of the basic convolutional neural networks we'll discuss So let's come up with a very trivial classifier to explain the concept of how we could go about it

you start to think about how to classify an image if you don't know any of these techniques This is perhaps the approach you would take Is you would subtract images So in order to know that an image of a cat is different than an

image of a dog You have to compare them When given those 2 images what's the way you compare them?

1 way you could do it is you just subtract it And then sum all the pixel wise differences in the image Just subtract the intensity of the image pixel by pixel Sum it up If that difference is really high that means the images are very different.

Using that metric, we can look at CIFAR-10 and use it as a classifier.

Saying, based on this difference function, I'm going to find 1 of the 10 bins for a new image That has the lowest difference

Find an image in this data set That is most like the image I have and put it in the same bin as that image is in So, there's 10 classes If we just flip a coin, the accuracy of our classifier will be 10% Using our image difference classifier We can actually do pretty good Much better than random, much better than 10%.

That's a classifier, we have our first classifier.

Instead of comparing it to just trying to find 1 image that's the closest in our data set, we try to find K closest and say what class do the majority of them belong to And we take that K and increase it from 1 to 2 to 3 to 4 to 5 And see how that changes the problem

With 7 nearest neighbors which is the optimal under this approach for CIFAR-10

We achieve 30% accuracy Human level is 95% accuracy And with convolutional neural networks We get very close to 100% That's Where neural networks shine This very task of binning images It all starts at this basic computational unit Signal in, each of the signals are weighed, summed, bias added

And put an input into a nonlinear activation function that produces an output.

The nonlinear activation function is key.

All of these put together, in more and more hidden layers, form a deep neural network.

And that deep neural network is trained, as we've discussed, by taking a forward pass On examples that have ground truth labels Seeing how close those labels are to the real ground truth And then punishing the weights That resulted in the incorrect decisions And rewarding the weights that resulted in correct decisions For the case of 10 examples the output of the network is 10 different values.

The input being handwritten digits from 0 to 9.

And we wanted our network to classify what is in this image Of a handwritten digit

way it's often done is there's 10 outputs of the network And each of the neurons on the output is Responsible for getting really excited when its number is called

And everybody else is supposed to be not excited.

Therefore, the number of classes is the number of outputs.

And you assign a class to the input image based on the highest, the neuron which produces the highest output.

a fully connected network that we've discussed on Monday.

There is in deep learning a lot of tricks that make things work that make training much more efficient on large class problems where there's a lot of classes on large data sets When the representation that the neural network is tasked with learning is extremely complex And that's where convolutional neural networks step in The trick they use is spatial invariance They use the idea that A cat in the top left corner of an image Is the same as a cat in

the bottom right corner of an image So we can learn the same features across the image

That's where the convolution operation steps in Instead of the fully connected networks Here There's a third dimension of depth So the blocks in this neural network As

input take 3D volumes And as output produce 3D volumes

They take a slice of the image, a window, and slide it across Applying the same exact weights and we'll go through an example The same exact weights as in the fully connected network on the edges that are used to map the input to the output Here are used to map the slice of an image This window of an image to the output And you can make several, many of such Convolutional filters, many layers many different options of what kind of features you look for in an image, what kind of window you slide across in order to extract all kinds of things, all kinds of edges, All kind of higher order patterns in the images The very important thing is The parameters on each of these filters These subset of the image, these windows Are shared If The feature that defines a cat is useful in the top left corner It's useful in the top right corner It's useful in every aspect of the image This is the trick that makes convolutional neural networks Save a lot of parameters, reduce parameters significantly

Is the reuse, the spatial sharing of features across the space of the image The depth of these 3D volumes is the number of

filters The stride is the skip of the filter, the step size How many pixels you skip when you apply the filter to the input And the padding is the padding, the 0 padding on the outside of the input to a convolutional layer.

So on the left here and the slides are now available online, you can follow them along and I'll step through this example.

On the left here is input volume of 3 channels.

The 3 squares there are the 3 channels and there's numbers inside those channels and then we have a filter in red.

2 of them, 2 channels of filters with a bias and we, those filters are 3 by 3.

Each 1 of them is size 3x3 and what we do is we take those 3x3 filters that are to be learned, These are our variables, our weights that we have to learn And then we slide it across an image To produce the output on the right, the green So by applying the filters in the red There's 2 of them And Within each 1 there's 1 for every input channel We go from the left to the right From the input volume on

the right And you can look You can pull up

you can't see the numbers on the screen But the operations are performed on the input To produce the single value that's highlighted there in the green in the output And we slide this convolution, no filter, along the image With a stride in this case, of 2 Skipping, skipping along They sum to the right, the 2 channel output in green.

That's it, that's the convolutional operation.

That's what's called the convolutional layer in neural networks.

And the parameters here, besides the bias, are the red values in the middle.

And there's a lot of interesting tricks we'll discuss today on top of those.

This is the spatially invariant sharing of parameters

that make convolutional neural networks able to efficiently learn and find patterns and images.

To build your intuition a little bit more about convolution, Here's an input image on the left and on

the right, the identity filter produces the output you see on the right.

And then there's different ways, you can, different kinds of edges you can extract with the activate, with the resulting activation map seen on the right So when applying the filters, those edge detection filters to the image on the left, you produce in white are the parts that activate the convolution The results of these filters

That's what we're trying to learn Any kind of edge Any kind of pattern you can move along in this window in this way that's shown here, you slide around the image and you produce the output you see on the right and depending on how many filters you have in every level you have Many of such slices that you see on the right The input on the left, the output on the right If you have dozens of filters You would have dozens of images on the right Each with different results that show Where each of the individual filter patterns were found And we learn what patterns are useful to look for In order to perform the classification task That's the task for the neural network To learn these filters And the filters have higher and higher order of representation.

Going from the very basic edges to the high semantic meaning that spans entire images.

And the ability to span images can be done in several ways But traditionally has been successfully done through max pooling, through pooling

Of taking the output of a convolutional operation and reducing the resolution of that by condensing that information by for example taking the maximum values, the maximum activations.

Therefore reducing the spatial resolution which has detrimental effects as we'll talk about in scene segmentation, but it's beneficial for finding higher order representations in the images that bring images together, That bring features together to form an entity that we're trying to identify and classify

Okay, so that forms a convolutional neural network Such convolutional layers stacked on top of each other is the only addition to a neural network that makes for a convolutional neural network.

And then at the end, the fully connected layers or any kind of other architectures allow us to apply particular domains.

In ImageNet, the data set, in ImageNet, the challenge, the task is classification.

As I mentioned in the first lecture, ImageNet is a data set,

With 14 million images, 21,000 categories.

And a lot of depth to many of the categories.

As I mentioned, 1,200 Granny Smith apples.

These allow to, these allow the neural networks to learn the rich representations in both pose, lighting variability and interclass class variation for the particular things, particular classes like Granny Smith apples.

So let's look through the various networks.

It started with AlexNet, the first really big successful GPU trained neural network on ImageNet that's achieved a significant boost over the previous year and moved on to VGGNet,

GoogleNet, AguLinet, ResNet, CUImage and SENet in 2017.

Again, the numbers will show for the accuracy are based on the top 5 error rate We get 5 guesses and it's a 1 or 0 If you get guess, if 1 of the 5 is correct You get a 1 for that particular guess Otherwise it's a 0

And human error is 5.1 When a human tries to achieve the same, tries to perform the same task as the machinist asks for doing, the error is 5.1 The human annotation is performed on the images based on binary classification Granny Smith Apple or not, cat or not.

The actual task that the machine has to perform and that the human competing has to perform is given an image is provide 1 of the many classes.

Under that human errors 5.1% which was surpassed in 2015 by ResNet to achieve 4% error.

I'll zoom in on the later networks They have some interesting insights But AlexNet and VGGnet Both followed a very similar architecture Very uniform throughout its depth

VGGnet in 2014 is convolution, convolution pooling convolution pooling, convolution pooling And fully connected layers at the end There's a certain kind of beautiful simplicity, uniformity to these architectures because you can just make it deeper and deeper and makes it very amenable to implementation in a layer stack kind of way in any of the deep learning frameworks.

In the case of VGGnet, 16 or 19 layers with 138 million parameters not many optimizations on these parameters therefore the number of parameters is much higher than the networks that followed it despite the layers not being that large.

GoogleNet introduced the inception module starting to do some interesting things with the small modules within these networks which allow for the training to be more efficient and effective.

The idea behind the inception module shown here with the previous layer on bottom and the convolutional layer here with the inception module on top produced on top is it used the idea that different size convolutions provide different value for the network Smaller convolutions are able to capture or propagate forward features that are very local High resolution in texture Larger convolutions are better able to represent and capture and catch highly abstracted features, higher order features.

So the idea behind the Inception module is to say well, as opposed to choosing in a hyperparameter tuning process or architecture design process, choosing which convolution size we want to go with Why not do all of them together?

In the case of the GoogleNet model There's the 1x1, 3x3 and 5x5 convolutions with the old trusty friend of Max Pooling still left in there as well which has lost favor more and more over time for

the image classification task And the result is there's fewer parameters that are required if you pick

the placing of these inception modules correctly.

The number of parameters required to achieve a higher performance is much lower.

ResNet, 1 of the most popular still to date architectures that we'll discuss in scene segmentation as well,

Came up and used the idea of a residual block The initial inspiring observation Which doesn't necessarily hold true as

it turns out But that network depth Increases representation power

So these residual blocks allow you to have much deeper networks And I'll explain why in a second here But

the thought was they work so well Because the networks are much deeper The key thing that makes these blocks so effective Is the same idea that's reminiscent of a current neural networks That I hope we get a chance to talk about The training of them is much easier They take a simple block Repeat it over and over And they pass the input along without transformation Along with the ability to transform it To learn the filters learn the weights So you're allowed to You're allowed every layer to not only take on the processing of previous layers But to take in the raw and transformed data and learn something new The ability to learn something new allows you to have much deeper networks.

And the simplicity of this block allows for more effective training.

The state-of-the-art in 2017, the winner is Squeeze and Excitation Networks

That unlike the previous year with CU Image Which simply took ensemble methods And combined a lot of successful approaches to take a marginal improvement.

SCNet got a significant improvement at least in percentages.

from 4% to 3% Something like that By using a very simple idea that I think is important to mention A simple insight

It added a parameter to each channel in the convolutional layer In the convolutional block

So the network can now adjust the weighting on each channel Based, for each feature map Based on the content, based on the input to the network This is kind of a take away to think about About any of the networks You talk about any of the architectures Is a lot of times Recurrent neural networks and convolutional neural networks have tricks that significantly reduce the number of parameters The bulk, the sort of low-hanging fruit They use spatial invariance, temporal invariance to reduce the number of parameters to represent the input data But They also leave certain things not parametrized.

They don't allow the network to learn it.

Allowing in this case the network to learn the weighting on each of the individual channels, so each of the individual filters is something that you learn as along with the filters, makes a huge boost.

The cool thing about this is it's applicable to any architecture.

This kind of block, this kind of what, the squeeze and excitation block, is applicable to any architecture.

And, because obviously it's just simply parameterizes the ability to choose which filter you go with based on the content.

And for future research, it inspires to think about what else can be parameterized in neural networks What else can be controlled as part of the learning process including higher and higher order hyperparameters

Which aspects of the training and the architecture of the network can be part of the learning.

Another network has been in development since the 90s, ideas, with Jeff Hinton, But really received, has been published on

and received significant attention in 2017 that I won't go into detail here.

We are going to release an online only video about capsule networks.

It's a little bit too technical, but they inspire a very important point

that we should always think about with deep learning whenever it's successful.

It's to think about what, as I mentioned, with the cat eating a banana, on a philosophical and the mathematical level, we have to consider what assumptions these networks make and what, through those assumptions, they throw away.

So neural networks due to the spatial, with convolutional neural networks, due to their spatial invariance, throw away information about the relationship

between the hierarchies, between the simple and the complex objects So the face on the left and the face on the right Looks the same to a convolutional neural network The presence of eyes and nose and mouth Is the essential aspect of what makes the classification task work for convolutional network.

Where it will fire and say this is definitely a face.

But the spatial relationship is lost, is ignored.

Which means There's a lot of implications to this, but for things like pose variation, that information is lost.

We're throwing that away completely and hoping that the pooling operation that's performed in these networks is able to sort of mesh everything together to come up with the features that are firing of the different parts of the face to then come up with the total classification that it's a face without representing really the relationship between these features at the low level and the high level at the low level of the hierarchy at the simple and the complex level.

This is a super exciting field now that's hopefully will spark developments of how we design neural networks that are able to learn the rotational, the orientation in variance as well.

Ok, so as I mentioned, you take these convolutional neural networks, chop off the final layer in order to apply to a particular domain.

And that is what we'll do with fully convolutional neural networks.

The ones that we tasked to segment the image at a pixel level.

As a reminder, these networks through the convolutional process Are really producing a heat map Different parts of the network are getting excited Based on the different aspects of the image and so it can be used to do the localization of detecting not just classifying the image but localizing the object and they could do so at a pixel level.

are doing the encoding process, they're taking the rich raw sensory information in the image and encoding them into an interpretable set of features, representation that can then be used for classification.

But we can also then use a decoder, up-sample that information and produce a map like this.

Fully convolutional neural network segmentation, semantic scene segmentation, image segmentation.

The goal is to, as opposed to classify the entire image, you classify every single pixel.

You color every single pixel with what that pixel What object that pixel belongs to in this 2D space of the image The 2D projection

in the image of a three-dimensional world.

So the thing is, there's been a lot of advancement in the last 3 years,

but it's still an incredibly difficult problem.

If you think about the amount of data that's used for training And the task of pixel level of megapixels here Of millions of pixels that are tasked with Having assigned a single label It's an extremely difficult problem Why is this interesting important problem to try to solve?

As opposed to bounding boxes around cats Well, it's whenever precise boundaries of objects are important Certainly medical applications when looking at imaging and detecting particular, for example, detecting tumors in medical imaging of different organs.

And in driving, in robotics, when objects are involved, it's a dense scene involved with vehicles, pedestrian, cyclists, we need to be able to not just have a loose estimate of where objects are, we need to be able to have the exact boundaries And then potentially through data fusion, fusing sensors together Fusing this rich textural information about pedestrian, cyclists and vehicles to light our data that's providing us the three-dimensional map of the world We have both the semantic meaning of the different objects and their exact three-dimensional location

A lot of this work successfully A lot of the work in the semantic segmentation Started with fully convolutional networks For semantic segmentation paper FCN, that's where

the name FCN came from In November 2014 Now go through a few papers here To give you some intuition where the field is gone And how that takes us to segfuse The segmentation competition So FCN repurposed the ImageNet pre-trained nets The nets that were trained to classify what's in an image The entire image And chopped off the fully connected layers And then added decoder parts that up-sampled the image To produce a heat map Here shown with a tabby cat a heat map of where the cat is in the image.

It's a much slower, much coarser resolution than the input image.

Skip connections to improve coarseness of upsampling There's a few tricks If you do the most naive approach The upsampling is going to be extremely coarse Because that's the whole point of the neural network The encoding part is you throw away all the useless data To the most essential aspects that represent that image So you're throwing away a lot of information that's necessary To then form a high-resolution image So there's a few tricks where you skip a few of the final pooling operations to go in similar ways as a residual block to go to the output, produce higher and higher resolution heatmap at the end SegNet in 2015

applied this to the driving context and really taking a Dikiti dataset and have shown a lot of interesting results And really explored the encoder-decoder formulation of the problem Really solidifying this, the place of the encoder-decoder framework For the segmentation task Dilated convolution, I'm taking you through a few components which are critical here to the state-of-the-art.

Dilated convolutions, so the convolution operation

See all Lex Fridman transcripts on Youtube

MIT 6.S094: Computer Vision