This is lecture 5 of course 6.S094: Deep Learning for Self-Driving Cars taught in Winter 2017.

INFO:
Slides: http://bit.ly/2HcHq6O
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Playlist: https://goo.gl/SLCb1y

Links to individual lecture videos for the course:

Lecture 1: Introduction to Deep Learning and Self-Driving Cars
https://youtu.be/1L0TKZQcUtA

Lecture 2: Deep Reinforcement Learning for Motion Planning
https://youtu.be/QDzM8r3WgBw

Lecture 3: Convolutional Neural Networks for End-to-End Learning of the Driving Task 
https://youtu.be/U1toUkZw6VI

Lecture 4: Recurrent Neural Networks for Steering through Time 
https://youtu.be/nFTQ7kHQWtc

Lecture 5: Deep Learning for Human-Centered Semi-Autonomous Vehicles 
https://youtu.be/ByZF8_-OJNI

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

So how do we turn this camera back in on the human?

So we've been talking about perception, how to detect cats and dogs, pedestrians, lanes, how to steer a vehicle based on the external environment.

The thing that's really fascinating and severely understudied is the human side.

So, you know, you talk about the Tesla, we have cameras in 17 Teslas driving around Cambridge because Tesla is 1 of the only vehicles allowing you to experience in the real way on the road the interaction between the human and the machine.

And the thing that we don't have, that deep learning needs on the human side of semi-autonomous vehicles and fully autonomous vehicles is video of drivers.

That's what my work is in, is looking at billions of video frames of human beings driving 60 miles an hour plus on the highway in their semi-autonomous Tesla.

What are the things that we want to know about the human?

If we are a deep learning therapist and we try to break apart the different things we can detect from this raw set of pixels, we can look here from the green to red is the different detection problems, the different computer vision detection problems.

Green means it's less challenging, it's feasible, even under poor lighting conditions, variable pose, noisy environment, poor resolution.

Red means it's really hard no matter what you do.

That's starting on the left with face detection body pose, 1 of the best studied and 1 of the easier computer vision problems.

And then there is micro saccades, the slight tremors of the eye that happen at 1, at a rate of a thousand times a second.

All right, let's look at, well first, like why do we even care about the human in the car?

So you think about it, to build trust, the car needs to have some awareness of the biological thing it's carrying inside, the human inside.

You kind of assume the car knows about you because you're like sitting there controlling it, but if you think about it, almost every single car on the road today has no sensors with which it's perceiving you.

It knows some cars have a pressure sensor on the steering wheel and a pressure sensor or some kind of sensor detecting that you're sitting in the seat.

That's the only thing it knows about you.

This same car that's driving 70 miles an hour on the highway autonomously, how is it supposed to build trust with you if it doesn't perceive you?

So if I'm constantly advocating something, it's that we should have a driver-facing camera in every car.

And that despite the privacy concerns, you have a camera on your phone and you don't have as much of a privacy concern there.

Despite the privacy concerns, the safety benefits are huge.

So let's start with the easy 1, body pose, detecting body pose.

There's these dummies, crash test dummies with which, which are used to design the safety system, the passive safety systems in our cars.

And they make certain assumptions about body shapes, male, female, child, body shapes.

But they also make assumptions about the position of your body in the seat.

They have the optimal position, the position they assume you take.

The reality is in a Tesla, when the car is driving itself, the variability, if you remember the cat, the deformable cat, you start doing a little bit more of that.

You start to reach back in the backseat, in your purse, your bag, for your cell phone, these kinds of things.

And we need to know how often that happens.

The car needs to know that you're in that position.

That's critical for that very serious moment when the actual crash happens.

Whenever you have these kind of tasks of detecting, for example, body poses, you're detecting points of the shoulders, points of the head, 5, 10 points along the arms, the skeleton.

You have a CNN, convolutional neural network, that takes this input image and takes as an output, it's a regressor.

It gives an XY position of the whatever you're looking for, the left shoulder or the right shoulder.

And then you have a cascade of regressors that give you all these points, that give you the shoulders, the arms and so on.

And then you have through time on every single frame you make that prediction and then you optimize, You know, you can make certain assumptions about physics that your arm can't be in this place in 1 frame and then the next frame be over here.

So under those constraints, you can then minimize the error, the temporal error from frame to frame.

Or you can just dump all the frames as if there are different channels, like RGB is 3 channels, you could think of those channels as in time.

You can dump all those frames together in what are called 3D convolutional neural networks.

You dump them all together and then you estimate the body pose in all the frames at once.

And there are some data sets for sports and we're building our own.

This is what we're recording in the Tesla.

There's in the bottom right, there's a blue icon that lights up automatically detected if it's operating under autopilot.

That means the car is currently driving itself.

There's 5 cameras, 1 of the forward roadway, 1 in the instrument cluster, 1 of the center stack, steering wheel, his face.

You dump the raw pixels into a convolutional neural network, have 6 classes, forward roadway, you're predicting where the person is looking, forward roadway, left, right, center stack, instrument cluster, rear view mirror, and you give it millions of frames for every class.

And it does incredibly well at predicting where the driver is looking.

And the process is the same for majority of the driver state problems that have to do with the face.

The face has so much information, Where you're looking, emotion, drowsiness, so different degrees of frustration, I'll fly through those as well.

There's noise, there's vibration from the vehicle.

So first you have to, video stabilization, you have to remove all that vibration, all that noise as best as you can.

There's a lot of algorithms, non-neural network algorithms, boring but they work, for removing the noise, removing the effects of sudden light variations and the vibrations of the vehicle.

So you have to estimate the frame of the camera, the position of the camera, and estimate the identity of the person you're looking at.

The more you can specialize the network to the identity of the person and the identity of the car the person is riding in, the better the performance for the different driver state classification.

You have a background model that works on everyone and you specialize each individual.

You specialize each individual network to that 1 individual.

There is face frontalization, fancy name, for the fact that no matter where they're looking you want to transfer that face of the eyes and nose of the exact same position in the image.

That way if you want to look at the eyes and you want to study the subtle movement of the eyes, the subtle blinking, the dynamics of the eyelid, the velocity of the eyelid, it's always in the same place.

You can really focus in, remove all effects of any other motion of the head.

And then you just, This is the beauty of deep learning, right?

You don't, there is some pre-processing because this is real-world data, but you just dump the raw pixels in.

You dump the raw pixels in and predict whatever you need.

You can have, so we had a study where people used a crappy and a good voice-based navigation system.

So crappy 1 got them really frustrated and they self-reported it as a frustrating experience or not on scale 1 to 10.

But it had a bunch of people use the system and you know, they put themselves as frustrated or not and so then we can predict, we can train a convolutional neural network to predict is this person frustrated or not.

Turns out smiling is a strong indication of frustration.

You can also predict drowsiness in this way, gaze estimation in this way, cognitive load.

I'll briefly look at that and the process is all the same.

You detect the face, you find the landmark points in the face for the face alignment, face frontalization, and then you dump the raw pixels in for classification, step 5.

You can use SVMs there or you can use what everyone uses now, convolutional neural networks.

This is the 1 part where CNNs have still struggled to compete, is the alignment problem.

This is why I talked about the cascade of regressors, is finding the landmarks on the eyebrows, the nose, the jawline, the mouth.

There's certain constraints there and so algorithms that can utilize those constraints effectively can often perform better than end-to-end regressors that just don't have any concept of what a face is shaped like.

And there's huge datasets and we're part of the awesome community that's building those datasets for face alignment.

Okay, so this is again the TA in his younger form.

This is live in the car, real-time system predicting where they're looking.

This is taking slow steps towards the exciting direction that machine learning is headed which is unsupervised learning.

The less you have to have humans look through the data and annotate that data, the more power these machine learning algorithms get.

So, currently supervised learning is what's needed.

You need human beings to label a cat and label a dog.

But if you can only have a human being label 1%, 1 tenth of a percent of a data set, only the hard cases, so the machine can come to the human and be like, I don't know what I'm looking at these pictures, because of the partial light occlusions.

We're not good at dealing with occlusions, whether it's your own arm or because of light conditions.

We're not good with crazy light drowning out the image.

This is what Google self-driving car actually struggle with when they're trying to use their vision sensors.

So just all kinds of occlusions are really hard for computer vision algorithms.

And in those case, we want a machine to step in and say, and pass that image on to the human.

And the other corner cases is, so in driving for example, 90 plus percent of the time, all you're doing is staring forward at the road in the same way.

That's where machine annotation, automated annotation shines.

Because it's seen that face for hundreds of millions of frames already in that exact position.

So it can do all the hard work of annotation for you.

It's in the transition away from those positions that it needs a little bit help.

Just to make sure that this person just start looking away from the road to the rear view and you bring those points up.

So you're using optical flow, putting the optical flow in the convolutional neural network.

You use that to predict when something has changed.

And when something has changed, you bring that to the machine for annotation.

All of this is to build a giant, billions of frames, annotated data set of ground truth.

I want you to train your driver state algorithms.

On the X-axis is the fraction of frames that a human has to annotate.

The more the human annotates, the higher the accuracy.

This is for the gaze classification task.

When with an 84 fold, so almost 2 orders of magnitude reduction in human annotation.

And the result is millions of images like this, video frames.

The frustrated driver is the 1 that's on the bottom.

So a lot of movement of the eyebrows and a lot of smiling.

And the happy, the satisfied, I don't say happy, the satisfied driver is cold and stoic.

And that's true for subject after subject because driving is a boring experience and you want it to stay that way.

So the comment was, my emotions might then have nothing to do with the driving experience.

You're an actor on the stage for others with your emotion.

So when you're alone, you might not express emotion.

You're really expressing emotion oftentimes for others.

Like you're frustrated, so it's like, oh, what the heck.

That's for the passenger, and that's absolutely right.

So 1 of the cool things we're doing, as I said, we now have over a billion video frames in the Tesla.

We're collecting huge amounts of data in the Tesla.

In this case, we know the ground truth, how frustrated they were.

In naturalistic data, when it's just people driving around, we don't know how they're really feeling at the moment.

We're not asking them to enter in an app, how are you feeling right now?

But We do know certain things, like we know that people sing a lot.

So that doesn't happen in this kind of data, because there's somebody sitting in the car.

And I think the expression of frustration is also the same.

Yes, so, yeah, the question is, yeah, so, or the comment is that the data set, the solo data set is probably gonna be very different from a data set that's non-solo with a passenger.

The tricky thing about driving, and this is why it's a huge challenge for self-driving cars, for the external facing sensors and for the internal facing sensors analyzing human behavior is like 99.9% of driving is the same thing.

So finding the interesting bits is actually pretty complicated.

So that has to do with emotion, That has to do with...

So whenever you're talking or singing, we can find that.

But how do you find the subtle expressions of emotion?

I mean similar to emotion, it's a little more concrete in a sense that there's a lot of good science and ways to measure cognitive load, cognitive workload, how occupied your mind is.

And so the window to the soul, the cognitive workload soul is the eyes.

So pupil, So first of all, the eyes move in 2 different ways.

They move a lot of ways, but 2 major ways is saccades.

They jump around whenever you look around the room.

When you read their eyes are jumping around.

And when you just follow this bottle with your eyes, that your eyes are actually going to move smoothly, smooth pursuit.

Somebody actually just told me today that probably has to do with our hunting background or as animals.

I don't know how that helps like frogs track flies really well, so that you have to like, I don't know.

Anyway, the point is there are smooth pursuit movements where the eyes move smoothly.

And those are all indications of certain aspects of cognitive load.

And then there is very subtle movements which are almost imperceptible for computer vision.

And these are micro saccades, these are tremors of the eye.

Here, work from here from Bill Freeman, magnifying those subtle movements.

And so cognitive load, When the pupil, that black dot in the middle, just in case you don't know what a pupil is, in that middle eye, when it gets larger, that's an indicator of high cognitive load.

But it also gets larger when the light is dim.

So we can't rely in the wild, outside, in the car, or just in general outdoors on using the pupil size.

Even though pupil size has been used effectively in a lab to measure cognitive load, it can't be reliably used in the car.

When there's a higher cognitive load, your blink rate decreases and your blink duration shortens.

Okay, I think I'm just repeating the same thing over and over, but you could imagine how we can predict cognitive load, right?

Here is the primary eye of the person the system is observing.

We take the sequence of a hundred, oh it's 90 images, so that's 6 seconds, 16 frames a second, 15 frames a second, and we dump that into a 3D convolutional neural network.

Again, that means it's 90 channels of 90 frames, gray scale, and then the prediction is 1 of 3 classes of cognitive load.

Low cognitive load, medium cognitive load, and high cognitive load.

And there's ground truth for that because we have people, over 500 different people do different tasks of various cognitive load.

And after some frontalization again, where you see the eyes are, no matter where the person is looking, the image of the face is transposed in such a way that the eyes, the corners of the eyes remain always in the same position.

After the frontalization, we find the eye, active appearance models, find 39 points of the eyelids, the iris, and 4 points on the pupil.

Putting all of that into a 3D CNN model, their positioned image eye sequence on the left, 3D CNN model in the middle, cognitive load prediction on the right.

This code by the way is freely available online.

All you have to do, dump a webcam from the video stream, CNN runs in faster than real time, predicts cognitive load.

Same process as detecting the identity of the face, same process as detecting where the driver is looking, same process detecting emotion.

And all of those require very little hyperparameter tuning on the convolutional neural networks.

And why do we care about detecting what the driver is doing?

Is On the, oh man, this is the comeback of the slide.

I was criticized for this being a very cheesy slide.

In the path towards full automation, We're likely to take gradual steps towards that.

And especially given that the, this is given today, our new president.

This is manually controlled vehicle country for quite a little while.

And control being given to somebody else, to the machine, will be a gradual process.

It's a gradual process of that machine earning trust.

And through that process, the machine like the Tesla, like the BMW, like the Mercedes, the Volvo that's now playing with these ideas is going to need to see what the human is doing.

And for that, to see what the human is doing, we have billions of miles of forward-facing data.

What we need is billions of miles of driver-facing data as well.

We're in the process of collecting that, and this is a pitch for automakers and everybody to buy cars that have a driver-facing camera.

So I said we need a lot of data, But I think this class has been, and through your own research, you'll find that we're in the very early stages of discovering the power of deep learning.

For example, as recently, like Jan LeCun said, that it seems that the deeper the network, the better the results in a lot of really important cases, even though the data is not increasing.

So why does a deeper network give better results?

This is a mysterious thing we don't understand.

There's these hundreds of millions of parameters and from them is emerging some kind of structure, some kind of representation of the knowledge that we're giving it.

1 of my favorite examples of this emergent concept is Conway's Game of Life.

For those of you who know what this is, will probably criticize me for it being as cheesy as a stairway slide, but I think it's actually such a simple and brilliant example of how, like a neuron in a neural network is a really simple computational unit and then incredible power emerges when you just combine a lot of them in a network.

In the same way, this is called a cellular automata.

And every single cell is operating under a simple rule.

You can think of it as a cell living and dying.

It's filled in black when it's alive and white when it's dead.

And when it has 2 or 3, if it's alive and has 2 or 3 neighbors, it survives to the next time slot.

And if it has exactly 3 neighbors, it's dead.

All it's doing is operating under this very local process.

It's, or in the way we're currently training neural networks, in this local gradient.

And what happens if you run this system operating under really local rules, when you get on the right, It's not, again, you have to go home, hopefully no drugs involved, but you have to open up your mind and see how amazing that is.

Because what happens is, It's a local computational unit that knows very little about the world but somehow really complex patterns emerge and we don't understand why.

In fact, under different rules, incredible patterns emerge and it feels like it's living creatures like communicating.

Like when you just watch it, not these examples.

But even these examples, this complex geometric patterns that emerge, It's incredible.

We don't understand why and we need to in order to see how these networks will be able to reason.

I encourage you to Read the deep learning book.

It's available online, deeplearningbook.org.

As I mentioned to a few people, you should, well first there's a ton of amazing papers every day coming out on archive.

I'll put these links up, but there's a lot of good collections of strong paper, list of papers.

There is the literally awesome list, the awesome deep learning papers on GitHub.

It's calling itself awesome but it happens to be awesome.

And there is a lot of blogs that are just amazing.

That's how I recommend you learn machine learning, is on blogs.

And if you're interested in the application of deep learning in the automotive space, you can come do research in our group.

See all Lex Fridman transcripts on Youtube

MIT 6.S094: Deep Learning for Human-Centered Semi-Autonomous Vehicles