This is lecture 5 of course 6.S094: Deep Learning for Self-Driving Cars (2018 version). This class is free and open to everyone. It is an introduction to the practice of deep learning through the applied theme of building a self-driving car.

OUTLINE:
0:00 - Intro
6:53 - Human Imperfections
22:57 - Pedestrian Detection
28:57 - Body Pose Estimation
35:40 - Glance Classification
47:13 - Emotion Recognition
53:24 - Cognitive Load Estimation
1:00:54 - Human-Centered Vision for Autonomous Vehicles

INFO:
Slides: http://bit.ly/2HbGuzx
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Playlist: https://goo.gl/SLCb1y

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- Show your support: https://www.patreon.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

LINKS:
Playlist: https://goo.gl/SLCb1y
Lecture 1: Deep Learning - https://youtu.be/-6INDaLcuJY
Lecture 2: Self-Driving Cars - https://youtu.be/_OCjqIgxwHw
Lecture 3: Deep Reinforcement Learning - https://youtu.be/MQ6pP65o7OM
Lecture 4: Computer Vision - https://youtu.be/CLOAswsxudo
Lecture 5: Deep Learning for Human Sensing - https://youtu.be/Z2GfE8pLyxc

Guest talk: Sacha Arnoud, Waymo - https://youtu.be/LSX3qdy0dFg
Guest talk: Emilio Frazolli, nuTonomy - https://youtu.be/dWSbItd0HEA
Guest talk: Sterling Anderson, Aurora - https://youtu.be/HKBhP9JISF0

2017:
Guest talk: Sertac Karaman, MIT - https://youtu.be/0fLSf3NO0-s
Guest talk: Chris Gerdes, Stanford - https://youtu.be/LDprUza7yT4

Today we will talk about how to apply the methods of deep learning to understanding the sense in the human being.

The focus will be on computer vision, the visual aspects of a human being.

Of course, we humans express ourselves visually, but also through audio, voice and through text.

We're just going to focus on computer vision.

How we can use computer vision to extract useful actionable information from video, images, video of human beings.

So, what are the requirements for successfully applying deep learning methods in the real world.

So when we talk about human sensing, we're not talking about basic face recognition of celebrity images.

We're talking about using computer vision, deep learning methods to create systems that operate in the real world.

And in order for them to operate in the real world, there are several things.

They sound simple, some are much harder than they sound.

First, and the most important, here from most to more to less critical, ordered, is data.

We need a lot of real-world data to form the data set on which these supervised learning methods can be trained.

I'll say this over and over throughout the day today, data is everything.

That means data collection is the hardest part and the most important part.

We'll talk about how that data collection is carried out here in our group at MIT.

All the different ways we capture human beings in the driving context, in the road user context, pedestrians, cyclists.

But the data, it starts and ends at data.

Okay, then once you have the data, Okay, data isn't everything, I lied.

Because you have to actually annotate it.

lidar, all the types of sensors we'll talk about to capture real-world

You have to reduce that into meaningful representative cases of what happens in that real world.

In driving, 99% of the time driving looks the same.

It's the 1%, the interesting cases that we're interested in.

And what we want is algorithms to train learning algorithms on those 1%.

So we have to collect the 100%, we have to collect all the data, and then figure out an automated, semi-automated ways to find the pieces of that data that could be used to train neural networks and that are representative of the general things, kinds of things that happen in this world.

Annotation isn't just about drawing bounding boxes on images of cats Annotation tooling is key to unlocking real-world performance Systems that successfully solve

some problem Accomplish some goal in real-world data That means designing annotation tools for a particular task Annotation tools that are used for glance classification for determining where drivers are looking is very different than annotation tools used for body pose estimation.

It's very different than the tooling that we use for psych fuse investing thousands of dollars for the competition for this class to annotate fully scene segmentation where every pixel is colored.

There needs to be tooling for each 1 of those elements and they're key.

It's how do we leverage human computation, the human brain to most effectively label images such that we can train neural networks on them.

In order to train these networks, in order to parse the data we collect, and we'll talk about, we have now over 5 billion images of data, of driving data.

In order to parse that, you can't do it on a single machine.

You have to do large-scale distributed compute and large-scale distributed storage.

And finally, the stuff that's the most exciting, That people, that this class and many classes and much of the literature is focused on is the algorithms.

The deep learning algorithms, the machine learning algorithms, the algorithms that learn from data.

Of course that's really exciting and important.

But what we find time and time again in real world systems is that as long as these algorithms learn from data so as long as it's deep learning the data is what's much more important Of course it's nice for the algorithms to be calibration free, meaning they learn to calibrate, self calibrate.

We don't need to have the sensors in an exact same position every time.

The robustness of the system is then generalizable across multiple vehicles, multiple scenarios.

And 1 of the key things that comes up time and time again and we'll mention today is a lot of the algorithms developed in deep learning are really focused for computer vision are focused on single images.

Now the real world happens in both space and time and we have to have algorithms that both capture the visual characteristics but also look at the sequence of images, sequence of those visual characteristics that form the temporal dynamics, the physics of this world.

So it's nice when those algorithms are able to capture the physics of the scene.

The big takeaway I would like if you leave with anything today, unfortunately

it's that the painful, boring stuff of collecting data, of cleaning that data, of annotating that data, in order to create successful systems is much more important than good algorithms or great algorithms.

It's important to have good algorithms as long as you have neural networks that learn from that data.

Okay, so today I'll talk, I'd like to talk about human imperfections

The pedestrian body pose, glance, emotion, cognitive load estimation that we can use to help those humans as they operate in a driving context.

with the idea, the vision, that fully autonomous vehicles, as some of our guest speakers have spoke about, and Sterling Ennis will speak about tomorrow is really far away.

That the humans will be an integral part of the operating, cooperating with AI systems.

And I will continue on that line of thought to try to motivate why we need to continuously approach the autonomous vehicle, the self-driving car paradigm in a human-centered way.

Okay, first before we talk about human imperfections, let's just pause and acknowledge that humans are amazing.

We're actually really good at a lot of things.

That's sometimes sort of fun to talk about how much, how terrible of drivers we are, how distracted we are, how irrational we are.

But we're actually really damn good at driving.

Here's a video of a state-of-the-art soccer player, Messi, the best soccer player in the world obviously.

And the state-of-the-art robot on the right, same thing Well, it's not playing but I assure you the American Ninja Warrior Casey is far superior to the DARPA humanoid robotics systems shown on the right Okay, so continuing on the line of thought to challenge us here that humans are amazing is, you know, there's a record high in 2016 in the United States.

More than 40,000 people died in car crashes in the United States.

But that's in 3.2 trillion miles traveled.

So that's 1 fatality per 80 million miles.

That's 1 in 625 chance of dying in a car crash in your lifetime.

Interesting side fact, for anyone in the United States, folks who live in Massachusetts are the least likely to die in

So for everyone that thinks of Boston Drives as terrible, maybe that adds some perspective.

Here's a visualization of Waze data across a period of a day showing you the rich blood of the city, that the traffic flow of the city, the people getting from A to B on a mass scale And doing it, surviving, doing it

of distraction with a smartphone, the eating, the secondary tasks of talking to other passengers, grooming, reading, using navigation system, yes, sometimes watching video and manually adjusting or adjusting the radio.

And 3,000 people were killed and 400,000 were injured in motor vehicle crashes involving distraction in 2014.

Distraction is a very serious issue for safety.

Texting Every day more and more people text.

Smartphones are proliferating our society.

170 billion text messages are sent in the United States every month.

Eyes off road for 5 seconds is the average time your eyes are off the road while texting.

If you're traveling 55 miles an hour in that 5 seconds, that's enough time to cover the length of a football field.

So you're blindfolded, you're not looking at the road.

In 5 seconds, the average time of texting, you're covering the entire football field.

So many things can happen in that moment of time.

31% of traffic fatalities involve a drunk driver.

23% of nighttime drivers tested positive for a legal prescription or over-the-counter medication.

Distracted driving, as I said, is a huge safety risk.

Drowsy driving, people driving tired, nearly 3% of all traffic fatalities involve a drowsy driver.

If you are uncomfortable with videos that involve risk, I urge you to look away.

These are videos collected by AAA of teenagers, a very large scale naturalistic driving data set, and is capturing clips of teenagers being distracted on their smartphone.

Once you take it in, the problem we're against.

So in that context of human imperfections,

we have to ask ourselves, is the human-centered approach to autonomy in systems, autonomous vehicles that are using artificial intelligence to aid the driving task, do we want to go, as I mentioned a couple of lectures ago, the human-centered way or the full autonomy way?

The tempting path is towards full autonomy.

Where we remove this imperfect, flawed human from the picture altogether.

And focus on the robotics problem of perception, control, and planning, and driving policy.

Or do we work together, human and machine, to improve the safety, to alleviate distraction, to bring driver attention back to the road and use artificial intelligence to increase safety through collaboration human robot interaction versus removing the human completely from the picture As I've mentioned As Sterling will certainly talk about tomorrow and rightfully so and yesterday or on Tuesday Emilio has talked about the L4 way is grounded in literature.

It's grounded in common sense in some sense.

You can count on the fact that humans, the natural flaws of human beings to overtrust, to misbehave, to be irrational about their risk estimates will result in improper use of the technology.

And that leads to what I've showed before, the public perception of what drivers do in semi-autonomous vehicles.

The moment the system works well, they begin to over trust.

They begin to do stuff they're not supposed to be doing in the car taking it for granted A recent video that somebody posted this is a common sort of more practical concern that people have is well, the traditional ways to ensure the physical engagement of the drivers by saying they should touch the wheel, the steering wheel every once in a while.

And of course there's ways to bypass the need to touch the steering wheel.

Some people hang objects like a can off of the steering wheel.

In this case, brilliantly, I have to say, they shove an orange into the wheel to make the touch sensor fire and therefore be able to take their hands off the autopilot.

And that kind of idea makes us believe that there's no way that humans will find a way to misuse this technology.

that's not giving the technology enough credit.

Artificial intelligence systems, if they're able to perceive the human being, are also able to work with the human being.

And that's what I'd like to talk about today.

Teaching cars to perceive the human being.

Data is everything in these real-world systems.

With the MIT naturalistic driving data set of 25 vehicles of which 21 are equipped with Tesla autopilot, We instrument them.

This is what we do with the data collection.

We'll see the cameras on the face, capturing high definition video of the face.

That's where we get the glance classification, the emotion recognition, cognitive load, everything coming from the face.

Then we have another camera, fish eye, that's looking at the body of the driver.

And from that comes the body pose estimation, hands-on wheel, activity recognition.

And then 1 video looking out for the full scene segmentation for all the scene perception tasks.

And everything is being recorded, synchronized together with GPS, with audio, with all the camcorder from the car on a single device.

Synchronization of this data is critical.

We have thousands like it, traveling hundreds of miles, sometimes hundreds of miles under automated control, an autopilot.

And from this data, we can both gain understanding of what people do, which is really important to understand how autonomy, successful autonomy can be deployed in the real world and to design algorithms for training, for training the deep learning, the deep neural networks in order to perform the perception task better.

25 vehicles, 21 Teslas, Model S, Model X and now Model 3.

Every single day we have thousands of miles in the Boston, Massachusetts area, driving around, all of that video being recorded.

There's several ways to look at autonomy.

We can create a perfectly safe system, we can create it right now, we've had it forever before even cars.

A car that never moves is a perfectly safe system, well not perfectly but almost.

And but it doesn't provide a service that's valuable.

It doesn't provide an enjoyable driving experience.

So okay, what about slow moving vehicles?

The reality is with these Tesla vehicles and L2 systems doing automated driving.

People are driving 33% of miles using Tesla Autopilot.

That means that people are getting value from it.

A large fraction of their driving is done

The glance classification algorithm we'll talk about today is used as 1 example that we use to understand what's in this data.

Shown with the bar graphs there in the red and the blue.

Red is during manual driving, blue is during autopilot driving and we look at glance classification, regions of where drivers are looking, on-road and off-road and if that distribution changes with automated driving or manual driving.

And with these glance classification methods, we can determine that there's not much difference, at least until you dig into the details, which we haven't done.

In the aggregate, there's not a significant difference.

That means people are getting value and enjoying using these technologies,

but yet they're staying attentive, or at least not attentive, but physically engaged.

When your eyes are on the road, you might not

be attentive, but you're at the very least physically, your body's positioned in such a way, your head is looking at the forward roadway, that you're physically in position to be alert and to take in the forward roadway.

So they're using it and they don't over trust it.

And that's I think the sweet spot that human robot interaction needs to achieve is the human gaining through experience, through exploration, through trial and error, exploring and understanding the limitations of the system to a degree that overtrust can occur.

That seems to be happening in this system.

And using the computer vision methods I'll talk about we can continue to explore how that can be achieved in other systems when the fraction of automated driving increases from 30% to 40% to 50% and so on It's all about the data and I'll harp on this again.

I will mention, of course, it's the same convolutional neural networks.

It's the same networks that take in raw pixels and extract features of interest.

It's 3D convolutional neural networks that take in a sequences of images and extract the temporal dynamics along with the visual characteristics of the individual images It's RNNs, LSTMs that use the convolutional neural networks to extract features and over time look at the dynamics of the images These are pretty basic architectures The same kind of deep neural network architectures But they rely fundamentally and deeply on the data On real-world data So let's start where perhaps on the human sensing side it all began, which is pedestrian detection.

To put it in context, pedestrian detection here, shown from left to right, on the left is green, showing the easier human sensing tasks.

Tasks of sensing some aspect of the human being.

But SGNU detection, which is detecting the full body of a human being in an image or video, is 1 of the easier computer vision tasks.

And on the right, in the red, micro saccades, these are tremors of the eye or measuring the pupil diameter or measuring the cognitive load or the fine blink dynamics of the eye, the velocity of the blink, micro glances and eye pose are much harder problems.

So you think body pose estimation, pedestrian detection, face classification detection, recognition, head pose estimation, all those are easier tasks.

Anything that starts getting smaller, looking at the eye and everything that start getting fine-grained there's much more difficult.

So we start at the easiest, pedestrian detection.

And as the usual challenges of all of computer vision we've talked about, as the various styles of appearance, so the inter-class variation, the different possible articulations of our bodies, superseded only perhaps by cats.

But us humans are pretty flexible as well.

The presence of occlusion from the accessories that we wear to occluding self-occlusion and occluding each other.

Crowded scenes have a lot of humans in them and they occlude each other and therefore being able to disambiguate to figure out each individual pedestrian is a very challenging problem.

Whether that was Harkascades, Hogg or CNN, the, through the decades,

the sliding window approach was used because the pedestrians can be small in an image or big so there's the problem of scale so you use a sliding window to detect where that pedestrian is.

You have a classifier that's given a single image says is this a pedestrian or not.

You take that classifier, you slide it across the image to find where all the pedestrians is seen are.

So you can use non neural network methods or you can use convolution neural networks for that classifier.

Then came along R-CNN, Fast R-CNN, Fast R-CNN.

These are networks that as opposed to doing a complete sliding window approach, are much more intelligent, clever about generating the candidates to consider.

So as opposed to considering every possible position of a window, different scales of the window, they generate a small subset of candidates that are more likely.

And finally using a CNN classified for those candidates whether there's a pedestrian or not, whether there's an object of interest or not, a face or not.

And using non-maximal suppression because there's overlapping bounding boxes to figure out what is the most likely bounding box around this pedestrian, around this object.

And there's a lot of variance now with Mask RCNN, really the state-of-the-art localization network.

Mask also adds to this, on top of the bounding box also performs segmentation.

There's VoxelNet which does three-dimensional and LiDAR data, uses localization and point clouds.

So it's not just using a 2D images but in 3D.

But it's all kind of grounded in the RCNN framework.

So we have large-scale data collection going on here in Cambridge.

If you've seen cameras or LiDAR at various intersections throughout MIT, we're part of that.

So for example, here's 1 of the intersections.

We're collecting about 10 hours a day instrumenting it with various sensors I'll mention but we see about 12,000 pedestrians a day

across that particular intersection using 4k cameras using

stereo vision cameras 360 now the insta 360 which is an 8k 360 camera GoPro, LiDAR various sizes, the 64 channel or the 16.

This is from the LiDAR data of the same intersection.

This is for the 4K camcorders pointing at a different intersection and the different, then capturing the entire 360 view with the vehicles approaching and the pedestrians making crossing decisions.

This is understanding the negotiation that pedestrians, the nonverbal negotiation that pedestrians perform in choosing to cross or not, especially when they're jaywalking, and everybody jaywalks.

Especially if you're familiar with this particular intersection, there's more jaywalkers than non-jaywalkers.

And so we record everything about the driver and everything about the pedestrians.

And again, RCNN, this is where it comes in, is you do bounding box detection of the pedestrians, here the vehicles as well.

And allows you to convert this raw data into hours of pedestrian crossing decisions And begin to interpret it That's pedestrian detection, bounding box For body pose estimation Is the more difficult task Body pose estimation is also finding the joints, the hands, the elbows, the shoulders, the hips, knees, feet, the landmark points in the image, X, Y position, that mark those joints.

It's important to determine the vertical position or the alignment of the driver.

The seat belts and the airbag testing is always performed and the seatbelt testing is performed with the dummy considering the frontal position in a standard dummy position.

With the greater and greater degrees of automation comes more capability and flexibility for the driver to get misaligned from the standard quote-unquote dummy position.

And so body pose or at least upper body pose estimation allows you to determine how often these drivers get out of

line from the standard position, general movement.

And then you can look at it at hands-on wheel Smartphone, smartphone detection Activity

And help add context to glance estimation That we will talk about So some of the more traditional methods were sequential It's detecting first the head and then stepping detecting the shoulders, the elbows, the hands The deep pose holistic view which has been a very powerful successful way for multi-person pose estimation,

is performing a regression of detecting body parts from the entire image.

It's not sequentially stitching bodies together it's detecting the left elbow, the right elbow, the hands individually.

It's performing that detection and then stitching everything together afterwards.

Allowing you to deal with the crazy deformations of the body that happen, the occlusions and so on, because you don't need all of the joints to be visible.

And with this cascade of pose regressors meaning these are convolutional neural networks that take in a raw image and produce an XY position of their estimate of each individual joint.

Input is an image, output is an estimate of a joint

of an elbow, shoulder, whatever, 1 of several landmarks.

And then you can build on top of that, every estimation zooms in on that particular area and

performs a finer and finer grain estimation of the exact position of

So through this process we can do part detection in

multi-person scene that contain multiple people.

So we can detect the head, the neck, here the hands, the elbows shown in the various images on the right that don't have an understanding of who the head, the elbows, the hands belong to.

It's just performing a detection without trying to do individual person detection first.

And then, finally, connecting, or not finally, but next step is connecting with part affinity fields is connecting those parts together.

See all Lex Fridman transcripts on Youtube

MIT 6.S094: Deep Learning for Human Sensing