Open source code and paper will be out soon on https://lex.mit.edu
We ran our new glance classification code on the driver-facing video from the tragic Uber crash in Arizona. Open source code and arXiv paper will be out soon. I hope it can help, in however small a way, make the roads safer for people inside and outside both manually-controlled and autonomous vehicles.

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

As we design and build autonomous vehicle systems here at MIT, and as we study real world, natural realistic driving, we begin to understand that it may be 1234 decades before we're able to build autonomous vehicles that can be fully autonomous without integrating the human being.

Before then we have to integrate the human being whether as a driver, as a safety driver, or a teleoperator.

For that at the very beginning, at the very least, the autonomous vehicle needs to be able to perceive the state of the driver.

As we look at the recent case, the tragic case of the pedestrian fatality in Arizona, we can see that the perception of what the driver is doing, whether they're looking on the road or not, is of critical importance for this environment.

So we'd like to show to you the Glance Region Classification algorithm running on the video of the driver's face.

And also in the near future, we're going to make the code open source, available to everybody, together with an archive submission, in hopes that companies And universities testing autonomous vehicles can integrate it into their testing procedures and make sure they're doing everything they can to make the testing process as safe as possible.

In the middle is the detection of the face region that is then fed to the glance classification algorithm as a sequence of images to the neural network and then the neural network produces an output, a prediction of where the driver is looking.

That's shown to the right, the current glance region that's predicted, whether it's road, left, right, rearview mirror, center stack, instrument cluster.

Then off-road glance duration, whenever the driver is looking off-road, that number increases.

We run this glance classification algorithm on this particular video to show that it is a powerful signal for an autonomous vehicle to have, especially in the case when a safety driver is tasked with monitoring the safe operation of the autonomous vehicle.


Speaker 0: As we design and build autonomous vehicle systems here at MIT, and as we study real world, natural realistic driving, we begin to understand that it may be 1234 decades before we're able to build autonomous vehicles that can be fully autonomous without integrating the human being. Before then we have to integrate the human being whether as a driver, as a safety driver, or a teleoperator. For that at the very beginning, at the very least, the autonomous vehicle needs to be able to perceive the state of the driver. As we look at the recent case, the tragic case of the pedestrian fatality in Arizona, we can see that the perception of what the driver is doing, whether they're looking on the road or not, is of critical importance for this environment. So we'd like to show to you the Glance Region Classification algorithm running on the video of the driver's face.

And also in the near future, we're going to make the code open source, available to everybody, together with an archive submission, in hopes that companies And universities testing autonomous vehicles can integrate it into their testing procedures and make sure they're doing everything they can to make the testing process as safe as possible. Let's take a look. To the left is the original video. In the middle is the detection of the face region that is then fed to the glance classification algorithm as a sequence of images to the neural network and then the neural network produces an output, a prediction of where the driver is looking. That's shown to the right, the current glance region that's predicted, whether it's road, left, right, rearview mirror, center stack, instrument cluster.

Then off-road glance duration, whenever the driver is looking off-road, that number increases. We run this glance classification algorithm on this particular video to show that it is a powerful signal for an autonomous vehicle to have, especially in the case when a safety driver is tasked with monitoring the safe operation of the autonomous vehicle. You

See all Lex Fridman transcripts on Youtube

Uber Self-Driving Car Glance Classification