Introducing our shared autonomy research vehicle with a demo of voice-based transfer of control based on whether the driver is paying attention to the road and on risk factors detected in the external environment. Our work aims to reimagine the self-driving car as a shared autonomy system built around the human being. See paper describing key concepts below. Thank you to Veoneer for their support of our research.

OUTLINE:
0:00 Intro
1:09 Description of components
3:09 Tweeting
5:56 Pedestrian
6:53 Outro

LINKS:
Paper: https://arxiv.org/abs/1810.01835
Website: https://hcai.mit.edu/hcav/
Semi-Autonomous Tweet: https://bit.ly/2NTL7Bn

CONNECT:
- Subscribe to this YouTube channel
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com
- AI Podcast: https://lexfridman.com/ai/

This is the Human-Centered Autonomous Vehicle.

1 of the main ideas underlying our work is that solving the task of autonomous driving is more complicated and more fascinating than the strictly robotics challenges of localization, mapping, perception, control, and planning.

You also have to enable the vehicle to perceive, predict, communicate, and collaborate with human beings.

The humans inside the car, like the driver and the passengers, and the humans outside the car, like the pedestrians, cyclists, the drivers of other vehicles, and even teleoperators.

The studies, the code, the data, and the demos we release all consider autonomous driving in this kind of human-centered way, where the control is transferred from human to machine and back to human based on the state of the external driving environment and the state of the driver.

What we'd like to demonstrate today is the basics voice-based transfer of control from human to machine based on whether the driver is paying attention to the road or not.

Inside we have 2 cameras on the driver, 1 on the driver's face, 1 on the driver's body.

We have 2 cameras look at the external roadway and we have a few other cameras for filming purposes.

There's a center stack display showing who's in control of the vehicle, human or machine.

So currently the human is in control of the vehicle.

On the center stack display, it shows the gear as drive.

The perception control and driver state sensing algorithms you see today are running in real time but the visualizations you're seeing in video are done in offline post-processing.

Our perception system today is vision-based using 2 neural networks.

1 is doing road segmentation, the other is doing object detection of vehicles, cyclists, pedestrians, traffic signs, traffic lights.

The acceleration braking and steering of the car is performed by PID controllers.

The driver state sensing that we're showing today is glance region classification, and that's performed using 3D convolutional neural networks.

High level planning decisions to transfer control or to stop the vehicle are performed by a decision fusion algorithm that combines risk factors in the external environment and driver state, whether the driver is paying attention to the road or not.

Safety for us is the number 1 priority, always.

The vehicles and pedestrians here today are all part of our team, all part of the demonstration.

There's another safety driver in the car that can stop the vehicle at any moment by pressing a single button.

Okay let's engage in a distracting activity.

This tweet while driving in the MIT semi-autonomous vehicle semi-autonomous vehicle on a A test track.

I am taking control of steering and braking.

The car is now in control as the center stack display shows.

That I'm not paying attention and has taken control after asking me nicely for it.

Very well might be the first tweet ever sent from an autonomous vehicle while it's driving itself.

Lex, pedestrian is blocking our lane of travel.

That was a demo of the basics, perception, motion planning, driver state sensing, transfer control and tweeting.

And we have a lot more to explore together.

Our team is working on various aspects of human-centered artificial intelligence toward our mission to save lives through effective human-robot collaboration.


Speaker 0: This is the Human-Centered Autonomous Vehicle. 1 of the main ideas underlying our work is that solving the task of autonomous driving is more complicated and more fascinating than the strictly robotics challenges of localization, mapping, perception, control, and planning. You also have to enable the vehicle to perceive, predict, communicate, and collaborate with human beings. The humans inside the car, like the driver and the passengers, and the humans outside the car, like the pedestrians, cyclists, the drivers of other vehicles, and even teleoperators. The studies, the code, the data, and the demos we release all consider autonomous driving in this kind of human-centered way, where the control is transferred from human to machine and back to human based on the state of the external driving environment and the state of the driver.

What we'd like to demonstrate today is the basics voice-based transfer of control from human to machine based on whether the driver is paying attention to the road or not.

Speaker 1: Inside we have 2 cameras on the driver, 1 on the driver's face, 1 on the driver's body. We have 2 cameras look at the external roadway and we have a few other cameras for filming purposes. There's a center stack display showing who's in control of the vehicle, human or machine. So currently the human is in control of the vehicle. Let's drive.

Let's put the car in drive. On the center stack display, it shows the gear as drive. The perception control and driver state sensing algorithms you see today are running in real time but the visualizations you're seeing in video are done in offline post-processing. Our perception system today is vision-based using 2 neural networks. 1 is doing road segmentation, the other is doing object detection of vehicles, cyclists, pedestrians, traffic signs, traffic lights.

The acceleration braking and steering of the car is performed by PID controllers. The driver state sensing that we're showing today is glance region classification, and that's performed using 3D convolutional neural networks. High level planning decisions to transfer control or to stop the vehicle are performed by a decision fusion algorithm that combines risk factors in the external environment and driver state, whether the driver is paying attention to the road or not. Safety for us is the number 1 priority, always. We are on a test track.

The vehicles and pedestrians here today are all part of our team, all part of the demonstration. There's another safety driver in the car that can stop the vehicle at any moment by pressing a single button. Okay let's engage in a distracting activity. Twitter and let's send a tweet. This tweet while driving in the MIT semi-autonomous vehicle semi-autonomous vehicle on a A test track.

Speaker 2: Lex, you appear distracted. Would you like me to take over?

Speaker 1: Yes, please.

Speaker 2: Great. I am taking control of steering and braking.

Speaker 1: The car is now in control as the center stack display shows. So I will continue with the tweet. That I'm not paying attention and has taken control after asking me nicely for it. Video out tomorrow. Okay here goes nothing.

It's posted. Very well might be the first tweet ever sent from an autonomous vehicle while it's driving itself.

Speaker 2: Elevated driving risk detected. Elevated driving risk detected. I am stopping for a pedestrian. Lex, pedestrian is blocking our lane of travel. Should I honk?

Speaker 1: No, please shift gear to park.

Speaker 2: Great, we are now in park.

Speaker 0: That was a demo of the basics, perception, motion planning, driver state sensing, transfer control and tweeting. And we have a lot more to explore together. Our team is working on various aspects of human-centered artificial intelligence toward our mission to save lives through effective human-robot collaboration.

See all Lex Fridman transcripts on Youtube

MIT Human-Centered Autonomous Vehicle