Arguing machines concept applied to Tesla Autopilot "arguing" with an end-to-end neural network. Disagreement is used to predict challenging situations. It's a simple but powerful method for human supervision of black box AI systems.

OUTLINE:
0:00 Outside intro
1:00 Concept overview
2:46 In-car description of components
3:45 On-road demonstration

LINKS:
Paper: https://arxiv.org/pdf/1710.04459.pdf
Web: https://hcai.mit.edu/arguing-machines/
GitHub: (available soon)

CONNECT:
- AI Podcast: https://lexfridman.com/ai/
- Subscribe to this YouTube channel
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

Our group at MIT is studying semi-autonomous vehicles.

Now that includes both inward-facing sensors for driver state sensing and outward-facing sensors for scene perception and the control planning, motion planning task.

Now today we'll look at the second part of that, at the perception and the control of the vehicle.

On the dashboard of the Tesla there's a Jetson TX2 with a camera sitting on top of it.

We have a neural network end-to-end running on the Jetson that's detecting the forward roadway taking it a sequence of images and producing steering commands.

We also have here a Tesla that has a perception control system on it in the form of autopilot.

It's making decisions based on this single video stream producing steering commands.

And we'll look at 2 systems arguing today.

Autopilot arguing against a neural network.

In this concept, Tesla Autopilot is the primary AI system, and the N2N neural network is the secondary AI system, and the disagreement between the 2 is used to detect challenging situations and seek human driver supervision.

It is important to clarify that this is not a criticism of Autopilot.

Of the 2, it is by far the superior perception control system.

The question is whether the argument between the 2 systems can create transparency that leverage the human driver as a supervisor of challenging driving scenarios, scenarios that may have not otherwise been caught by Autopilot alone.

This is a general framework for supervision of black box AI systems that we hope can help save human lives.

In the paper accompanying this video, we show that we can predict driver-initiated disengagement of autopilot with a simple threshold on the disagreement of steering decisions.

We believe this is a very surprising and powerful result that hopefully may be useful for human supervision of any kind of AI system that operates in the real world and makes decisions where errors may result in loss of human life.

A quick note that we use the intensity of red color on the disagreement detected text as the visualization of disagreement magnitude.

In retrospect, this is not an effective visualization because visually it looks like the 2 systems are constantly disagreeing.

The intent of the on-road demo is to show successful real-time operation of the Argio machines framework.

The paper that goes along with this approach, on the other hand, is where we show the predictive power of this approach on large-scale naturalistic data.

Inside the car, we have a screen over the center stack and a Jetson TX2 with a camera on top of it.

The camera is feeding a video stream into the Jetson.

On the Jetson is a neural network that's predicting the steering command, taking in end-to-end the video stream from the forward roadway and as an output for the neural network giving a steering command.

The pink line is the steering suggested by the neural network.

Cyan line is the steering of the car of the Tesla that we're getting from the cam bus.

So when I move the steering wheel around we see that live in real time mapped on this graphic here showing in cyan the steering position of the car.

And up top is whenever the 2 disagree significantly, the disagreement detected red sign appears showing that there's a disagreement.

We're now driving on the highway with the Tesla being controlled by autopilot and the Jetson TX2 on the dashboard with the camera plugged in has a neural network running on it end to end.

And the input to the neural network is a sequence of images and the output is steering commands.

Now there's 2 perception control systems working here.

1 is autopilot, the other 1 is an end-to-end neural network.

Both, the steering commands from both, are being visualized on the center stack here.

In pink is the output from the neural network and cyan is the output from autopilot.

And whenever there is some disagreement or a lot of disagreement Up on top there's a disagreement detected text that becomes more intensely read the greater the disagreement.

At the bottom of the screen is the input to the neural network That is a sequence of images that are subtracted from each other, capturing the temporal dynamics of the scene.

There's 2 perception control systems, 2 AI systems, taking in the external world using a monocular camera and making a prediction, making steering commands to control the vehicle.

Now whenever those 2 systems disagree that's interesting for many reasons.

1, the disagreement is an indicator that from a visual perspective, from a perception perspective, the situation is challenging for those systems.

Therefore, you might want to bring the driver's attention to the situation so they take control back from the vehicle.

It's also interesting for validating systems.

So if you propose a new perception control system, you can imagine putting it into a car to go along with autopilot or with other similar systems to see how well that new system works with autopilot, when it disagrees, when it doesn't.

And the disagreement from the computer vision aspect is also really interesting for detecting edge cases.

So the challenging thing about driving or for building autonomous vehicles is that most of the driving is really boring.

So 1 of the ways to detect those interesting bits, the edge cases, is to look at the disagreement between these perception systems.

To look at cases when the 2 perception systems diverge and therefore they struggle with that situation.

Finally, when the driver is controlling and takes control of the vehicle, which I am doing now, and when my steering decisions, my turning of the steering wheel is such that the neural network disagrees, it perhaps means that I am either distracted or the situation is visually challenging, therefore I should pay extra attention.

So it makes sense for the system to warn you about that situation.

Now the interesting thing about Tesla and the Autopilot system is that if we instrument a lot of these vehicles, as we have, we've instrumented 20 Teslas as part of the MIT autonomous vehicle study and are collecting month after month, year after year now, data video in and video out.

We can use that data to train better systems, to train perception systems, control, motion planning, and the end-to-end network that we're showing today.

We have the large-scale data to train the learning-based perception and control algorithms.

Now an important thing to mention is that these systems were designed to work on the highway at highway speeds.

So the kind of disagreement it's trained to detect is disagreement between autopilot and the neural network in highway situations.

So the visual characteristics of lane markings deteriorating or construction zones and so on.

Now the details and if you're interested in more can be found in a paper titled arguing machines.


Speaker 0: Our group at MIT is studying semi-autonomous vehicles. Now that includes both inward-facing sensors for driver state sensing and outward-facing sensors for scene perception and the control planning, motion planning task. Now today we'll look at the second part of that, at the perception and the control of the vehicle. On the dashboard of the Tesla there's a Jetson TX2 with a camera sitting on top of it. We have a neural network end-to-end running on the Jetson that's detecting the forward roadway taking it a sequence of images and producing steering commands.

We also have here a Tesla that has a perception control system on it in the form of autopilot. It's using a monocular camera. This is the hardware version 1. It's making decisions based on this single video stream producing steering commands. And we'll look at 2 systems arguing today.

Autopilot arguing against a neural network. And we'll see what comes out.

Speaker 1: In this concept, Tesla Autopilot is the primary AI system, and the N2N neural network is the secondary AI system, and the disagreement between the 2 is used to detect challenging situations and seek human driver supervision. It is important to clarify that this is not a criticism of Autopilot. Of the 2, it is by far the superior perception control system. The question is whether the argument between the 2 systems can create transparency that leverage the human driver as a supervisor of challenging driving scenarios, scenarios that may have not otherwise been caught by Autopilot alone. This is a general framework for supervision of black box AI systems that we hope can help save human lives.

In the paper accompanying this video, we show that we can predict driver-initiated disengagement of autopilot with a simple threshold on the disagreement of steering decisions. We believe this is a very surprising and powerful result that hopefully may be useful for human supervision of any kind of AI system that operates in the real world and makes decisions where errors may result in loss of human life. A quick note that we use the intensity of red color on the disagreement detected text as the visualization of disagreement magnitude. In retrospect, this is not an effective visualization because visually it looks like the 2 systems are constantly disagreeing. They are not.

The intent of the on-road demo is to show successful real-time operation of the Argio machines framework. The paper that goes along with this approach, on the other hand, is where we show the predictive power of this approach on large-scale naturalistic data.

Speaker 2: Inside the car, we have a screen over the center stack and a Jetson TX2 with a camera on top of it. The camera is feeding a video stream into the Jetson. On the Jetson is a neural network that's predicting the steering command, taking in end-to-end the video stream from the forward roadway and as an output for the neural network giving a steering command. That's being shown as pink on

Speaker 3: this display. The pink line is the steering suggested by the neural network. Cyan line is the steering of the car of the Tesla that we're getting from the cam bus. So when I move the steering wheel around we see that live in real time mapped on this graphic here showing in cyan the steering position of the car. And up top is whenever the 2 disagree significantly, the disagreement detected red sign appears showing that there's a disagreement.

And I'll demonstrate that on road. We're now driving on the highway with the Tesla being controlled by autopilot and the Jetson TX2 on the dashboard with the camera plugged in has a neural network running on it end to end. And the input to the neural network is a sequence of images and the output is steering commands. Now there's 2 perception control systems working here. 1 is autopilot, the other 1 is an end-to-end neural network.

Both, the steering commands from both, are being visualized on the center stack here. In pink is the output from the neural network and cyan is the output from autopilot. And whenever there is some disagreement or a lot of disagreement Up on top there's a disagreement detected text that becomes more intensely read the greater the disagreement. At the bottom of the screen is the input to the neural network That is a sequence of images that are subtracted from each other, capturing the temporal dynamics of the scene. All right, so why is this interesting?

There's 2 perception control systems, 2 AI systems, taking in the external world using a monocular camera and making a prediction, making steering commands to control the vehicle. Now whenever those 2 systems disagree that's interesting for many reasons. 1, the disagreement is an indicator that from a visual perspective, from a perception perspective, the situation is challenging for those systems. Therefore, you might want to bring the driver's attention to the situation so they take control back from the vehicle. It's also interesting for validating systems.

So if you propose a new perception control system, you can imagine putting it into a car to go along with autopilot or with other similar systems to see how well that new system works with autopilot, when it disagrees, when it doesn't. And the disagreement from the computer vision aspect is also really interesting for detecting edge cases. So the challenging thing about driving or for building autonomous vehicles is that most of the driving is really boring. The interesting bits happen rarely. So 1 of the ways to detect those interesting bits, the edge cases, is to look at the disagreement between these perception systems.

To look at cases when the 2 perception systems diverge and therefore they struggle with that situation. Finally, when the driver is controlling and takes control of the vehicle, which I am doing now, and when my steering decisions, my turning of the steering wheel is such that the neural network disagrees, it perhaps means that I am either distracted or the situation is visually challenging, therefore I should pay extra attention. So it makes sense for the system to warn you about that situation. Now the interesting thing about Tesla and the Autopilot system is that if we instrument a lot of these vehicles, as we have, we've instrumented 20 Teslas as part of the MIT autonomous vehicle study and are collecting month after month, year after year now, data video in and video out. We can use that data to train better systems, to train perception systems, control, motion planning, and the end-to-end network that we're showing today.

We have the large-scale data to train the learning-based perception and control algorithms. Now an important thing to mention is that these systems were designed to work on the highway at highway speeds. So the kind of disagreement it's trained to detect is disagreement between autopilot and the neural network in highway situations. So the visual characteristics of lane markings deteriorating or construction zones and so on. Now the details and if you're interested in more can be found in a paper titled arguing machines.

See all Lex Fridman transcripts on Youtube

Arguing Machines: Tesla Autopilot vs Neural Network