More information: http://lexfridman.com/driverfrustration
Paper PDF: http://ijcai.org/Proceedings/16/Papers/195.pdf

We present a method for detecting driver frustration from both video and audio streams captured during the driver's interaction with an in-vehicle voice-based navigation system. The video is of the driver's face when the machine is speaking, and the audio is of the driver's voice when he or she is speaking. We analyze a dataset of 20 drivers that contains 596 audio epochs (audio clips, with duration from 1 sec to 15 sec) and 615 video epochs (video clips, with duration from 1 sec to 45 sec). The dataset is balanced across 2 age groups, 2 vehicle systems, and both genders. The model was subject-independently trained and tested using 4-fold cross-validation. We achieve an accuracy of 77.4% for detecting frustration from a single audio epoch and 81.2% for detecting frustration from a single video epoch. We then treat the video and audio epochs as a sequence of interactions and use decision fusion to characterize the trade-off between decision time and classification accuracy, which improved the prediction accuracy to 88.5% after 9 epochs.

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- Show your support: https://www.patreon.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

This video accompanies our paper presented at IJCAI, the International Joint Conference on Artificial Intelligence, where we propose a system for detecting driver frustration from the fusion of 2 data streams, first the audio of the driver's voice and second the video of the driver's face.

These are video snapshots of 2 drivers using the in-car voice-based navigation system.

Which 1 of them looks more frustrated with the interaction?

To help answer that question, let's take a look at an example interaction involving the driver on the right.

Our proposed approach uses the audio of the driver's voice when the quote-unquote human is speaking and the video of the driver's face when he's listening to the machine speak.

What you're seeing and hearing is the driver attempting to instruct the car's voice-based navigation system to navigate to 177 Massachusetts Ave, Cambridge, Massachusetts.

177 Massachusetts Ave, Cambridge, Massachusetts.

177 Massachusetts Avenue, Cambridge, Massachusetts.

On a scale of 1 to 10, with 1 being completely satisfied and 10 being completely frustrated, the smiling driver reported his frustration level with this interaction to be a 9.

We use self-reported level of frustration as the ground truth for the binary classification of satisfied versus frustrated.

When the driver is speaking, we extract the Geneva Minimalistic Acoustic Parameter Set, GMAPS, features from their voice, which measures basic physiological changes in voice production.

When the driver is listening, we extract 14 facial actions using the Aftek system from the video of the driver's face.

The classifier decisions are fused together to produce an accuracy of 88.5% on an on-road data set of 20 subjects.

There are 2 takeaways from this work that may go beyond just detecting driver frustration.

First, self-reported emotion state may be very different than 1 assigned by a group of external annotators, so we have to be careful when using such annotations as a ground truth for other effective computing experiments.

Second, detection of emotion may require considering not just facial actions or voice acoustics, but also context of the interaction, and the target of the effective communication.

For more information or to contact the authors, please visit the following website.


Speaker 0: This video accompanies our paper presented at IJCAI, the International Joint Conference on Artificial Intelligence, where we propose a system for detecting driver frustration from the fusion of 2 data streams, first the audio of the driver's voice and second the video of the driver's face. Let's ask an illustrative question. These are video snapshots of 2 drivers using the in-car voice-based navigation system. Which 1 of them looks more frustrated with the interaction? To help answer that question, let's take a look at an example interaction involving the driver on the right.

Our proposed approach uses the audio of the driver's voice when the quote-unquote human is speaking and the video of the driver's face when he's listening to the machine speak. What you're seeing and hearing is the driver attempting to instruct the car's voice-based navigation system to navigate to 177 Massachusetts Ave, Cambridge, Massachusetts.

Speaker 1: 177 Massachusetts Ave, Cambridge, Massachusetts.

Speaker 2: Man of the above.

Speaker 1: 177 Massachusetts Avenue, Cambridge, Massachusetts. I'm on call.

Speaker 3: So, Massachusetts, say what? So, Massachusetts, say what? You know, the election is coming up. Madam Delpho. Cambridge, Massachusetts.

Thank you. Now, the next station is Cambridge.

Speaker 2: Man of the above. I did not

Speaker 3: catch that. It was too much about airplanes.

Speaker 2: Man of the above.

Speaker 3: See, he has to reach an address.

Speaker 1: Cambridge, Massachusetts.

Speaker 0: So there's your answer. On a scale of 1 to 10, with 1 being completely satisfied and 10 being completely frustrated, the smiling driver reported his frustration level with this interaction to be a 9. We use self-reported level of frustration as the ground truth for the binary classification of satisfied versus frustrated. When the driver is speaking, we extract the Geneva Minimalistic Acoustic Parameter Set, GMAPS, features from their voice, which measures basic physiological changes in voice production. When the driver is listening, we extract 14 facial actions using the Aftek system from the video of the driver's face.

The classifier decisions are fused together to produce an accuracy of 88.5% on an on-road data set of 20 subjects. There are 2 takeaways from this work that may go beyond just detecting driver frustration. First, self-reported emotion state may be very different than 1 assigned by a group of external annotators, so we have to be careful when using such annotations as a ground truth for other effective computing experiments. Second, detection of emotion may require considering not just facial actions or voice acoustics, but also context of the interaction, and the target of the effective communication. For more information or to contact the authors, please visit the following website.

See all Lex Fridman transcripts on Youtube

Detecting Driver Frustration from Audio and Video (IJCAI 2016)