More details (paper and code snippets) for this work can be found here: http://lexfridman.com/carsync

This is a demo for the paper "Automated Synchronization of Driving Data Using Vibration and Steering Events". It provides a simple way to synchronize driving data to ~13 ms precision automatically (meaning: no manual checking/tuning by a human necessary).

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- Show your support: https://www.patreon.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

This is a quick demo of how car vibration and steering events can be used to synchronize driving data.

The video itself is a visualization of the data streams we're working with.

The audio you're hearing in the background besides my voice is from a shotgun microphone placed behind the rear right tire.

The middle column has 3 images each from a different webcam.

The dashboard video has an overlaid steering wheel icon that is visualizing the position of the steering wheel as supported by the CAN network.

The top left image shows the dense optical flow in the video of the forward roadway.

The bottom left just shows our location on a map.

We're in beautiful Cambridge, Massachusetts.

And the rest are plots showing the tenth-second window around the current measurement of various sensors.

On the left are the horizontal optical flow in the front video and the steering wheel position.

On the right are the audio energy from the shotgun microphone, the Y component of the optical flow from 3 webcams, and finally the Z axis of the accelerometer.

What we would like to do is to synchronize all of these sensors, either online or offline as a post-processing step.

We do this by first synchronizing the video of the forward roadway with a CAN network by looking at steering events.

When you make a turn like the 1 coming up here, the horizontal optical flow will be negative if it's a right turn and positive if it's a left turn.

Coming up here is a left turn and you will see in the top left image, the dense optical flow will light up all the same color.

It will be a positive value since the left turn.

We can then determine the optimal shift for the synchronization between the steering wheel and the forward video by computing the cross-correlation function, the maximum value for the cross-correlation function, to determine the shift.

And in the same way, we synchronize the rest of the sensors with the video of the forward roadway using vibration events.

On the right are 5 plots showing the audio energy, the y-component of the optical flow for the 3 webcams, and the z-axis of the accelerometer, each capturing the vibration of the car caused by the road.

A few examples are coming up shortly here.

So steering and vibration gives us a signal that we can use for passive synchronization.

The result is a synchronized dataset, which is important both for the analysis of driver behavior and for the design of ADAS systems that use decision fusion to make real-time prediction based on multiple sensor streams.

The paper along with the sample dataset and source code are available in the description.


Speaker 0: This is a quick demo of how car vibration and steering events can be used to synchronize driving data. The video itself is a visualization of the data streams we're working with. The audio you're hearing in the background besides my voice is from a shotgun microphone placed behind the rear right tire. The middle column has 3 images each from a different webcam. Front, dashboard and face.

The dashboard video has an overlaid steering wheel icon that is visualizing the position of the steering wheel as supported by the CAN network. The top left image shows the dense optical flow in the video of the forward roadway. The bottom left just shows our location on a map. We're in beautiful Cambridge, Massachusetts. And the rest are plots showing the tenth-second window around the current measurement of various sensors.

On the left are the horizontal optical flow in the front video and the steering wheel position. On the right are the audio energy from the shotgun microphone, the Y component of the optical flow from 3 webcams, and finally the Z axis of the accelerometer. What we would like to do is to synchronize all of these sensors, either online or offline as a post-processing step. We do this by first synchronizing the video of the forward roadway with a CAN network by looking at steering events. When you make a turn like the 1 coming up here, the horizontal optical flow will be negative if it's a right turn and positive if it's a left turn.

Coming up here is a left turn and you will see in the top left image, the dense optical flow will light up all the same color. It will be a positive value since the left turn. We can then determine the optimal shift for the synchronization between the steering wheel and the forward video by computing the cross-correlation function, the maximum value for the cross-correlation function, to determine the shift. And in the same way, we synchronize the rest of the sensors with the video of the forward roadway using vibration events. On the right are 5 plots showing the audio energy, the y-component of the optical flow for the 3 webcams, and the z-axis of the accelerometer, each capturing the vibration of the car caused by the road.

A few examples are coming up shortly here. So steering and vibration gives us a signal that we can use for passive synchronization. The result is a synchronized dataset, which is important both for the analysis of driver behavior and for the design of ADAS systems that use decision fusion to make real-time prediction based on multiple sensor streams. The paper along with the sample dataset and source code are available in the description.

See all Lex Fridman transcripts on Youtube

Automated Synchronization of Driving Data: Video, Audio, IMU, and Telemetry