This is a talk by Sacha Arnoud for course 6.S094: Deep Learning for Self-Driving Cars (2018 version). Sacha is the Director of Engineering at Waymo and his talk is titled "The rise of machine learning in self-driving cars." This class is free and open to everyone. It is an introduction to the practice of deep learning through the applied theme of building a self-driving car.

INFO:
Course website: https://selfdrivingcars.mit.edu
Contact: deepcars@mit.edu

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- Show your support: https://www.patreon.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

LINKS:
Playlist: https://goo.gl/SLCb1y
Lecture 1: Deep Learning - https://youtu.be/-6INDaLcuJY
Lecture 2: Self-Driving Cars - https://youtu.be/_OCjqIgxwHw
Lecture 3: Deep Reinforcement Learning - https://youtu.be/MQ6pP65o7OM
Lecture 4: Computer Vision - https://youtu.be/CLOAswsxudo
Lecture 5: Deep Learning for Human Sensing - https://youtu.be/Z2GfE8pLyxc

Guest talk: Sacha Arnoud, Waymo - https://youtu.be/LSX3qdy0dFg
Guest talk: Emilio Frazolli, nuTonomy - https://youtu.be/dWSbItd0HEA
Guest talk: Sterling Anderson, Aurora - https://youtu.be/HKBhP9JISF0

2017:
Guest talk: Sertac Karaman, MIT - https://youtu.be/0fLSf3NO0-s
Guest talk: Chris Gerdes, Stanford - https://youtu.be/LDprUza7yT4

Today we have the Director of Engineering, Head of Perception at Waymo, a company that's recently driven over 4 million miles autonomously, and in so doing inspired the world in what artificial intelligence and good engineering can do.

So please give a warm welcome to Sacha Arnoux.

Thanks a lot for giving me the opportunity to be able to come and share my passion with self-driving cars and be able to share with you all the great work we've been doing at Waymo over the last 10 years, and give you more details on the recent milestones we've reached.

So as you see, we'll cover a lot of different topics, some more technical, some more about context.

But when I over the content, I have 3 main objectives that I'd like to convey today.

So keep that in mind as we go through the presentation.

My first 1 is to give you some background around the self-driving space, and what's happening there, and what it takes to build self-driving cars, but also give you some behind-the-scenes views and tidbits on the history of machine learning, deep learning, and how it all came together within the big alphabet family from Google to Waymo.

Another piece, obviously, another objective I have is to give you some technical meat around the techniques that are working today on our self-driving cars.

So I think during the class you hear a lot, you've heard a lot about different deep learning techniques, models, architectures, algorithms, and I'll try to put that in a current hole so that you can see how those pieces fit together to build the system we have today.

And last but not least, I think as Lex mentioned, it takes a lot more, actually, than algorithms to build a sophisticated system such as our self-driving cars.

And fundamentally, it takes a full industrial project to make that happen.

And I'll try to give you some color, which hopefully is a little different from what you've heard during the week.

I'll try to give you some color on what it takes to actually plan out such an industrial project in real life, and essentially productionize machine learning.

It's a very hot topic, and for very good reasons.

Actually, only a year ago, in January 2017, Waymo became its own company.

So that was a major milestone and a testimony to the robustness of the solution, so that we could move to a productization phase.

So what you see on the picture here is our latest generation self-driving vehicle.

I'll come back to that and give you more insights on what

So self-driving indeed draws a lot of attention, and for a very good reason.

I personally believe, and I think you will agree with me, that self-driving really has the potential to deeply change the way we look about mobility and the way we move people and things around.

So only to cover a few aspects here, obviously, that I don't want to go into too many details, but safety is 1 of the main motivations.

of US crashes today involve human errors.

A lot of those errors are around distraction and things that could be avoided.

Accessibility and access to mobility is also a big motivation of ours.

So obviously, the self-driving technology has the potential to make it very available and cheaper for more people to be able to move around.

And last but not least is efficiency, collective efficiency.

So not only we spend a lot of time in our cars, in long commute hours, I personally spend a lot

of time in long commute hours, And that time we spend

in traffic probably could be better spent doing something else than having to drive the car in complicated situations.

Beyond traffic, obviously, self-driving technology has the potential to deeply change the way we think about traffic, parking spots, urban environments, city design.

So that's why it's a very exciting topic.

So that's why we made it our mission at Weimo, is fundamentally to make it safe and easy to move people and things around.

a nice mission, and we've been on it for a

So actually, The whole adventure started close to 10 years ago, in

And at the time, that started under the umbrella of a Google project that you may have heard of called Chauffeur.

And back in those days, so remember, we were before the deep learning days, at least

the first objective of the project was to try and assemble a first prototype vehicle, take off-the-shelf sensors, assemble them together, and try to go and decide if self-driving is even a possibility.

It's like, it's 1 thing to have some prototype somewhere, but is that even a thing that is worth pursuing?

Which is a very common way for Google to tackle problems.

So the genesis for that work was to come up with a pretty aggressive objective.

So the team, the first milestone for the team was to essentially assemble 10 100-mile loops in Northern California around Mountain View and try and figure out, so for a total of 1,000 miles, and try and see if they could build a first system that would be able to go and drive those loops autonomously.

So those loops went through some very aggressive patterns.

So you see that Some of those loops go through the Santa Cruz Mountains, which is an area in California that, as you'll see, I'll show you a video, that has very small roads and two-way traffic and cliffs with negative obstacles and complicated patterns like that.

Some of those paths were going on highways, and 1 of the busiest highways.

Some of those routes were going around Lake Tahoe, which is in the Sierras in California, where you can encounter different kinds of weather and again different kinds of roads conditions.

Those routes were going around bridges and the Bay Area has quite a few bridges to go through.

Though some of them were even going through a dense urban area.

So you can see San Francisco being driven, you can see Monterey, some of the Monterey centers being driven.

And as you'll see on the video, those truly bring dense urban area challenges.

So since I promised it, so here you're gonna see some pictures of the driving, it's kind of working.

So here with better quality, so here you see the roads I was talking about on the Santa Cruz mountains, driving in the night, animals crossing the street, freeway driving, going through a pay toll, that's the Montre area that is fairly dense.

There's the aquarium there, a pretty popular 1.

That's the famous Lombard Street in San Francisco that you may have heard of, which in San Francisco always brings its unique set of challenges between fog and slopes and in that case even sharp turns.

So those 10 loops were successfully completed

So on the heels of that success, the team decided, and Google decided that self-driving was worth pursuing, and moved forward with the development of the technology and testing.

So we've been at it for all those years and have been working very hard on it.

Historically, Waymo and I think all the other companies out there have been relying on what we call safety drivers to still sit behind the wheels, even if the car is driving autonomously.

You still have a safety driver who is able to take over at any time and make sure that we have very safe operations.

And we've been accumulating miles and knowledge and developing the system, many iterations of the system, across all those years.

We reached a major milestone, as Lex mentioned, back in November, where for the first time we reached a level of confidence and maturity in a system that we felt confident and proved to ourselves that it was safe to remove the safety driver.

As you can imagine, that's a major milestone because it takes a very high level of confidence to not have that backup solution of a safety driver to take over where something to arise.

So here I'm gonna show you a small video, a quick quick capture

Since then we've been continuously operating driverless cars, self-driving cars in the Phoenix area in

We're acting as passengers, getting on a back seat.

You can notice that there is no driver on the driver's seat.

So here we are running a car-hailing kind of service.

So the passengers simply press a button, the application knows where they want to go, and the car goes.

So we started with a fairly constrained geographical area in Chandler, close to Phoenix, Arizona.

And we are hard-working to expand the testing and the scope of our operating area since then.

So that goes well beyond a single car a single day.

Not only we do that continuously, but we also have a growing fleet of self-driving cars that we are deploying there all the way and looking for a product launch pretty quickly.

So I talked about 2010, and we are in 2018, and we are getting there, But it took quite a bit of time.

So I think 1 of the key ideas that I'd like to convey here today, and that I will go back to during the presentation, is how much work, and how much work it takes to really take a demo or something that's working in the lab into something that you feel safe to put on the roads, and get all the way to that depth of understanding, that depth of perfection in your technology, that you operate safely.

So 1 way to say that is that when you're 90% done, you still have 90% to go, right?

So 90% of the technology takes only 10% of the time, right?

You need to 10x the the capabilities of your technology.

You need to 10x your team size and find ways for more engineers and more researchers to collaborate together.

You need to 10x the capabilities of your sensors.

You need to 10x fundamentally the overall quality of the system, right?

And your testing practices, as we'll see, and a lot of the aspects of the program.

So, Beyond the context of self-driving cars, I want to spend a little bit of time to give you kind of an insider view of the rise of deep learning.

deep learning was not really available yet in full capacity in the industry.

And so over those years, actually, it took a lot of breakthroughs to be able to reach that stage.

And 1 of them was the algorithm breakthrough that deep learning gave us.

And I give you a little bit of a backstage view on what happened at Google during those years.

So as you know, Google has committed itself to machine learning and deep learning very early on.

You may have heard of the Google Brain, what we call internally the Google Brain Team, which is a team fundamentally hard at work to lead the bleeding edge of research, which is known, but also leading the development of the tools and infrastructure of the whole machine learning ecosystem at Google and Waymo to essentially allow many teams to develop machine learning at scale all the way to successful products.

So they've been working and pushing, the deep learning technology has been pushing the field in many directions, from computer vision to speech understanding to NLP, And all those directions are things that you can see in Google products today.

So whether you're talking Google Assistant, or Google Photos, speech recognition, or even Google Maps, you can see the impact of deep learning in all those areas.

And actually, many years ago, I was part of, I myself was part of the Street View team, and I was leading the, what, an internal program, an internal project that we called StreetSmart.

And the goal we had at StreetSmart was to use deep learning and machine learning techniques to go and analyze street view imagery.

And as you know, that's a fairly big and varied corpus, so that we could extract elements that are core to our mapping strategy, and that way build better Google Maps.

So for instance, in that picture, so that's a piece of a panorama from Street View imagery, and you can see that there are a lot of pieces in there that if you could find and properly localize, would drastically help you build better maps.

So street numbers, obviously, that are really useful to map addresses, street names that when combined even on similar techniques from Arial views, will help you properly draw all the routes and give a name to them.

And those 2 combines actually allow you to do very high-quality address lookups, which

more specifically text on business facades that allow you to not only maybe localize business listings that you may have gotten by other means to actual physical locations, but also build some of those local listings directly from scratch.

And more traffic-oriented patterns, whether it's traffic lights, traffic signs, that can be used then for ETA, navigation ETA predictions, and stuff like that.

1 of the, as I mentioned, 1 of the hard pieces to do is to map addresses at scale.

And so you can imagine that we had the breakthrough when we first were able to properly find those street numbers out of the street view imagery and out of the facade.

Solving that problem actually requires a lot of pieces.

Not only do you need to find where the street number is on the facade, which is, if you think about it, a fairly hard semantic problem.

What's the difference between a street number versus another kind of number versus other text?

But then obviously read it, because there's no point having pixels if you cannot understand the number that's on the facade, all the way to properly geolocalizing it so that you can then put it on Google Maps.

And so the first deep learning application that succeeded in production, and that's all the way back to 2012, that we had the first system in production, was really the first breakthrough that we had across Alphabet on our ability to properly understand real scene situations.

So here I'm going to show you a video that kind of sums it up.

So look, every 1 of those segments is actually a view, starting from the car, going to the physical number of all those house numbers that we've been able to detect and transcribe.

So here that's in Sao Paolo, and where you can see that when all that data is put together, it gives you a very consistent view of the addressing scheme.

Similar things, obviously we have more, that's in Paris, where we have even more imagery, so more views of those physical numbers that, if you're able to triangulate, you're able to localize them very accurately, and have very accurate maps.

So the last example I'm going to show is in Cape Town in South Africa, where again the impact of that deep learning work has been huge in terms of quality.

So many countries today actually have up to more than 95% of addresses mapped that way.

So obviously you can see a lot of parallelism between that work on street view imagery and doing the same on the real scene on the car.

But obviously, doing that on the car is even harder.

It's even harder because you need to do that real time and very quickly with low latency.

And you also need to do that in an embedded system.

So the cars have to be entirely autonomous.

You cannot rely on a connection to a Google data center, and first you don't have the time, in terms of latency, to bring data back and forth.

But also you cannot rely on a connection for the safe operation of your system.

So you need to do the processing within the car.

But very so that's a paper that you can read that dates all the way back to 2014, where for the first time, by using slightly different techniques, we were able to put deep learning at work inside that constrained real-time environment, and start to have impact, and in that case around the pedestrian detection.

So as I said, there are a lot of analogies.

You can see that to properly drive that scene, Like Street View, you need to see the traffic light, you need to understand if the light is red or green, and that's what essentially will allow you to do that processing.

Obviously driving is even more challenging beyond the real time, and if you saw the cyclist going through, so we have real stuff happening on the scene that you need to detect and properly understand, interpret, and predict.

And at the same time, he explicitly took a night driving example

to show you that While you can choose when you take pictures of Street View and do it in daytime and in perfect conditions, driving requires you to take the conditions as they are, and you have to deal with it.

So there has been, from the very early beginning, there's been a lot of cross-punnelization between the real scene work.

So here I took a few papers that we did in Street View that obviously if you read them you'll see directly apply to some of the stuff we do on the cars.

But obviously that collaboration between Google Research and Waymo historically went well beyond Studio only and across all the research groups.

And that still is a very strong collaboration going on that enables us to be, to stay on the bleeding edge, right, of what we can do.

a little bit at how things happened, I want to

spend more time and go into more of the details of what's going on in the cars today, and how deep learning is actually impacting our current system.

So I think during the, if I looked at the cursors properly, I think during the week you went through the major pieces that you need to master to make a self-driving car.

So I'm sure you heard about mapping, localization, so putting the car within those maps and understanding where you are with pretty good accuracy.

Perception, scene understanding, which is a higher level semantic understanding of what's going on in the scene, starting to predict what the agents are gonna do around you so that you can do better motion planning.

Obviously, there's a whole robotics aspect.

At the end of the day, the car, in many ways, acts like a robot, whether it's around the sensor data or even the control interfaces to the car.

And for everyone who was dealt with hardware and robotics, you will agree with me that it's not a perfect world, and you need to deal with those errors.

Other pieces that you may have talked about is around simulation and essentially validation of whatever system you put together.

So obviously, machine learning and deep learning have been having a deep impact in a growing set of those areas.

But for the next minutes here, I'm gonna focus more on the perception piece, which is a core element of what a self-driving car needs to do.

So fundamentally, perception is a system in the car that needs to build an understanding of the world around it.

a little silly to have to recompute the actual location of the road, the actual interconnectivity of the intersections, of every intersection once you get on the scene, because those things you can pre-compute.

You can pre-compute in advance and save your on-board computing for other tasks that are more critical.

So really, that's often referred to as the mapping exercise, but really it's about reducing the computation you're gonna have to do on the car once it drives.

The other big input, obviously, is what sensors are gonna give you once you get on the spot.

So sensor data is the signal that's gonna tell you what is not like what you mapped.

And the things is the traffic light red or green.

So as we saw on the initial picture, we have quite a set of sensors on our self-driving cars.

So they go from vision systems, radar, and LIDAR, are the 3 big families of sensors we have.

1 point to note here is that they are designed to be complementary.

So they are designed to be complementary first in their localization on the car, So we don't put them in the same spot, because obviously blind spots is a major issue, and you want to have good coverage of the field of view.

The other piece is that they are complementary in their capabilities.

So for instance, to give you an example, cameras are going to be very good to give you a dense representation.

It's like, it's a very dense set of information.

It contains a lot of semantic information.

You can really see a large number of details, but for instance they are not really good to give you depth.

It's much harder, and computationally expensive, to get depth information out of camera systems.

So systems like a LiDAR, for instance, will give you very good, very good, when you hit objects, will give you a very good depth estimation.

But obviously they're gonna lack a lot of the semantic information that you will find on camera systems.

So all those sensors are designed to be complementary in terms of their capabilities.

It goes without saying that the better your sensors are, the better your perception system is going to be.

So that's why at Waymo we took the path of designing our own sensors in-house and enhancing what's available off the shelf today, because it's important for us to go all the way to be able to build a self-driving system that we could believe in.

It takes those 2 inputs and builds a representation of the scene.

So at the end of the day, you have to realize that in nature, that work of perception is really what differentiates, deeply differentiates, what you need to do in a self-driving system as opposed to a lower level driving assistance system.

In many cases, for instance if you do speed cruise, or if you do a lot of lower level driver assistance, A lot of the strategies can be around not bumping into things.

If you see things moving around, you group them, you segment them appropriately in blocks of moving things, and you don't hit them, you're good enough in most cases.

When you don't have a driver on the driver's seat, obviously the challenge totally changes scale.

So to give you an example, for instance, if you're on the lane and you see a bicyclist going more slowly on the lane right of you, and there's a car next to you, you need to understand that there's a chance that that car is going to want to avoid, that bicyclist is going to swerve, and you need to anticipate that behavior so that you can properly decide whether you want to slow down, give space for the car, or speed up and have the car go behind you.

Those are the kinds of behaviors that go well beyond not bumping into things and that require a much deeper understanding of the world that's going on around you.

So let me put it in picture and we come back to that example in the court case.

So here is a typical scene that we encountered at least.

So so here obviously you have a police car pulled over, probably pulled over someone there.

You have a cyclist on the road moving forward, and we need to drive through that situation.

So the first thing you can do, you have to do obviously, is the basics.

So Out of your sensor data, understand that a set of point clouds and pixels belong to the cyclist.

Find that you have 2 cars on the scene, the police car and the car parked in front of it.

Understand the policeman as a pedestrian.

Obviously you need a little more than that.

Obviously you need, if you understand that the flashing lights are on, you understand that the police car is becoming an EV and is performing something on the scene, if you understand that this car is parked, we see that's a valuable piece of information that's going to tell you whether you can pass it or not.

Something you may have not noticed is that there are so cones.

So there are cones here on the scene that would prevent you, for instance, to go and drive that pathway if you wanted to.

Next level of getting closer to behavior prediction, Obviously, if you also understand that actually the police car has an open door, then all of a sudden you can start to expect a behavior where someone is gonna get out of that car, right?

And the way you swerve, even if you were to decide to swerve, or the way someone getting out of that car would impact the trajectory of the cyclist is something you need to understand in order to properly and safely drive.

And only then, only when you have that depth of understanding, you can start to come up with realistic behavior predictions and trajectory predictions for all those agents on the scene, so that you can come up with a proper strategy for your planning control.

So how is deep learning playing into that whole space?

And how is deep learning impacting used to solve many of those problems.

So remember when I said when you're 90% done, you still have 90% to go?

So I think that's that starts to beat us.

I also talked about how robotics and having sensors in real life is not a perfect world.

So actually it is a big piece of the puzzle.

So I wish sensors would give us perfect data all the time, and would give us a perfect picture that we can do reliably use to do a deep learning, but unfortunately that's not how it works.

So here for instance, You see an example where you have a pickup truck, so the imagery doesn't show it, but you have smoke coming out of the exhaust, and you have exhaust that's triggering LIDAR laser points.

Not very relevant for any behavior prediction or for your driving behavior.

So those points, obviously, and it's safe to go and drive through them.

So those are very safe to ignore in terms of scene understanding.

So filtering the whole bunch of data coming off your sensors is a very important task, because that reduces the computation you're going to have to do, but also key to operate safely.

A more subtle 1, but an important 1, are around reflections.

So here we are driving a scene, there's a car here, on the camera picture the car is reflected in a bus, and if you just do a naive detection, you're gonna, especially if the bus moves along with you, which is very typical, and everything moves, then all of a sudden you're gonna have 2 cars on the scene.

And if you take that car too seriously, all the way to impacting your behavior, obviously you're gonna make mistakes.

So here I showed you an example of reflections on the visual range, but obviously that affects all sensors in slightly different manners.

But you could have the same effect, for instance, with a LiDAR data, where, for instance, you drive a freeway and you have a road sign on top of the freeway, that will reflect in the back window of the car in front of you, and then showing a reflected sign on the road.

See all Lex Fridman transcripts on Youtube

Sacha Arnoud, Director of Engineering, Waymo - MIT Self-Driving Cars