Introductory lecture of the MIT Self-Driving Cars series (6.S094) with an overview of the autonomous vehicle industry in 2018 and looking forward to 2019, including Waymo, Tesla, Cruise, Ford, GM, and out-of-the-box ideas of boring tunnels, flying cars, connected vehicles, and more. This covers the state of the art in terms of industry developments and not the perception and planning algorithm development. The latter will be covered in detail in future lectures. For more lecture videos on deep learning, reinforcement learning (RL), artificial intelligence (AI & AGI), and podcast conversations, visit our website or follow TensorFlow code tutorials on our GitHub repo.

INFO:
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Slides: http://bit.ly/2HLcQ4s 
Playlist: http://bit.ly/2S1MVdy

OUTLINE:
0:00 - Introduction
1:53 - 2018 in review
4:49 - Fatalities
8:29 - Taxi services
10:54 - Predictions
16:55 - Human-centered autonomy
19:42 - Levels of autonomy and proliferation strategies
24:48 - Out-of-the-box ideas
27:28 - Who will be first?
29:26 - Historical context
31:05 - Underlying beliefs of the industry and public
32:32 - Driving is hard
35:32 - Humans are amazing
37:10 - Humans and automation don't mix well?
41:55 - Two approaches: Lidar vs Vision
49:54 - In the meantime… data
52:49 - The road ahead

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman

Today I'd like to talk about the state of the art of autonomous vehicles, how I see the landscape, how others see the landscape, what we're all excited about, ways to solve the problem, and what to look forward to in 2019 as we also get to hear from the different perspectives and the various leaders in industry and autonomous vehicles in

the next few, next couple of weeks and next few days.

So the problem, the mission, the dream, the thing that we're trying to solve for many may be about entrepreneurial possibilities of making money and so on, but really it's about improving access to mobility, moving people around the world that don't have that ability, whether it has to do with age or purely access of where you live.

We want to increase the efficiency of how people move about.

The ability to be productive in the time we spend in traffic and transportation.

1 of the most hated things in terms of stress and motion.

The thing in our lives that if we could just with a snap of a finger remove is traffic.

So the ability to convert that into efficiency, into a productive aspect, into a positive aspect of life.

And really the most important thing, at least for me and for many of us working in this space, is to save lives.

Prevent crashes that lead to injuries, prevent crashes that lead to fatalities.

Here's a counter, every 23 seconds, somebody in the world dies in a car auto crash.

It should be a sobering, it is for me, thing that I think about every single day.

You go to bed, you wake up, you work out all the deep learning methods, all the different papers we're publishing, everything we're trying to push forward is really to save lives at the beginning and at the end that is the main goal.

So with that groundwork, with that idea, with that base of mission that we're all working towards from the different ideas and different perspectives, I would like to review what happened in 2018.

So first, Waymo has done incredible work in deploying and testing their vehicles in various domains and have in October reached the mark of 10 million miles driven autonomously, which is an incredible accomplishment.

It's truly a big step for fully autonomous vehicles in terms of deployment.

And obviously it's growing and growing by day.

And we'll have Jaguar here from Waymo to talk about their work there.

Then on the L2, on the semi-autonomous side that's the pair, that's the mirror side of this equation.

The other incredible number that's perhaps less talked about is the 1000000000 mile mark reached by Tesla in the semi-autonomous driving of Autopilot.

Autopilot is a system that's able to control its position in the lane, center itself in the lane, it's able to control the longitudinal movement, so not follow a vehicle when there's a vehicle in front and so on.

But the degree of its ability to do so is the critical thing here, is the ability to do so for many minutes at a time, even hours at a time, especially on highway driving.

And the fact that they've reached 1000000000 with a B miles is an incredible accomplishment.

All of that from the machine learning perspective is data.

And all of the autopilot miles are driven with the primary sensor being a camera.

And how does computer vision work in modern day?

Especially with the second iteration of autopilot hardware, there's a neural network, there's a set of neural networks behind it.

That is probably the largest deployment of neural networks in the world that has a direct impact on a human life that's able to decide, that's able to make life critical decisions many times a second, over and over.

You go from the step of image classification on ImageNet and you sit there with a TensorFlow and you're very happy there.

You're able to achieve a 99.3 accuracy with a state of the art algorithm.

You take from that a step towards, there's a human life.

Your parents driving, your grandparents driving this, your children driving this system.

And there's a neural network making the decision of whether they live.

So that 1000000000 mark is an incredible accomplishment.

And on the sobering side, from various perspectives, the fatalities.

There's been 2 fatalities that happened in March of 2018.

1 in the fully autonomous side of things with Uber in Tempe, Arizona hitting a pedestrian and leading to a pedestrian fatality.

And on the semi-autonomous side of Tesla Autopilot, the third fatality that Tesla Autopilot led to, and the 1 in 2018, is in Mountain View, California, when Tesla slammed into a divider, killing his driver.

Now, The 2 aspects here that are sobering and really important to think about as we talk about the progression of autonomous vehicles, proliferation in our world, is our response as a public, as from the general public to the engineers, to the media and so on, how we think about these fatalities.

And obviously there's a disproportionate amount of attention given to these fatalities.

And that's something as engineers, you have to also think about.

That the bar is much higher on every level in terms of performance.

So in order to success, as I'll argue, in order to design successful autonomous vehicles, those vehicles will have to take risks.

And when the risks don't pan out, the public, if the public doesn't understand the general problem that we're tackling, the goal, the mission, that those risks, when they don't, the risks that are taken can have significant detrimental effect to the progress in this autonomous vehicle space.

So that's something we really have to think about.

So the question was, do we know the rate of fatalities per mile of vehicle driven, which is at the crudest level how people think about safety.

So there's about 80, 90, 100 million miles driven in manually controlled cars at every fatality.

So 1 fatality per, depending on which numbers you look at, it's 80 to 100 million miles.

And the Tesla vehicle, for example, the fatality is, well we could just take the 1000000000 and divide it by 3.

Now, it's apples and oranges in comparison.

And that's something actually that we're working on to make sure that we compare it correctly.

Compare the aspects of manual miles that directly are comparable to the autopilot miles.

So autopilot is a modern vehicle that's much safer.

Tesla is a modern vehicle that's much safer than the general population of manually driven vehicles.

Autopilot is driven on only particular kinds of roads, on the highway, primarily most of the miles.

The kinds of people that drive Autopilot, all these kinds of factors need to be considered when you compare the 2.

But when you just look at the numbers, Tesla autopilot is 3 times safer than manually driven vehicles, but that's not the right way to look at it.

And for anyone that's ever taken a statistics class, 3 fatalities is not does not, it's not a large number by which to make any significant conclusions.

Nevertheless, that doesn't stop the media, the New York Times, and everybody from responding to a single fatality,

which PR and marketing aspects of these different companies are very sensitive to, which is of course troubling and concerning for an engineer that wants to save lives, but it's something that we have to think about.

There's been a lot of announcements or rather actual launches of public testing of autonomous taxi services.

So companies that are on public roads have been delivering real people from 1 location to another.

In many of these cases, it's very small scale, just a few vehicles.

In most cases, it's very low speed in a constrained environment, in a constrained community.

And almost always, really always with a safety driver.

There's a few exceptions for demonstration purposes, but there's always an actual driver in the seat.

Some of the brilliant folks representing these companies will speak in this course.

Is Voyage doing it in an isolated community.

Awesome work they're doing in villages in Florida.

Optimus Ride here in Boston, doing in the community in Union Point.

Drive AI in Texas, Maine Mobility, expanding beyond Detroit, but really most operations in Detroit.

Waymo has launched its service, Waymo 1, that's gotten some publicity in Phoenix, Arizona.

Neuro doing 0 occupancy deliveries of groceries

has to be delivering humans, it's delivering groceries autonomously.

Uber is quietly or not so quietly resumed its autonomous vehicle taxi service testing in Pittsburgh in a very careful, constrained way.

Aptiv, after acquiring Carl and Yema, Neutonomy, has been doing extensive, large-scale taxi service testing everywhere from Vegas to Boston here, to Pittsburgh and in Singapore, of course.

Aurora that spoke here last time, the head of Tesla Autopilot that launched Aurora, and the Chris Hermsen behind this young upstart company is doing testing in San Francisco and Pittsburgh, and then Cruz Kyle will be here to talk from GM is doing testing in San Francisco, Arizona and Michigan.

So when we talk about predictions, I'll talk about a few people predicting when we're going to have autonomous vehicles and when you yourself think about what it means, when will they be here?

When will autonomous vehicles arrive such that the Uber that you call will be autonomous and not with a populated by a driver?

So the thing we have to think about is what we think about how we define autonomous, what that experience looks like.

And most importantly, in these discussions, we have to think about scale.

So we here at MIT, our group, MIT Human-Centered Autonomous Vehicle, We have a fully autonomous vehicle that people can get in if you would like and it will give you a ride in a particular location.

But that's 1 vehicle, it's not a service and it only works on particular roads.

In some ways it's not much different than most of the companies that we're talking about today.

Now scale here, there's a magic number, I'm not sure what it is, but for the purpose of this conversation, it says 10,000, where there's a meaningful deployment.

When it's truly going beyond that prototype demo mode to where everything is under control to where it's really touching the general population in a fundamental way.

Scale is everything here and it starts let's say at 10,000.

Just to give you for reference there's 46,000 active Uber drivers in New York City.

of the Uber drivers in New York City all of a sudden become passengers.

I'm not a marketing PR person, so I don't understand why everybody has to make a prediction but they all seem to.

All the major automakers have made a prediction of when they'll have a deployed, when they will be able to deploy autonomous vehicles.

Tesla has made in early 2017, a prediction that they will have autonomous vehicles 2018.

In 2018, they've now adjusted the prediction to 2019.

Nissan, Honda, Toyota have made prediction for 2020 under certain constraints in highway urban.

Hyundai and Volvo has in 2021, the BMW and Ford, Ford saying at scale.

So there is the predictions that are extremely optimistic that are perhaps driven by the instinct that the company has to declare that they're at the cutting edge of innovation.

And then there is many of the leading engineers behind the leading these teams, including Carl and Yama and Gil Pratt from MIT who injects a little bit of caution and grounded

ideas about how difficult it is to remove the human from

So Carl says that basically teleoperation kind of gives this analogy of an elevator.

You know, an elevator is fully autonomous, but there's still a button to call for help if something happens.

And that's how he thinks about autonomous vehicles.

Even with greater and greater degree of automation, there's still going to have to be a human in the loop.

There's still going to be a way to contact a human to get help.

And Gil Pratt and Toyota, and they're making some announcements at CES, basically saying that the human in the loop is the fundamental aspect that we need to approach this problem.

And removing the human from consideration is really, really far away.

And Gil, historically and currently, is 1 of the sort of the great roboticists in the world that defined a lot of the DARPA challenges and a lot of our progress, historically speaking, up to this point.

So really the full spectrum, we can think of it as the Elon Rodney spectrum of optimism versus pessimism.

The Elon Musk who's extremely bold and optimistic about his predictions.

I often connect with this kind of thinking because sometimes you have to believe the impossible is possible in order to make it happen.

And then there's Rodney, 1 also 1 of the great roboticists, the former head of the C-Cell, the AI laboratory here is a little bit on the pessimistic side.

So for Elon, the fully autonomous vehicle will be here in 2019.

For Rodney, the vehicles are really fully autonomous or beyond 2050.

But he believes in the 30s, there will be a significant, a major city will be able to allocate a significant region of, significant region of that city where manual driving is fully banned.

Which is the way he believes those vehicles, autonomous vehicles can really proliferate when you ban manually driven vehicles in certain parts.

And in the 40s, 2045 or beyond, majority of US cities will ban manually driven vehicles.

Of course, the quote from Elon Musk in 2017 is that my guess is that in probably 10 years it will be very unusual for cars to be built that are not fully autonomous.

So we also have to think about the long tail of the fact that many people drive cars that are 10 years old, 20 years old.

So even when you have every car is built that's fully autonomous, it's still going to take time for that dissipation of vehicles to happen.

to take a little pause into the ridiculous and the fun to explain the view.

Yes, that is me playing guitar in our autonomous vehicle.

Now the point of this ridiculous video and embarrassing I should never played it.

Yeah, okay, I think it's gonna be over soon.

Now for those of you born in the 90s, that's classic rock.

So the point I'm trying to make beyond predictions

is that autonomous vehicles will not be adopted by human beings in the near term, in the next 10, 15 years, because they're safer.

Safety is not going to, they may be safer, but they're not going to be so much safer that that's going to be the reason you adopt.

It's not gonna be because they get you to the location faster.

Everything we see with autonomy is they're going to be slower until majority of the fleet is autonomous.

They're cautious and therefore slower and therefore more annoying in the way we think about actually how we navigate this world.

We take risk, we drive assertively with speed over the speed limit all the time.

That is not how autonomous vehicles today operate.

So they're not going to get us there faster.

And for every promise, every hope that they're going to be cheaper, really there's still significant investment going into them and there's not good economics in the near term of how to make them obviously significantly cheaper.

What I think Uber and Lyft has taken over the taxi service because of the human experience.

In the same way, autonomy will only take over if not take over, be adopted by human beings if it creates a better human experience.

If there's something about the experience that you enjoy the heck out of.

This video and many others that we're putting out shows that the natural language communication, the interaction with the car, the ability of the car to sense everything you're doing, from the activity of the driver, to the driver's attention, and being able to transfer control back and forth in a playful way, but really in a serious way, also that's personalized

the human experience, the richness of the human experience, that is what we need to also solve.

That's something you have to think about because many of the people that'll be speaking at this class and many of the people that are working on this problem are not focused on the human experience.

It's a kind of afterthought that once we solve the autonomous vehicle problem, it'll be fun as hell to be in that car.

I believe you first have to make it fun as hell to be in the car and then solve the autonomous vehicle problem, jointly.

So in the language that we're talking about here, there's several levels of autonomy that are defined from level 0 to level 4, level 0, no automation, 4 and 5, level 3, 4 and 5, increasing automation.

So level 2 is when the driver is still responsible.

Level 3, 4, 5 is when there's less and less responsibility.

But really in 3, 4, 5 there's parts of the driving where the liability is on the car.

So there's only really 2, as far as I'm concerned, levels.

Human-centered autonomy and full autonomy.

Human-centered means the human is responsible.

Full autonomy means the car is responsible.

Both on the legal side, the experience side, and the algorithm side.

That means full autonomy does not allow for teleoperation.

So it doesn't allow for the human to step in and remotely control the vehicle because that means the human is still in

It doesn't allow for the 10-second rule that it's going to be fully autonomous but once it starts warning you you have 10 seconds to take over.

We cannot guarantee safety in any situation.

It has to be able to, if the driver doesn't respond in 10 seconds, it has to be able to find safe harbor.

It has to be able to pull off to the side of

the road without hurting anybody else to find safety.

So that's the fully autonomous challenge.

And so how do we envision these 2 levels of automation proliferating society, getting deployed at

On the fully autonomous side, the way to think about it, with the predictions that we're talking about here, is there's several different possibilities of how to deploy these vehicles.

1 is last-mile delivery of goods and service like the groceries.

These are zero-occupancy vehicles delivering groceries or delivering human beings at the last mile.

What the last mile means is, it's slow moving transport to the destination, where most of the tricky driving along the way is done manually, and then the last mile delivery in the city, in the urban environment, is done by 0 occupancy autonomous vehicles.

Trucking on the highway, possibly with platooning, where a sequence of trucks follow each other.

So in this, what people think about as a pretty well-defined problem of highway driving with lanes well marked, well mapped, routes throughout the United States and globally on the highway driving is automatable.

The specific urban routes, kind of like what a lot of these companies are working on, defining this taxi service and a personalized public transport.

There's certain pickup locations you're allowed to go to, there are certain drop-off locations, that's it.

It's kind of like taking the train here, but as opposed to getting on the train with a hundred other people you're getting or bus you're getting on the car with when you're alone or with 1 other person.

The closed communities something Oliver Cameron with Voyage is working on defining and Optimus Ride, defining a particular community that you now have a monopoly over, that you define the constraints, you define the customer base, and then you just deliver the vehicles, you map the entire road, you have slow moving transport that gets people from A to B anywhere in that community.

And then there's the world of 0 occupancy ride sharing delivery.

So the Uber that comes to you as opposed to having you drive it yourself and it comes to you autonomously with nobody in there.

So imagine a world where we have empty vehicles driving around, delivering themselves to you.

Semi-autonomous side is thinking about a world where teleoperation plays a really crucial role, where it's fully autonomous under certain constraints in the highway, but a human can always step in.

High autonomy on the highway, kind of like what Tesla is working towards most recently, is on-ramp to off-ramp.

Now the driver is still responsible liability-wise and in terms of just observing the vehicle and algorithmically speaking, but the autonomy is pretty high level to a point where much of the highway driving could be done fully autonomously and low autonomy unrestricted travel as an advanced driver assistance system meaning that the car kind of like the Tesla, the Volvo S90s, or the SuperCruise and the Cadillacs, all these kinds of L2 systems that are able to keep you in the lane, you know, 10 to 30% of the miles that you drive, and some fraction of

the time take some of the stress of driving off.

Vehicle to vehicle communication and vehicle to infrastructure communication enabling us to navigate, for example, intersection efficiently without stopping, removing all traffic lights.

So here shown on the bottom is our conventional approach of there's a queuing system that forms because of traffic lights that turn red, green, yellow

and without traffic lights and with communication to

the infrastructure and between the vehicles, you can actually optimize that to significantly increase the traffic flow through a city.

of tunnels under cities Layers of tunnels under cities

Tunnels all the way down Autonomous vehicles Basically by the design of the tunnel constraining the problem to such a degree that an autonomous, I mean, the idea of autonomy just is completely transformed.

That you're basically, a car is able to transform itself into a mini train, into a mini public transit entity for a particular period of time.

So you get into that tunnel, you drive at 200 miles an hour and not necessarily drive, be driven 200 miles an hour and then you get out of the tunnel.

Of course There's the flying cars, personalized flying car vehicles.

Rodney, as I mentioned before, does believe that we'll have them in 2050.

There's a lot of people that are seriously actually thinking about this.

This problem is there's a level of autonomy obviously that's required here for a regular person, like I don't know, somebody without a pilot's license for example, to be able to take off and land.

You know, making that experience accessible to regular people means that there's going to be a significant amount of autonomy involved.

1 of the people really, 1 of the companies really seriously working on this is Uber, with the Uber Elevate, Uber Air, I think it's called.

And the idea is that you would meet your vehicle not on the street, but at a roof.

You take an elevator, you meet them at the roof of the, of a building.

And they're seriously addressing this problem.

Many of the great solutions to the world's problems have been laughed at at some point.

So let's not laugh too loud at these possibilities.

Back in my day, we used to drive in the street.

I sort of out of curiosity asked, did a little public poll, 3,000 people responded, asked who will be first to deploy 10,000 fully autonomous cars operating on public roads without a safety driver.

And several options percolated with Tesla getting 50%, 57% of the vote, and Waymo getting 21% of the vote, and 14% someone else, and 8% the curmudgeons and the engineers saying no 1 in the next 50 years will do it.

in 1998, when Google came along, the leaders of the space were Ask Jeeves and InfoSeek and Excite, all services I've used and probably some people in this room have used, Lycos, Yahoo.

Obviously, they were the leaders in the space and Google disrupted that space completely.

See all Lex Fridman transcripts on Youtube

Self-Driving Cars: State of the Art (2019)