Oliver Cameron is the Co-Founder and CEO of Voyage. Before that he was the lead of the Udacity Self-Driving Car program that made ideas in autonomous vehicle research and development accessible to the world. For more lecture videos on deep learning, reinforcement learning (RL), artificial intelligence (AI & AGI), and podcast conversations, visit our website or follow TensorFlow code tutorials on our GitHub repo.

INFO:
Website: https://deeplearning.mit.edu
GitHub: https://github.com/lexfridman/mit-deep-learning
Playlist: http://bit.ly/2S1MVdy

OUTLINE:
0:00 - Lex introducing Oliver
0:39 - Oliver background
4:40 - Udacity self-driving car engineer nanodegree program
14:10 - Autonomous trip from Mountain View to San Francisco
23:11 - Open source challenges
26:48 - Birth of Voyage
31:58 - Retirement communities
38:35 - Sensor and technology stack
40:45 - Example challenge for perception (foliage)
41:58 - Survey of recent perception research
45:51 - Lessons learned
48:45 - Q&A

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman

All right, welcome back to 6.094, Deep Learning for Self-Driving Cars.

He's the co-founder and the CEO of Voyage.

Before that, he was the lead of the Udacity Self-Driving Car Program that made ideas in autonomous vehicle research and development accessible to the entire world.

He has a passion for the topic and a genuine open nature that makes him 1 of my favorite people in general and 1 of my favorite people working in this space and I think thousands of people agree with that So please give Oliver a warm welcome.

And thank you all for having me here today.

But in reality, the kind of thing I wanna share today is kinda like this title says, how to start a self-driving car startup.

Rarely do you kinda get an inside scoop of how a startup is formed.

You kinda hear all the PR, all the kind of very lovey-dovey press releases out there.

I want to share kind of the inside of how at least Voyage came to be, which was a little unconventional compared to your average self-driving car startup.

They always tell you that the path to a startup, getting to the goal you want is kind of a zigzag.

I'm not very good at learning in a classroom.

For me, I found learning by doing, by building, has always been the thing that's worked best for me.

So, going all the way back to when I was a teenager, software just in general was my passion.

This idea that you can make something out of absolutely nothing, and then all of a sudden, millions, and in Facebook's case, billions of people can be using that thing.

And after building lots of crazy stuff, and perhaps not being too popular in high school because that's all I did, I started a company.

I won't bore you with all the details, but learned a lot during the experience and went through Y Combinator, which I believe started right here in Cambridge, which is very cool.

And then this very pivotal moment happened to me.

I heard about this online class, which was generating a whole bunch of scandal and lots of controversy.

And it was from this guy called Sebastian Thrun.

He'd taken this Stanford class he taught in artificial intelligence and just said, screw it, we're gonna put the whole thing online.

And back then, and this was around 2011, this was a very controversial thing to do.

Today, MIT and many others do this all the time.

But back then, there was a hell of a lot of controversy around doing something like this.

But this learning format really just appealed to me.

Being able to sit in front of my laptop, learn at my own pace, build, build, build was something that really resonated with me.

And I took this class in 2013, Artificial Intelligence for Robotics.

And this again was just this pivotal moment.

My head exploded, all the enthusiasm I'd had for software kind of transferred to artificial intelligence and and robotics and just became addicted to the format of what are now called MOOCs, massively open online courses.

And I loved them so much that I decided, hey, I want to go do this and help others learn this stuff.

So, hey, let's go join Udacity and build more classes like this.

So I did that for 4 years, led our machine learning robotics and eventually our self-driving car curriculum, which was a lot of fun.

And I got to learn directly from 2 great company builders, like truly great company builders.

He was the operator extraordinaire at Udacity, understood how to build a company, how to build a culture, how to incentivize, and how to do all those things that we don't often talk about.

He, of course, founded the Google Self-Driving Car Project in its early days.

And right now, I believe he's building flying cars.

Just in general, I learned so much from him, but this idea that you are literally in control of your destiny, you can build absolutely anything if you put your mind to it, was always pretty inspirational.

Today, of course, I build self-driving cars at Voyage, and we'll talk more about what makes us special compared to the other self-driving car companies you may have heard of in this class and beyond.

Can you raise your hand if you've heard of Udacity?

Udacity, like I said, was founded by Sebastian Thrun.

He took this class online and it all just exploded and he built a company around it.

Udacity's real focus is on increasing the world's GDP.

This idea that talent is everywhere, that it isn't now just constrained to the best schools in the world, that because of this proliferation of content, there are talented students all over the world.

And all they need is the content in which to be able to build crazy cool world changing things.

And what I see as my job today is to go out into the world and find these ridiculously talented people.

And then put them to work on the hardest problems that exist.

And Udacity to me felt like the perfect place to do this.

As a kind of prelude to this, about 3 years into Udacity, we had had this real focus, like I said, on machine learning and robotics.

But we really want to take it to the next step.

And we came up with this kind of concept internally that we called Only at Udacity.

What if we taught the things that other places weren't teaching?

What if people all around the world could come learn from what may appear to be niche topics, but were just being taught at the right time because that industry is about to blow up.

And the first 1 we did of this, and we've done some after, including flying cars, a much more in-depth curriculum on artificial intelligence, was self-driving cars.

So this is a quick video that introduces it and this is of course Sebastian Thrun, robotics legend.

If we can build a safer car, we would save a lot of lives.

Just imagine, instead of owning a car, you have a little phone and you start the car, and an empty car comes to you, you just jump in, and lets you go somewhere, and when you don't need it anymore, it disappears.

There is an enormous market for self-driving car engineers.

Lots and lots of companies that you wouldn't suspect have entered that field and are massively high-end.

I challenge everybody to be part of this.

We're launching the University in another degree program called Self-Driving Cars so that everyone in the world can become a self-driving car engineer.

It was to accelerate the deployment of self-driving cars.

Like Sebastian says in that video, there's a number of reasons why self-driving cars are transformational.

And at the time, this was around 2016, it felt like self-driving cars were just taking a little bit too long.

We rewind to that particular spot in time.

Google was really the only main effort going on.

And what we believed is that it needed to happen faster and that 1 of the reasons it wasn't happening fast enough is because there wasn't enough talent in the space.

So what we decided to do is, like I said, build something quite special.

We wanted to pair up a world-class curriculum, an actual self-driving car, which we'll talk about more, and what we called our open source challenges.

And all of that would come together to build this quite special curriculum.

1 of our beliefs was that partnering with industry was the right way to go.

That was because it felt, and I believe this, that the knowledge of how to build a self-driving car was not necessarily trapped in academia, it was trapped in industry.

So we had to go straight to industry, work with engineers that were already challenging themselves with these problems, and get them on camera.

Have them teach the concepts that they know and build day in, day out, and have that be transplanted to thousands of minds around the world.

So these are just some of those partners.

But we had a real focus on finding these engineers wherever they may be and getting those folks on camera.

We also built an incredibly talented team.

This is just a small snippet of the curriculum team, but of course, Sebastian Thrun was a big part of this curriculum.

When I told folks that I'd gotten the chance to work with him on specifically self-driving cars, he likened it to getting basketball lessons from Michael Jordan, which I thought was pretty fun.

And they were probably just as entertaining.

But some really, truly great folks working on this curriculum and still doing that to this day, who deserve all of the credit, frankly.

Here's a quick photo of our first lecture recordings with eventual Voyage co-founders Eric and Mac.

Eric, who's on the left, hates this picture, and here's why.

He still isn't at max height, but he still has that box on his desk.

And we built a whole 12-month curriculum to take an intermediate software engineer who may be in consumer software or just some other part of the software world and take them into self-driving cars.

We wanted to cover perception, prediction, planning, localization, controls even, just the whole breadth of the industry.

And the reason we wanted to do that is because we saw the best fit for a Udacity student not necessarily being a specialist in a niche, for example, just perception, although there's been a whole bunch of folks doing that as well, but that the skills of a Udacity student tend to pair themselves well with being a generalist, someone that can contribute all across the stack.

So we try to give these folks that breadth of knowledge.

So here's another quick video of just the curriculum that we built with some previews.

In the first term, you'll build projects on deep learning and computer vision.

For example, you'll build a behavioral cloning project where you drive a car yourself in a simulator, kind of like in a video game, and then you use data from your own driving in the simulator to train a neural network to drive that car for you.

This is the type of project that cutting-edge Silicon Valley startups are working on right now, and it puts you at the forefront of the deep learning in the autonomous vehicle industry.

You'll also build a project to detect and track vehicles in a video stream, just like real autonomous vehicles have to do out in the highway.

In term 2, you'll learn about sensor fusion, localization, and control.

This is hardcore robotics that every self-driving car engineer needs to know in order to actuate and move the vehicle through space.

In the localization module, you'll build a kidnap vehicle project, which takes a vehicle that's lost and figures out where it is in the world with the help of sensor readings and a map.

This is exactly what real self-driving cars have to do every time they turn on in order to figure out where they are in the world.

In the control module, you'll build a model predictive controller, which is a really advanced type of controller that's actually how most self-driving cars move through the world and use the steering wheel, throttle, and brake to follow a set of waypoints or a trajectory to get from 1 point to the next.

In term 3, you'll learn about path planning, you'll have an elective month, and you'll learn about system integration.

Path planning is really the brains of a self-driving car.

It's how the car figures out how to get from 1 point to another, as well as how to react when you meet obstacles in terms of seeing.

I'm going to give you a sneak preview of how path planning works, and this is something that nobody's ever seen before.

There's prediction, which is figuring out what the other vehicles are going to do around us.

There's behavior, which is figuring out what we want to do.

This goes on for a while, so we'll pause it there.

The impact of this curriculum was bigger than we thought it would be.

When we pitched as a small team this idea to Sebastian and to Vish at Udacity, there was a lot of skepticism that something like this was going to be successful.

And the reason that there was that skepticism is that 1 of the kind of formulas that you'd actually looked at to determine the impact of building a certain type of content was the number of open jobs available.

If there was, for example, in web development, mobile development, all that good stuff, there was millions of jobs open.

So it felt like there was a massive opportunity to impact the area.

But if you were to in 2016 search for self-driving car engineers or the different disciplines that exist within, it was just Google.

So it was very interesting just to see the instantaneous reaction that we had to launching this curriculum.

Today over 14,000 successful students from all around the world, as you can see.

Probably the most exciting thing is to see what students have done with this sort of curriculum.

For example, I learned recently that a set of our students here are building a self-driving truck startup in India.

Another set of students in South Korea are building a perception engine for self-driving cars.

Just a whole bunch of folks building truly amazing things.

Not only that, they've gotten jobs at Cruise, Zooks, Waymo, Argo, all the big names and are actively impacting those companies today.

So, we also decided to make that curriculum extra special And we decided to do that by building an actual self-driving car.

And whenever I talked about this internally, you'd ask me people ask me why.

Why go to the, you know, the length of building an actual self-driving car?

And selfishly, some of it was just a personal want to, you know, build a self-driving car.

But the reasoning that I use is that what better way to prove to these students that are putting their faith in us, that we know what we're doing, than to build our own self-driving car.

And also, what better way to collaborate with these students on an area that is really infantile than, again, by having this platform that students could actually run code on a car.

So we decide to buy a car, and we'll talk more about that in a second, but we set ourselves a milestone for our self-driving car.

It was to drive from Mountain View to San Francisco, 32 miles of driving with 0 disengagements.

It won't be 0 disengagements every single time because otherwise we've got an actual self-driving car.

But in a short period of time, how much progress can we make towards this stated goal?

Raise your hand if you've been on El Camino Real in that sort of region.

You probably understand it's got a lot of traffic lights.

In fact, in our route, about 130 traffic lights.

It's multi-lane, 3 lanes, speed limit of about 40, 45, something like that.

So it's fairly complex, but it's also got some constraining factors, which is what we're looking for, too.

You're probably very familiar if you follow self-driving cars with the Lincoln MKZs of the world.

They're everywhere and there's a reason for that in terms of the drive-by-wire nature of the vehicle and other stuff.

And we outfitted a whole bunch of sensors, some cameras, some LIDARs, all that good stuff.

We affectionately called this the Periscope.

I don't know why it's in slow motion, but this was not our final design.

The goal was to accomplish that milestone within 6 months.

So, we of course had to work fast, assembled a dream team of folks that I'd worked with on different projects at Udacity, that also wanted to come dabble in this.

Folks that worked on the machine learning curriculum, robotics curriculum, etc.

And we did this at the Shoreline Amphitheater parking lot, which actually now is a very popular place to test self-driving cars in the Bay Area because Google used to do it in the past.

We saw what I believe to be a motorcycle gang.

We kept iterating, kept building, and it started to come together.

In fact, some stuff that we thought wouldn't work, surprisingly, just started to work.

So Mac discovered that we shouldn't have stopped at that traffic light, but we did.

And of course, We learned a lot by going in this route, the different behaviors of drivers.

1 of the things that we were worried about is vehicles cutting us off.

And when we say cutting us off, it means a vehicle pulling out in front of us, even a few hundred feet in front.

The car was doing very well driving itself.

We built some cool algorithms to change lanes when necessary.

Somewhat you see with Tesla autopilot these days.

We collaborate with some students on a traffic light classifier, which was integrated into ROS there.

can tell Eric was surprised that it was just fine.

And we also had a penchant for building, or for recording themed videos, like you saw maybe from Elon Musk and the Tesla team with Paint It Black.

But we always wanted to test most of the day, just to get the most learnings out of everything.

This video was made at 2.30 AM, driving from Mountain View to San Francisco, all 32 miles.

So it's easier because there's less traffic, right?

This is kind of cheating, and didn't count as the milestone, just to be clear.

You'll see that we eventually hit the 32 miles, and Mac, who's in the driver's seat, is pretty excited about that.

I'm just a boy, I'm a man Everybody's gonna need me I'm just a boy, I'm a man Everybody's gonna need me

But of course, that didn't count because it's in the middle of the night and that's not going to be a very useful route.

But it was an awesome accomplishment just to even make it 32 miles with no disengagements when this traffic lights lane changes, all that good stuff.

But after 4 months, this is in the daytime, this began I believe at like 6, sorry 7 AM, we accomplished it.

That small team had come together and built something pretty cool that could handle, again, multi-lane roadways, varying speed limits, traffic lights, objects, all that good stuff.

And the thing that really brought this home to me is that the industry was now ready.

It felt like this feeling I had in software where someone in their bedroom can go and build something and launch it, almost feeling overnight, could now, not quite the same, but close to the same happen in self-driving cars.

But we'll, we'll talk more about what this led to in a little bit.

We also got the same question, why do this?

And it was clear to me that for something like self-driving cars, which was so formative, we had to collaborate with students to figure out the best stuff.

Because even the folks that were at Udacity were not necessarily the world's leading experts in these topics.

So we wanted to use this hive mind of activity from around the world to teach the best stuff.

So just through a period of a year, these are all the different challenges we launched.

There was prizes and leaderboards and all this sort of fun stuff.

The 1 that I'll focus most on today is using deep learning to predict steering angles.

It was that given a single camera frame, you have to predict the appropriate steering angle of the vehicle.

If anyone had read NVIDIA's end-to-end papers in

this stuff was all the rage, and it felt like 1 of those areas that was just begging for more exploration.

And again, let's use this, all these students from around the world to do it.

And we did have students from all around the world.

There was over a 100 teams, people self-organized into these little groups to go and build this.

And over the course of about 4 months, we had a whole bunch of submissions all taking incredibly different approaches to the problem.

We released data sets, validation sets, all that good stuff.

Here you'll see our V winning model And I later found out that the author of this model actually went on to lead the self-driving car team at Yandex.

Which if you've been following CES is doing some pretty cool stuff in self-driving cars today.

But you'll see this is on a route from the Bay Area to Half Moon Bay, a very windy road, and you'll see that the prediction matches pretty closely to the actual, which is nice.

If you read his description of his solution, It's a pretty cool solution.

I think the most exciting thing was just the number of different approaches to the problem, all resulting in some awesome stuff.

Again, in true Voyage fashion, we recorded a video of what this model performed like on our car.

It wasn't perfect as any first model and just the general approach of camera only driving had its faults.

1 of the main ones that we realized after trying all this stuff out is that, of course, a car, when steered by such an input, performs differently in a car than it does on your desk in a simulator or through prerecorded camera frames.

So adjusting for those corrections that might need to be made is something that students after the fact added, which was pretty cool.

So after all of these things, building that curriculum, building a self-driving car, launching these challenges, It felt like it was time for something new.

It was awesome to go and collaborate with all these students.

And it felt like I had to go build something.

So gathered that same team that had built this curriculum, and we said, we're going to go build a self-driving car.

This is from my pitch at Coastal Ventures.

You can kind of see the pitch deck there a little bit.

Our pitch has changed somewhat through time, but that's still pretty accurate.

And we started what is now called Voyage.

And our goal really was that we wanted to, again, build a self-driving car.

We didn't want to follow the same formula that we felt we'd seen from some of the other folks in the field.

See all Lex Fridman transcripts on Youtube

Oliver Cameron (CEO, Voyage) - MIT Self-Driving Cars