Sertac Karaman is a professor at MIT, co-founder of the autonomous vehicle company Optimus Ride, and is one of top roboticists in the world, including robots that drive and robots that fly.

Support this podcast by signing up with these sponsors:
- Cash App - use code "LexPodcast" and download:
- Cash App (App Store): https://apple.co/2sPrUHe
- Cash App (Google Play): https://bit.ly/2MlvP5w

EPISODE LINKS:
Sertac's Website: http://sertac.scripts.mit.edu/web/
Sertac's Twitter: https://twitter.com/sertackaraman
Optimus Ride: https://www.optimusride.com/

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
1:44 - Autonomous flying vs autonomous driving
6:37 - Flying cars
10:27 - Role of simulation in robotics
17:35 - Game theory and robotics
24:30 - Autonomous vehicle company strategies
29:46 - Optimus Ride
47:08 - Waymo, Tesla, Optimus Ride timelines
53:22 - Achieving the impossible
53:50 - Iterative learning
58:39 - Is Lidar is a crutch?
1:03:21 - Fast autonomous flight
1:18:06 - Most beautiful idea in robotics

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Sertac Karaman, a professor at MIT, co-founder of the autonomous vehicle company Optimus Ride, and is 1 of the top roboticists in the world, including robots that drive and robots that fly.

To me, personally, he has been a mentor, a colleague, and a friend.

He's 1 of the smartest, most generous people I know.

So it was a pleasure and honor to finally sit down with him for this recorded conversation.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, review it with 5 Stars and Apple Podcasts, support it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D-M-A-N.

few minutes of ads now and never any ads in the middle that can break the flow of the conversation.

I hope that works for you and doesn't hurt the listening experience.

This show is presented by Cash App, the number 1 finance app in the App Store.

When you get it, use the code LEXPODCAST.

Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1.

Since Cash App allows you to send and receive money digitally, let me mention a surprising fact about physical money.

It costs 2.4 cents to produce a single penny.

In fact, I think it costs $85 million annually to produce them.

That's a crazy little fact about physical money.

So again, if you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10 and Cash App will also donate $10 to Thirst, an organization that is helping to advance robotics and STEM education for young people around the world.

And now, here's my conversation with Sertac Karaman.

Since you have worked extensively on both, what is the more difficult task?

I think that autonomous flying, just kind of doing it for consumer drones and so on, the kinds of applications that we're looking at right now is probably easier.

And so I think that that's maybe 1 of the reasons why it took off literally a little earlier than the autonomous cars.

But I think if you look ahead, I would think that the real benefits of autonomous flying, unleashing them in transportation, logistics, and so on, I think it's a lot harder than autonomous driving.

So I think my guess is that, we've seen a few kind of machines fly here and there, but we really haven't yet seen any kind of machine, like at massive scale, large scale being deployed and flown and so on.

And I think that's going to be after we kind of resolve some of the large scale deployments of autonomous driving.

What's your intuition behind why at scale when consumer facing drones are tough?

So I think in general, at scale, it's tough.

Like for example, when you think about it, we have actually deployed a lot of robots in the, let's say, the past 50 years.

We as academics or we business entrepreneurs?

And I think that, well, when you think about it, robots, they're autonomous.

They work on their own, but they are either in isolated environments or they are in They may be at scale, but they're really confined to a certain environment that they don't interact so much with humans.

And so they work in, I don't know, factory floors, warehouses, they work on Mars, they are fully autonomous over there.

But I think that the real challenge of our time is to take these vehicles and put them into places where humans are present.

So now I know that there's a lot of human robot interaction type of things that need to be done.

And so that's 1 thing, but even just from the fundamental algorithms and systems and the business cases, or maybe the business models, even like architecture, planning, societal issues, legal issues, there's a whole bunch of pack of things that are related to us putting robotic vehicles into human present environments.

And these humans, they will not potentially be even trained to interact with them.

They may not even be using the services that are provided by these vehicles.

They may not even know that they're autonomous.

They're just doing their thing, living in environments that are designed for humans, not for robots.

And that I think is 1 of the biggest challenges I think of our time, to put vehicles there.

And to go back to your question, I think doing that at scale, meaning you go out in a city and you have thousands or tens of thousands of autonomous vehicles that are going around.

It is so dense to the point where if you see 1 of them, you look around, you see another 1.

And that density, we've never done anything like that before.

And I would bet that that kind of density will first happen with autonomous cars because I think we can bend the environment a little bit.

We can especially kind of making them safe is a lot easier when they're like on the ground.

When they're in the air, it's a little bit more complicated.

But I don't see that there's gonna be a big separation.

I think that there will come a time that we're gonna quickly see these things unfold.

Do you think there will be a time where there's tens of thousands of delivery drones that fill the sky?

You know, I think it's possible, to be honest.

Delivery drones is 1 thing, but you can imagine, for transportation, like an important use case is, we're in Boston, you wanna go from Boston to New York, and you wanna do it from the top of this building to the top of another building in Manhattan, and you're going to do it in 1 and a half hours.

So like you and me be a friend, like almost like an Uber.

In our work in autonomous vehicles, I see that.

So there's kind of like a bit of a need for 1 person transport, but also like a few people.

So you and I could take that trip together.

I think kind of sounds crazy, maybe even sounds a bit cheesy, but I think that those kinds of things are some of the real opportunities.

And I think, you know, it's not like the typical airplane and the airport would disappear very quickly, but I would think that, you know, many people would feel like they would spend an extra $100 on doing that and cutting that 4 hour travel down to 1 1⁄2 hours.

It's like when people imagine the future for 50 plus years, they think flying cars.

It's like all technologies, it's cheesy to think about now because it seems so far away, but overnight it can change.

But just technically speaking, in your view, how feasible is it to make that happen?

I'll get to that question, but just 1 thing is that I think sometimes we think about what's going to happen in the next 50 years.

I mean, we could get what's going to happen in transportation in the next 50 years.

I think there's a 50-50 chance that you can build machines that can ionize the air around them and push it down with magnets and they would fly like a flying saucer.

So it's a bit hard to guess, like when you think about 50 years before.

But I would think that there's this kind of notion where there's a certain type of airspace that we call the agile airspace, and there's good amount of opportunities in that airspace.

So that would be the space that is kind of a little bit higher than the place where you can throw a stone, because that's a tough thing when you think about it.

It takes a kid and a stone to take an aircraft down and then what happens?

But Imagine the airspace that's high enough so that you cannot throw a stone, but it is low enough that you're not interacting with the very large aircraft that are flying several thousand feet above.

And that airspace is underutilized, Or it's actually kind of not utilized at all.

So there's like recreational people kind of fly every now and then, but it's very few.

Like if you look up in the sky, you may not see any of them at any given time.

Every now and then you'll see 1 airplane utilizing that space and you'll be surprised.

And the moment you're outside of an airport a little bit, it just kind of flies off and then it goes out.

And I think utilizing that airspace, the technical challenges there is building in autonomy and ensuring that that kind of autonomy is safe.

Ultimately, I think it is going to be building in complex software, complicated so that it's maybe a few orders of magnitude more complicated than what we have on aircraft today.

And at the same time, just like we insure on aircraft, insuring that it's safe.

And so building that kind of complicated hardware and software becomes a challenge, especially when you build that hardware, I mean you build that software with data.

And so, of course there's some rule-based software in there that kind of do a certain set of things, but then there's a lot of training there.

So you think machine learning will be key to delivering safe vehicles in the future, especially flight?

Not maybe the safe part, but I think the intelligent part.

I mean, there are certain things that we do it with machine learning and it's just there's like right now no other way.

And I don't know how else they could be done.

I mean, we could maybe gather billions of programmers, humans, who program perception algorithms that detect things in the sky and whatever, or we, I don't know, we maybe even have robots like learning a simulation environment and transfer.

And they might be learning a lot better in a simulation environment than a billion humans put their brains together and try to program.

So what's the role of simulations with drones?

How promising, just the very thing you said just now, how promising is the possibility of training and developing a safe flying robot in simulation and deploying it and having that work pretty well in the real world.

I think that a lot of people, when they hear simulation, they will focus on training immediately.

But I think 1 thing that you said, which was interesting, it's developing.

I think simulation environments actually could be key and great for development.

Like for example, there's people in the automotive industry have been using dynamic simulation for like decades now.

And it's pretty standard that you would build and you would simulate.

If you want to build an embedded controller, you plug that kind of embedded computer into another computer, that other computer would simulate and so on.

And I think, fast forward these things, you can create pretty crazy simulation environments.

Like For instance, 1 of the things that has happened recently and that we can do now is that we can simulate cameras a lot better than we used to simulate them.

And that's, I think we just hit the elbow on that kind of improvement.

I would imagine that with improvements in hardware especially and with improvements in machine learning, I think that we would get to a point where we can simulate cameras very, very

Simulate cameras means simulate how a real camera would see the real world, therefore you can explore the limitations of that, you can train perception algorithms on that in simulation, all that kind of stuff.

Exactly, so It has been easier to simulate what we would call interceptive sensors, like internal sensors.

So for example, inertial sensing has been easy to simulate.

It has also been easy to simulate dynamics, like physics that are governed by ordinary differential equations.

I mean, like how a car goes around, maybe how it rolls on the road, how it interacts with the road, or even an aircraft flying around, like the dynamic physics of that.

What has been really hard has been to simulate extraceptive sensors, sensors that kind of like look out from the vehicle.

Laser range finders are a little bit easier.

Cameras, radars are a little bit tougher.

I think once we nail that down, the next challenge I think in simulation will be to simulate human behavior.

Even when you imagine how a human driven car would act around, even that is hard.

But imagine trying to simulate a model of a human just doing a bunch of gestures and so on.

It's not captured with motion capture, but it is simulated.

In fact, today, I get involved a lot with this kind of very high-end rendering projects.

And I have this test that I pass it to my friends or my mom.

I send 2 photos, 2 kind of pictures, and I say, which 1 is rendered, which 1 is real?

And it's pretty hard to distinguish, except I realized, except when we put humans in there.

It's possible that our brains are trained in a way that we recognize humans extremely well, but we don't so much recognize the built environments because built environments sort of came after per se we evolved into sort of being humans, but humans were always there.

Same thing happens, for example, you look at monkeys and you can't distinguish 1 from another, but they sort of do.

And it's very possible that they look at humans, it's kind of pretty hard to distinguish 1 from another, but we do.

And so our eyes are pretty well trained to look at humans and understand if something is off, we will get it.

So in my typical friend test or mom test, what would happen is that we'd put like a human walking in a, in a, in a thing.

And they, they say, you know, this is not right.

I can take the human, and I can show you inside of a building or an apartment.

And it will look like, if we had time to render it, it will look great.

A lot of movies that people are watching, it's all computer generated.

Nowadays, even you watch a drama movie, and like there's nothing going on action wise, but it turns out it's kind of like cheaper, I guess, to render the background, and so they would.

How do we get a human that would pass the mom slash friend test, a simulation of a human walking.

Do you think that's something we can creep up to by just doing kind of a comparison learning where you have humans annotate what's more realistic and not just by watching?

It's hard because a lot of the other things that I mentioned to you, including simulating cameras, it is, the thing there is that we know the physics, we know how it works in the real world and we can write some rules and we can do that.

Like for example, simulating cameras, there's this thing called ray tracing.

I mean, you literally just kind of imagine it's very similar to, it's not exactly the same, but it's very similar to tracing photon by photon.

They're going around, bouncing on things and coming to your eye.

But human behavior, developing a dynamic, like a model of that, that is mathematical so that you can put it into a processor that would go through that, that's going to be hard.

Or another thing that you can do is that, you know, you can show the front test.

You know, you can say this or that and this or that, and that will be labeling.

Anything that requires human labeling, ultimately we're limited by the number of humans that, you know, we have available at our disposal and the things that they can do.

They have to do a lot of other things than also labeling this data.

So that modeling human behavior part is, I think, we're going to realize it's very tough.

And I think that also affects our development of autonomous vehicles.

So you're building self-driving, at the first time, like right after Urban Challenge, I think everybody focused on localization, mapping and localization.

Slam algorithms came in, Google was just doing that.

And so building these HD maps, basically that's about knowing where you are.

And then 5 years later in 2012, 2013 came the kind of coding code AI revolution and that started telling us what everybody else is.

But we're still missing what everybody else is gonna do next.

And so you wanna know where you are, you wanna know what everybody else is, hopefully you know what you're gonna do next and then you wanna predict what other people are going to do and that last bit has been a real challenge.

What do you think is the role, your own, of your, the ego vehicle, the robot, the you, the robotic you, in controlling, in having some control of how the future unrolls, of what's gonna happen in the future.

That seems to be a little bit ignored in trying to predict the future is how you yourself can affect that future by being either aggressive or less aggressive or signaling in some kind of way, sort of this kind of game theoretic dance seems to be ignored for the moment.

I mean, it's quite interesting actually like how we interact with things versus we interact with humans.

So if you see a vehicle that's completely empty and it's trying to do something, all of a sudden it becomes a thing.

And so you can throw your backpack or you can kick it, put your feet on it and things like that.

But when it's a human, there's all kinds of ways of interacting with a human.

So if you and I are face to face, we're very civil, We talk and we understand each other for the most part.

But the thing is that, like for example, you and I might interact through YouTube comments and the conversation may go at a totally different angle.

And so I think people kind of abusing these autonomous vehicles is a real issue in some sense.

And so when you're an ego vehicle, you're trying to coordinate your way, make your way, it's actually kind of harder than being a human.

You not only need to be as smart as humans are, but you also, you're a thing.

So they're going to abuse you a little bit.

So you need to make sure that you can get around and do something.

So I, in general, believe in that sort of game theoretic aspects.

I've actually personally have done quite a few papers, both on that kind of game theory and also this kind of understanding people's social value orientation, for example.

Some people are aggressive, some people not so much.

And a robot could understand that by just looking at how people drive.

And as they kind of come and approach, you can actually understand if someone is gonna be aggressive or not as a robot.

Well, in terms of predicting what they're going to do, the hard question is you as a robot, should you be aggressive or not when faced with an aggressive robot?

Right now it seems like aggressive is a very dangerous thing to do because it's costly from a societal perspective, how you're perceived.

People are not very accepting of aggressive robots in modern society.

And so, I'm not entirely sure how to go about, but I know for a fact that how these robots interact with other people in there is going to be...

And that interaction is always going to be there.

I mean, you could be interacting with other vehicles or other just people kind of like walking around.

And like I said, the moment there's nobody in the seat, it's like an empty thing just rolling off the street.

It becomes no different than any other thing that's not human.

And maybe abuse is the wrong word, but people, maybe rightfully even, they feel like this is a human present environment.

It's designed for humans to be, and they want to own it.

And then the robots, they would need to understand it and they would need to respond in a certain way.

And I think that this actually opens up quite a few interesting societal questions for us as we deploy, like we talk, robots at large scale.

So what would happen when we try to deploy robots at large scale, I think, is that we can design systems in a way that they're very efficient, or we can design them that they're very sustainable.

But ultimately the sustainability efficiency trade-offs, they're going to be right in there and we're going to have to make some choices.

We're not going to be able to just kind of put it aside.

So for example, we can be very aggressive and we can reduce transportation delays, increase capacity of transportation, or we can be a lot nicer and allow other people to kind of quote unquote, own the environment and live in a nice place and then efficiency will drop.

So when you think about it, I think sustainability gets attached to energy consumption or environmental impact immediately.

And those are there, but livability is another sustainability impact.

So you create an environment that people want to live in.

And if robots are going around being aggressive, you don't want to live in that environment, maybe.

However, you should note that if you're not being aggressive, then you're probably taking up some delays in transportation and this and that.

And I think this choice has always been there in transportation, but I think the more autonomy comes in, the more explicit the choice becomes.

Yeah, and when it becomes explicit, then we can start to optimize it, and then we'll get to ask the very difficult societal questions of what do we value more, efficiency or sustainability?

I think we're gonna have to like, I think that the interesting thing about the whole autonomous vehicles question, I think, is also kind of, I think a lot of times we have focused on technology development, like hundreds of years, and the products somehow followed and then we got to make these choices and things like that.

But this is a good time that you even think about autonomous taxi type of deployments and the systems that would evolve from there.

And you realize the business models are different.

Urban planning, you get into regulations.

And then you get into these issues that you didn't think about before, but sustainability and ethics is right in the middle of it.

Even testing autonomous vehicles, think about it.

You're testing autonomous vehicles in human present environments.

The risk may be very small, but still, it's a strictly greater than 0 risk that you're putting people into.

And so then you have that innovation risk trade-off that you're in that somewhere.

And we understand that pretty well now is that if we don't test, at least the development will be slower.

I mean, it doesn't mean that we're not going to be able to develop.

I think it's going to be pretty hard actually.

But the thing is that those kinds of trade-offs we already are making.

And as these systems become more ubiquitous, I think those trade-offs will just really hit.

So you are 1 of the founders of Optimus Ride and Thomas vehicle company.

We'll talk about it, but let me on that point, ask maybe good examples, keeping Optimus Ride out of this question, sort of exemplars of different strategies on the spectrum of innovation and safety or caution.

So like Waymo, Google self-driving car, Waymo represents maybe a more cautious approach.

And then you have Tesla on the other side, headed by Elon Musk, that represents a more, however, which adjective you wanna use, aggressive, innovative, I don't know.

But What do you think about the difference between the 2 strategies in your view?

What's more likely, what's needed and is more likely to succeed in the short term and the long term?

See all Lex Fridman transcripts on Youtube

Sertac Karaman: Robots That Fly and Robots That Drive | Lex Fridman Podcast #97