Full episode with Jim Keller (Feb 2020): https://www.youtube.com/watch?v=Nb2tebYAaOA
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Jim Keller is a legendary microprocessor engineer, having worked at AMD, Apple, Tesla, and now Intel. He's known for his work on the AMD K7, K8, K12 and Zen microarchitectures, Apple A4, A5 processors, and co-author of the specifications for the x86-64 instruction set and HyperTransport interconnect.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

All the cost is in the equipment to do it.

And the trend on equipment is, once you figure out how

to build the equipment, the trend of cost is 0.

Elon said, first you figure out what configuration you want the atoms in, and then how to put them there, right?

Because, well, here's the, you know, his great insight is people are how constrained.

I have this thing, I know how it works, and then little tweaks to that will generate something, as opposed to what do I actually want and then figure out how to build it.

Well, let me ask on that topic, you were 1 of the key early people in the development of autopilot, at least in the hardware side.

Elon Musk believes that autopilot and vehicle autonomy, if you just look at that problem, can follow this kind of exponential improvement.

In terms of the how question that we're talking about, there's no reason why it can't.

What are your thoughts on this particular space of vehicle autonomy and your part of it and Elon Musk's and Tesla's vision for

Well the computer you need to build was straightforward.

And you could argue, well does it need to be 2 times faster or 5 times or 10 times?

But that's just a matter of time or price in the short run.

You don't have to be especially smart to drive a car.

I mean the big problem with safety is attention, which computers are really good at, not skills.

Well, let me push back on 1, you see everything you said is correct, but we as humans tend to take for granted how incredible our vision system is.

You can drive a car with 20, 50 vision and you can train a neural network to extract the distance of any object and the shape of any surface from a video and data.

Because it's not just detecting objects, it's understanding the scene, and it's being able to do it in a way that doesn't make errors.

So the beautiful thing about the human vision system and our entire brain around the whole thing is we're able to fill in the gaps.

It's not just about perfectly detecting cars, it's inferring the occluded cars, it's understanding the physics.

So you think what data would compute with improvement of computation, with improvement in collection?

is a, you know, when you're driving a car and somebody cuts you off, your brain has theories about why they did it.

You know, they're a bad person, they're distracted, they're dumb, you know, you can listen to yourself.

So, you know, if you think that narrative is important to be able to successfully drive a car, then current autopilot systems can't do it.

But if cars are ballistic things with tracks and probabilistic changes of speed and direction, and roads are fixed and given by the way, they don't change dynamically.

Right, you can map the world really thoroughly.

You can place every object really thoroughly.

Right, you can calculate trajectories of things really thoroughly.

But everything you said about really thoroughly has a different degree of difficulty.

And you could say at some point, computer autonomous systems will be way better at things that humans are lousy at.

Like, they'll be better at attention, they'll always remember there was a pothole in the road that humans keep forgetting about.

They'll remember that this set of roads has these weirdo lines on it that the computers figured out once.

And especially if they get updates so if somebody changes a given, the key to robots and stuff somebody said is to maximize the givens.

So having a robot pick up this bottle cap is way easier if you put a red dot on the top.

Because then you have to figure out, if you want to do a certain thing with it, maximize the givens is the thing.

And autonomous systems are happily maximizing the givens.

Like humans, When you drive someplace new, you remember it because you're processing it the whole time.

And after the 50th time you drove to work, you get to work, you don't know how you got there.

But the cars have no theories about why they got cut off or why they're in traffic.

Right, so I tend to believe you do have to have theories, meta models of other people, especially with pedestrian and cyclists, but also with other cars.

So everything you said is actually essential to driving.

Driving is a lot more complicated than people realize, I think, so to push back slightly, but to-

You can't just wait for a gap, you have to be somewhat aggressive.

You'd be surprised how simple a calculation for that is.

I may be on that particular point, but there's a, maybe I should have to push back.

You know what, yeah, I'll just say where I stand.

I would be very surprised, but I think you might be surprised how complicated it is.

I tell people, it's like progress disappoints in the short run, surprises in the long run.

I suspect in 10 years it'll be just taken for granted.

like- It's gonna be a $50 solution that nobody cares about.

It's like GPS is like, wow, GPS, we have satellites in space that tell you where your location is.

But I do think that systems that involve human behavior are more complicated than we give them credit for.

So we can do incredible things with technology that don't involve humans, but when you...

I think humans are less complicated than people frequently ascribed.

We tend to operate out of large numbers of patterns and just keep doing it over and over.

But I can't trust you because you're a human.

But My hope is on the point you've made is, even if, no matter who's right, I'm hoping that there's a lot of things that humans aren't good at that machines are definitely good at, like you said, attention, things like that, where they'll be so much better that the overall picture of safety and autonomy will be obviously cars will be safer, even if they're not as good at understanding.

I mean, there are already the current safety systems like cruise control that doesn't let you run into people and lane keeping.

There are so many features that you just look at the Pareto of accidents and knocking off like 80% of them is super doable.

Just to linger on the autopilot team and the efforts there, it seems to be that there's a very intense scrutiny by the media and the public in terms of safety, the pressure, the bar put before autonomous vehicles.

What are your, sort of as a person there working on the hardware and trying to build a system that builds a safe vehicle and so on.

Is it unfair, is it expected of new technology?

I was interested, I talked to both American and European regulators, And I was worried that the regulations would write into the rules technology solutions, like modern brake systems imply hydraulic brakes.

So if you read the regulations, to meet the letter of the law for brakes, it sort of has to be hydraulic, right?

And the regulator said they're interested in the use cases, like a head-on crash, an offset crash, don't hit pedestrians, don't run into people, don't leave the road, Don't run a red light or a stoplight.

And they had all the data about which scenarios injured or killed the most people.

And for the most part, those conversations were like, what's the right thing to do to take the next step.

Now Elon's very interested also in the benefits of autonomous driving are freeing people's time and attention as well as safety.

And I think that's also an interesting thing, but building autonomous systems so they're safe and safer than people seemed, since the goal is to be 10x safer than people, having the bar to be safer than people and scrutinizing accidents seems philosophically correct.

It's different than the things that you worked at, the Intel, AMD, Apple, with autopilot chip design and hardware design.

What are interesting or challenging aspects of building this specialized kind of computing system in the automotive space?

I mean, there's 2 tricks to building like an automotive computer.

1 is the software team, the machine learning team, is developing algorithms that are changing fast.

So as you're building the accelerator, you have this worry or intuition that the algorithms will change enough that the accelerator will be the wrong 1, right?

And there's a generic thing, which is, if you build a really good general purpose computer, say its performance is 1, and then GPU guys will deliver about 5X to performance for the same amount of silicon because instead of discovering parallelism, you're given parallelism.

And then special accelerators get another 2 to 5X on top of a GPU because you say, I know the math is always 8 bit integers into 32 bit accumulators and the operations are the subset of mathematical possibilities.

So, you know, AI accelerators have a claimed performance benefit over GPUs because in the narrow math space, you're nailing the algorithm.

Now, you still try to make it programmable, but the AI field is changing really fast.

So there's a little creative tension there of, I want the acceleration afforded by specialization without being over specialized so that the new algorithm is so much more effective that you'd have been better off on a GPU.

To build a good computer for an application like automotive, there's all kinds of sensor inputs and safety processors and a bunch of stuff.

So 1 of Elon's goals is to make it super affordable.

So some of the recent startups you look at, they have a server in the trunk because they're saying, I'm going to build this autopilot computer, replace it as a driver.

So their cost budget is $10,000 or $20,000.

And Elon's constraint was, I'm gonna put 1 in every car whether people buy autonomous driving or not.

So the cost constraint he had in mind was great.

Right, and to hit that, you had to think about the system design, that's complicated, it's fun.

You know, it's like, it's craftsman's work.

You can say, Stradivarius is this incredible thing, the musicians are incredible, but the guy making the violin, you know, picked wood and sanded it, and then he cut it, you know, and he glued it, and you know, and he waited for the right day so that when he put the finish on it, it didn't, you know, do something dumb.

You may be a genius craftsman because you have the best techniques and you discover a new 1, but most engineers are craftsmen's work.

I used to, I dug ditches when I was in college I got really good at it, satisfying.

So there's an expression called complex mastery behavior.

So when you're learning something, that's fun because you're learning something.

When you do something and throw it in simple, it's not that satisfying.

But if the steps that you have to do are complicated and you're good at them, it's satisfying to do them.

And then if you're intrigued by it all, as you're doing them, you sometimes learn new things that you can raise your game.

And engineers, like engineering is complicated enough that you have to learn a lot of skills.

And then a lot of what you do is then craftsman's work, which is fun.

Autonomous driving, building a very resource constrained computer.

So a computer has to be cheap enough that it's put in every single car.

That's essentially boils down to craftsman's work.

There's thoughtful decisions and problems to solve and trade-offs to make.

You need 10 camera in portrait 8, you know, you're building for the current car or the next 1, you know, how do you do the safety stuff?

You know, there's a whole bunch of details, but it's fun, but it's not like I'm building a new type of neural network which has a new mathematics and a new computer to work.

But the rejection to practice, once you pick the architecture, you look inside and what do you see?

Adders and multipliers and memories and the basics.

So computers is always this weird set of abstraction layers of ideas and thinking that reduction to practice is transistors and wires and pretty basic stuff.

By the way, like factory work, lots of people think factory work is road assembly stuff.

Like the people who work there really like it.

And the car is moving and the parts are moving and sometimes the parts are damaged and you have to coordinate putting all the stuff together and people are good at it.

And I remember 1 day I went to work and The line was shut down for some reason and some of the guys sitting around were really bummed because they had reorganized a bunch of stuff and they were gonna hit a new record for the number of cars built that day and they were all gung-ho to do it.

But what they did was complicated and you couldn't do it.

Well, after a while you could, but you'd have to work your way up because putting the bright, what's called the brights, the trim on a car on a moving assembly line where it has to be attached 25 places in a minute and a half is unbelievably complicated.

And human beings can do it, it's really good.

I think that's harder than driving a car, by the way.

Well, we'll get you in the factory someday and then we'll see how you do.

No, not for us humans driving a car is easy.

I'm saying building a machine that drives a car is not easy.

Driving a car is easy for humans because we've been evolving for billions of years.

Now you join the rest of the unit in mocking me.

I didn't really mock you, I was just intrigued by your anthropology.

Okay, but in general, what have you learned in terms of thinking about passion, craftsmanship, tension, chaos, The whole mess of it, what have you learned, have taken away from your time working with Elon Musk, working at Tesla, which is known to be a place of chaos, innovation, craftsmanship, and

Like, you think you have an understanding about what first principles of something is, and then you talk to Elon about it, and you didn't scratch the surface.

You know, he has a deep belief that no matter what you do, it's a local maximum.

Right, and I had a friend, he invented a better electric motor, and it was like a lot better than what we were using, and 1 day he came by, he said, you know, I'm a little disappointed because this is really great, and you didn't seem that impressed, and I said, you know when the super intelligent aliens come, are they gonna be looking for you?

But doing interesting work that's both innovative and let's say craftsman's work on the current thing is really satisfying and it's good.

And then Elon was good at taking everything apart.

You know, that ability to look at it without assumptions and how constraints is super wild.

And that's super fun, and he's into it too.

Like when they first landed 2 SpaceX rockets to Tesla, we had a video projector in the big room and like 500 people came down and when they landed everybody cheered and some people cried.

And then people say, well, it's chaotic, really?

You think that's not gonna be unbelievably painful?

Do people look back on it and say, boy, I'm really happy I had that experience to go take apart that many layers of assumptions?

So it could be emotionally and intellectually painful, that whole process of just stripping away assumptions.

Yeah, imagine 99% of your thought process is protecting your self-conception.

How do you think you're feeling when you get back into that 1 bit that's useful and now you're open and you have the ability to do something different?

Now, for a long time I've suspected you could get better.

Like you can think better, you can think more clearly, you can take things apart.

Well, here's the other thing is like, I joke like I read books and people think, oh, you read books?

Well, no, I've read a couple of books a week for 55 years.

Well, maybe 50, because I didn't learn to read until I was 8 or something.

And it turns out when people write books, they often take 20 years of their life where they passionately did something, reduced it to 200 pages.

And then you go online and you can find out who wrote the best books and who, you know, that's kind of wild.

So there's this wild selection process and then you can read it and for the most part understand it.

Like I went to 1 company, I thought, I haven't managed much before, so I read 20 management books and I started talking to them and basically compared to all the VPs running around, I'd read 19 more management books than anybody else.

And half the stuff worked, like first time.

But at the core of that is questioning the assumptions or sort of entering the thinking, first principles thinking, sort of looking at the reality of the situation and using that knowledge, applying that knowledge.

Yeah, so I would say my brain has this idea that you can question first assumptions.

And, But I can go days at a time and forget that and you have to kind of like circle back that observation.

Well, it's hard to just keep it front and center because you operate on so many levels all the time and getting this done takes priority or being happy takes priority or screwing around takes priority.

Like how you go through life is complicated.

And then you remember, oh yeah, I could really think first principles.

But you do for a while and that's kind of cool.

So just as a last question in your sense from the big picture, from the first principles, do you think, you kind of answered it already, but do you think autonomous driving is something we can solve on a timeline of years?

So 1235, 10 years, as opposed to a century?

Just to linger on it a little longer, where's the confidence coming from?

Is it the fundamentals of the problem, the fundamentals of building the hardware and the software?

As a computational problem, understanding ballistics, roles, topography, it seems pretty solvable.

And you can see this, like speech recognition for a long time, people are doing frequency and domain analysis and all kinds of stuff, and that didn't work at all, right?

And then they did deep learning about it, and it worked great.

And it took multiple iterations, and autonomous driving is way past the frequency analysis point.

You know, use radar, don't run into things.

And the data gathering is going up and the computation is going up and the algorithm understanding is going up and there's a whole bunch of problems getting solved like that.

The data side is really powerful, but I disagree with both you and Elon.

I'll tell Elon once again as I did before that when you add human beings into the picture, it's no longer a ballistics problem.

It's something more complicated, but I could be very well proven wrong.

Cars are highly damped in terms of rate of change.

Like the steering system's really slow compared to a computer.

The acceleration, the acceleration's really slow.

On a ballistics time scale, but human behavior, I don't know.

Weirdly we operate half a second behind reality.

Nobody really understands that 1 either, it's pretty funny.

And I think with the rate of improvement in all aspects on both the compute and the software and the hardware, there's gonna be pleasant surprises all over the place.

See all Lex Fridman transcripts on Youtube

Jim Keller: Elon Musk and Tesla Autopilot | AI Podcast Clips