Jim Keller is a legendary microprocessor engineer, previously at AMD, Apple, Tesla, Intel, and now Tenstorrent. Please support this podcast by checking out our sponsors:
- Athletic Greens: https://athleticgreens.com/lex and use code LEX to get 1 month of fish oil
- Brooklinen: https://brooklinen.com and use code LEX to get $25 off + free shipping
- ExpressVPN: https://expressvpn.com/lexpod and use code LexPod to get 3 months free
- Belcampo: https://belcampo.com/lex and use code LEX to get 20% off first order

EPISODE LINKS:
Jim's Twitter: https://twitter.com/jimkxa
Jim's Wiki: https://en.wikipedia.org/wiki/Jim_Keller_(engineer)
Tenstorrent: https://www.tenstorrent.com/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
1:33 - Good design is both science and engineering
7:33 - Javascript
11:40 - RISC vs CISC
15:39 - What makes a great processor?
17:09 - Intel vs ARM
18:58 - Steve Jobs and Apple
21:36 - Elon Musk and Steve Jobs
27:21 - Father
31:03 - Perfection
37:18 - Modular design
42:52 - Moore's law
49:50 - Hardware for deep learning
56:44 - Making neural networks fast at scale
1:04:22 - Andrej Karpathy and Chris Lattner
1:08:36 - How GPUs work
1:12:43 - Tesla Autopilot, NVIDIA, and Mobileye
1:17:23 - Andrej Karpathy and Software 2.0
1:23:43 - Tesla Dojo
1:26:20 - Neural networks will understand physics better than humans
1:28:33 - Re-engineering the human brain
1:33:26 - Infinite fun and the Culture Series by Iain Banks
1:35:20 - Neuralink
1:40:43 - Dreams
1:44:37 - Ideas
1:54:49 - Aliens
1:59:46 - Jordan Peterson
2:04:44 - Viruses
2:07:52 - WallStreetBets and Robinhood
2:15:55 - Advice for young people
2:17:45 - Human condition
2:20:14 - Fear is a cage
2:25:04 - Love
2:31:27 - Regrets

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Jim Keller, his second time in the podcast.

Jim is a legendary microprocessor architect and is

widely seen as 1 of the greatest engineering minds of the computing age.

In a peculiar twist of space-time in our simulation.

Jim is also a brother-in-law of Jordan Peterson.

We talk about this and about computing, artificial intelligence, consciousness, and life.

Athletic Greens All-in-One Nutrition Drink, Brooklyn and Sheets, ExpressVPN, and Belcampo Grass-Fed Meat.

Click the sponsor links to get a discount and to support this podcast.

say that Jim is someone who, on a personal level, inspired me to be myself.

There was something in his words, on and off the mic, or perhaps that he even paid attention to me at all, that almost told me, you're all right, kid.

A kind of pat on the back that can make the difference between a mind that flourishes and a mind that is broken down by the cynicism of the world.

So I guess that's just my brief few words of thank you to Jim and in general, gratitude for the people who have given me a chance on this podcast, in my work, and in life.

If you enjoy this thing, subscribe on YouTube, review it on Apple Podcasts, follow on Spotify, support it on Patreon, or connect with me on Twitter, Alex Friedman.

And now, here's my conversation with Jim Keller.

What's the value and effectiveness of theory versus engineering, this dichotomy, in building good software or hardware systems?

By engineering do you mean, you know, reduction of practice of known methods?

And then science is the pursuit of discovering things that people don't understand.

Definitions are interesting here, but I was thinking more in theory, constructing models that kind of generalize about how things work.

And engineering is actually building stuff, the pragmatic, like, okay, we have these nice models, but how do we actually get things to work?

Like economists have all these models of how the economy works and how different policies will have an effect, but then there's the actual, okay, let's call it engineering, of like actually deploying the policies.

So computer design is almost all engineering and reduction of practice in known methods.

Now, because of the complexity of the computers we built, you know, you could think you're, well, we'll just go write some code, and then we'll verify it, and then we'll put it together.

And then you find out that the combination of all that stuff is complicated.

And then you have to be inventive to figure out how to do it.

Right, so that's definitely happens a lot.

And then, every so often some big idea happens, but it might be 1 person.

And that idea is in what, in the space of engineering or is it in the space of- Well,

So 1 of the limits of computer performance is branch prediction.

So, and there's a whole bunch of ideas about how good you could predict a branch.

And people said, there's a limit to it, it's an asymptotic curve.

And somebody came up with a better way to do branch prediction.

And every computer in the world now uses it.

So the engineers who build branch prediction hardware were happy to drop the 1 kind of training array and put it in another 1.

And branch prediction is 1 of the key problems underlying all of sort of the lowest level of software.

Computers are limited by, you know, single-thread computers are limited by 2 things.

The predictability of the path of the branches and a predictability of the locality of data.

So we have predictors that now predict both of those pretty well.

So memory is a couple hundred cycles away, local cache is a couple cycles away.

When you're executing fast, virtually all the data has to be in the local cache.

So a simple program says, add 1 to every element in an array.

It's really easy to see what the stream of data will be.

But you might have a more complicated program that says, get an element of this array, look at something, make a decision, go get another element, it's kind of random.

And you can think that's really unpredictable.

And then you make this big predictor that looks at this kind of pattern and you realize, well, if you get this data and this data, then you probably want that 1.

And if you get this 1 and this 1 and this 1, you probably want that 1.

And is that theory or is that engineering?

Like the paper that was written, was it asymptotic kind of discussion or is it more like here's a hack that works well?

Like there's information theory in it, I think, somewhere.

Yeah, but once you know the method, implementing it is an engineering problem.

Now there's a flip side of this, which is in a big design team, what percentage of people think their plan or their life's work is engineering versus inventing things.

So lots of companies will reward you for filing patents.

Some many big companies get stuck because to get promoted you have to come up with something new.

And then what happens is everybody's trying to do some random new thing,

of which doesn't matter, and the basics get neglected.

They think like the cell library and the basic CAD tools, you know, or basic, you know, software validation methods.

You know, they want to work on the exciting stuff.

And then they spend lots of time trying to figure out how to patent something.

But the breakthroughs are in the simple stuff.

No, no, you have to do the simple stuff really well.

If you're building a building out of bricks, you want great bricks.

So you go to 2 places that sell bricks, and 1 guy says, yeah, they're over there in an ugly pile.

And the other guy is like lovingly tells you about the 50 kinds of bricks and how hard they are and how beautiful they are and how square they are.

And which 1 are you gonna buy bricks from?

So you're talking about the craftsman, the person who understands bricks, who loves bricks, who loves

And when you start thinking engineering is about invention and set up a system that rewards invention, the craftsmanship gets neglected.

Okay, so maybe 1 perspective is the theory, the science over-emphasizes invention, and engineering emphasizes craftsmanship, and therefore, so if you, it doesn't matter what you do, theory, engineering.

They're always talking about some breakthrough or innovation, and everybody thinks that's the most important thing.

But the number of innovative ideas is actually relatively low.

And innovation creates a whole new opportunity.

Like when some guy invented the internet, right?

The million people that wrote software against that were mostly doing engineering software writing.

So the elaboration of that idea was huge.

I don't know if you know Brendan Eich, he wrote JavaScript in 10 days, and that's an interesting story.

It makes me wonder, and it was famously for many years considered to be a pretty crappy programming language, Still is perhaps, it's been improving sort of consistently.

But the interesting thing about that guy is, you know, he doesn't get any awards.

You don't get a Nobel Prize or a Fields Medal or...

That is currently the number 1 programming language in the world and runs, now is increasingly running the back end of the internet.

Was it the right thing at the right time?

Because like when stuff like JavaScript came out, like there was a move from, you know, writing C programs and C++ to let's call it what they call managed code frameworks, where you write simple code, it might be interpreted, it has lots of libraries, productivity is high, you don't have to be an expert.

So Java was supposed to solve all the world's problems, it was complicated, JavaScript came out after a bunch of other scripting languages.

I'm not an expert on it, but was it the right thing at the right time, or was there something clever, because he wasn't the only 1.

And maybe if he figured out what it was, then he'd get a prize.

Like that's, Yeah, you know, maybe his problem is he hasn't defined this.

Well, I think there was a bunch of blog posts written about it, which is like, wrong is right, which is like doing the crappy thing fast, just hacking together the thing that answers some of the needs and then iterating over time, listening to developers, like listening to people who actually use the thing.

This is something you can do more in software.

You have to have a good instinct of when is the right time for the right tool and make it super simple and just get it out there.

The problem is, this is true with hardware, this is less true with software, is there's a backward compatibility that just drags behind you as you try to fix all the mistakes of the past.

But the timing, there's something about that.

You have to give yourself over to the, you have to have this broad sense of what's needed now, both scientifically and the community.

And it's just like, it was obvious that there was no, The interesting thing about JavaScript is everything that ran in the browser at the time, like Java and I think other like scheme, other programming languages, they were all in a separate external container.

And then JavaScript was literally just injected into the webpage.

It was the dumbest possible thing running in the same thread as everything else.

So JavaScript code is inserted as a comment in the HTML code.

It's so- Right, so it has no apparatus for a virtual machine and container, it just executed in the framework of the program

And then because something about that accessibility, the ease of its use, resulted in then developers innovating of how to actually use it.

I mean, I don't even know what to make of that, but it does seem to echo across different software, like stories of different software.

PHP has the same story, really crappy language.

Well, we used to have a joke that the random length instructions, variable length instructions, that's always won, even though they're obviously worse.

X86 is arguably the worst architecture on the planet.

Well I mean isn't that also the story of risk versus sys?

There's something about simplicity that us in this evolutionary process is valued.

If it's simple, it spreads faster, it seems like.

Yeah, it could be simple is good, but too simple is bad.

What's RISC, what's CISC, and who's gonna win in that space, in these instruction sets?

AI software's gonna win, but there'll be little computers that run little programs like normal all over the place.

But we're going through another transformation, so.

But you think instruction sets underneath it all will change?

I mean, the limits of performance are, you know, predictability of instructions and data.

And then the usability of it is some, you know, quality of design, quality of tools, availability.

Like right now, x86 is proprietary with Intel and AMD, but they can change it any way they want independently.

Arm is proprietary to ARM, and they won't let anybody else change it.

And RISC-V is open source, so anybody can change it, which is super cool.

But that also might mean it gets changed in too many random ways that there's no common subset of it that people can use.

Like if you were to bet all your money on 1 or the other, risk 5 versus it?

Well, x86, oddly enough, when Intel first started developing it, they licensed it to like 7 people.

And then they move faster than others and also bought 1 or 2 of them.

But there was 7 different people making x86 because at the time there was 6502 and Z80s and

And you could argue everybody thought Z80 was the better instruction set, but that was proprietary to 1 place.

So there's like 4 or 5 different microprocessors.

Intel went open, got the market share, because people felt like they had multiple sources from it.

And then over time it narrowed down to 2 players.

So why, you as a historian, why did Intel win for so long with their processors?

So it's just looking back to JavaScript and Brennan Eich is a Microsoft and Netscape and all these internet browsers.

Microsoft won the browser game because they aggressively stole other people's ideas, like right after they did it.

You know, I don't know if Itel was stealing other people's ideas.

They started making RAMs, random access memories.

And then at the time when the Japanese manufacturers came up, they were getting out competed on that and they pivoted the microprocessors and they made the first integrated microprocessor for programs.

Right, and then they led semiconductor industry.

IBM, all kinds of big companies had boatloads of money.

Their processor designs were pretty good.

I think the Core 2 was probably the first 1 I thought was great.

It was a really fast processor, and then Haswell was great.

And both- What makes a great processor in that?

Oh, if you just look at it, it's performance versus everybody else.

It's the size of it, the usability of it.

So it's not specific, some kind of element that makes it beautiful, it's just like literally just raw performance.

Well, there's the fastest in an environment.

for years you made the fastest 1 you could and then people started to have power limits.

So then you made the fastest at the right PowerPoint.

And then when we started doing multi-processors, like if you could scale your processors more than the other guy, you could be 10% faster on like a single thread, but you have more threads.

And then ARM really explored like, you know, they have the A-series and the R-series and the M-series, like a family of processors for all these different design points from like unbelievably small and simple.

And so then when you're doing the design, it's sort of like this big palette of CPUs.

Like they're the only ones with a credible, you know, top to bottom palette.

What do you mean a credible top to bottom?

Well, there's people who make microcontrollers that are small, but they don't have a fast 1.

There's people who make fast processors, but don't have a medium 1 or a small 1.

So what's the difference in the ARM folks and Intel in terms of the way they're approaching this problem?

Well, Intel, almost all their processor designs were very custom, high-end, for the last 15, 20 years.

And the architecture, they're really good.

But the company itself was fairly insular to what's going on in the industry with CAD tools and stuff.

And there's this debate about custom design versus synthesis, and how do you approach that?

I'd say Intel was slow on getting to synthesized processors.

See all Lex Fridman transcripts on Youtube

Jim Keller: The Future of Computing, AI, Life, and Consciousness | Lex Fridman Podcast #162