Full episode with Jim Keller (Feb 2020): https://www.youtube.com/watch?v=Nb2tebYAaOA
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Jim Keller is a legendary microprocessor engineer, having worked at AMD, Apple, Tesla, and now Intel. He's known for his work on the AMD K7, K8, K12 and Zen microarchitectures, Apple A4, A5 processors, and co-author of the specifications for the x86-64 instruction set and HyperTransport interconnect.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

For over 50 years now, Moore's Law has served, for me and millions of others, as an inspiring beacon of what kind of amazing future brilliant engineers can build.

I'm just making your kids laugh all of today.

So first, in your eyes, what is Moore's Law if you could define for people who don't know?

Well, the simple statement was, from Gordon Moore, was double the number of transistors every 2 years.

And then my operational model is, we increase the performance of computers by 2x every 2 or 3 years.

And it's wiggled around substantially over time.

And also in how we deliver performance has changed.

So the foundational idea was to access the transistors every 2 years.

The current cadence is something like, they call it a shrink factor, like .6 every 2 years, which is not

But that's referring strictly, again, to the original definition of just.

A shrink factor, just getting them smaller, smaller, smaller.

If you make the transistors smaller by .6, then you get 1 over .6 more transistors.

What's the broader, what do you think should be the broader definition of Moore's Law?

When you mentioned how you think of performance, just broadly, what's a good way to think about Moore's Law?

Well, first of all, I've been aware of Moore's Law for 30 years.

You're just watching it before your eyes kind of thing.

Well, and somewhere where I became aware of it, I was also informed that Moore's Law was gonna die in 10 to 15 years.

And I thought that was true at first, but then after 10 years, it was gonna die in 10 to 15 years.

And then at 1 point it was gonna die in 5 years, and then it went back up to 10 years, and at some point I decided not to worry about that particular prognostication for the rest of my life, which is fun.

And then I joined Intel, and everybody said Moore's Law is dead.

And I thought that's sad because it's the Moore's Law company.

And it's not dead, and it's always been gonna die.

And humans like these apocryphal kind of statements like we'll run out of food or we'll run out of air or run out of room or run out of something.

Right, but it's still incredible that it's lived for as long as it has.

And yes, there's many people who believe now that Moore's Law is dead.

You know, they can join the last 50 years of people who had the same idea.

Yeah, there's a long tradition, but why do you think, if you can try to understand it, why do you think it's not dead currently?

Let's just think, People think Moore's law is 1 thing, transistors get smaller.

But actually under the sheet, there's literally thousands of innovations.

And almost all those innovations have their own diminishing return curves.

So if you graph it, it looks like a cascade of diminishing return curves.

I don't know what to call that, but the result is an exponential curve.

So if you're an expert in 1 of the things on a diminishing return curve, and you can see its plateau, you will probably tell people.

Well, this is this is done Meanwhile, some other pilot people are or doing something different So that's that's just normal.

So then there's the observation of how small could a switching device be?

So a modern transistor is something like a thousand by a thousand by a thousand atoms, right?

And you get quantum effects down around 2 to 10 atoms.

So you can imagine a transistor as small as 10 by 10 by 10.

And then the quantum computational people are working away at how to use quantum effects.

Well a fin, like a modern transistor, if you look at the fin, it's like 120 atoms wide, but we can make that thinner, and then there's a gate wrapped around it, and then there's spacing, there's a whole bunch of geometry and a competent transistor designer could count both atoms in every single direction.

Like there's techniques now to already put down atoms in a single atomic layer.

It's just, you know, from a manufacturing process, if placing an atom takes 10 minutes and you need to put 10 to the 23rd atoms together to make a computer, it would take a long time.

So the methods are both shrinking things and then coming up with effective ways to control what's happening.

There's equipment, there's optics, there's chemistry, there's physics, there's material science, there's metallurgy, there's lots of ideas about when you put different materials together, how do they interact, are they stable, is it stable over temperature, are they repeatable?

There's literally thousands of technologies involved.

But just for the shrinking, you don't think we're quite yet close to the fundamental limits of physics.

I did a talk on Moore's Law and I asked for a roadmap to a path of 100 and after 2 weeks, they said we only got to 50.

To 50 and I said, why don't you give it another 2 weeks?

Well, here's the thing about Moore's Law, right?

So I believe that the next 10 or 20 years of shrinking is gonna happen, right?

Now, as a computer designer, you have 2 stances.

You think it's going to shrink, in which case you're designing and thinking about architecture in a way that you'll use more transistors.

Or conversely, not be swamped by the complexity of all the transistors you get.

So you're open to the possibility and waiting for the possibility of a whole new army of transistors ready to work.

I'm expecting more transistors every 2 or 3 years by a number large enough that how you think about design, how you think about architecture has to change.

Like imagine you build buildings out of bricks, and every year the bricks are half the size, or every 2 years.

Well, if you kept building bricks the same way, you know, so many bricks per person per day, the amount of time to build a building would go up exponentially.

But if you said, I know that's coming, so now I'm gonna design equipment that moves bricks faster, uses them better, because maybe you're getting something out of the smaller bricks, more strength, thinner walls, you know, less material efficiency out of that.

So once you have a roadmap with what's going to happen, transistors, we're going to get more of them, then you design all this collateral around it to take advantage of it, and also to cope with it.

Like that's the thing people don't understand, it's like if I didn't believe in Moore's Law, and then Moore's Law transistors showed up, my design teams were all drowned.

So what's the hardest part of this influx of new transistors?

I mean, even if you just look historically, throughout your career, what's the thing, what fundamentally changes when you add more transistors in the task of designing an

By the way, there's some science showing that we do get smarter because of nutrition, whatever.

Nobody understands it, nobody knows if it's still going on.

I would believe for the most part, people aren't getting much smarter.

So human beings, we're really good in teams of 10, up to teams of 100, they can know each other.

Beyond that, you have to have organizational boundaries.

So you're kind of, you have, those are pretty hard constraints, right?

So then you have to divide and conquer, like as the designs get bigger, you have to divide it into pieces.

You know, the power of abstraction layers is really high.

We used to build computers out of transistors.

Now we have a team that turns transistors into logic cells, and another team that turns them into functional units, and another 1 that turns them into computers.

Right, so we have abstraction layers in there.

And you have to think about when do you shift gears on that.

We also use faster computers to build faster computers.

So some algorithms run twice as fast on new computers, but a lot of algorithms are N squared.

So a computer with twice as many transistors in it might take 4 times as long to run, so you have to refactor the software.

Like simply using faster computers to build bigger computers doesn't work.

So you have to think about all these things.

So in terms of computing performance and the exciting possibility that more powerful computers bring, is shrinking the thing we've just been talking about.

1 of the, for you, 1 of the biggest exciting possibilities of advancement in performance, or is there other directions that you're interested in?

Like, like, in the direction of sort of enforcing given parallelism, or like doing massive parallelism in terms of many, many CPUs, stacking CPUs on top of each other, that kind of parallelism, or any kind of parallelism?

So, old computers, you know, slow computers, you said A equal B plus C times D.

And then we made faster computers with vector units, and you can do proper equations and matrices, right?

And then modern like AI computations or like convolutional neural networks where you convolve 1 large data set against another.

And so there's sort of this hierarchy of mathematics, you know, from simple equation to linear equations to matrix equations to deeper kind of computation.

And the data sets are getting so big that people are thinking of data as a topology problem.

You know, data is organized in some immense shape.

And then the computation, which sort of wants to be, get data from immense shape and do some computation on it.

So what computers have allowed people to do is have algorithms go much, much further.

So that paper you referenced, the Sutton paper, they talked about, you know, like when AI started, it was apply rule sets to something.

That's a very simple computational situation.

And then when they did first chess thing, they solved deep searches.

So have a huge database of moves and results, deep search, but it's still just a search.

Now we take large numbers of images and we use it to train these weight sets that we convolve across.

It's a completely different kind of phenomena.

Now they're doing the next generation, and if you look at it, they're going up this mathematical graph, right?

And then computations, both computation and data sets support going up that graph.

Yeah, the kind of computation that might, I mean, I would argue that all of it is still a search.

Just Like you said, a topology problem of data sets, he's searching the data sets for valuable data and also the actual optimization of neural networks is a kind of search for the...

I don't know, if you had looked at the inner layers of finding a cat, it's not a search.

So, you know, a projection, here's a shadow of this phone.

And then you can have a shadow of that onto something, a shadow on that of something.

If you look in the layers, you'll see this layer actually describes pointy ears and round eyedness and fuzziness.

But the computation to tease out the attributes is not search.

The inference part might be search, but the training is not search.

Then in deep networks, they look at layers and they don't even know it's represented.

And yet if you take the layers out, it doesn't work.

But you'd have to talk to a mathematician about what that actually is.

Well, we could disagree, but it's just semantics, I think, it's not, but it's certainly not.

I would say it's absolutely not semantics, but.

Okay, all right, well, if you wanna go there.

So optimization to me is search, And we're trying to optimize the ability of a neural network to detect cat ears.

And the difference between chess and the space, the incredibly multi-dimensional, hundred thousand dimensional space that neural networks are trying to optimize over is nothing like the chessboard database.

So it's a totally different kind of thing.

Okay, in that sense you can say it loses the meaning.

The funny thing is, it's the difference between given search space and found search space.

But You're saying, what's your sense in terms of the basic mathematical operations and the architectures, compute hardware that enables those operations?

Do you see the CPUs of today still being a really core part of executing those mathematical operations?

Yes, well the operations continue to be add, subtract, load, store, compare, and branch.

So it's interesting that the building blocks of computers are transistors, and under that, atoms.

So you got atoms, transistors, logic gates, computers, right, functional units of computers.

The building blocks of mathematics at some level are things like adds and subtracts and multiplies, but the space mathematics can describe is I think essentially infinite.

But the computers that run the algorithms are still doing the same things.

Now, a given algorithm might say, I need sparse data, or I need 32-bit data, or I need, you know, like a convolution operation that naturally takes eight-bit data, multiplies it and sums it up a certain way.

So the, like the data types in TensorFlow imply an optimization set, but when you go write down a look at the computers, it's Anand Orgait's doing adds and multiplies.

Now the quantum researchers think they're gonna change that radically.

And then there's people who think about analog computing because you look in the brain and it seems to be more analogish, that maybe there's a way to do that more efficiently.

But we have a million X on computation and I don't know the relationship between computational, let's say intensity and ability to hit mathematical abstractions.

I don't know any way to describe that, but just like you saw in AI, you went from rule sets to simple search, to complex search, to say found search.

Like those are orders of magnitude more computation to do.

And as we get the next 2 orders of magnitude, like a friend Roger Godori said, like every order of magnitude changes the computation.

Fundamentally changes what the computation is doing.

Oh, you know the expression, the difference in quantity is the difference in kind.

The difference between ant and anthill, right?

You know, there's this indefinable place where the quantity changed to quality, right?

And we've seen that happen in mathematics multiple times and you know, my guess is it's going to keep happening.

So in your sense, yeah, if you focus head down and shrinking the transistor.

Well, it's not just head down, we're aware of the software stacks that are running and the computational loads, and we're kind of pondering what do you do with a petabyte of memory that wants to be accessed in a sparse way and have the kind of calculations AI programmers want.

So there's a dialogue interaction, but when you go in the computer chip, you find adders and subtractors and multipliers.

So if you zoom out then with, as you mentioned, Rich Sutton, the idea that most of the development in the last many decades in AI research came from just leveraging computation and just simple algorithms waiting for the computation to improve.

Well, software guys have a thing that they call the problem of early optimization.

So you write a big software stack, and if you start optimizing the first thing you write, the odds of that being the performance limiter is low.

But when you get the whole thing working, can you make it 2X faster by optimizing the right things?

While you're optimizing that, could you have written a new software stack, which would have been a better choice?

But the whole time as you're doing the writing, that's the software we're talking about.

If Moore's law is going to continue, then your AI research should expect that to show up.

And then you make a slightly different set of choices.

And from here, it's just us rewriting algorithms.

That seems like a failed strategy for the last 30 years of Moore's Law's death.

I think you've answered it, but I'll just ask

So why do you think Moore's Law is not going to die?

Which is the most promising, exciting possibility of why it won't die in the next 5, 10 years?

So is it the continued shrinking of the transistor or is it another S-curve that steps in and it totally sort of.

Well, shrinking the transistor is literally thousands of innovations.

There's a whole bunch of S-curves just kind of running their course and being reinvented and new things.

You know, the semiconductor fabricators and technologists have all announced what's called nanowires, so they took a fin which had a gate around it and turned that into little wires so you have better control of that and they're smaller.

And then from there there's some obvious steps about how to shrink that.

So the metallurgy around wire stacks and stuff has very obvious abilities to shrink.

And there's a whole combination of things there to do.

Your sense is that we're going to get a lot if this innovation from just that shrinking.

Yeah, like a factor of a hundred, it's a lot.

Now you're smart and you might know, but to me it's totally unpredictable of what that 100x would bring in terms of the nature of the computation that people would be.

So for a long time, it was mainframes, minis, workstation, PC, mobile.

Moore's Law drove faster, smaller computers.

And then when we were thinking about Moore's Law, Rajagirdari said every 10x generates a new computation.

So scalar, vector, matrix, topological computation.

And if you go look at the industry trends, there was mainframes and mini computers and PCs, and then the internet took off, and then we got mobile devices, and now we're building 5G wireless with 1 millisecond latency.

And people are starting to think about the smart world where everything knows you, recognizes you.

How does it make you feel that you're 1 of the key architects of this kind of future?

So we're not talking about the architects of the high level people who build the Angry Bird apps and Snapchat.

Who knows, maybe that's the whole point of the universe.

gonna take a stand at that and the attention distracting nature of mobile phones.

The side effects of smartphones or the attention distraction, which part?

Well, who knows where this is all leading?

My parents used to yell at my sisters for hiding in the closet with a wired phone with a dial on it.

Now my wife yells at my kids for talking to their friends all day on text.

See all Lex Fridman transcripts on Youtube

Moore's Law is Not Dead (Jim Keller) | AI Podcast Clips