David Patterson is a Turing award winner and professor of computer science at Berkeley. He is known for pioneering contributions to RISC processor architecture used by 99% of new chips today and for co-creating RAID storage. The impact that these two lines of research and development have had on our world is immeasurable. He is also one of the great educators of computer science in the world. His book with John Hennessy "Computer Architecture: A Quantitative Approach" is how I first learned about and was humbled by the inner workings of machines at the lowest level.

Support this podcast by signing up with these sponsors:
- Jordan Harbinger Show: https://jordanharbinger.com/lex/
- Cash App - use code "LexPodcast" and download:
- Cash App (App Store): https://apple.co/2sPrUHe
- Cash App (Google Play): https://bit.ly/2MlvP5w

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
3:28 - How have computers changed?
4:22 - What's inside a computer?
10:02 - Layers of abstraction
13:05 - RISC vs CISC computer architectures
28:18 - Designing a good instruction set is an art
31:46 - Measures of performance
36:02 - RISC instruction set
39:39 - RISC-V open standard instruction set architecture
51:12 - Why do ARM implementations vary?
52:57 - Simple is beautiful in instruction set design
58:09 - How machine learning changed computers
1:08:18 - Machine learning benchmarks
1:16:30 - Quantum computing
1:19:41 - Moore's law
1:28:22 - RAID data storage
1:36:53 - Teaching
1:40:59 - Wrestling
1:45:26 - Meaning of life

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with David Patterson, Turing Award winner and professor of computer science at Berkeley.

He's known for pioneering contributions to risk processor architecture used by 99% of new chips today and for co-creating RAID storage.

The impact that these 2 lines of research and development have had on our world is immeasurable.

He's also 1 of the great educators of computer science in the world.

His book with John Hennessey is how I first learned about and was humbled by the inner workings of machines at the lowest level.

2 sponsors, the Jordan Harbinger Show and Cash App.

Please consider supporting the podcast by going to jordanharbinger.com slash Lex and downloading Cash App and using code LEXPODCAST.

It's the best way to support this podcast and in general, the journey I'm on in my research and startup.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, review it, the 5 Stars on Apple Podcast, support it on Patreon, or connect with me on Twitter at Lex Friedman, spelled without the E, just F-R-I-D-M-A-N.

As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation.

This episode is supported by the Jordan Harbinger Show.

On that page, there's links to subscribe to it on Apple Podcasts, Spotify, and everywhere else.

He gets the best out of his guests, dives deep, calls them out when it's needed, and makes the whole thing fun to listen to.

He's interviewed Kobe Bryant, Mark Cuban, Neil deGrasse Tyson, Garry Kasparov, and many more.

I recently listened to his conversation with Frank Abagnale, author of Catch Me If You Can, 1 of the world's most famous con men.

Perfect podcast length and topic for a recent long distance run that I did.

Again, go to jordanharbinger.com slash Lex.

To give him my love and to support this podcast, subscribe also on Apple Podcasts, Spotify, and everywhere else.

This show is presented by Cash App, the greatest sponsor of this podcast ever, and the number 1 finance app in the App Store.

money to friends, buy Bitcoin, and invest in the stock market with as little as $1.

Since Cash App allows you to buy Bitcoin, let me mention that cryptocurrency in the context of the history of money is fascinating.

I recommend Ascent of Money as a great book on this history.

Debits and credits on Ledger started around 30,000 years ago.

The US dollar created over 200 years ago, and the first decentralized cryptocurrency released just over 10 years ago.

So given that history, cryptocurrency is still very much in its early days of development, but it's still aiming to, and just might, redefine the nature of money.

So again, if you get Cash App from the App Store or Google Play, and use the code LEXPODCAST, you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world.

And now, here's my conversation with David Patterson.

Let's start with the big historical question.

How have computers changed in the past 50 years at both the fundamental architectural level and in general in your eyes?

Well, the biggest thing that happened was the invention of the microprocessor.

So computers that used to fill up several rooms could fit inside your cell phone.

And not only did they get smaller, they got a lot faster.

So they're a million times faster than they were 50 years ago, and they're much cheaper, and they're ubiquitous.

There's 7.8 billion people on this planet, probably half of them have cell phones right now, just remarkable.

It's probably more microprocessors than there are people.

Sure, I don't know what the ratio is, but I'm sure it's above 1.

Maybe it's 10 to 1 or some number like that.

So a way to say what a microprocessor is is to tell you what's inside a computer.

So a computer forever has classically had 5 pieces.

There's input and output, which kind of naturally, as you'd expect, is input is like speech or typing and output is displays.

There's a memory and like the name sounds, it remembers things.

So it's integrated circuits whose job is you put information in them, when you ask for it, it comes back out.

And the third part is the processor, where the microprocessor comes from.

And that has 2 pieces as well, and that is the control, which is kind of the brain of the processor, and what's called the arithmetic unit, it's kind of the brawn of the computer.

So if you think of the, as a human body, the arithmetic unit, the thing that does the number crunching is the body and the control is the brain.

So those 5 pieces, input, output, memory, arithmetic unit and control have been in computers since the very dawn, and the last 2 are considered the processor.

So a microprocessor simply means a processor that fits on a microchip, and that was invented about 40 years ago, was the first microprocessor.

It's interesting that you refer to the arithmetic unit as the, like you connect it to the body and the controllers of the brain.

So I guess, I never thought of it that way.

It's a nice way to think of it because most of the actions the microprocessor does in terms of literally sort of computation.

Microprocessor does computation, it processes information.

And most of the thing it does is basic arithmetic operations.

So there are add instructions, subtract instructions, multiply and divide.

And kind of the brilliance of the invention of the computer or the processor is that it performs very trivial operations, but it just performs billions of them per second.

And what we're capable of doing is writing software that can take these very trivial instructions and have them create tasks that can do things better than human beings can do today.

Just looking back through your career, did you anticipate the kind of how good we would be able to get at doing these small basic operations?

How many surprises along the way where you just kind of sat back and said, wow, I didn't expect it to go this fast, this good?

Well, the fundamental driving force is what's called Moore's Law, which was named after Gordon Moore, who's a Berkeley alumnus.

And he made this observation very early in what are called semiconductors.

And semiconductors are these ideas, you can build these very simple switches and you can put them on these microchips.

And he made this observation over 50 years ago.

He looked at a few years and said, I think what's going to happen is the number of these little switches called transistors is going to double every year for the next decade.

And in 1975 he said, well, maybe it's gonna double every 2 years.

And that, what other people since named that Moore's Law, guided the industry.

And when Gordon Moore made that prediction, he wrote a paper back in, I think, in the 70s and said, not only did this gonna happen, he wrote what would be the implications of that.

he shows ideas like computers being in cars and computers being in something that you would buy in the grocery store and stuff like that.

So he kind of not only called his shot, he called the implications of it.

So if you were in the computing field and if you believed Moore's prediction, he kind of said what would be happening in the future.

So it's not kind of, it's at 1 sense, this is what was predicted and you could imagine, it was easy to believe that Moore's Law was gonna continue, and so this would be the implications.

On the other side, there are these kind of shocking events in your life.

Like I remember driving in a Marine across the bay in San Francisco and seeing a bulletin board at a local civic center and it had a URL on it.

And it was like, for the people at the time, these first URLs, and that's the, you know, www select stuff with the HTP, People thought it looked like alien writing, right?

You'd see these advertisements and commercials or bulletin boards that had this alien writing on it.

So for the lay people, it was like, what the hell is going on here?

And for those people in industry, it was, oh my God, this stuff is getting so popular, it's actually leaking out of our nerdy world into the real world.

So that, I mean, there was events like that.

I think another 1 was, I remember with the, in the early days of the personal computer, when we started seeing advertisements in magazines for personal computers, like it's so popular that it's made the newspapers.

So, at 1 hand, you know, Gordon Moore predicted it and you kind of expected it to happen, but when it really hit and you saw it affecting society, it was shocking.

So maybe taking a step back and looking both engineering and philosophical perspective, what do you see as the layers of abstraction in a computer?

Do you see a computer as a set of layers of abstractions?

Yeah, I think that's 1 of the things that computer science fundamentals is these things are really complicated in the way we cope with complicated software and complicated hardware is these layers of abstraction.

And that simply means that we suspend disbelief and pretend that the only thing you know is that layer and you don't know anything about the layer below it.

And that's the way we can make very complicated things.

And probably it started with hardware, that that's the way it was done, but it's been proven extremely useful.

And I would say in a modern computer today, there might be 10 or 20 layers of abstraction, and they're all trying to kind of enforce this contract is, all you know is this interface, there's a set of commands that you can, are allowed to use, and you stick to those commands, and we will faithfully execute that, and it's like peeling the layers of a onion, you get down, there's a new set of layers and so forth.

So for people who wanna study computer science, the exciting part about it is you can keep peeling those layers.

You take your first course and you might learn to program in Python and then you can take a follow-on course and you can get it down to a lower level language like C and you can go and then you can, if you want to, you can start getting into the hardware layers and you keep getting down all the way to that transistor that I talked about that Gordon Moore predicted, and you can understand all those layers all the way up to the highest level application software.

If you're interested, you can go into any depth and keep going.

In particular, what's happening right now, or what's happened in software last 20 years and recently in hardware, there's getting to be open source versions of all of these things.

So What open source means is what the engineer, the programmer designs, it's not secret, the belonging to a company, it's out there on the worldwide web, so you can see it.

So you can look at, for lots of pieces of software that you use, you can see exactly what the programmer does if you wanna get involved.

Recently, there's been an efforts to make open source hardware and those interfaces open so you can see that.

So instead of before you had to stop at the hardware, you can now start going layer by layer below that and see what's inside there.

So it's a remarkable time that for the interested individual can really see in great depth what's really going on in the computers that power everything that we see around us.

Are you thinking also when you say open source at the hardware level, is this going to the design, architecture, instruction set level or is it going to literally the, you know, the manufacturer of the actual hardware, of the actual chips, whether that's ASICs, specialized to a particular domain or the general.

Yeah, so let's talk about that a little bit.

So when you get down to the bottom layer of software, the way software talks to hardware is in a vocabulary.

And what we call that vocabulary, we call that, the words of that vocabulary are called instructions.

And the technical term for the vocabulary is instruction set.

So those instructions are like what we talked about earlier.

There can be instructions like add, subtract, and multiply, divide.

There's instructions to put data into memory, which is called a store instruction, and to get data back, which is called a load instructions.

And those simple instructions go back to the very dawn of computing.

In 1950, the commercial computer had these instructions.

So That's the instruction set that we're talking about.

So up until, I'd say 10 years ago, these instruction sets were all proprietary.

So a very popular 1 is owned by Intel, the 1 that's in the cloud and in all the PCs in the world.

There have been a sequence of ones that, the first number was called 8086, and since then there's been a lot of numbers, but they all end in

So there's been that kind of family of instruction sets.

The other 1 that's very popular is from ARM, that kind of powers all the cell phones in the world, all the iPads in the world, and a lot of things that are so-called Internet of Things devices, ARM, and that 1 is also proprietary.

ARM will license it to people for a fee, but they own that.

So the new idea that got started at Berkeley kind of unintentionally 10 years ago is early in my career, we pioneered a way to do these vocabularies instruction sets that was very controversial at the time.

At the time, in the 1980s, conventional wisdom was these vocabularies instruction sets should have powerful instructions, So polysyllabic kind of words, you can think of that.

And so instead of just add, subtract, and multiply, they would have polynomial divide or sort a list.

And the hope was of those powerful vocabularies that make it easier for software.

So we thought that didn't make sense for microprocessors.

There was people at Berkeley and Stanford and IBM who argued the opposite.

And we called that was a reduced instruction set computer.

And the abbreviation was RISC, and typical for computer people, we use the abbreviations to start pronouncing it, so RISC was the thing.

So We said for microprocessors, which with Gordon's Moore is changing really fast, we think it's better to have a pretty simple set of instructions, reduced set of instructions.

That that would be a better way to build microprocessors since they're gonna be changing so fast due to Moore's Law.

And then we'll just use standard software to generate more of those simple instructions.

And 1 of the pieces of software that's in that software stack, going between these layers of abstractions is called a compiler.

And it's basically translates, it's a translator between levels.

So the technical question was, well, since there are these reduced instructions, you have to execute more of them.

They're simpler so they could go faster, but you have to do more of them.

And it ended up that we ended up executing maybe 50% more instructions, maybe a third more instructions, but they ran 4 times faster.

So this risk, controversial risk ideas proved to be maybe factors of 3 or 4 better.

I love that this idea was controversial and almost kind of like rebellious.

So that's in the context of what was more conventional is the complex instructional set computing.

And believe it or not, this sounds very, very, who cares about this, right?

It was violently debated at several conferences.

And people thought risk was de-evolution.

We're gonna make software worse by making those instructions simpler, and there are fierce debates at several conferences in the 1980s, and then later in the 80s, it kind of settled

It's not completely intuitive to me why risk has, for the most part, won.

Yeah, yeah, and maybe I can sort of say a bunch of dumb things that could lay the land for further commentary.

So to me, this is kind of an interesting thing.

With modern compilers, you really could write faster code with C++.

So relying on the compiler to reduce your complicated code into something simple and fast.

So to me, comparing RISC, maybe this is a dumb question, but why is it that focusing the definition, the design of the instruction set on very few simple instructions, in the long run, provide faster execution?

Versus coming up with, like you said, a ton of complicated instructions that over time, you know, years, maybe decades, you come up with compilers that can reduce those into simple instructions for you?

Yeah, so let's try and split that into 2 pieces.

So if the compiler can do that for you, if the compiler can take a complicated program and produce simpler instructions, then the programmer doesn't care, right?

Programmer, I don't care just how fast is the computer I'm using, how much does it cost?

And so what happened kind of in the software industry is right around before the 1980s, critical pieces of software were still written not in languages like C or C++, they were written in what's called assembly language, where there's this kind of humans writing exactly at the instructions at the level that a computer can understand.

So they were writing add, subtract, multiply, you know, instructions.

But the belief was to write this lowest level of software that people use, which are called operating systems, they had to be written in assembly language because these high-level languages were just too inefficient.

They were too slow or the programs would be too big.

So that changed with a famous operating system called Unix, which is kind of the grandfather of all the operating systems today.

So Unix demonstrated that you could write something as complicated as an operating system in a language like C.

So once that was true, then that meant we could hide the instruction set from the programmer.

And so that meant then it didn't really matter.

The programmer didn't have to write lots of these simple instructions.

So that was part of our arguments for risk, is if you were still writing assembly languages, maybe a better case for CISC instructions.

But if the compiler can do that, it's gonna be, you know, that's done once, the computer translates it once, and then every time you run the program, it runs at this potentially simpler instructions.

And people would acknowledge that the simpler instructions could lead to a faster computer.

You can think of monosyllabic instructions.

You could say them, you know, if you think of reading, you could probably read them faster or say them faster than long instructions.

The same thing, that analogy works pretty well for hardware.

And as long as you didn't have to read a lot more of those instructions, you could win.

But it's interesting that in that discussion of UNIX and C that there's only 1 step of levels of abstraction from the code that's really the closest to the machine, to the code that's written by human.

It's, at least to me again, perhaps a dumb intuition, but it feels like there might have been more layers, sort of different kinds of humans stacked on top of each other.

So what's true and not true about what you said is several of the layers of software, so if you, 2 layers would be, suppose we just talk about 2 layers.

That would be the operating system, like you get from Microsoft or from Apple, like iOS or the Windows operating system.

And let's say applications that run on top of it, like Word or Excel.

So both the operating system could be written in C and the application could be written in C.

But you could construct those 2 layers and the applications absolutely do call upon the operating system.

And the change was that both of them could be written in higher level languages.

So it's 1 step of a translation, but you can still build many layers of abstraction of software on top of that.

So Still today, many of the layers that you'll deal with, you may deal with debuggers, you may deal with linkers.

There's libraries, many of those today will be written in C++, say, even though that language is pretty ancient, and even the Python interpreter is probably written in C or C++, so lots of layers there are probably written in these, some old-fashioned efficient languages that still take 1 step to produce these instructions, produce risk instructions, but they're composed, each layer of software invokes 1 another through these interfaces, and you can get 10 layers of software that way.

So in general, the risk was developed here at Berkeley?

It was kind of the 3 places that were these radicals that advocated for this against the rest of the community were IBM, Berkeley, and Stanford.

You're 1 of these radicals, and how radical did you feel?

How doubtful were you that risk might be the right approach?

Because you can also intuit that it is kind of taking a step back into simplicity, not forward into simplicity.

Yeah, no, it was easy to make, yeah, it was easy to make the argument against it.

Well, this was my colleague, John Hennessey, at Stanford, and I, we were both assistant professors, And for me, I just believed in the power of our ideas.

I thought what we were saying made sense.

The other thing that I didn't mention is 1 of the surprises of these complex instruction sets.

You could certainly write these complex instructions if the programmer is writing them themselves.

It turned out to be kind of difficult for the compiler to generate those complex instructions.

Kind of ironically, you'd have to find the right circumstances that just exactly fit this complex instruction.

It was actually easier for the compiler to generate these simple instructions.

So not only did these complex instructions make the hardware more difficult to build, often the compiler wouldn't even use them.

And so it's harder to build, the compiler doesn't use them that much, the simple instructions go better with Moore's law, that, you know, the number of transistors is doubling every 2 years, so we're going to have, you know, you want to reduce the time to design the microprocessor, that may be more important than the number of instructions.

So I think we believed in the, that we were right, that this was the best idea.

Then the question became in these debates, well, yeah, that's a good technical idea, but in the business world, this doesn't matter.

It's like arguing that if there's a standard with the railroad tracks, and you've come up with a better width, but the whole world is covered in railroad tracks so your ideas have no chance of success.

It was technically right, but commercially it'll be insignificant.

the history of human civilization is full of good ideas that lost because somebody else came along first with a worse idea.

And it's good that in the computing world, at least some of these have, well, you could, I mean, there's probably still CISC people that say.

Well, And what happened was, what was interesting, Intel, a bunch of the companies with CISC instruction sets of vocabulary, they gave up.

What Intel did, to its credit, because Intel's vocabulary was in the personal computer.

And so that was a very valuable vocabulary because the way we distribute software is in those actual instructions.

It's in the instructions of that instruction set.

So you don't get that source code, what the programmers wrote, you get after it's been translated into the lowest level, that's if you were to get a floppy disk or download software, it's in the instructions of that instruction set.

So the x86 instruction set was very valuable.

So what Intel did cleverly and amazingly is they had their chips in hardware do a translation step.

See all Lex Fridman transcripts on Youtube

David Patterson: Computer Architecture and Data Storage | Lex Fridman Podcast #104