Chris Lattner is a world-class software & hardware engineer, leading projects at Apple, Tesla, Google, and SiFive. Please support this podcast by checking out our sponsors:
- Blinkist: https://blinkist.com/lex and use code LEX to get a free week of premium
- Neuro: https://www.getneuro.com and use code LEX to get 15% off
- MasterClass: https://masterclass.com/lex to get 15% off annual sub
- Cash App: https://cash.app/ and use code LexPodcast to get $10

EPISODE LINKS:
Chris's Twitter: https://twitter.com/clattner_llvm
Chris's Website: http://nondot.org/sabre/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:25 - Working with Elon Musk, Steve Jobs, Jeff Dean
7:55 - Why do programming languages matter?
13:55 - Python vs Swift
24:48 - Design decisions
30:06 - Types
33:54 - Programming languages are a bicycle for the mind
36:26 - Picking what language to learn
42:25 - Most beautiful feature of a programming language
51:50 - Walrus operator
1:01:16 - LLVM
1:06:28 - MLIR compiler framework
1:10:35 - SiFive semiconductor design
1:23:09 - Moore's Law
1:26:22 - Parallelization
1:30:50 - Swift concurrency manifesto
1:41:39 - Running a neural network fast
1:47:16 - Is the universe a quantum computer?
1:52:57 - Effects of the pandemic on society
2:10:09 - GPT-3
2:14:28 - Software 2.0
2:27:54 - Advice for young people
2:32:37 - Meaning of life

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Chris Lattner, his second time on the podcast.

the most brilliant engineers in modern computing, having created LLVM compiler infrastructure project, the Clang compiler, the Swift programming language, a lot of key contributions to TensorFlow and TPUs as part of Google.

He served as Vice President of Autopilot Software at Tesla, was a software innovator and leader at Apple, and now is at Sci-5 as Senior Vice President of Platform Engineering, looking to revolutionize chip design to make it faster, better, and cheaper.

Quick mention of each sponsor, followed by some thoughts related to the episode.

First sponsor is Blinkist, an app that summarizes key ideas from thousands of books.

I use it almost every day to learn new things or to pick which books I want to read or listen to next.

Second is Neuro, the maker of functional sugar-free gum and mints that I use to supercharge my mind with caffeine, L-theanine, and B vitamins.

Third is Masterclass, online courses from the best people in the world on each of the topics covered from rockets, to game design, to poker, to writing, and to guitar.

And finally, Cash app, the app I use to send money to friends for food, drinks, and unfortunately lost bets.

Please check out the sponsors in the description to get a discount and to support this podcast.

As a side note, let me say that Chris has been an inspiration to me on a human level because he is so damn good as an engineer and leader of engineers and yet he's able to stay humble, especially humble enough to hear the voices of disagreement and to learn from them.

He was supportive of me and this podcast from the early days, and for that, I'm forever grateful.

To be honest, most of my life, no 1 really believed that I would amount to much.

So when another human being looks at me, it makes me feel like I might be someone special.

The weird kid in the corner with a dream is someone who might need your love and support in order for that dream to flourish.

If you enjoy this thing, subscribe on YouTube, review it with 5 Stars on Apple Podcasts, follow on Spotify, support on Patreon, or connect with me on Twitter at Lex Friedman.

And now, here's my conversation with Chris Latner.

What are the strongest qualities of Steve Jobs, Elon Musk, and the great and powerful Jeff Dean since you've gotten a chance to work with each?

You're starting with an easy question there.

I guess you could do maybe a pairwise comparison between them instead of a group comparison.

So if you look at Steve Jobs and Elon, I worked a lot more with Elon than I did with Steve.

They're both very demanding in their own way.

My sense is Steve is much more human factor focused, where Elon is more technology focused.

Steve's trying to build things that feel good, that people love, that affect people's lives, how they live.

He's looking into the future a little bit in terms of what people want, where I think that Elon focuses more on learning how exponentials work and predicting the development of those.

That was 1 of the things that reading the biography and how can a designer essentially talk to engineers and get their respect?

My sense is that he pushed people really hard.

But then when he got an explanation that made sense to him, then he would let go.

And he did actually have a lot of respect for engineering.

And when you can read people well, you can know when they're holding back and when you can get a little bit more out of them.

I mean, if you compare the other folks, so Jeff Dean, right?

Jeff is a really, really, really nice guy, well-meaning.

He combines it with brilliance, so he can pull people together in a really great way.

I don't think you'd even want to be that.

Jeff is an amazing engineer today, right?

So it's really hard to compare Jeff to either of those 2.

I think that Jeff leads through technology and building it himself and then pulling people in and inspiring them.

And so I think that that's 1 of the amazing things about Jeff, but each of these people, with their pros and cons, all are really inspirational and have achieved amazing things.

So it's been, I've been very fortunate to get to work with these guys.

For yourself, you've led large teams, you've done some of the incredible difficult technical challenges.

Is there something you've picked up from them about how to lead?

Yeah, well, so I mean, I think leadership is really hard.

It really depends on what you're looking for there.

I think you really need to know what you're talking about.

So being grounded on the product, on the technology, on the business, on the mission is really important.

Being, understanding what people are looking for, why they're there.

1 of the most amazing things about Tesla is the unifying vision, right?

People are there because they believe in clean energy and electrification, all these kinds of things.

The other is to understand what really motivates people, how to get the best people, how to build a plan that actually can be executed, right?

There's so many different aspects of leadership and it really depends on the time, the place, the problems.

You know, there's a lot of issues that don't need to be solved.

And so if you focus on the right things and prioritize well, that can really help move things.

1 is you really have to know what you're talking about.

How you've worked on a lot of very challenging technical things.

So I kind of assume you were born technically savvy, but assuming that's not the case, how did you develop technical expertise?

Like even at Google, you worked on I don't know how many projects, but really challenging, very varied.

Compilers, TPUs, hardware, cloud stuff, bunch of different things.

The thing that I've become comfortable with as I've gained experience is being okay with not knowing.

And so a major part of leadership is actually, it's not about having the right answer, it's about getting the right answer.

And so if you're working in a team of amazing people, right, and many of these places, many of these companies all have amazing people.

It's the question of how do you get people together?

How do you get people to be vulnerable sometimes with an idea that maybe isn't good enough, but it's the start of something beautiful?

How do you provide an environment where you're not just like top-down, that shall do the thing that I tell you to do, but you're encouraging people to be part of the solution, and providing a safe space where if you're not doing the right thing, they're willing to tell you about it.

Well, I've been in the hardware realm recently, and I don't know much at all about how chips are designed.

I know some of the principles and the ars technical level of this.

But It turns out that if you ask a lot of dumb questions, you get smarter really quick.

And when you're surrounded by people that want to teach and learn themselves, it can be a beautiful thing.

So let's talk about programming languages, if it's okay.

At the highest absurd philosophical level, because I-

I will forever get romantic and torture you, I apologize.

Why do programming languages even matter?

So you're saying why should you care about any 1 programming language or why do we care about programming computers?

No, why do we care about programming language design, creating effective programming languages, choosing 1 programming language versus another programming language?

Why we keep struggling and improving through the evolution of these programming languages.

So, I mean, I think you have to come back to what are we trying to do here, right?

So we have these, these beasts called computers that are very good at specific kinds of things, and we think it's useful to have them do it for us, right?

Now, you have this question of how best to express that, because you have a human brain still that has an idea in its head, and you wanna achieve something, right?

So, well, there's lots of ways of doing this you can go directly to the machine and speak assembly language and then you can express directly what the computer understands that's fine you can then have higher and higher and higher levels of abstraction up until machine learning and you're designing a neural net to do the work for you.

The question is where along this way do you want to stop and what benefits do you get out of doing so?

And so programming languages in general, you have C, you have Fortran, Java, and Ada, Pascal, Swift, you have lots of different things.

They'll have different trade-offs, and they're tackling different parts of the problems.

Now, 1 of the things that most programming languages do is they're trying to make it so that you have pretty basic things like portability across different hardware.

So you've got, I'm gonna run on an Intel PC, I'm going to run on a RISC-V PC, I'm going to run on ARM phone or something like that, fine.

I want to write 1 program and have it portable, and this is something that assembly doesn't do.

Now, when you start looking at the space of programming languages, This is where I think it's fun because programming languages all have trade-offs and most people will walk up to them and they look at the surface level of syntax and say, oh I like curly braces or I like tabs or I like you know semicolons or not or whatever right.

Subjective, fairly subjective, very shallow things.

But programming languages, when done right, can actually be very powerful.

And the benefit they bring is expression.

Okay, and If you look at programming languages, there's really 2 different levels to them.

1 is the down in the dirt, nuts and bolts of how do you get the computer to be efficient, stuff like that, how they work, type systems, compiler stuff, things like that.

The UI for programming language is really a design problem, and a lot of people don't think about it that way.

The UI, you mean all that stuff with the braces and-

Yeah, all that stuff is the UI, and what it is, and UI means user interface.

What's really going on is it's the interface between the guts and the human.

Humans have feelings, they have things they like, they have things they don't like, and a lot of people treat programming languages as though humans are just kind of abstract creatures that cannot be predicted.

But it turns out that actually there is better and worse.

People can tell when a program language is good or when it was an accident.

1 of the things with Swift in particular is that a tremendous amount of time by tremendous number of people have been put into really polishing and making it feel good.

But it also has really good nuts and bolts underneath it.

You said that SWIFT makes a lot of people feel good.

So how do you predict that tens of thousands, hundreds of thousands of people are going to enjoy using this, the user experience of this programming language?

Well, you can look at it in terms of better and worse, right?

So if you have to write lots of boilerplate or something like that, you will feel unproductive.

If like C, for example, is what's called a memory unsafe language.

And so you get dangling pointers and you get all these kind of bugs that then you have spent tons of time debugging and it's a real pain in the butt, and you feel unproductive.

And so by subtracting these things from the experience, you get happier people.

If you look at the people, people that are most productive on Stack Overflow, they have a set of priorities that may not always correlate perfectly with the experience of the majority of users.

You know, if you look at the most upvoted, quote unquote, correct answer on Stack Overflow, is usually really sort of prioritizes like safe code, proper code, stable code, you know, that kind of stuff.

As opposed to like if I want to use go-to statements in my basic, right?

I want to use go-to state, like, what if 99% of people want to use go-to statements?

So you use completely improper, you know, unsafe syntax.

I don't think that people actually, like if you boil it down and you get below the surface level, people don't actually care about go-tos or if statements or things like this.

So the real question is, I want to set up a web server and I want to do a thing, whatever.

From a programming language perspective, there's really 2 things that matter there.

Then how quickly can you put it together?

What are the tools around that look like?

When you want to build a library that's missing, what do you do?

Now, this is where you see huge divergence in the force between worlds.

Python is really good at assembling things, but it's not so great at building all the libraries.

So what you get, because of performance reasons, other things like this, is you get Python layered on top of C, for example, and that means that doing certain kinds of things, it doesn't really make sense to do in Python.

Instead, you do it in C, and then you wrap it, and then you're living in 2 worlds, and 2 worlds never is really great because tooling and the debugger doesn't work right, and all these kinds of things.

Can you clarify a little bit what you mean by Python is not good at building libraries, meaning it doesn't make it conducive?

Meaning like it's not conducive to developers to come in and add libraries, or is it the duality of the, it's a dance between Python and C and-

I did not mean to say that Python is bad for libraries.

What I meant to say is there are libraries that Python's really good at that you can write in Python.

But there are other things, like if you want to build a machine learning framework, you're not going to build a machine learning framework in Python because of performance, for example.

Or you want GPU acceleration or things like this.

Instead, what you do is you write a bunch of C or C++ code or something like that, and then you talk to it from Python.

This is because of decisions that were made in the Python design, and those decisions have other counterbalancing forces.

But the trick when you start looking at this from a programming language perspective is you start to say, okay, cool.

How do I build this catalog of libraries that are really powerful?

And how do I make it so that then they can be assembled into ways that feel good and they generally work the first time?

Because when you're talking about building a thing, you have to include the debugging, the fixing, the turnaround cycle, the development cycle, all that kind of stuff into the process of building the thing.

It's not just about pounding out the code.

And so this is where things like, you know, Catching bugs at compile time is valuable, for example.

But if you dive into the details in this, Swift, for example, has certain things like value semantics, which is this fancy way of saying that when you treat a variable like a value, it acts like a mathematical object would.

Okay, so you have used PyTorch a little bit.

Tensors are n-dimensional grid of numbers.

Very simple, You can do plus and other operators on them.

But why do you need to clone a tensor sometimes?

It's the usual object thing that's in Python.

So in Python, and just like with Java and many other languages, this isn't unique to Python.

In Python, it has a thing called reference semantics, which is the nerdy way of explaining this.

And what that means is you actually have a pointer do a thing instead of the thing.

Now, this is due to a bunch of implementation details that you don't want to go into.

But in Swift, you have this thing called value semantics.

And so when you have a tensor in Swift, it is a value.

If you copy it, it looks like you have a unique copy.

And if you go change 1 of those copies, then it doesn't update the other 1 because you just made a copy of this thing.

So that's like highly error prone in at least computer science, math-centric disciplines about Python.

That the thing you would expect to behave.

It doesn't behave like math and in fact, quietly doesn't behave like math and then can ruin the entirety of your math.

Well, and then it puts you in debugging land again.

Right now, you just want to get something done and you're like, wait a second, where do I need to put clone and what level of the stack, which is very complicated, which I thought I was reusing somebody's library and now I need to understand it to know where to clone a thing.

And so this is where programming languages really matter.

So in Swift, having value semantics so that both you get the benefit of math working like math, but also the efficiency that comes with certain advantages there, certain implementation details there, really benefit you as a programmer.

Like, how do you know that a thing should be treated like a value?

Yeah, so Swift has a pretty strong culture and good language support for defining values.

And so if you have an array, so tensors are 1 example that the machine learning folks are very used to.

Just think about arrays, same thing, where you have an array, you put, you create an array, you put 2 or 3 or 4 things into it, and then you pass it off to another function.

What happens if that function adds some more things to it?

Well, you'll see it on the side that you pass it in.

Now, what if you pass an array off to a function?

It scrolls it away in some dictionary or some other data structure somewhere.

Well, it thought that you just handed it that array, then you return back and that reference to that array still exists in the caller and they go and put more stuff in it.

The person you handed it off to may have thought they had the only reference that and so they didn't know that this was going to change underneath the covers.

This is where you end up having to do clone.

I was past a thing, I'm not sure if I have the only version of it, so now I have to clone it.

What value semantics does is it allows you to say, hey, I have a SunSwift, it defaults to value semantics.

So defaults to value semantics and then because most things should be treated like values, then it makes sense for that to be

And 1 of the important things about that is that arrays and dictionaries and all these other collections that are aggregations of other things also have value semantics.

And so when you pass this around to different parts of your program, you don't have to do these defensive copies.

It's great because you define away the bug, which is a big deal for productivity, the number 1 thing most people care about.

But it's also good for performance because when you're doing a clone, so you pass the array down to the thing, it was like, I don't know if anybody else has it, I have to clone it.

Well, you just did a copy of a bunch of data, it could be big.

Then it could be that the thing that called you is not keeping track of the old thing.

So you just made a copy of it and you may not have had to.

So the way the value semantics work in Swift is it uses this thing called copy on write, which means that you get the benefit of safety and performance.

It has another special trick because if you think, certain languages like Java, for example, they have immutable strings.

What they're trying to do is they provide value semantics by having pure immutability.

Functional languages have pure immutability in lots of different places, and this provides a much safer model and it provides value semantics.

The problem with this is if you have immutability, everything is expensive.

For example, in Java, if you have a string X and a string Y, you append them together, we have to allocate a new string to hold X, Y.

Well, and strings in Java are immutable, and if there's optimizations for short runs, and it's complicated, but generally think about them as a separate allocation and so when you append them together you have to go allocate a third thing because somebody might have a pointer to either of the other ones right and you can't go change them So you have to go allocate a third thing.

Because of the beauty of how the Swift Values Max system works out, if you have a string on Swift, you say, hey, put in x, right?

And they say, append on YZW, it knows that there's only 1 reference to that.

And so you're not allocating tons of stuff on the side.

You're not, you don't have all those problems.

When you pass it off, you can know you have the only reference.

If you pass it off to multiple different people, but nobody changes it, they can all share the same thing.

And so you get a really nice sweet spot that I haven't seen in other languages.

Like I thought there was going to be a philosophical like narrative here that you're gonna have to pay a cost for it.

It sounds like, I think value semantics is beneficial for easing of debugging or minimizing the risk of errors, like bringing the errors closer to the source, bringing the symptom of the error closer to the source of the error, however you say that.

But you're saying there's not a performance cost either if you implement

Yeah, Well, so there's trade-offs with everything.

And so if you are doing very low-level stuff, then sometimes you can notice the cost.

But then what you're doing is you're saying, what is the right default?

So coming back to user interface, when you talk about programming languages, 1 of the major things that Swift does that makes people love it, that is not obvious when it comes to designing a language, is this UI principle of progressive disclosure of complexity.

So Swift, like many languages, is very powerful.

The question is, when do you have to learn the power as a user?

See all Lex Fridman transcripts on Youtube

Chris Lattner: The Future of Computing and Programming Languages | Lex Fridman Podcast #131