The following is a conversation with Chris Latner.

Currently, he's a senior director at Google, working on several projects, including CPU, GPU, TPU accelerators for TensorFlow, Swift for TensorFlow, and all kinds of machine learning compiler magic going on behind the scenes.

He's 1 of the top experts in the world on compiler technologies, which means he deeply understands the intricacies of how hardware and software come together to create efficient code.

He created the LLVM compiler infrastructure project and the Clang compiler.

He led major engineering efforts at Apple, including the creation of the Swift programming language.

He also briefly spent time at Tesla as vice president of Autopilot software during the transition from Autopilot hardware 1 to hardware 2 when Tesla essentially started from scratch to build an in-house software infrastructure for Autopilot.

I could have easily talked to Chris for many more hours.

Compiling code down across the levels of abstraction is 1 of the most fundamental and fascinating aspects of what computers do.

And he is 1 of the world experts in this process.

It's rigorous science and it's messy, beautiful art.

This conversation is part of the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Chris Ladner.

What was the first program you've ever written?

I think I started as a kid and my parents got a basic programming book.

And so when I started, it was typing out programs from a book and seeing how they worked and then typing them in wrong and trying to figure out why they were not working right, that kind of stuff.

So BASIC, what was the first language that you remember yourself maybe falling in love with, like really connecting with?

I don't know, I mean, I feel like I've learned a lot along the way and each of them have a different special thing about them.

So I started in BASIC and then went like GWBASIC, which was the thing back in the DOS days, and then upgraded to QBASIC and eventually QWICBASIC, which are all slightly more fancy versions of Microsoft Basic, made the jump to Pascal and started doing machine language programming and assembly in Pascal, which was really cool.

Eventually got into C, C++, and then kind of did lots of other weird things.

I feel like you took the dark path, which is the, you could have gone lisp.

You could have gone higher level, sort of functional, philosophical, hippie route.

Instead you went into like the dark arts of the sea.

So started with basic Pascal net assembly and then wrote a lot of assembly.

And I eventually did small talk and other things like that.

So and then that was really about trying to be able to do more powerful things than what Pascal could do, and also to learn a different world.

So he was really confusing to me with pointers and the syntax and everything, and it took a while.

But Pascal's much more principled in various ways, sees more, I mean, it has its historical roots, but it's not as easy to learn.

With pointers, there's this memory management thing that you have to become conscious of.

Is that the first time you start to understand that there's resources that you're supposed to manage?

Well, so you have that in Pascal as well, but in Pascal, these like the caret instead of the star, and there's some small differences like that, but it's not about a pointer arithmetic.

And in C, you end up thinking about how things get laid out in memory a lot more.

And so in Pascal, you have allocating and deallocating and owning the memory, but just the programs are simpler and you don't have to...

Well, for example, Pascal has a string type.

And so you can think about a string instead of an array of characters which are consecutive in memory.

So it's a little bit of a higher level abstraction.

Let's talk about LLVM, Clang, and compilers.

So can you tell me first what LLVM and Clang are, and how is it that you find yourself the creator and lead developer, 1 of the most powerful compiler optimization systems in use today?

So the way I look at this is you have a two-sided problem of you have humans that need to write code.

And then you have machines that need to run the program that the human wrote.

And for lots of reasons, the humans don't want to be writing in binary and want to think about every piece of hardware.

And so at the same time that you have lots of humans, you also have lots of kinds of hardware.

And So compilers are the art of allowing humans to think at a level of abstraction that they want to think about, and then get that program, get the thing that they wrote, to run on a specific piece of hardware.

And the interesting and exciting part of all this is that there's now lots of different kinds of hardware, chips like x86 and PowerPC and ARM and things like that, but also high-performance accelerators for machine learning and other things like that are also just different kinds of hardware, GPUs.

And at the same time, on the programming side of it, you have your basic, you have C, you have JavaScript, you have Python, you have Swift, you have like lots of other languages that are all trying to talk to the human in a different way to make them more expressive and capable and powerful.

And so Compilers are the thing that goes from 1 to the other.

And so you go from what the human wrote, and programming languages end up being about expressing intent, not just for the compiler and the hardware, but the programming language's job is really to capture an expression of what the programmer wanted that then can be maintained and adapted and evolved by other humans, as well as interpreted by the compiler.

So when you look at this problem, you have on the 1 hand humans, which are complicated, and you have hardware, which is complicated.

And so compilers typically work in multiple phases.

And so the software engineering challenge that you have here is try to get maximum reuse out of the amount of code that you write, because these compilers are very complicated.

And so the way it typically works out is that you have something called a front end or a parser that is language specific.

And so you'll have a C parser, and that's what Clang is, or C++ or JavaScript or Python or whatever, that's the front end.

Then you'll have a middle part, which is often the optimizer, and then you'll have a late part, which is hardware specific.

And so compilers end up, There's many different layers often, but these 3 big groups are very common in compilers.

And what LLVM is trying to do is trying to standardize that middle and last part.

And so 1 of the cool things about LLVM is that there are a lot of different languages that compile through to it.

And so things like Swift, but also Julia, Rust, Clang for C, C++, Objective-C, like these are all very different languages and they can all use the same optimization infrastructure, which gets better performance, and the same code generation infrastructure for hardware support.

And so LLVM is really that layer that is common that all these different specific compilers can use.

And is it a standard, like a specification, or is it literally an implementation?

And so it's, I think there's a couple of different ways of looking at it, right?

Because it depends on which angle you're looking at it from.

LLVM ends up being a bunch of code, okay?

So it's a bunch of code that people reuse and they build compilers with.

We call it a compiler infrastructure because it's kind of the underlying platform that you build a concrete compiler on top of.

And the LLVM community is hundreds of people that all collaborate, and 1 of the most fascinating things about LLVM over the course of time is that we've managed somehow to successfully get harsh competitors in the commercial space to collaborate on shared infrastructure.

And so you have Google and Apple, you have AMD and Intel, you have Nvidia and AMD on the graphics side, you have Cray and everybody else doing these things.

And like all these companies are collaborating together to make that shared infrastructure really, really great.

And they do this not out of the goodness of their heart, but they do it because it's in their commercial interest of having really great infrastructure that they can build on top of, and facing the reality that it's so expensive that no 1 company, even the big companies, no 1 company really wants to implement it all themselves.

That's a great point because it's also about the skill sets.

And these, the skill sets are very hard to find.

It always seems like with open source projects, the kind, and LLVM is open source?

It's about, it's 19 years old now, so it's fairly old.

It seems like the magic often happens within a very small circle of people.

Yes, So LLVM came from a university project.

And so I was at the University of Illinois.

And there was myself, my advisor, and then a team of 2 or 3 research students in the research group.

And we built many of the core pieces initially.

I then graduated and went to Apple and at Apple brought it to the products first in the OpenGL graphics stack, but eventually to the C compiler realm and eventually built Clang and eventually built Swift and these things, along the way building a team of people that are really amazing compiler engineers that help build a lot of that.

And so as it was gaining momentum, and as Apple was using it, being open source and public and encouraging contribution, many others, for example, at Google came in and started contributing.

And in some cases, Google effectively owns Clang now because it cares so much about C++ and the evolution of that ecosystem.

And so it's investing a lot in the C++ world and the tooling and things like that.

And so likewise, NVIDIA cares a lot about CUDA.

And so CUDA uses Clang and uses LLVM for graphics and GPU.

And so when you first started as a master's project, I guess, did you think it was gonna go as far as it went?

It seems like a really difficult undertaking, a brave 1.

Yeah, no, no, no, it was nothing like that.

So I mean, my goal when I went to the University of Illinois was to get in and out with a non-thesis master's in a year and get back to work.

So I was not planning to stay for 5 years and build this massive infrastructure.

And a lot of it was because LLVM was fun.

I was building cool stuff and learning really interesting things and facing both software engineering challenges but also learning how to work in a team and things like that.

I had worked at many companies as interns before that but it was really a different thing to have a team of people that are working together and trying to collaborate in version control and it was just a little bit different.

Like I said, I just talked to Don Knuth and he believes that 2% of the world population have something weird with their brain, that they're geeks, they understand computers, they're connected with computers.

He says, I can't prove it, but it's very empirically there.

Is there something that attracts you to the idea of optimizing code?

I mean, it seems like that's 1 of the biggest, coolest things about LLVM.

Yeah, that's 1 of the major things it does.

So I got into that because of a person, actually.

So when I was in my undergraduate, I had an advisor, or a professor named Steve Vegdal.

And he, I went to this little tiny private school.

There were, I think, 7 or 9 people in my computer science department, students in my class.

So it was a very tiny, very small school.

It was kind of a wart on the side of the math department, kind of a thing at the time.

I think it's evolved a lot in the many years since then, but Steve Vegdal was a compiler guy and he was super passionate and his passion rubbed off on me.

And 1 of the things I like about compilers is that they're large, complicated software pieces.

And so 1 of the culminating classes that many computer science departments, at least at the time, did was to say that you would take algorithms and data structures in all these core classes, but then the compilers class was 1 of the last classes you take because it pulls everything together and then you work on 1 piece of code over the entire semester.

And so you keep building on your own work, which is really interesting.

It's also very challenging because in many classes, if you don't get a project done, you just forget about it and move on to the next 1 and get your B or whatever it is.

But here you have to live with the decisions you make and continue to reinvest in it.

And so I did a extra study project with him the following semester and he was just really great.

And he was also a great mentor in a lot of ways.

And so from him and from his advice, he encouraged me to go to graduate school.

I wasn't super excited about going to grad school.

I wanted the master's degree, but I didn't want to be an academic.

And, but like I said, I kind of got tricked into saying and was having a lot of fun and I definitely do not regret it.

Well, the aspects of compilers were the things you connected with.

So, LLVM, there's also the other part that's just really interesting, if you're interested in languages, is parsing and just analyzing, like, yeah, analyzing the language, breaking it down, parsing, and so on.

Was that interesting to you or were you more interested in optimization?

For me it was more, so I'm not really a math person.

I understand some bits of it when I get into it.

But math is never the thing that attracted me.

And so a lot of the parser part of the compiler has a lot of good formal theories that Don, for example, knows quite well.

But I just like building a thing and seeing what it could do and exploring and getting it to do more things and then setting new goals and reaching for them.

And in the case of LLVM, when I started working on that, my research advisor that I was working for was a compiler guy.

And so he and I specifically found each other because we were both interested in compilers and so I started working with them and taking his class and a lot of LLVM initially was, it's fun implementing all the standard algorithms and all the things that people had been talking about and were well known and they were in the curricula for advanced studies in compilers.

And so just being able to build that was really fun.

And I was learning a lot by, instead of reading about it, just building.

So you said compilers are these complicated systems.

Can you even just, with language, try to describe, you know, how you turn a C++ program into code?

So I'll give you examples of the hard parts along the way.

So C++ is a very complicated programming language.

It's something like 1,400 pages in the spec.

What makes the language complicated in terms of what's syntactically?

So it's what they call syntax, so the actual how the characters are arranged, yes.

It's also, in the case of C++, there's a huge amount of history.

You play that forward, and then a bunch of suboptimal, in some cases, decisions were made, and they compound.

And then more and more and more things keep getting added to C++ and it will probably never stop.

But the language is very complicated from that perspective.

And so the interactions between subsystems is very complicated.

And when you talk about the front end, 1 of the major challenges which Clang as a project, the C C++ compiler that I built, I and many people built, 1 of the challenges we took on was we looked at GCC.

Okay GCC at the time was like a really good industry standardized compiler that had really consolidated a lot of the other compilers in the world and was a standard, but it wasn't really great for research.

The design was very difficult to work with and it was full of global variables and other things that made it very difficult to reuse in ways that it wasn't originally designed for.

And so with Clang, 1 of the things that we wanted to do is push forward on better user interface.

So make error messages that are just better than GCCs.

And that's actually hard because you have to do a lot of bookkeeping in an efficient way to be able to do that.

And so compile time is about making it efficient, which is also really hard when you're keeping track of extra information.

So refactoring tools and other analysis tools that the GCC never supported, also leveraging the extra information we kept, but enabling those new classes of tools that then get built into IDEs.

And so that's been 1 of the areas that Clang has really helped push the world forward in, is in the tooling for C and C++ and things like that.

But C++ in the front end piece is complicated and you have to build syntax trees and you have to check every rule in the spec and you have to turn that back into an error message to the human that the human can understand when they do something wrong.

But then you start doing the, what's called lowering.

So going from C++ and the way that it represents code down to the machine.

And when you do that, there's many different phases you go through.

Often There are, I think LLVM has something like 150 different, what are called passes in the compiler, that the code passes through, and these get organized in very complicated ways, which affect the generated code, and the performance, and compile time, and many other things.

So after you do the clang parsing, what's the graph?

Yeah, so in the parser, it's usually a tree.

And so the idea is you have a node for the plus that the human wrote in their code, or the function call, you'll have a node for call with the function that they call and the arguments they pass, things like that.

This then gets lowered into what's called an intermediate representation.

And intermediate representations are like LLVM has 1.

And there it's what's called a control flow graph.

And so you represent each operation in the program as a very simple, like this is gonna add 2 numbers, this is gonna multiply 2 things, this maybe will do a call, but then they get put in what are called blocks.

And so you get blocks of these straight line operations, where instead of being nested like in a tree, it's straight line operations, And so there's a sequence and an ordering to these operations.

And so it's a straight line sequence of operations within the block.

And then you have branches, like conditional branches, between blocks.

And so when you write a loop, for example, in a syntax tree, you would have a for node, like for a for statement in a C-like language, you'd have a for node.

And you have a pointer to the expression for the initializer, a pointer to the expression for the increment, a pointer to the expression for the comparison, a pointer to the body.

In a control flow graph, you get a block for the code that runs before the loop, so the initializer code.

Then you have a block for the body of the loop.

And so the body of the loop code goes in there, but also the increment and other things like that.

And then you have a branch that goes back to the top and a comparison and a branch that goes out.

And so it's more of a assembly level kind of representation.

But the nice thing about this level of representation is it's much more language independent.

And so there's lots of different kinds of languages with different kinds of, you know, JavaScript has a lot of different ideas of what is false, for example, and all that can stay in the front end, but then that middle part can be shared across all those.

How close is that intermediate representation to neural networks, for example?

Is it, are they, because everything you describe is a kind of, echoes of a neural network graph.

They're quite different in details, but they're very similar in idea.

So 1 of the things that neural networks do is they learn representations for data at different levels of abstraction, right?

And then they transform those through layers, right?

So the compiler does very similar things, but 1 of the things the compiler does is it has relatively few different representations.

a neural network often, as you get deeper, for example, you get many different representations and each layer or set of ops is transforming between these different representations.

In a compiler, often you get 1 representation and they do many transformations to it.

And these transformations are often applied iteratively.

And For programmers, there's familiar types of things.

For example, trying to find expressions inside of a loop and pulling them out of a loop, so that they execute for your times, or find redundant computation, or find constant folding or other simplifications, turning 2 times X into X shift left by 1, and things like this are all the examples of the things that happen.

But compilers end up getting a lot of theorem proving and other kinds of algorithms that try to find higher level properties of the program that then can be used by the optimizer.

Cool, so what's like the biggest bang for the buck with optimization?

At the very beginning, the 80s, I don't know.

Yeah, so for the 80s, a lot of it was things like register allocation.

So the idea of in a modern, like a microprocessor, what you'll end up having is you'll end up having memory, which is relatively slow, and then you have registers that are relatively fast.

But registers, you don't have very many of them.

Okay, And so when you're writing a bunch of code, you're just saying like, compute this, put in a temporary variable, compute this, compute this, compute this, put in a temporary variable, I have a loop, I have some other stuff going on.

Well, now you're running on an x86, like a desktop PC or something.

Well, it only has, in some cases, some modes, 8 registers, right?

And so now the compiler has to choose what values get put in what registers at what points in the program.

So if you think about, you have a loop, an inner loop that executes millions of times maybe.

If you're doing loads and stores inside that loop, then it's gonna be really slow.

But if you can somehow fit all the values inside that loop and registers, now it's really fast.

And so getting that right requires a lot of work because there's many different ways to do that.

And often what the compiler ends up doing is it ends up thinking about things in a different representation than what the human wrote.

Well, the compiler thinks about that as 4 different values, each which have different lifetimes across the function that it's in, And each of those could be put in a register or memory or different memory or maybe in some parts of the code recomputed instead of stored and reloaded.

And there are many of these different kinds of techniques that can be used.

So it's adding almost like a time dimension to, it's trying to optimize across time.

So it's considering when you're programming, you're not thinking in

See all Lex Fridman transcripts on Youtube

Chris Lattner: Compilers, LLVM, Swift, TPU, and ML Accelerators | Lex Fridman Podcast #21