Chris Lattner is a legendary software and hardware engineer, leading projects at Apple, Tesla, Google, SiFive, and Modular AI, including the development of Swift, LLVM, Clang, MLIR, CIRCT, TPUs, and Mojo. Please support this podcast by checking out our sponsors:
- iHerb: https://lexfridman.com/iherb and use code LEX to get 22% off your order
- Numerai: https://numer.ai/lex
- InsideTracker: https://insidetracker.com/lex to get 20% off

EPISODE LINKS:
Chris's Twitter: https://twitter.com/clattner_llvm
Chris's Website: http://nondot.org/sabre/
Mojo programming language: https://www.modular.com/mojo
Modular AI: https://modular.com/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:20 - Mojo programming language
12:37 - Code indentation
21:04 - The power of autotuning
30:54 - Typed programming languages
47:38 - Immutability
59:56 - Distributed deployment
1:34:23 - Mojo vs CPython
1:50:12 - Guido van Rossum
1:57:13 - Mojo vs PyTorch vs TensorFlow
2:00:37 - Swift programming language
2:06:09 - Julia programming language
2:11:14 - Switching programming languages
2:20:40 - Mojo playground
2:25:30 - Jeremy Howard
2:36:16 - Function overloading
2:44:41 - Error vs Exception
2:52:21 - Mojo roadmap
3:05:23 - Building a company
3:17:09 - ChatGPT
3:23:32 - Danger of AI
3:27:27 - Future of programming
3:30:43 - Advice for young people

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

On 1 axis you have more hardware coming in, on the other hand you have an explosion of innovation in AI.

And so what happened with both TensorFlow and PyTorch is that the explosion of innovation in AI has led to, it's not just about matrix multiplication and convolution, these things have now like 2, 000 different operators.

And on the other hand you have, I don't know how many pieces of hardware there are, it's a lot.

Part of my thesis, part of my belief of where computing goes, if you look out 10 years from now, is it's not going to get simpler.

Physics isn't going back to where we came from, it's only going to get weirder from here on out.

And so to me, the exciting part about what we're building is it's about building that universal platform, which the world can continue to get weird, because again, I don't think it's avoidable.

But we can help lift people's scale, do things with it, and they don't have to rewrite their code every time a new device comes out.

The following is a conversation with Chris Lattner, his third time on this podcast.

As I've said many times before, he's 1 of the most brilliant engineers in modern computing.

Having created LLM compiler infrastructure project, the Clang compiler, the Swift programming language, a lot of key contributions to TensorFlow and TPUs as part of Google.

He served as vice president of autopilot software at Tesla, was a software innovator and leader at Apple, and now he co-created a new full stack AI infrastructure for distributed training, inference, and deployment on all kinds of hardware called modular, and a new programming language called Mojo.

That is a superset of Python, giving you all the usability of Python, but with a performance of C, C++.

In many cases, Mojo code has demonstrated over 30, 000x speed up over Python.

If you love machine learning, if you love Python, you should definitely give Mojo a try.

This programming language, this new AI framework and infrastructure, and this conversation with Chris is mind-blowing.

It gets pretty technical at times, so I hope you hang on for the ride.

To support it, please check out our sponsors in the description.

And now, dear friends, here's Chris Latner.

It's been, I think, 2 years since we last talked, and in that time, you somehow went and co-created a new programming language called Mojo.

So it's optimized for AI, it's a superset of Python.

For Mojo, well, so I mean, I think you have to zoom out.

So I've been working on a lot of related technologies for many, many years.

So I've worked on LLVM and a lot of things and mobile and servers and things like this.

And what's happened with AI is we have new GPUs and new machine learning accelerators and other ASICs and things like that that make AI go real fast.

That's 1 of the biggest, largest scale deployed systems that exist for AI.

And really what you see is if you look across all of the things that are happening in the industry, there's this new compute platform coming.

And it's not just about CPUs or GPUs or TPUs or NPUs or IPUs or whatever, all the PUs.

It's about how do we program these things.

And so for software folks like us, it doesn't do us any good if there's this amazing hardware that we can't use.

And 1 of the things you find out really quick is that having the theoretical capability of programming something and then having the world's power and the innovation of all the smart people in the world get unleashed on something can be quite different.

And so really where Mojo came from was starting from a problem of we need to be able to take machine learning, take the infrastructure underneath it and make it way more accessible, way more usable, way more understandable by normal people and researchers and other folks that are not themselves like experts in GPUs and things like this.

And then through that journey, we realized, hey, we need syntax for this.

So 1 of the main features of the language, I say so fully in jest, is that it allows you to have the file extension to be an emoji, or the fire emoji, which is 1 of the first emojis used as a file extension I've ever seen in my life.

And then you ask yourself the question, why in the 21st century are we not using Unicode for file extensions?

I think clearly the most important decision you made, the most, but you could also just use MOJO as the file extension.

I mean, come on, Lex, you think that the world's ready for this?

This is a big moment in the world, right?

This is, we're releasing this onto the world.

Emojis are such a big part of our daily lives.

Well, and like you take a step back and look at what file extensions are, right?

And so why are we spending all the screen space on them and all this stuff?

Also, you know, you have them stacked up next to text files and PDF files and whatever else, like if you're going to do something cool, you want to stand out, right?

Emojis are colorful, they're visual, they're beautiful.

What's been the response so far from, is there support on Windows on operating systems in displaying like File Explorer?

The 1 problem I've seen is that Git doesn't escape it right.

And so it thinks that the fire emoji is unprintable and so it like prints out weird hex things if you use the command line git tool.

But everything else as far as I'm aware works fine.

And I have faith that git can be improved.

Visual Studio Code, Windows, like all this stuff, totally ready.

Because people have internationalization in their normal part of their paths.

So this is just like taking the next step.

Somewhere between, oh wow, that makes sense.

This can never be, I can never handle this.

This is something where I think that the world will get there.

We don't have to bet the whole farm on this.

I think we can provide both paths, but I think it'll be great.

When can we have emojis as part of the code, I wonder?

Yeah, so I mean, lots of languages provide that.

So I think that we have partial support for that.

It's probably not fully done yet, but yeah, you can do that.

For example, in Swift, you can do that for sure.

So an example we gave at Apple was the dog cow.

So that's a classical Mac heritage thing.

And so you use the dog and the cow emoji together and that could be your variable name.

But of course the internet went and made pile of poop for everything.

So if you wanna name your function pile of poop, then you can totally go to town and see how that gets through code review.

Okay, so let me just ask a bunch of random questions.

So is Mojo primarily designed for AIs or is it a general purpose programming language?

So it's AI first, and so AI is driving a lot of the requirements, and so Modular is building and designing and driving Mojo forward, and it's not because it's an interesting project theoretically to build, it's because we need it.

And so at Modular, we're really tackling the AI infrastructure landscape and the big problems in AI, and the reasons that it is so difficult to use and scale and adopt and deploy and like all these big problems in AI.

And so we're coming at it from that perspective.

Now when you do that, when you start tackling these problems, you realize that the solution to these problems isn't actually an AI-specific solution.

And so while we're doing this, we're building Mojo to be a fully general programming language.

And that means that you can obviously tackle GPUs and CPUs and like these AI things, but it's also a really great way to build NumPy and other things like that.

Or, you know, just if you look at what many Python libraries are today, often they're a layer of Python for the API, and they end up being C and C++ code underneath them.

That's very true in AI, that's true in lots of other domains as well.

Anytime you see this pattern, that's an opportunity for Mojo to help simplify the world and help people have 1 thing.

To optimize through simplification by having 1 thing.

Mojo's the programming language, modular is the whole software stack.

So just over a year ago, we started this company called Modular.

Okay, what Modular's about is, It's about taking AI and up-leveling it into the next generation.

And so if you take a step back, what's gone on in the last 5678 years is that we've had things like TensorFlow and PyTorch and these other systems come in.

And what's happened is these things have grown like crazy.

It's being used to power so many systems.

It used to be controversial years ago, but now it's a thing.

But the challenge with these systems is that they haven't always been thought out with current demands in mind.

And so you think about it, where were LLMs 8 years ago?

And a lot of what people are doing today are very different than when these systems were built.

And meanwhile, the hardware side of this has gotten into a huge mess.

There's tons of new chips and accelerators and every big company is announcing a new chip every day, it feels like.

And so between that, you have this moving system on 1 side, moving system on the other side, and it just turns into this gigantic mess, which makes it very difficult for people to actually use AI, particularly in production deployment scenarios.

And so what Modular's doing is we're helping build out that software stack to help solve some of those problems, so then people can be more productive and get more AI research into production.

Now, what Mojo does is it's a really, really, really important piece of that.

And so that is part of that engine and part of the technology that allows us to solve these problems.

So Mojo is a programming language that allows you to do the high level programming, the low level programming, they do all kinds of programming in that spectrum that gets you closer and closer to the hardware.

You're a guy who knows love, I know this.

How it feels like I'm writing natural language English.

How when I can not just write, but read other people's codes, somehow I can understand it faster.

It's more condensed than other languages, like ones I'm really familiar with, like C++ and C.

We'll probably talk about some of them, but list comprehensions and stuff like this.

Well, so, And don't forget the entire ecosystem of all the packages.

Because there's always something, if you want to do anything, there's always a package.

Yeah, so it's not just the ecosystem of the packages and the ecosystem of the humans that do it.

That's a really, that's an interesting dynamic.

think something about the usability and the ecosystem makes the thing viral, it grows and it's a virtuous cycle.

Well, there's many things that went into that.

Like, so I think that ML was very good for Python.

And so I think that TensorFlow and PyTorch and these systems embracing Python really took and helped Python grow.

But I think that the major thing underlying it is that Python's like the universal connector.

It really helps bring together lots of different systems so you can compose them and build out larger systems without having to understand how it works.

But then what is the problem with Python?

Well, I guess you could say several things, but probably that it's slow.

I think that's usually what people complain about, right?

And so, I mean, other people complain about tabs and spaces versus curly braces or whatever.

But I mean, those people are just wrong because it is actually just better to use indentation.

Listen, I've recently left Emacs for VS Code.

Because on the way to doing that, I also said I've considered Vim.

You're touching on deep religions, right?

Anyway, tabs is an interesting design decision.

You've really written a new programming language here.

Yes, it is a superset of Python, but you can make a bunch of different interesting decisions here.

And you chose actually to stick with Python as in terms of some of the syntax.

So, I mean, you can explain this in many rational ways.

I think that the annotation is beautiful, but that's not a rational explanation, but I can defend it rationally.

First of all, Python 1 has millions of programmers.

Second of all, if you look at it, C code, C++ code, Java, whatever, Swift, curly brace languages also run through formatting tools and get indented.

And so if they're not indented correctly, first

There's notorious bugs that have happened across time where the indentation was wrong or misleading and it wasn't formatted right, and so it turned into an issue, right?

And so what ends up happening in modern, large-scale code bases is people run automatic formatters.

So now what you end up with is indentation and curly braces.

Well, if you're gonna have the notion of grouping, why not have 1 thing, right?

And get rid of all the clutter and have a more beautiful thing.

Also, you look at many of these languages, it's like, okay, well, you can have curly braces or you can omit them if there's 1 statement or you just like enter this entire world of complicated design space that objectively you don't need if you have Python style indentation.

Yeah, I would love to actually see statistics on errors made because of indentation Like how many errors are made in Python versus in C++ that have to do with basic formatting all that kind of stuff I would love to see

I think it's probably pretty minor because once you get Like you use VS code I do too.

So if you get VS Code set up, it does the indentation for you generally.

So it's actually really nice to not have to fight it.

Then what you can see is the editor is telling you how your code will work by indenting it, which I think is pretty cool.

I honestly don't think I've ever, I don't remember having an error in Python because I intended stuff wrong.

So I mean, I think that there's, again, this is a religious thing.

And I love to kind of, you know, I realized that this is such a polarizing thing and everyone wants to argue about it.

So I like poking at the bear a little bit.

But frankly, right, come back to the first point, Python 1.

It's huge, it's in AI, it's the right thing.

For us, we see Mojo as being an incredible part of the Python ecosystem.

We're not looking to break Python or change it or quote-unquote fix it.

Our view is that Python is just not done yet.

And so if you look at, you know, you mentioned Python being slow.

Well, there's a couple of different things that go into that which we can talk about if you want.

But 1 of them is that it just doesn't have those features that you would use to do C-like programming.

And so if you say, okay, well, I'm forced out of Python into C for certain use cases, well, then what we're doing is we're saying, okay, well, Why is that?

Can we just add those features that are missing from Python back up to Mojo?

And then you can have everything that's great about Python, all the things that you're talking about that you love, plus not be forced out of it when you do something a little bit more computationally intense or weird or hardware-y or whatever it is that you're doing.

Well, a million questions I want to ask, but high level again, is it compiled or is it an interpreted language?

So Mojo, the complicated answer, does all the things.

So it's interpreted, it's JIT compiled, and it's statically compiled.

So 1 of the things that makes Python beautiful is that it's very dynamic.

And because it's dynamic, 1 of the things they added is that it has this powerful metaprogramming feature.

And so if you look at something like PyTorch or TensorFlow or, I mean, even a simple use case like you define a class that has the plus method, right?

You can overload the Dunder methods like Dunder add, for example, and then the plus method works on your class.

It has very nice and very expressive dynamic metaprogramming features.

In Mojo, we want all those features come in.

We don't want to break Python, we want it all to work.

But the problem is, is you can't run those super dynamic features on an embedded processor or on a GPU, or if you could, you probably don't want to just because of the performance.

And so we entered this question of saying, okay, how do you get the power of this dynamic metaprogramming into a language that has to be super efficient in specific cases?

And so what we did was we said, okay, we'll take that interpreter.

Take that interpreter and allow it to run at compile time.

And so now what you get is you get compile time metaprogramming.

And so this is super interesting and super powerful because 1 of the big advantages you get is you get Python-style expressive APIs.

You get the ability to have overloaded operators.

And if you look at what happens inside of PyTorch, for example, with automatic differentiation, and eager mode, and all these things, they're using these really dynamic and powerful features at runtime.

But we can take those features and lift them so that they run at compile time.

So you're, because C++ has meta programming with templates, but it's really messy.

It's super messy, it's always, it was accidentally, I mean, different people have different interpretations.

My interpretation is that it was made accidentally powerful.

It was not designed to be Turing complete, for example, but that was discovered kind of along the way, accidentally.

And so there've been a number of languages in the space.

And so they usually have templates or code instantiation, code copying features of various sorts.

Some more modern languages or some more newer languages, let's say, they're fairly unknown like ZIG, for example.

Says, okay, well, let's take all of those types that you can run it, all those things you can do at runtime and allow them to happen at compile time.

So 1 of the problems with C++, I mean, which is 1 of the problems with C++.

I mean, everybody hates me for a variety of reasons anyways, I'm sure.

I've written enough C++ code to earn a little bit of grumpiness with C++.

But 1 of the problems with it is that the metaprogramming system templates is just a completely different universe from the normal runtime programming world.

And so if you do metaprogramming and programming, it's just like a different universe, different syntax, different concepts, different stuff going on.

And so again, 1 of our goals with Mojo is to make things really easy to use, easy to learn.

And so as you do this, you say, OK, well, I have to do programming at runtime.

I have to do programming at compile time.

Because that sounds, to me as a fan of metaprogramming in C++ even, How hard is it to pull that off?

That sounds really, really exciting because you can do the same style programming at compile time and at run time.

And so, I mean, in terms of the compiler implementation details, it's hard.

It requires, I mean, what Mojo has underneath the covers is a completely new approach to the design of the compiler itself.

This builds on these technologies like MLIR that you mentioned, but it also includes other caching and other interpreters and JIT compilers and other stuff like that.

And so it really takes the standard model of programming languages and kind of twists it and unifies it with the runtime model, which I think is really cool.

And to me, the value of that is that, again, many of these languages have metaprogramming features.

Like they grow macros or something, right?

And so, you know, if you go back to Lisp, 1 of the most powerful things about it is that it said that the metaprogramming and the programming are the same, right?

And so that made it way simpler, way more consistent, way easier to understand and reason about, and it made it more composable.

So if you build a library, you can use it both at runtime and compile time, which is pretty cool.

Yeah, and for machine learning, I think metaprogramming, I think we could generally say is extremely useful.

See all Lex Fridman transcripts on Youtube

Chris Lattner: Future of Programming and AI | Lex Fridman Podcast #381