Lex Fridman Podcast full episode: https://www.youtube.com/watch?v=cdiD-9MMpb0
Please support this podcast by checking out our sponsors:
- Eight Sleep: https://www.eightsleep.com/lex to get special savings
- BetterHelp: https://betterhelp.com/lex to get 10% off
- Fundrise: https://fundrise.com/lex
- Athletic Greens: https://athleticgreens.com/lex to get 1 month of fish oil

GUEST BIO:
Andrej Karpathy is a legendary AI researcher, engineer, and educator. He's the former director of AI at Tesla, a founding member of OpenAI, and an educator at Stanford.

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

Looking back, what is the most beautiful or surprising idea in deep learning or AI in general that you've come across?

You've seen this field explode and grow in interesting ways.

Just what cool ideas, like what made you sit back and go, small, big or small?

Well, the 1 that I've been thinking about recently, the most probably is the transformer architecture.

So basically neural networks have a lot of architectures that were trendy have come and gone for different sensory modalities, like for vision, audio, text, you would process them with different looking neural nets.

And recently we've seen this convergence towards 1 architecture, the transformer.

And you can feed it video, or you can feed it, you know, images or speech or text, and it just gobbles it up.

And it's kind of like a bit of a general purpose computer that is also trainable and very efficient to run on our hardware.

And So this paper came out in 2016, I want to say.

You criticized the paper title in retrospect that it wasn't, it didn't foresee the bigness of the impact

Yeah, I'm not sure if the authors were aware of the impact that paper would go on to have.

But I think they were aware of some of the motivations and design decisions behind the transformer, and they chose not to, I think, expand on it in that way in the paper.

And so I think they had an idea that there was more than just the surface of just like, Oh, we're just doing translation and here's a better architecture.

This is like a really cool, differentiable, optimizable, efficient computer that you've proposed.

And maybe they didn't have all of that foresight, but I think it's really interesting.

Isn't it funny, sorry to interrupt, that that title is memeable, that they went for such a profound idea, they went with a, I don't think anyone used that kind of title before, right?

Yeah, it's like a meme or something, basically.

That 1, like, maybe if it was a more serious title,

Honestly, I, yeah, there is an element of me that honestly agrees with you and prefers it this way.

promise and then under deliver potentially.

So you want to just meme your way to greatness.

So you tweeted that Transformer is a magnificent neural network architecture because it is a general purpose differentiable computer.

It is Simultaneously expressive in the forward pass, optimizable via back propagation gradient descent, and efficient, high parallelism compute graph.

Can you discuss some of those details, expressive, optimizable, efficient, from memory or in general, whatever comes to your heart.

You want to have a general purpose computer that you can train on arbitrary problems, like say the task of next word prediction or detecting if there's a cat in an image or something like that.

And you want to train this computer, so you want to set its weights.

And I think there's a number of design criteria that sort of overlap in the transformer simultaneously that made it very successful.

And I think the authors were kind of deliberately trying to make this a really powerful architecture.

And so basically it's very powerful in the forward pass because it's able to express very general computation as sort of something that looks like message passing.

You have nodes, and they all store vectors.

And these nodes get to basically look at each other, and it's each other's vectors.

And basically nodes get to broadcast, hey, I'm looking for certain things.

And then other nodes get to broadcast, hey, these are the things I have.

Transformer is much more than just the attention component.

It's got many pieces architectural that went into it, the residual connection, the way it's arranged, there's a multi-layer perceptron in there, the way it's stacked, and so on.

But basically, there's a message passing scheme where nodes get to look at each other, decide what's interesting, and then update each other.

And so I think the, when you get to the details of it, I think it's a very expressive function.

So it can express lots of different types of algorithms in forward pass.

Not only that, but the way it's designed with the residual connections, layer normalizations, the softmax attention and everything, it's also optimizable.

This is a really big deal because there's lots of computers that are powerful that you can't optimize or they're not easy to optimize using the techniques that we have, which is backpropagation and gradient descent.

These are first order methods, very simple optimizers really.

And so you also need it to be optimizable.

And then lastly, you want it to run efficiently in our hardware.

Our hardware is a massive throughput machine like GPUs.

So you don't want to do lots of sequential operations.

You want to do a lot of operations serially.

And the Transformer is designed with that in mind as well.

And so it's designed for our hardware and it's designed to both be very expressive in a forward pass, but also very optimizable in the backward pass.

And you said that the residual connections support a kind of ability to learn short algorithms faster than first, and then gradually extend them longer during training.

What's the idea of learning short algorithms?

Think of it as a, so basically a transformer is a series of blocks, right?

And these blocks have a tension and a little multilayer perceptron.

And so you go off into a block and you come back to this residual pathway and then you go off and you come back and then you have a number of layers arranged sequentially.

And so the way to look at it I think is because of the residual pathway in the backward path the gradients sort of flow along it uninterrupted because addition distributes the gradient equally to all of its branches.

So the gradient from the supervision at the top just floats directly to the first layer.

And all the residual connections are arranged so that in the beginning during initialization, they contribute nothing to the residual pathway.

So what it kind of looks like is, imagine the transformer is kind of like a Python function, like a def.

And you get to do various kinds of lines of code.

Say you have 100 layers deep transformer.

Typically, they would be much shorter, say 20.

So you have 20 lines of code and you can do something in them.

And so think of during the optimization, basically what it looks like is first you optimize the first line

of code and then the second line of code can kick in and the third line of code can kick in.

And I kind of feel like because of the residual pathway and the dynamics of the optimization, you can learn a very short algorithm that gets the approximate answer, but then the other layers can kick in and start to create a contribution.

And at the end of it, you're optimizing over an algorithm that is 20 lines of code.

Except these lines of code are very complex, because it's an entire block of a transformer.

What's really interesting is that this transformer architecture actually has been remarkably resilient.

Basically, the transformer that came out in

is the transformer you would use today, except you reshuffle some of the layer norms.

The layer normalizations have been reshuffled to a pre-norm formulation.

And so it's been remarkably stable, but there's a lot of bells and whistles that people have attached on it and tried to improve it.

I do think that basically it's a big step in simultaneously optimizing for lots of properties of a desirable neural network architecture.

And I think people have been trying to change it, but it's proven remarkably resilient.

But I do think that there should be even better architectures potentially.

But it's, you admire the resilience here.

There's something profound about this architecture that leads to resilience.

So maybe we can, everything could be turned into a, into a problem that transformers can solve.

Currently, it definitely looks like the transformer is taking over AI and you can feed basically arbitrary problems into it and it's a general differentiable computer and it's extremely powerful and this conversions in AI has been really interesting to watch for me personally.

What else do you think could be discovered here about transformers?

Or is it a stable, I want a stable place.

Is there something interesting we might discover about Transformers, like aha moments, maybe has to do with memory, maybe knowledge representation, that kind of stuff.

Definitely the Zeitgeist today is just pushing, like basically right now the Zeitgeist is do not touch the Transformer, Touch everything else.

So people are scaling up the data sets, making them much, much bigger.

They're working on the evaluation, making the evaluation much, much bigger.

And they're basically keeping the architecture unchanged.

And that's how we've, that's the last 5 years of progress in AI, kind of.


Speaker 0: Looking back, what is the most beautiful or surprising idea in deep learning or AI in general that you've come across? You've seen this field explode and grow in interesting ways. Just what cool ideas, like what made you sit back and go, small, big or small?

Speaker 1: Well, the 1 that I've been thinking about recently, the most probably is the transformer architecture. So basically neural networks have a lot of architectures that were trendy have come and gone for different sensory modalities, like for vision, audio, text, you would process them with different looking neural nets. And recently we've seen this convergence towards 1 architecture, the transformer. And you can feed it video, or you can feed it, you know, images or speech or text, and it just gobbles it up. And it's kind of like a bit of a general purpose computer that is also trainable and very efficient to run on our hardware.

And So this paper came out in 2016, I want to say.

Speaker 0: Attention is all you need.

Speaker 1: Attention is all you need.

Speaker 0: You criticized the paper title in retrospect that it wasn't, it didn't foresee the bigness of the impact

Speaker 1: that it was going to have. Yeah, I'm not sure if the authors were aware of the impact that paper would go on to have. Probably they weren't. But I think they were aware of some of the motivations and design decisions behind the transformer, and they chose not to, I think, expand on it in that way in the paper. And so I think they had an idea that there was more than just the surface of just like, Oh, we're just doing translation and here's a better architecture.

You're not just doing translation. This is like a really cool, differentiable, optimizable, efficient computer that you've proposed. And maybe they didn't have all of that foresight, but I think it's really interesting.

Speaker 0: Isn't it funny, sorry to interrupt, that that title is memeable, that they went for such a profound idea, they went with a, I don't think anyone used that kind of title before, right?

Speaker 1: Attention is all you need. Yeah, it's like a meme or something, basically.

Speaker 0: Isn't that funny? That 1, like, maybe if it was a more serious title,

Speaker 1: it

Speaker 0: wouldn't have the impact.

Speaker 1: Honestly, I, yeah, there is an element of me that honestly agrees with you and prefers it this way. Yes. Ah.

Speaker 2: If it

Speaker 1: was too grand, it would over

Speaker 2: promise and then under deliver potentially.

Speaker 1: So you want to just meme your way to greatness.

Speaker 0: That should be a t-shirt. So you tweeted that Transformer is a magnificent neural network architecture because it is a general purpose differentiable computer. It is Simultaneously expressive in the forward pass, optimizable via back propagation gradient descent, and efficient, high parallelism compute graph. Can you discuss some of those details, expressive, optimizable, efficient, from memory or in general, whatever comes to your heart.

Speaker 1: You want to have a general purpose computer that you can train on arbitrary problems, like say the task of next word prediction or detecting if there's a cat in an image or something like that. And you want to train this computer, so you want to set its weights. And I think there's a number of design criteria that sort of overlap in the transformer simultaneously that made it very successful. And I think the authors were kind of deliberately trying to make this a really powerful architecture. And so basically it's very powerful in the forward pass because it's able to express very general computation as sort of something that looks like message passing.

You have nodes, and they all store vectors. And these nodes get to basically look at each other, and it's each other's vectors. And they get to communicate. And basically nodes get to broadcast, hey, I'm looking for certain things. And then other nodes get to broadcast, hey, these are the things I have.

Those are the keys and the values.

Speaker 0: So it's not just attention.

Speaker 1: Yeah, exactly. Transformer is much more than just the attention component. It's got many pieces architectural that went into it, the residual connection, the way it's arranged, there's a multi-layer perceptron in there, the way it's stacked, and so on. But basically, there's a message passing scheme where nodes get to look at each other, decide what's interesting, and then update each other. And so I think the, when you get to the details of it, I think it's a very expressive function.

So it can express lots of different types of algorithms in forward pass. Not only that, but the way it's designed with the residual connections, layer normalizations, the softmax attention and everything, it's also optimizable. This is a really big deal because there's lots of computers that are powerful that you can't optimize or they're not easy to optimize using the techniques that we have, which is backpropagation and gradient descent. These are first order methods, very simple optimizers really. And so you also need it to be optimizable.

And then lastly, you want it to run efficiently in our hardware. Our hardware is a massive throughput machine like GPUs. They prefer lots of parallelism. So you don't want to do lots of sequential operations. You want to do a lot of operations serially.

And the Transformer is designed with that in mind as well. And so it's designed for our hardware and it's designed to both be very expressive in a forward pass, but also very optimizable in the backward pass.

Speaker 0: And you said that the residual connections support a kind of ability to learn short algorithms faster than first, and then gradually extend them longer during training. What's the idea of learning short algorithms?

Speaker 1: Right. Think of it as a, so basically a transformer is a series of blocks, right? And these blocks have a tension and a little multilayer perceptron. And so you go off into a block and you come back to this residual pathway and then you go off and you come back and then you have a number of layers arranged sequentially. And so the way to look at it I think is because of the residual pathway in the backward path the gradients sort of flow along it uninterrupted because addition distributes the gradient equally to all of its branches.

So the gradient from the supervision at the top just floats directly to the first layer. And all the residual connections are arranged so that in the beginning during initialization, they contribute nothing to the residual pathway. So what it kind of looks like is, imagine the transformer is kind of like a Python function, like a def. And you get to do various kinds of lines of code. Say you have 100 layers deep transformer.

Typically, they would be much shorter, say 20. So you have 20 lines of code and you can do something in them. And so think of during the optimization, basically what it looks like is first you optimize the first line

Speaker 2: of code and then the second line of code can kick in and the third line of code can kick in.

Speaker 1: And I kind of feel like because of the residual pathway and the dynamics of the optimization, you can learn a very short algorithm that gets the approximate answer, but then the other layers can kick in and start to create a contribution. And at the end of it, you're optimizing over an algorithm that is 20 lines of code. Except these lines of code are very complex, because it's an entire block of a transformer.

Speaker 2: You can do a lot in there.

Speaker 1: What's really interesting is that this transformer architecture actually has been remarkably resilient. Basically, the transformer that came out in

Speaker 0: 2016

Speaker 1: is the transformer you would use today, except you reshuffle some of the layer norms. The layer normalizations have been reshuffled to a pre-norm formulation. And so it's been remarkably stable, but there's a lot of bells and whistles that people have attached on it and tried to improve it. I do think that basically it's a big step in simultaneously optimizing for lots of properties of a desirable neural network architecture. And I think people have been trying to change it, but it's proven remarkably resilient.

But I do think that there should be even better architectures potentially.

Speaker 0: But it's, you admire the resilience here. Yeah. There's something profound about this architecture that leads to resilience. So maybe we can, everything could be turned into a, into a problem that transformers can solve.

Speaker 1: Currently, it definitely looks like the transformer is taking over AI and you can feed basically arbitrary problems into it and it's a general differentiable computer and it's extremely powerful and this conversions in AI has been really interesting to watch for me personally.

Speaker 0: What else do you think could be discovered here about transformers? Like what's surprising thing? Or is it a stable, I want a stable place. Is there something interesting we might discover about Transformers, like aha moments, maybe has to do with memory, maybe knowledge representation, that kind of stuff.

Speaker 1: Definitely the Zeitgeist today is just pushing, like basically right now the Zeitgeist is do not touch the Transformer, Touch everything else. So people are scaling up the data sets, making them much, much bigger. They're working on the evaluation, making the evaluation much, much bigger. And they're basically keeping the architecture unchanged. And that's how we've, that's the last 5 years of progress in AI, kind of.

Speaker 0: You

See all Lex Clips transcripts on Youtube

Transformers: The best idea in AI | Andrej Karpathy and Lex Fridman