Ilya Sutskever is the co-founder of OpenAI, is one of the most cited computer scientist in history with over 165,000 citations, and to me, is one of the most brilliant and insightful minds ever in the field of deep learning. There are very few people in this world who I would rather talk to and brainstorm with about deep learning, intelligence, and life than Ilya, on and off the mic.

Support this podcast by signing up with these sponsors:
- Cash App - use code "LexPodcast" and download:
- Cash App (App Store): https://apple.co/2sPrUHe
- Cash App (Google Play): https://bit.ly/2MlvP5w

EPISODE LINKS:
Ilya's Twitter: https://twitter.com/ilyasut
Ilya's Website: https://www.cs.toronto.edu/~ilya/

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:23 - AlexNet paper and the ImageNet moment
8:33 - Cost functions
13:39 - Recurrent neural networks
16:19 - Key ideas that led to success of deep learning
19:57 - What's harder to solve: language or vision?
29:35 - We're massively underestimating deep learning
36:04 - Deep double descent
41:20 - Backpropagation
42:42 - Can neural networks be made to reason?
50:35 - Long-term memory
56:37 - Language models
1:00:35 - GPT-2
1:07:14 - Active learning
1:08:52 - Staged release of AI systems
1:13:41 - How to build AGI?
1:25:00 - Question to AGI
1:32:07 - Meaning of life

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Ilya Saskever, co-founder and chief scientist of OpenAI, 1 of the most cited computer scientists in history with over 165,000 citations.

And to me, 1 of the most brilliant and insightful minds ever in the field of deep learning.

There are very few people in this world who I would rather talk to and brainstorm with about deep learning, intelligence, and life in general than Ilya, on and off the mic.

This conversation was recorded before the outbreak of the pandemic.

For everyone feeling the medical, psychological, and financial burden of this crisis, I'm sending love your way.

Stay strong, we're in this together, we'll beat this thing.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, review it with 5 Stars and Apple Podcast, support it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D-M-A-N.

As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation.

I hope that works for you and doesn't hurt the listening experience.

This show is presented by Cash App, the number 1 finance app in the App store.

Cash app lets you send money to friends, buy Bitcoin, invest in the stock market with as little as $1.

Since cash app allows you to buy Bitcoin, let me mention that cryptocurrency in the context of the history of money is fascinating.

I recommend A Scent of Money as a great book on this history.

Debits and credits on ledgers started around 30,000 years ago.

The US dollar created over 200 years ago, and Bitcoin, the first decentralized cryptocurrency, released just over 10 years ago.

So given that history, cryptocurrency is still very much in its early days of development, but it's still aiming to, and just might, redefine the nature of money.

So again, if you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10 and Cash App will also donate $10 to FIRST, an organization that is helping advance robotics and STEM education for young people around the world.

And now, here's my conversation with Ilya Sutskever.

You were 1 of the 3 authors with Alex Khrushchevsky, Jeff Hinton of the famed Alex Ned paper that is arguably the paper that marked the big catalytic moment that launched the deep learning revolution.

At that time, take us back to that time, what was your intuition about neural networks, about the representational power of neural networks?

And maybe you can mention how did that evolve over the next few years up to today, over the 10 years?

At some point, we realized that we can train very large, I shouldn't say very, you know, they were tiny by today's standards, but large and deep neural networks end to end with back propagation.

At some point, different people obtained this result.

The first moment in which I realized that deep neural networks are powerful was when James Martens invented the Hessian free optimizer in 2010 and he trained a 10-layer neural network end-to-end without pre-training from scratch and when that happened I thought this is it because if you can train a big neural network a big neural network can represent very complicated function.

Because if you have a neural network with 10 layers, it's as though you allow the human brain to run for some number of milliseconds.

Neuron firings are slow, And so in maybe 100 milliseconds, your neurons only fire 10 times.

And in 100 milliseconds, you can perfectly recognize any object.

So I thought, so I already had the idea then that we need to train a very big neural network on lots of supervised data, and then it must succeed, because we can find the best neural network.

And then there's also theory that if you have more data than parameters, you won't overfit.

Today we know that actually this theory is very incomplete and you won't overfit even if you have less data than parameters, but definitely if you have more data than parameters, you won't overfit.

So the fact that neural networks were heavily over-parameterized wasn't discouraging to you?

So you were thinking about the theory that the number of parameters, the fact there's a huge number of parameters is okay?

I mean, there was some evidence before that it was okay-ish, but the theory was that if you had a big dataset and a big neural net, it was going to work.

The overparameterization just didn't really figure much as a problem.

I thought, well, with images, you just go and add some data augmentation and it's going to be okay.

The main doubt was can we train a big enough...

Will we have enough compute to train a big enough neural net?

The thing which wasn't clear was whether there would be enough compute to get a very convincing result.

And then at some point Alex Kirzhevsky wrote these insanely fast CUDA kernels for training convolutional neural nets.

And that was bam, let's do this, let's get ImageNet, and it's gonna be the greatest thing.

Was your intuition, most of your intuition from empirical results by you and by others?

So like just actually demonstrating that a piece of program can train a 10-layer neural network, or was there some pen and paper, or marker and whiteboard thinking, intuition?

Because you just connected a 10-layer large neural network to the brain, So you just mentioned the brain.

So in your intuition about neural networks, does the human brain come into play as a intuition builder?

I mean, you know, you gotta be precise with these analogies between artificial neural networks and the brain, but there's no question that the brain is a huge source of intuition and inspiration for deep learning researchers since all the way from Rosenblatt in the 60s Like if you look at the whole idea of a neural network is directly inspired by the brain You had people like McCallum and Pitts who were saying, hey, you got these neurons in the brain and hey, we recently learned about the computer and automata, can we use some ideas from the computer and automata to design some kind of computational object that's going to be simple, computational and kind of like the brain?

Then you had the convolutional neural network from Fukushima, and then later, Jan Lekan, who said, hey, if you limit the receptive fields of a neural network, it's gonna be especially suitable for images, as it turned out to be true.

So there was a very small number of examples where analogies to the brain were successful.

And I thought, well, probably an artificial neuron is not that different from the brain if you squint hard enough, so let's just assume it is and roll with it.

So we're now at a time where deep learning is very successful, so let us squint less and say, let's open our eyes and say, what to you is an interesting difference between the human brain, now I know you're probably not an expert, neither a neuroscientist nor a biologist, but loosely speaking, what's the difference between the human brain and artificial neural networks?

That's interesting to you for the next decade or 2.

What is an interesting difference between the brain and our artificial neural networks.

So I feel like today, artificial neural networks, so we all agree that there are certain dimensions in which the human brain vastly outperforms our models.

But I also think that there are some ways in which our artificial neural networks have a number of very important advantages over the brain.

Looking at the advantages versus disadvantages is a good way to figure out what is the important difference.

So the brain uses spikes, which may or may not be important.

Yes, That's a really interesting question.

That's 1 big architectural difference between artificial neural networks.

It's hard to tell, but my prior is not very high, and I can say why.

You know, there are people who are interested in spiking neural networks, And basically what they figured out is that they need to simulate the non-spiking neural networks in spikes And that's how they're gonna make them work.

If you don't simulate the non-spiking neural networks in spikes It's not going to work because the question is why should it work and that connects to questions around backpropagation and questions around Deep learning.

Why should the learning rule work at all?

It's not a self-evident question, especially if you, let's say, if you were just starting in the field and you read the very early papers, you can say, hey, people are saying, let's build neural networks.

That's a great idea because the brain is a neural network, so it would be useful to build neural networks.

It should be possible to train them probably, but how?

And so the big idea is the cost function.

The cost function is a way of measuring the performance of the system according to some measure.

By the way, that is a big, actually let me think.

Is that 1, a difficult idea to arrive at, and how big of an idea is that?

Is supervised learning a difficult concept to come to?

All concepts are very easy in retrospect.

Yeah, that's what, it seems trivial now, but I, so, because the reason I ask that, and we'll talk about it, because is there other things?

Is there things that don't necessarily have a cost function, or maybe have many cost functions, or maybe have dynamic cost functions, or maybe a totally different kind of architectures.

Because we have to think like that in order to arrive at something new, right?

So the good examples of things which don't have clear cost functions are GANs.

So instead of thinking of a cost function, where you want to optimize, where you know that you have an algorithm gradient descent, which will optimize the cost function.

And then you can reason about the behavior of your system in terms of what it optimizes.

With GAN you say, I have a game and I'll reason about the behavior of the system in terms of the equilibrium of the game.

But it's all about coming up with these mathematical objects that help us reason about the behavior of our system.

Yeah, so GAN is the only 1, it's kind of a, the cost function is emergent from the comparison.

I don't know if it's meaningful to talk about the cost function of a GAN.

It's kind of like the cost function of biological evolution or the cost function of the economy.

You can talk about regions to which it will go towards, but I don't think, I don't think the cost function analogy is the most useful.

So if evolution doesn't, that's really interesting.

So if evolution doesn't really have a cost function, like a cost function based on its, something akin to our mathematical conception of a cost function, then do you think cost functions in deep learning are holding us back?

Yeah, so you just kind of mentioned that cost function is a nice first profound idea.

So self-play starts to touch on that a little bit in reinforcement learning systems.

Self-play and also ideas around exploration where you're trying to take action that surprise a predictor.

I think cost functions are great and they serve us really well.

And I think that whenever we can do things with cost functions, we should.

And you know, maybe there is a chance that we will come up with some yet another profound way of looking at things that will involve cost functions in a less central way.

But I don't know, I think cost functions are, I mean, I would not bet against cost functions.

Is there other things about the brain that pop into your mind that might be different and interesting for us to consider in designing artificial neural networks?

I mean, 1 thing which may potentially be useful, I think people, neuroscientists have figured out something about the learning rule of the brain, or I'm talking about spike time independent plasticity, and it would be nice if some people were to study that in simulation.

Wait, sorry, spike time independent plasticity?

STD, it's a particular learning rule that uses spike timing to figure out how to, to determine how to update the synapses.

So it's kind of like, if a synapse fires into the neuron before the neuron fires, then it's strengthen the synapse.

And if the synapse fires into the neurons shortly after the neuron fired, then it weakens the synapse.

I'm 90% sure it's right, so if I said something wrong here, don't

But you sounded brilliant while saying it.

But the timing, that's 1 thing that's missing.

I think that's like a fundamental property of the brain is the timing of the signals.

Well, you have recurrent neural networks.

But you think of that as this, I mean that's a very crude, simplified, what's that called?

There's a clock, I guess, to recurrent neural networks.

It seems like the brain is the continuous version of that, the generalization, where all possible timings are possible and then within those timings is contained some information.

You think recurrent neural networks, the recurrence in recurrent neural networks can capture the same kind of phenomena as the timing that seems to be important for the brain, in the firing of neurons in the brain?

I mean, I think recurrent neural networks are amazing and they can do, I think they can do anything we'd want them to, we'd want a system to do.

Right now, recurrent neural networks have been superseded by transformers, but maybe 1 day they'll make a comeback.

Let me, on a small tangent, say, do you think they'll be back?

So, so much of the breakthroughs recently that we'll talk about on natural language processing and language modeling has been with transformers that don't emphasize recurrence.

Do you think recurrence will make a comeback?

Well, some kind of recurrence, I think very likely.

Recurrent neural networks for process, as they're typically thought of for processing sequences, I think it's also possible.

What is to you a recurrent neural network?

In generally speaking, I guess, what is a recurring neural network?

You have a neural network which maintains a high-dimensional hidden state.

And then when an observation arrives, it updates its high-dimensional hidden state through its connections in some way.

you know, that's what like expert systems did, right?

Symbolic AI, the knowledge-based, growing a knowledge base is maintaining a hidden state, which is its knowledge base and is growing it by sequentially processing.

Do you think of it more generally in that way?

Or is it simply, is it the more constrained form of a hidden state with certain kind of gating units that we think of as today with LSTMs and that?

I mean, the hidden state is technically what you described there, the hidden state that goes inside the LSTM or the RNN or something like this.

But then what should be contained, if you want to make the expert system analogy, I'm not, I mean, you could say that the knowledge is stored in the connections and then the short-term processing is done in the hidden state.

So, sort of, do you think there's a future of building large scale knowledge bases within the neural networks.

So we're gonna pause in that confidence because I wanna explore that.

But let me zoom back out and ask back to the history of ImageNet.

Neural networks have been around for many decades, as you mentioned.

What do you think were the key ideas that led to their success, that ImageNet moment and beyond, the success in the past 10 years.

Okay, so the question is, to make sure I didn't miss anything, the key ideas that led to the success of deep learning over the past 10 years.

Exactly, even though the fundamental thing behind deep learning has been around for much longer.

idea about deep learning, or rather the key fact about deep learning before deep learning started to be successful, is that it was underestimated.

People who worked in machine learning simply didn't think that neural networks could do much.

People didn't believe that large neural networks could be trained.

People thought that, well, there was a lot of debate going on in machine learning about what are the right methods and so on.

And people were arguing because there was no way to get hard facts.

And by that, I mean, there were no benchmarks which were truly hard that if you do really well on them, then you can say, look, here's my system.

That's when this field becomes a little bit more of an engineering field.

So in terms of deep learning, to answer the question directly, the ideas were all there.

The thing that was missing was a lot of supervised data and a lot of compute.

Once you have a lot of supervised data and a lot of compute.

Once you have a lot of supervised data and a lot of compute, then there is a third thing which is needed as well, and that is conviction.

Conviction that if you take the right stuff, which already exists, and apply and mix it with a lot of data and a lot of compute, that it will in fact work.

It was you had the, you needed the data, you needed the compute, which showed up in terms of GPUs, and you needed the conviction to realize that you need to mix them together.

So I guess the presence of compute and the presence of supervised data allowed the empirical evidence to do the convincing of the majority of the computer science community.

So I guess there's a key moment with Jitendra Malik and Alex, Alyosha Efros, who were very skeptical, right?

And then there's a Geoffrey Hinton that was the opposite of skeptical.

And there was a convincing moment, and I think ImageNet served as that moment.

And they represented this kind of, where the big pillars of computer vision community kinda, the wizards got together, and then all of a sudden there was a shift and it's not enough for the ideas to all be there and the computer to be there, it's for it to convince the cynicism that existed.

It's interesting that people just didn't believe for a couple of decades.

It's kind of, when put this way, it sounds like, well, you know, those silly people who didn't believe what were they missing.

But in reality, things were confusing because neural networks really did not work on anything And they were not the best method on pretty much anything as well.

And it was pretty rational to say, yeah, this stuff doesn't have any traction.

And that's why you need to have these very hard tasks, which are, which produce undeniable evidence.

And that's why the field is making progress today because we have these hard benchmarks which represent true progress.

And this is why we are able to avoid endless debate.

So incredibly, you've contributed some of the biggest recent ideas in AI, in computer vision, language, natural language processing, reinforcement learning, sort of everything in between.

There may not be a topic you haven't touched, and of course the fundamental science of deep learning.

What is the difference to you between vision, language, and as in reinforcement learning, action, as learning problems, and what are the commonalities?

Do you see them as all interconnected, or are they fundamentally different domains that require different approaches?

Machine learning is a field with a lot of unity, a huge amount of unity.

In fact, there's only 1 or 2 or 3 principles which are very, very simple.

And then they apply in almost the same way, in almost the same way to the different modalities to the different problems.

And that's why today, when someone writes a paper on improving optimization of deep learning in vision, it improves the different NLP applications and it improves the different reinforcement learning applications.

Reinforcement learning, so I would say that computer Vision and NLP are very similar to each other.

Today, they differ in that they have slightly different architectures.

We use transformers in NLP, and we use convolutional neural networks in Vision.

But it's also possible that 1 day this will change and everything will be unified with a single architecture.

Because if you go back a few years ago in natural language processing, there were a huge number of architectures for every different tiny problem had its own architecture.

Today, there's just 1 transformer for all those different tasks.

And if you go back in time even more, you had even more and more fragmentation and every little problem in AI had its own little sub-specialization and sub, you know, little set of collection of skills, people who would know how to engineer the features.

Now it's all been subsumed by deep learning.

And so I expect vision to become unified with natural language as well.

Or rather I shouldn't say expect, I think it's possible.

I don't want to be too sure because I think on the conventional neural net it's very computationally efficient.

RL does require slightly different techniques because you really do need to take action.

You really do need to do something about exploration.

But I think there is a lot of unity even there.

And I would expect for example that at some point there will be some broad unification between RL and supervised learning, where somehow the RL will be making decisions to make the supervised learning go better.

And it will be, I imagine 1 big black box and you just throw everything, you know, you shovel things into it and it just figures out what to do with whatever you shovel at it.

I mean, reinforcement learning has some aspects of language and vision combined almost.

There's elements of a long-term memory that you should be utilizing and there's elements of a really rich sensory space.

So it seems like the, it's like the union of the 2 or something like that.

I'd say that reinforcement learning is neither, but it naturally interfaces and integrates with the 2 of them.

You think action is fundamentally different?

So yeah, what is interesting about, what is unique about policy of learning to act?

Well, so 1 example, for instance, is that when you learn to act, you are fundamentally in a non-stationary world.

Because as your actions change, the things you see start changing.

You experience the world in a different way.

And this is not the case for the more traditional static problem where you have some distribution and you just apply a model to that distribution.

You think it's a fundamentally different problem or is it just a more difficult, it's a generalization of the problem of understanding?

I mean, it's a question of definitions almost.

There is a huge amount of commonality for sure.

You take gradients, you try to approximate gradients in both cases.

In the case of reinforcement learning, you have some tools to reduce the variance of the gradients.

You use the same neural net in both cases.

You compute the gradient, you apply Adam in both cases.

So, I mean, there's lots in common for sure, but there are some small differences which are not completely insignificant.

It's really just a matter of your point of view, what frame of reference, how much do you wanna zoom in or out as you look at these problems?

So people like Noam Chomsky believe that language is fundamental to everything.

See all Lex Fridman transcripts on Youtube

Ilya Sutskever: Deep Learning | Lex Fridman Podcast #94