Full episode with Ilya Sutskever (May 2020): https://www.youtube.com/watch?v=13CZPWmke6A
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Ilya Sutskever is the co-founder of OpenAI, is one of the most cited computer scientist in history with over 165,000 citations, and to me, is one of the most brilliant and insightful minds ever in the field of deep learning. There are very few people in this world who I would rather talk to and brainstorm with about deep learning, intelligence, and life than Ilya, on and off the mic.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

So incredibly, you've contributed some of the biggest recent ideas in AI, in computer vision, language, natural language processing, reinforcement learning, sort of everything in between, maybe not GANs.

Is there, there may not be a topic you haven't touched and of course the fundamental science of deep learning.

What is the difference to you between vision, language and as in reinforcement learning, action as learning problems and what are the commonalities?

Do you see them as all interconnected, or are they fundamentally different domains that require different approaches?

Machine learning is a field with a lot of unity, a huge amount of unity.

In fact, there's only 1 or 2 or 3 principles which are very, very simple.

And then they apply in almost the same way, in almost the same way to the different modalities, to the different problems.

And that's why today, when someone writes a paper on improving optimization of deep learning in vision, it improves the different NLP applications and it improves the different reinforcement learning applications.

Reinforcement learning, So I would say that computer vision and NLP are very similar to each other.

Today they differ in that they have slightly different architectures.

We use transformers in NLP and we use convolutional neural networks in vision.

But it's also possible that 1 day this will change and everything will be unified with a single architecture Because if you go back a few years ago in natural language processing There were a huge number of architectures for every different tiny problem had its own architecture Today there's just 1 transformer for all those different tasks.

And if you go back in time even more, you had even more and more fragmentation and every little problem in AI had its own little sub-specialization and sub, you know, little set of collection of skills, people who would know how to engineer the features.

Now it's all been subsumed by deep learning.

And so I expect vision to be communified with natural language as well.

Or rather I shouldn't say expect, I think it's possible.

I don't want to be too sure because I think on the conventional neural net it's very computationally efficient.

RL does require slightly different techniques because you really do need to take action.

You really do need to do something about exploration.

Your variance is much higher, But I think there is a lot of unity even there.

And I would expect, for example, that at some point there will be some broad unification between RL and supervised learning, where somehow the RL will be making decisions to make the supervised learning go better.

And there will be, I imagine, 1 big black box and you just throw every, you know, you shovel things into it and it just figures out what to do with whatever you shovel in it.

I mean, reinforcement learning has some aspects of language and vision combined, almost.

There's elements of a long-term memory that you should be utilizing and there's elements of a really rich sensory space.

So it seems like the, it's like the union of the 2 or something like that.

I'd say that reinforcement learning is neither, but it naturally interfaces and integrates with the 2 of them.

You think action is fundamentally different?

So yeah, what is interesting about, what is unique about policy of learning to act?

Well, so 1 example, for instance, is that when you learn to act you are fundamentally in a non-stationary world.

Because as your actions change, the things you see start changing.

You experience the world in a different way.

And this is not the case for the more traditional static problem where you have some distribution and you just apply a model to that distribution.

You think it's a fundamentally different problem or is it just a more difficult, it's a generalization of the problem of understanding?

I mean, it's a question of definitions almost.

There is a huge amount of commonality for sure.

You take gradients, you try to approximate gradients in both cases.

In the case of reinforcement learning, you have some tools to reduce the variance of the gradients.

There's lots of commonality, you use the same neural net in both cases.

You compute the gradient, you apply Adam in both cases.

So, I mean, there's lots in common for sure, but there are some small differences which are not completely insignificant.

It's really just a matter of your point of view, what frame of reference, how much do you want to zoom in or out as you look at these problems.

So people like Noam Chomsky believe that language is fundamental to everything.

Do you think language understanding is harder than visual scene understanding or vice versa?

I think that asking if a problem is hard is slightly wrong.

I think the question is a little bit wrong and I wanna explain why.

So what does it mean for a problem to be hard?

Okay, the non-interesting dumb answer to that is there's a benchmark and there's a human level performance on that benchmark And how is the effort required to reach the human level benchmark?

So from the perspective of how much until we get to human level on a very good benchmark?

Yeah, I understand what you mean by that.

So what I was going to say that a lot of it depends on, you know, once you solve a problem, it stops being hard.

And so, whether something is hard or not depends on what our tools can do today.

So, you know, you say today, true human level language understanding and visual perception are hard in the sense that there is no way of solving the problem completely in the next 3 months.

Beyond that, I'm just, my guess would be as good as yours, I don't know.

Oh, okay, so you don't have a fundamental intuition about how hard language understanding is.

I'd say language is probably going to be harder.

Like if you mean absolute, top-notch, 100% language understanding, I'll go with language.

But then if I show you a piece of paper with letters on it, is that, you see what I mean?

You have a vision system, you say it's the best human level vision system.

I show you, I open a book and I show you letters.

Will it understand how these letters form into word and sentences and meaning?

Where does vision end and language begin?

Yeah, so Chomsky would say it starts at language.

So vision is just a little example of the kind of structure and fundamental

hierarchy of ideas that's already represented in our brain somehow, that's represented through language.

But where does vision stop and language begin?

So 1 possibility is that it's impossible to achieve really deep understanding in either images or language without basically using the same kind of system.

So you're going to get the other for free.

I think it's pretty likely that yes, if we can get 1, our machine learning is probably that good that we can get the other.

And also, I think a lot of it really does depend on your definitions.

Because reading is vision, but should it count?

Yeah, to me, so my definition is if a system looked at an image and then a system looked at a piece of text and then told me something about that and I was really impressed,

You'll be impressed for half an hour and then you're gonna say, well, I mean all the systems do that, but here's the thing they don't do.

Well, The ones, okay, so I'm a fan of monogamy, so I like the idea of marrying somebody, being with them for several decades.

So I believe in the fact that yes, it's possible to have somebody continuously giving you pleasurable, interesting, witty, new ideas, friends.

The surprise, it's that injection of randomness.

It seems to be a nice source of, yeah, continued inspiration, like the wit, the humor.

I think, yeah, that would be, it's a very subjective test, but I think if you have enough humans in the room.

Yeah, I feel like I misunderstood what you meant by impressing you.

I thought you meant to impress you with its intelligence, with how well it understands an image.

I thought you meant something like, I'm gonna show it a really complicated image and it's gonna get it right and you're gonna say, wow, that's really cool.

Our systems of January 2020 have not been doing that.

Yeah, no, I think it all boils down to like the reason people click like on stuff on the internet, which is like, it makes them laugh.


Speaker 0: So incredibly, you've contributed some of the biggest recent ideas in AI, in computer vision, language, natural language processing, reinforcement learning, sort of everything in between, maybe not GANs. Is there, there may not be a topic you haven't touched and of course the fundamental science of deep learning. What is the difference to you between vision, language and as in reinforcement learning, action as learning problems and what are the commonalities? Do you see them as all interconnected, or are they fundamentally different domains that require different approaches?

Speaker 1: Okay, that's a good question. Machine learning is a field with a lot of unity, a huge amount of unity. In fact- What

Speaker 0: do you mean by unity?

Speaker 1: Like overlap of ideas? Overlap of ideas, overlap of principles. In fact, there's only 1 or 2 or 3 principles which are very, very simple. And then they apply in almost the same way, in almost the same way to the different modalities, to the different problems. And that's why today, when someone writes a paper on improving optimization of deep learning in vision, it improves the different NLP applications and it improves the different reinforcement learning applications.

Reinforcement learning, So I would say that computer vision and NLP are very similar to each other. Today they differ in that they have slightly different architectures. We use transformers in NLP and we use convolutional neural networks in vision. But it's also possible that 1 day this will change and everything will be unified with a single architecture Because if you go back a few years ago in natural language processing There were a huge number of architectures for every different tiny problem had its own architecture Today there's just 1 transformer for all those different tasks. And if you go back in time even more, you had even more and more fragmentation and every little problem in AI had its own little sub-specialization and sub, you know, little set of collection of skills, people who would know how to engineer the features.

Now it's all been subsumed by deep learning. We have this unification. And so I expect vision to be communified with natural language as well. Or rather I shouldn't say expect, I think it's possible. I don't want to be too sure because I think on the conventional neural net it's very computationally efficient.

RL is different. RL does require slightly different techniques because you really do need to take action. You really do need to do something about exploration. Your variance is much higher, But I think there is a lot of unity even there. And I would expect, for example, that at some point there will be some broad unification between RL and supervised learning, where somehow the RL will be making decisions to make the supervised learning go better.

And there will be, I imagine, 1 big black box and you just throw every, you know, you shovel things into it and it just figures out what to do with whatever you shovel in it.

Speaker 0: I mean, reinforcement learning has some aspects of language and vision combined, almost. There's elements of a long-term memory that you should be utilizing and there's elements of a really rich sensory space. So it seems like the, it's like the union of the 2 or something like that.

Speaker 1: I'd say something slightly differently. I'd say that reinforcement learning is neither, but it naturally interfaces and integrates with the 2 of them.

Speaker 0: You think action is fundamentally different? So yeah, what is interesting about, what is unique about policy of learning to act?

Speaker 1: Well, so 1 example, for instance, is that when you learn to act you are fundamentally in a non-stationary world. Because as your actions change, the things you see start changing. You experience the world in a different way. And this is not the case for the more traditional static problem where you have some distribution and you just apply a model to that distribution.

Speaker 0: You think it's a fundamentally different problem or is it just a more difficult, it's a generalization of the problem of understanding?

Speaker 1: I mean, it's a question of definitions almost. There is a huge amount of commonality for sure. You take gradients, you try to approximate gradients in both cases. In the case of reinforcement learning, you have some tools to reduce the variance of the gradients. You do that.

There's lots of commonality, you use the same neural net in both cases. You compute the gradient, you apply Adam in both cases. So, I mean, there's lots in common for sure, but there are some small differences which are not completely insignificant. It's really just a matter of your point of view, what frame of reference, how much do you want to zoom in or out as you look at these problems.

Speaker 0: Which problem do you think is harder? So people like Noam Chomsky believe that language is fundamental to everything. So it underlies everything. Do you think language understanding is harder than visual scene understanding or vice versa?

Speaker 1: I think that asking if a problem is hard is slightly wrong. I think the question is a little bit wrong and I wanna explain why. So what does it mean for a problem to be hard?

Speaker 0: Okay, the non-interesting dumb answer to that is there's a benchmark and there's a human level performance on that benchmark And how is the effort required to reach the human level benchmark?

Speaker 1: So from the perspective of how much until we get to human level on a very good benchmark? Yeah, I understand what you mean by that. So what I was going to say that a lot of it depends on, you know, once you solve a problem, it stops being hard. And that's always true. And so, whether something is hard or not depends on what our tools can do today.

So, you know, you say today, true human level language understanding and visual perception are hard in the sense that there is no way of solving the problem completely in the next 3 months. So I agree with that statement. Beyond that, I'm just, my guess would be as good as yours, I don't know.

Speaker 0: Oh, okay, so you don't have a fundamental intuition about how hard language understanding is.

Speaker 1: I think, I know I changed my mind. I'd say language is probably going to be harder. I mean, it depends on how you define it. Like if you mean absolute, top-notch, 100% language understanding, I'll go with language. But then if I show you a piece of paper with letters on it, is that, you see what I mean?

You have a vision system, you say it's the best human level vision system. I show you, I open a book and I show you letters. Will it understand how these letters form into word and sentences and meaning? Is this part of the vision problem? Where does vision end and language begin?

Yeah, so Chomsky would say it starts at language. So vision is just a little example of the kind of structure and fundamental

Speaker 0: hierarchy of ideas that's already represented in our brain somehow, that's represented through language. But where does vision stop and language begin? That's a really interesting question. So 1 possibility is that it's impossible to achieve really deep understanding in either images or language without basically using the same kind of system. So you're going to get the other for free.

Speaker 1: I think it's pretty likely that yes, if we can get 1, our machine learning is probably that good that we can get the other. But I'm not 100% sure. And also, I think a lot of it really does depend on your definitions. Definitions of? Of like perfect vision.

Because reading is vision, but should it count?

Speaker 0: Yeah, to me, so my definition is if a system looked at an image and then a system looked at a piece of text and then told me something about that and I was really impressed,

Speaker 1: That's relative. You'll be impressed for half an hour and then you're gonna say, well, I mean all the systems do that, but here's the thing they don't do.

Speaker 0: Yeah, but I don't have that with humans. Humans continue to impress me. Is that true? Well, The ones, okay, so I'm a fan of monogamy, so I like the idea of marrying somebody, being with them for several decades. So I believe in the fact that yes, it's possible to have somebody continuously giving you pleasurable, interesting, witty, new ideas, friends.

Yeah, I think so. They continue to surprise you. The surprise, it's that injection of randomness. It seems to be a nice source of, yeah, continued inspiration, like the wit, the humor. I think, yeah, that would be, it's a very subjective test, but I think if you have enough humans in the room.

Speaker 1: Yeah, I understand what you mean. Yeah, I feel like I misunderstood what you meant by impressing you. I thought you meant to impress you with its intelligence, with how well it understands an image. I thought you meant something like, I'm gonna show it a really complicated image and it's gonna get it right and you're gonna say, wow, that's really cool. Our systems of January 2020 have not been doing that.

Speaker 0: Yeah, no, I think it all boils down to like the reason people click like on stuff on the internet, which is like, it makes them laugh. So it's like humor or wit

Speaker 1: or insight. I'm sure we'll get that as well.

See all Lex Fridman transcripts on Youtube

Language or Vision - What's Harder? (Ilya Sutskever) | AI Podcast Clips