Yann LeCun is the Chief AI Scientist at Meta, professor at NYU, Turing Award winner, and one of the seminal researchers in the history of machine learning. Please support this podcast by checking out our sponsors:
- Public Goods: https://publicgoods.com/lex and use code LEX to get $15 off
- Indeed: https://indeed.com/lex to get $75 credit
- ROKA: https://roka.com/ and use code LEX to get 20% off your first order
- NetSuite: http://netsuite.com/lex to get free product tour
- Magic Spoon: https://magicspoon.com/lex and use code LEX to get $5 off

EPISODE LINKS:
Yann's Twitter: https://twitter.com/ylecun
Yann's Facebook: https://www.facebook.com/yann.lecun
Yann's Website: http://yann.lecun.com/
Books and resources mentioned:
Self-supervised learning (article): https://bit.ly/3Aau1DQ

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
0:36 - Self-supervised learning
10:55 - Vision vs language
16:46 - Statistics
22:33 - Three challenges of machine learning
28:22 - Chess
36:25 - Animals and intelligence
46:09 - Data augmentation
1:07:29 - Multimodal learning
1:19:18 - Consciousness
1:24:03 - Intrinsic vs learned ideas
1:28:15 - Fear of death
1:36:07 - Artificial Intelligence
1:49:56 - Facebook AI Research
2:06:34 - NeurIPS
2:22:46 - Complexity
2:31:11 - Music
2:36:06 - Advice for young people

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Jan Lekun, his second time on the podcast.

He is the chief AI scientist at Meta, formerly Facebook, professor at NYU, Turing Award winner, 1 of the seminal figures in the history of machine learning and artificial intelligence, and someone who is brilliant and opinionated in the best kind of way, and so is always fun to talk to.

To support it, please check out our sponsors in the description.

And now, here's my conversation with Jan Lekun.

You co-wrote the article, "'Self-supervised learning, the dark matter of intelligence." Great title, by the way, with Ishan Misra.

So let me ask, what is self-supervised learning, and why is it the dark matter of intelligence?

There is obviously a kind of learning that humans and animals are doing that we currently are not reproducing properly with machines or with AI, right?

So the most popular approaches to machine learning today are, or paradigms, I should say, are supervised learning and reinforcement learning.

Supervised learning requires many samples for learning anything.

And reinforcement learning requires a ridiculously large number of trial and errors for a system to learn anything.

And that's why we don't have self-driving cars.

Okay, so that to solve difficult problems, you have to have a lot of human annotation for supervised learning to work.

And to solve those difficult problems with reinforcement learning, you have to have some way to maybe simulate that problem such that you can do that large scale kind of learning that reinforcement learning requires.

Right, so how is it that, you know, most teenagers can learn to drive a car in about 20 hours of practice, whereas even with millions of hours of simulated practice, a self-driving car can't actually learn to drive itself properly.

And so obviously we're missing something, right?

And it's quite obvious for a lot of people that, you know, the immediate response you get from many people is, well, you know, humans use their background knowledge to learn faster.

Now, how was that background knowledge acquired?

So now you have to ask, you know, how do babies in the first few months of life learn how the world works?

Mostly by observation, because they can hardly act in the world.

And they learn an enormous amount of background knowledge about the world that may be the basis of what we call common sense.

This type of learning, it's not learning a task, it's not being reinforced for anything, it's just observing the world and figuring out how it works.

Building world models, learning world models.

And how do we reproduce this in machines?

So self-supervised learning is, you know, 1 instance or 1 attempt at trying to reproduce this kind of learning.

Okay, so you're looking at just observation, so not even the interacting part of a child, it's just sitting there watching mom and dad walk around, pick up stuff, all of that.

That's what you mean about background knowledge.

Perhaps not even watching mom and dad, just watching the world go by.

Just having eyes open or having eyes closed, or the very act of opening and closing eyes that the world appears and disappears, all that basic information.

And you're saying in order to learn to drive, like the reason humans are able to learn to drive quickly, some faster than others, is because of the background knowledge they were able to watch cars operate in the world in the many years leading up to it, the physics of basic objects, all that kind

I mean, the basic physics of objects, you don't even know, you don't even need to know, you know, how a car works, right?

Because that you can learn fairly quickly.

I mean, the example I use very often is you're driving next to a cliff and you know in advance because of your, you know, understanding of intuitive physics that if you turn the wheel to the right, the car will veer to the right, will run off the cliff, fall off the cliff, and nothing good will come out of this, right?

But if you are a sort of tabula rasa reinforcement learning system that doesn't have a model of the world, you have to repeat falling off this cliff thousands of times before you figure out it's a bad idea.

And then a few more thousand times before you figure out how to not do it.

And then a few more million times before you figure out how to not do it in every situation you ever encounter.

So self-supervised learning still has to have some source of truth being told to it by somebody.

So you have to figure out a way without human assistance or without significant amount of human assistance to get that truth from the world.

So the mystery there is how much signal is there?

How much truth is there that the world gives you?

Whether it's the human world, like you watch YouTube or something like that, or it's the more natural world.

There is way more signal in sort of a self-supervised setting than there is in either a supervised or reinforcement setting.

And this is going to my analogy of the cake.

Le cake, as someone has called it, where when you try to figure out how much information you ask the machine to predict and how much feedback you give the machine at every trial, in reinforcement learning, you give the machine a single scalar, you tell the machine you did good, you did bad, and you only tell this to the machine once in a while.

When I say you, it could be the universe telling the machine, right?

And so as a consequence, you cannot possibly learn something very complicated without many, many, many trials where you get many, many feedbacks of this type.

Supervised learning, you give a few bits to the machine at every sample.

Let's say you're training a system on recognizing images on ImageNet, there is 1,000 categories, that's a little less than 10 bits of information per sample.

But self supervised learning here is the setting.

Ideally, we don't know how to do this yet, but ideally you would show a machine a segment of video, and then stop the video and ask the machine to predict what's going to happen next.

So you let the machine predict and then you let time go by and show the machine what actually happened and hope the machine will learn to do a better job at predicting next time around.

There's a huge amount of information you give the machine because it's an entire video clip of the future after the video clip you fed it in the first place.

So both for language and for vision, there's a subtle, seemingly trivial construction, but maybe that's representative of what is required to create intelligence, which is filling the gap.

It sounds dumb, but can you, it is possible you could solve all of intelligence in this way, just for both language, just give a sentence and continue it, or give a sentence and there's a gap in it, some words blanked out, and you fill in what words go there.

For vision, you get a sequence of images and predict what's gonna happen next, or you fill in what happened in between.

Do you think it's possible that formulation alone as a signal for self-supervised learning can solve intelligence for vision and language?

I think that's our best shot at the moment.

So whether this will take us all the way to human level intelligence or something, or just cat level intelligence is not clear.

But among all the possible approaches that people have proposed, I think it's a bad shot.

So I think this idea of an intelligence system filling in the blanks, either, you know, predicting the future, inferring the past, filling in missing information.

I'm currently filling the blank of what is behind your head and what your head looks like from the back, because I have basic knowledge about how humans are made.

And I don't know if you're gonna, what you're gonna say, at which point you're gonna speak, whether you're gonna move your head this way or that way, which way you're gonna look, but I know you're not gonna just dematerialize and reappear 3 meters down the hall, because I know what's possible and what's impossible according to intuitive physics.

So- Well, you have a model of what's possible, what's impossible, and then you'd be very surprised if it happens, then you'll have to reconstruct your model.

It's what tells you, you know, what fills in the blanks.

So given your partial information about the state of the world, given by your perception, your model of the world fills in the missing information and that includes predicting the future, retradicting the past, filling in things you don't immediately perceive.

And that doesn't have to be purely generic vision or visual information or generic language.

You can go to specifics like predicting what control decision you make when you're driving in a lane, you have a sequence of images from a vehicle, and then you could, you have information, if you record it on video, where the car ended up going, so you can go back in time and predict where the car went based on the visual information.

But the question is whether we can come up with sort of a generic method for, you know, training machines to do this kind of prediction or filling in the blanks.

So Right now, this type of approach has been unbelievably successful in the context of natural language processing.

Every modern natural language processing is pre-trained in a self-supervised manner to fill in the blanks.

You show it a sequence of words, you remove 10% of them, and then you train some gigantic neural net to predict the words that are missing.

And once you've pre-trained that network, you can use the internal representation learned by it as input to something that you trained, supervised, or whatever.

Not so successful in images, although it's making progress.

And it's based on sort of manual data augmentation.

We can go into this later, but what has not been successful yet is training from video.

So getting a machine to learn to represent the visual world, for example, by just watching video.

Nobody has really succeeded in doing this.

Okay, well, let's kind of give a high-level overview.

What's the difference in kind and in difficulty between vision and language.

So you said people haven't been able to really kind of crack the problem of vision open in terms of self-supervised learning, but that may not be necessarily because it's fundamentally more difficult.

Maybe like when we're talking about achieving, like passing the Turing test in the full spirit of the Turing test in language might be harder than vision.

That's not obvious, so in your view, which is harder or perhaps are they just the same problem?

When the farther we get to solving each, the more we realize it's all the same thing.

I think what I'm looking for are methods that make them look essentially like the same cake, but currently they're not.

And The main issue with learning world models or learning predictive models is that the prediction is never a single thing because the world is not entirely predictable.

We can get into the philosophical discussion about it, but even if it's deterministic, it's not entirely predictable.

And so if I play a short video clip and then I ask you to predict what's going to happen next, there's many, many plausible continuations for that video clip, And the number of continuation grows with the interval of time that you're asking the system to make a prediction for.

And so 1 big question with self-supervised learning is how you represent this uncertainty, how you represent multiple discrete outcomes, how you represent a sort of continuum of possible outcomes, et cetera.

And, you know, if you are a sort of a classical machine learning person, you say, oh, you just represent a distribution, right?

And that we know how to do when we're predicting words, missing words in the text, because you can have a neural net give a score for every word in a dictionary.

It's a big list of numbers, maybe 100,000 or so.

And you can turn them into a probability distribution that tells you when I say a sentence, the cat is chasing the blank in the kitchen.

There are only a few words that make sense there.

It could be a mouse, it could be a lizard spot, or something like that.

And if I say the blank is chasing the blank in the savannah, you also have a bunch of plausible options for those 2 words, right?

Because you have kind of an underlying reality that you can refer to to sort of fill in those blanks.

So you cannot say for sure in the savanna, if it's a lion or a cheetah or whatever, you cannot know if it's a zebra or a goo or, you know, whatever, wildebeest, the same thing.

But you can represent the uncertainty by just a long list of numbers.

Now, if I do the same thing with video and I ask you to predict a video clip, it's not a discrete set of potential frames.

You have to have some way of representing a sort of infinite number of plausible continuations of multiple frames in a high dimensional continuous space.

And we just have no idea how to do this properly.

Just like the words, they try to get it down to a small finite set of like under a million, something like that.

it's kind of ridiculous that we're doing a distribution over every single possible word for language and it works.

It feels like that's a really dumb way to do it.

Like there seems to be like there should be some more compressed representation of the distribution of the words.

And so do you have any interesting ideas about how to represent all of reality in a compressed way such that you can form a distribution over it?

That's 1 of the big questions, you know, how do you do that?

But I mean, what's kind of, you know, another thing that really is stupid about, I shouldn't say stupid, but like simplistic about current approaches to self-supervised learning in NLP in text is that not only do you represent a giant distribution over words, but for multiple words that are missing, those distributions are essentially independent of each other.

And you don't pay too much of a price for this.

So the system, in the sentence that I gave earlier, if it gives a certain probability for a lion and a cheetah, and then a certain probability for a gazelle, wildebeest and zebra, those 2 probabilities are independent of each other.

And it's not the case that those things are independent.

Lions actually attack bigger animals than cheetahs.

So there's a huge independence hypothesis in this process, which is not actually true.

The reason for this is that we don't know how to represent properly distributions over combinatorial sequences of symbols, essentially, because the number grows exponentially with the length of the symbols.

And so we have to use tricks for this, but those techniques can get around, like don't even deal with it.

So the big question is, would there be some sort of abstract, latent representation of text that would say that, you know, when I switch lion for gazelle, lion for cheetah, I also have to switch zebra for gazelle.

Yeah, so this independence assumption, let me throw some criticism at you that I often hear and see how you respond.

So this kind of filling in the blanks is just statistics.

You're not learning anything, like the deep underlying concepts.

You're just mimicking stuff from the past, you're not learning anything new such that you can use it to generalize about the world.

Or, okay, let me just say the crude version, which is, it's just statistics, It's not intelligence.

What do you usually say to that if you kind of hear this kind of thing?

I don't get into those discussions because they are kind of pointless.

So first of all, it's quite possible that intelligence is just statistics.

It's just statistics of a particular kind.

It's kind of, is it possible that intelligence is just statistics?

So if you are asking the question, are the models of the world that we learn, do they have some notion of causality?

So if the criticism comes from people who say, you know, current machine learning systems don't care about causality, which by the way is wrong, you know, I agree with that.

Your model of the world should have your actions as 1 of the inputs, and that will drive you to learn causal models of the world where you know what intervention in the world will cause what result.

Or you can do this by observation of other agents acting in the world and observing the effect, other humans, for example.

So I think, you know, at some level of description, intelligence is just statistics, but that doesn't mean you don't, you know, you won't have models that have, you know, deep mechanistic explanation for what goes on.

Because A lot of people who actually voice their criticism say that those mechanistic model have to come from someplace else.

They have to come from human designers, they have to come from I don't know what.

Or if we don't learn them as an individual, nature learned them for us using evolution.

So regardless of what you think, those processes have been learned somehow.

So if you look at the human brain, just like when we humans introspect about how the brain works, it seems like when we think about what is intelligence, we think about the high level stuff, like the models we've constructed, concepts like cognitive science, like concepts of memory and reasoning module, almost like these high level modules.

Like, are we ignoring the dark matter, the basic low level mechanisms, just like we ignore the way the operating system works, we're just using the high level software.

We're ignoring that at the low level, the neural network might be doing something like statistics.

Like, meaning, sorry to use this word probably incorrectly and crudely, but doing this kind of fill in the gap kind of learning and just kind of updating the model constantly in order to be able to support the raw sensory information, to predict it and then adjust to the prediction when it's wrong.

But like when we look at our brain at the high level, it feels like we're doing, like we're playing chess.

Like we're like playing with high level concepts and we're stitching them together and we're putting them into long term memory.

But really what's going underneath is something we're not able to introspect, which is this kind of simple, large neural network that's just filling in the gaps.

Right, well, okay, so there's a lot of questions and a lot of answers there.

Okay, so first of all, there's a whole school of thought in neuroscience, computational neuroscience in particular, that likes the idea of predictive coding, which is really related to the idea I was talking about in self-supervised learning.

The essence of intelligence is the ability to predict, and everything the brain does is trying to predict everything from everything else.

That's really the underlying principle, if you want, that self-supervised learning is trying to kind of reproduce this idea of prediction as kind of an essential mechanism of task-independent learning, if you want.

The next step is, what kind of intelligence are you interested in reproducing?

And of course, we all think about trying to reproduce high-level cognitive processes in humans, but with machines, we're not even at the level of even reproducing the learning processes in a cat brain.

You know, the most intelligent or intelligent systems don't have as much common sense as a house cat.

And you know, cats don't do a whole lot of reasoning.

They certainly have, because, you know, many cats can figure out like how they can act on the world to get what they want.

They certainly have a fantastic model of intuitive physics, certainly of the dynamics of their own bodies, but also of praise and things like that, right?

They only do this with about 800 million neurons.

We are not anywhere close to reproducing this kind of thing.

So to some extent, I could say, let's not even worry about the high-level cognition and long-term planning and reasoning that humans can do until we figure out, can we even reproduce what cats are doing?

Now that said, this ability to learn world models, I think is the key to the possibility of learning machines that can also reason.

So whenever I give a talk, I say there are 3 main challenges in machine learning.

The first 1 is, you know, getting machines to learn to represent the world and proposing self-supervised learning.

The second is getting machines to reason in ways that are compatible with essentially gradient-based learning, because this is what deep learning is all about, really.

And the third 1 is something we have no idea how to solve, or at least I have no idea how to solve, is can we get machines to learn hierarchical representations of action plans?

We know how to train them to learn hierarchical representations of perception, you know, with convolutional nets and things like that and transformers.

Can we get them to spontaneously learn good hierarchical representations of actions?

All of that, you know, needs to be somewhat differentiable so that you can apply sort of gradient based learning, which is really what deep learning is about.

So it's background knowledge, ability to reason in a way that's differentiable, that is somehow connected, deeply integrated with that background knowledge, or builds on top of that background knowledge, and then given that background knowledge, be able to make hierarchical plans

So if you take classical optimal control, there's something in classical optimal control called model predictive control.

And it's been around since the early 60s.

NASA uses that to compute trajectories of rockets.

And the basic idea is that you have a predictive model of the rocket, let's say, or whatever system you intend to control, which given the state of the system at time t and given an action that you're taking on the system, So for a rocket to be thrust and, you know, all the controls you can have.

It gives you the state of the system at time t plus delta t, right?

So basically a differential equation, something like that.

And if you have this model and you have this model in the form of some sort of neural net or some sort of set of formula that you can back propagate gradient through, you can do what's called model predictive control or gradient-based model predictive control.

So you have, you can unroll that model in time.

You feed it a hypothesized sequence of actions, and then you have some objective function that measures how well at the end of the trajectory the system has succeeded or matched what you wanted to do.

Have you grasped the object you want to grasp?

If it's a rocket, you know, are you at the right place near the space station?

And by back propagation to a time, and again, this was invented in the 1960s by optimal control theorists, you can figure out what is the optimal sequence of actions that will get my system to the best final state.

And a lot of planning systems in robotics are actually based on this.

And you can think of this as a form of reasoning.

So, to take the example of the teenager driving a car again, you have a pretty good dynamical model of the car.

But you know, again, that if you turn the wheel to the right and there is a cliff, you're going to run off the cliff, right?

You don't need to have a very accurate model to predict that.

And you can run this in your mind and decide not to do it for that reason, because you can predict in advance that the result is going to be bad.

So you can imagine different scenarios and then employ or take the first step in the scenario that is most favourable and then repeat the process of planning.

That's called receding horizon model predictive control.

So all those things have names going back decades.

And so if you're not in classical optimal control, the model of the world is not generally learned.

There's, you know, sometimes a few parameters you have to identify, that's called systems identification, but, but generally the model is mostly deterministic and mostly built by hand.

So the big question of AI, I think the big challenge of AI for the next decade is how do we get machines to learn predictive models of the world that deal with uncertainty and deal with the real world in all this complexity.

So it's not just the trajectory of a rocket, which you can reduce to first principles.

It's not even just the trajectory of a robot arm, which again, you can model by careful mathematics, but it's everything else, everything we observe in the world, people, behavior, physical systems that involve collective phenomena like water or trees and branches in a tree or something, or like complex things that humans have no trouble developing abstract representations and predictive model for, but we still don't know how to do with machines.

Where do you put in these 3, maybe in the planning stages, the game theoretic nature of this world where your actions not only respond to the dynamic nature of the world, the environment, but also affect it.

So if there's other humans involved, is this point number 4 or is it somehow integrated into the hierarchical representation of action in your view?

It's just that now your model of the world has to deal with, you know, it just makes it more complicated, right?

The fact that humans are complicated and not easily predictable, that makes your model of the world much more complicated, that much more complicated.

Well, there's a chess, I mean, I suppose chess is an analogy.

I mean, there is a, I go, you go, I go, you go.

Like, Andrej Karpathy recently gave a talk at MIT about car doors.

I think there's some machine learning too, but mostly car doors.

And there's a dynamic nature to the car, like the person opening the door checking.

He was talking about the perception problem of what the ontology of what defines a car door, this big philosophical question.

But to me it was interesting, because it's obvious that the person opening the car doors, they're trying to get out, like here in New York, trying to get out of the car, you slowing down is going to signal something, you speeding up is gonna signal something, and that's a dance.

See all Lex Fridman transcripts on Youtube

Yann LeCun: Dark Matter of Intelligence and Self-Supervised Learning | Lex Fridman Podcast #258