Ishan Misra is a research scientist at FAIR working on self-supervised visual learning. Please support this podcast by checking out our sponsors:
- Onnit: https://lexfridman.com/onnit to get up to 10% off
- The Information: https://theinformation.com/lex to get 75% off first month
- Grammarly: https://grammarly.com/lex to get 20% off premium
- Athletic Greens: https://athleticgreens.com/lex and use code LEX to get 1 month of fish oil

EPISODE LINKS:
Ishan's twitter: https://twitter.com/imisra_
Ishan's website: https://imisra.github.io
Ishan's FAIR page: https://ai.facebook.com/people/ishan-misra/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
2:27 - Self-supervised learning
11:02 - Self-supervised learning is the dark matter of intelligence
14:54 - Categorization
23:28 - Is computer vision still really hard?
27:12 - Understanding Language
36:51 - Harder to solve: vision or language
43:36 - Contrastive learning & energy-based models
47:37 - Data augmentation
51:57 - Fixed audio spike by lowering sound with pen tool
1:00:10 - Real data vs. augmented data
1:03:54 - Non-contrastive learning energy based self supervised learning methods
1:07:32 - Unsupervised learning (SwAV)
1:10:14 - Self-supervised Pretraining (SEER)
1:15:21 - Self-supervised learning (SSL) architectures
1:21:21 - VISSL pytorch-based SSL library
1:24:15 - Multi-modal
1:31:43 - Active learning
1:37:22 - Autonomous driving
1:48:49 - Limits of deep learning
1:52:57 - Difference between learning and reasoning
1:58:03 - Building super-human AI
2:05:51 - Most beautiful idea in self-supervised learning
2:09:40 - Simulation for training AI
2:13:04 - Video games replacing reality
2:14:18 - How to write a good research paper
2:18:45 - Best programming language for beginners
2:19:39 - PyTorch vs TensorFlow
2:23:03 - Advice for getting into machine learning
2:25:09 - Advice for young people
2:27:35 - Meaning of life

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Ishan Misra, research scientist at Facebook AI Research who works on self-supervised machine learning in the domain of computer vision.

Or in other words, making AI systems understand the visual world with minimal help from us humans.

Transformers and self-attention has been successfully used by OpenAI's GPT-3 and other language models to do self-supervised learning in the domain of language.

Ishan, together with Yan LeCun and others, is trying to achieve the same success in the domain of images and video.

The goal is to leave a robot watching YouTube videos all night and in the morning, come back to a much smarter robot.

I read the blog post, Self-Supervised learning, the dark matter of intelligence by Ishan and Yann LeCun, and then listen to Ishan's appearance on the excellent machine learning street talk podcast.

By the way, If you're interested in machine learning and AI, I cannot recommend the ML Street Talk podcast highly enough.

Onnit, The Information, Grammarly, and Athletic Greens.

Check them out in the description to support this podcast.

As a side note, let me say that for those of you who may have been listening for quite a while, this podcast used to be called Artificial Intelligence Podcast because my life passion has always been, will always be artificial intelligence, both narrowly and broadly defined.

My goal with this podcast is still to have many conversations with world-class researchers in AI, math, physics, biology, and all the other sciences.

But I also want to talk to historians, musicians, athletes, and of course occasionally comedians.

In fact, I'm trying out doing this podcast 3 times a week now to give me more freedom with guest selection and maybe get a chance to have a bit more fun.

Speaking of fun, in this conversation, I challenged the listener to count the number of times the word banana is mentioned.

Ishan and I use the word banana as the canonical example at the core of the hard problem of computer vision and maybe the hard problem of consciousness.

This is the Lex Friedman Podcast, and here is my conversation with Ishan Misra.

And maybe even give the bigger basics of what is supervised and semi-supervised learning.

And maybe why is self-supervised learning a better term than unsupervised learning?

So typically for machine learning systems, the way they're trained is You get a bunch of humans, the humans point out particular concepts.

So if it's in the case of images, you want the humans to come and tell you what is present in the image, draw boxes around them, draw masks of like things, pixels, which are of particular categories or not.

For NLP, again, there are like lots of these particular tasks, say about sentiment analysis, about entailment, and so on.

Typically, for supervised learning, we get a big corpus of such annotated or labeled data, and then we feed that to a system, and the system is really trying to mimic.

It's taking this input of the data and then trying to mimic the output.

So it looks at an image and the human has tagged that this image contains a banana.

And now the system is basically trying to mimic that.

And so for supervised learning, we try to gather lots of such data and we train these machine learning models to imitate the input output.

And the hope is basically by doing so, now on unseen or like new kinds of data, this model can automatically learn to predict these concepts.

So this is a standard sort of supervised setting.

For semi-supervised setting, the idea typically is that you have, of course, all of the supervised data, but you have lots of other data which is unsupervised or which is like not labeled.

Now the problem basically with supervised learning and why you actually have all of these alternate sort of learning paradigms is supervised learning does just does not scale.

So if you look at for computer vision, the sort of largest 1 of the most popular data sets is ImageNet.

So the entire ImageNet data set has about 22,000 concepts, and about 14 million images.

So these concepts are basically just nouns and they're annotated on images.

And this entire data set was a mammoth data collection effort.

It actually gave rise to a lot of powerful learning algorithms.

It's credited with like sort of the rise of deep learning as well.

But this data set took about 22 human years to collect, to annotate.

And it's not even that many concepts, right?

It's not even that many images 14 million is nothing really.

Like you have about I think 400 million images or so or even more than that uploaded to most of the popular sort of social media websites today.

So now Supervised learning just doesn't scale.

If I want to now annotate more concepts, if I want to have various types of fine grained concepts, then it won't really scale.

So now you come up to these sort of different learning paradigms, for example, semi-supervised learning.

Where the idea is, of course, you have this annotated corpus of supervised data.

And you have lots of these unlabeled images.

And the idea is that the algorithm should basically try to measure some kind of consistency, or really try to measure some kind of signal on this sort of unlabeled data to make it self more confident about what it's really trying to predict.

So by access to this lots of unlabeled data, the idea is that the algorithm actually learns to be more confident and actually gets better at predicting these concepts.

And now we come to the other extreme, which is like self-supervised learning.

The idea basically is that the machine or the algorithm should really discover concepts or discover things about the world or learn representations about the world, which are useful without access to explicit human supervision.

So the word supervision is still in the term self-supervised.

And maybe that perhaps is when Jan-Lakoon and you argue that unsupervised is the incorrect terminology here.

So what is the supervision signal when the humans aren't part of the picture or not a big part of the picture.

So self-supervised, the reason it has the term supervised in itself is because you're using the data itself as supervision.

So because the data serves as its own source of supervision, it's self-supervised in that way.

Now, the reason a lot of people, I mean, we did it in that blog post with Jan, but a lot of other people have also argued for using this term self supervised.

So starting from like 94 from Virginia de Sars group, I think UCSD and now she's at UCSD.

Tendra Malik has said this a bunch of times as well.

And then unsupervised basically means everything which is not supervised.

But that includes stuff like semi-supervised, that includes other, like transductive learning, lots of other sort of settings.

So that's the reason like now people are preferring this term self-supervised because it explicitly says what's happening.

The data itself is the source of supervision and any sort of learning algorithm which tries to extract just sort of data supervision signals from the data itself is a self-supervised learning algorithm.

But there is within the data, a set of tricks which unlock the supervision.

And there's innovation, ingenuity required to unlock that supervision.

The data doesn't just speak to you some ground truth.

So I don't know what your favorite domain is.

So you specifically specialize in visual learning, but is there favorite examples, maybe in language or other domains?

Perhaps the most successful applications have been in NLP, language processing.

So the idea basically being that you can train models that can, you have a sentence and you mask out certain words and now these models learn to predict the masked out words.

So if you have like the cat jumped over the dog, so you can basically mask out cat And now you are essentially asking the model to predict what was missing, what did I mask out?

So the model is going to predict basically a distribution over all the possible words that it knows.

And probably it has, if it's a well-trained model, it has a sort of higher probability density for this word cat.

For vision, I would say the sort of more, I mean, the easier example, which is not as widely used these days, is basically say, for example, video prediction.

So if you have a video of say 10 seconds, you can feed in the first 9 seconds to a model and then ask it, hey, what happens basically in the 10th second?

And the idea basically is because the model is predicting something about the data itself.

Of course, you didn't need any human to tell you what was happening because the 10 second video was naturally captured.

Because the model is predicting what's happening there, it's going to automatically learn something about the structure of the world, how objects move, object permanence, and these kinds of things.

So like, if I have something at the edge of the table, it'll fall down.

Things like these, which you really don't have to sit and annotate.

In a supervised learning setting, I would have to sit and annotate.

And now I move this cup, it's still a cup.

And then it falls down and this is a fallen down cup.

So I won't have to annotate all of these things in a self-supervised setting.

Isn't that kind of a brilliant little trick of taking a series of data that is consistent and removing 1 element in that series and then teaching the algorithm to predict that element?

Isn't that, first of all, that's quite brilliant.

It seems to be applicable in anything that has the constraint of being a sequence that is consistent with the physical reality.

The question is, are there other tricks like this that can generate the self supervision signal?

So sequence is possibly the most widely used 1 in NLP.

For vision, the 1 that is actually used for like images, which is very popular these days, is basically taking an image and now taking different crops of that image.

So you can basically decide to crop say the top left corner and you crop say the bottom right corner and asking a network to basically present it with a choice saying that okay, Now you have this image, you have this image, are these the same or not?

And so the idea basically is that because different, like in an image, different parts of the image are going to be related.

So for example, if you have a chair and a table, basically these things are going to be close by.

Versus if you take, again, if you have like a zoomed in picture of a chair, if you're taking different crops, it's going to be different parts of the chair.

So the idea basically is that different crops of the image are related.

And so the features or the representations that you get from these different crops should also be related.

So this is possibly the most like widely used trick these days for self supervised learning in computer vision.

So again, using the consistency that's inherent to physical reality in visual domain, that's, you know, parts of an image are consistent.

And then in the language domain or anything that has sequences like language or something that's like a time series that you can chop up parts in time.

It's similar to the story of RNNs and CNNs, of RNNs and Covenants.

You and Jan LeCun wrote the blog post in March, 2021, titled, Self-supervised learning, the dark matter of intelligence.

Can you summarize this blog post and maybe explain the main idea or set of ideas?

The blog post was mainly about sort of just telling, I mean, this is really an accepted fact, I would say for a lot of people now that self supervised learning is something that is going to play an important role for machine learning algorithms that come in the future and even now.

Let me just comment that we don't yet have a good understanding what dark matter is.

Maybe the metaphor doesn't exactly transfer, but maybe it's actually perfectly transfers that we don't know.

We have an inkling that it'll be a big part of whatever solving intelligence looks like.

Right, so I think self-supervised learning, the way it's done right now is, I would say like the first step towards what it probably should end up like learning or what it should enable us to do.

So the idea for that particular piece was self supervised learning is going to be a very powerful way to learn common sense about the world or like stuff that is really hard to label.

For example, like is this piece of heavier than the cup.

Now, for all these kinds of things, you'll have to sit and label these things.

So supervised learning is clearly not going to scale.

So what is the thing that's actually going to scale?

It's probably going to be an agent that can either actually interact with it to lift it up or observe me doing it.

So if I'm basically lifting these things up, it can probably reason about, hey, this is taking him more time to lift up or the velocity is different.

Whereas the velocity for this is different, probably this 1 is heavier.

So essentially by observations of the data, you should be able to infer a lot of things about the world without someone explicitly telling you, this is heavy, this is not, this is something that can pour, this is something that cannot pour, this is somewhere that you can sit, this is not somewhere that you can sit.

But you just mentioned ability to interact with the world.

There's so many questions that are yet to be, that are still open, which is how do you select a set of data over which the self-supervised learning process works?

How much interactivity, like in the active learning or the machine teaching context is there?

Like how much actual interaction there is with the physical world, that kind of thing.

And then on top of that, which I have a million questions about which we don't know the answers to, but it's worth talking about is how much reasoning is involved, how much accumulation of knowledge versus something that's more akin to learning or whether that's the same thing.

But so we're like, It is truly dark matter.

But we are, I mean, a lot of us are actually convinced that it's going to be a sort of major thing in machine learning.

Let me reframe it then, that human supervision cannot be at large scale, the source of the solution to intelligence.

So there has, the machines have to discover the supervision in the natural signal of the world.

I mean, the other thing is also that humans are not particularly good labelers.

For example, like what's the difference between a dining table and a table?

Is it just the fact that 1, like, if you just look at a particular table, what makes us say 1 is dining table and the other is not?

Humans are not particularly consistent, they're not like very good sources of supervision for a lot of these kind of edge cases.

So it may be also the fact that if we want an algorithm or want a machine to solve a particular task for us, we can maybe just specify the end goal and like the stuff in between.

We really probably should not be specifying because we're not maybe going to confuse it a lot actually.

Well, humans can't even answer the meaning of life.

So I'm not sure if we're good supervisors of the end goal either.

Humans are not very good at telling the difference between what is and isn't a table, like you mentioned.

Do you think it's possible, let me ask you like a pretend you're Plato.

Is it possible to create a pretty good taxonomy of objects in the world?

It seems like a lot of approaches in machine learning kind of assume a hopeful vision that it's possible to construct a perfect taxonomy, or it exists perhaps out of our reach, but we can always get closer and closer to it.

So the thing is for any particular categorization that you create, if you have a discrete sort of categorization, I can always take the nearest 2 concepts, or I can take a third concept and I can blend it in and I can create a new category.

So if you were to enumerate n categories, I will always find an n plus 1 category for you.

That's not going to be in the n categories.

And I can actually create not just n plus 1, I can very easily create far more than n categories.

The thing is, a lot of things we talk about are actually compositional.

So it's really hard for us to come and sit and enumerate all of these out.

And they compose in various weird ways, right?

Like you have like a croissant and a donut come together to form a cronut.

So if you were to like enumerate all the foods up until, I don't know, whenever the cronut was about 10 years ago or 15 years ago, then this entire thing called cronut would not exist.

Yeah, I remember there was the most awesome video of a cat wearing a monkey costume.

It's a very difficult philosophical question.

So there is a concept of similarity between objects.

Just kind of getting a good function, a good way to tell which parts of things are similar and which parts of things are very different?

So you don't necessarily need to name everything or assign a name to everything to be able to use it.

I mean, lots of like, for example, animals, right.

They don't have necessarily a well-formed like syntactic language, but they're able to go about their day perfectly.

So, I mean, we probably look at things and we figure out, oh, this is similar to something else that I've seen before and then I can probably learn how to use it.

So I haven't seen all the possible doorknobs in the world.

But if you show me like I was able to get into this particular place fairly easily.

I've never seen that particular doorknob.

So I of course related to all the doorknobs that I've seen and I know exactly how it's going to open.

I have a pretty good idea of how it's going to open.

And I think this kind of translation between experiences only happens because of similarity.

Because I'm able to relate it to a doorknob.

If I related it to a hairdryer, I would probably be stuck still outside, not able to get it.

Again, a bit of a philosophical question, but it's can similarity take us all the way to understanding a thing?

Can having a good function that compares objects get us to understand something profound about singular objects?

Well, let me tell you what that's similar to.

I, So there's an idea of sort of reasoning by analogy kind of thing.

I think understanding is the process of placing that thing in some kind of network of knowledge that you have, that it perhaps is fundamentally related to other concepts.

So it's not like understanding is fundamentally related by like composition of other concepts and maybe in relation to other concepts.

And maybe like deeper and deeper understanding is maybe just adding more edges to that, to that graph somehow.

So maybe it is a composition of similarities.

I mean, ultimately, it is, I suppose it is a kind of embedding in that wisdom space.

So similarity does get you very, very far.

I mean, I don't even know what everything is, but it's going to take us really far.

And I think the thing is, things are similar in very different contexts, right?

So an elephant is similar to, I don't know, another sort of wild animal, let's just pick I don't know, lion, in a different way, because they're both 4 legged creatures.

But of course, they're very different in a lot of different ways.

So elephants are like herbivores, lions are not.

So similarity does similarity and particularly dissimilarity also actually helps us understand a lot about things.

And so that's actually why I think discrete categorization is very hard.

Just like forming this particular category of elephant and a particular category of lion, maybe it's good for like just like taxonomy, biological taxonomies.

But when it comes to like other things, which are not as maybe, for example, like grilled cheese.

I have a grilled cheese, I dip it in tomato and I keep it outside.

Now, is that still a grilled cheese or is that something else?

All right, so categorization is still very useful for solving problems, but is your intuition then sort of the self-supervised should be the, to borrow Jan LeCun's terminology, should be the cake and then categorization, the classification, maybe the supervised like layer should be just like the thing on top, the cherry or the icing or whatever.

So if you make it the cake, it gets in the way of learning.

If you make it the cake, then you won't be able to sit and annotate everything.

Like that's my very practical view on it.

It's just, I mean, in my PhD, I sat down and annotated like a bunch of cards for 1 of my projects.

And very quickly, I was just like, it was in a video, and I was basically drawing bound boxes around all these cards.

And I think I spent about a week doing all of that.

And basically, this was, I think, my first year of my PhD, or like, second year of my master's.

And then by the end of it, I'm like, okay, this is just hopeless.

And when I'd done that, someone came up to me and they basically told me, oh, this is a pickup truck, this is not a car.

And that's like, aha, this actually makes sense because a pickup truck is not really, like what was I annotating?

Was I annotating anything that is mobile?

By the way, the annotation was bounding boxes?

There's so many deep, profound questions here that you're almost cheating your way out of by doing self-supervised learning, by the way, which is like what makes for an object.

As opposed to solve intelligence, maybe you don't ever need to answer that question.

I mean, this is the question that anyone that's ever done annotation because it's so painful gets to ask, like, why am I doing a drawing very careful line around this object?

I remember when I first saw semantic segmentation where you have like instant segmentation where you have a very exact line around the object in a 2D plane of a fundamentally 3D object projected on a 2D plane.

So you're drawing a line around a car that might be occluded.

There might be another thing in front of it, but you're still drawing the line of the part of the car that you see.

See all Lex Fridman transcripts on Youtube

Ishan Misra: Self-Supervised Deep Learning in Computer Vision | Lex Fridman Podcast #206