Jitendra Malik is a professor at Berkeley and one of the seminal figures in the field of computer vision, the kind before the deep learning revolution, and the kind after. He has been cited over 180,000 times and has mentored many world-class researchers in computer science.

Support this podcast by supporting our sponsors:
- BetterHelp: http://betterhelp.com/lex
- ExpressVPN at https://www.expressvpn.com/lexpod

EPISODE LINKS:
Jitendra's website: https://people.eecs.berkeley.edu/~malik/
Jitendra's wiki: https://en.wikipedia.org/wiki/Jitendra_Malik

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
3:17 - Computer vision is hard
10:05 - Tesla Autopilot
21:20 - Human brain vs computers
23:14 - The general problem of computer vision
29:09 - Images vs video in computer vision
37:47 - Benchmarks in computer vision
40:06 - Active learning
45:34 - From pixels to semantics
52:47 - Semantic segmentation
57:05 - The three R's of computer vision
1:02:52 - End-to-end learning in computer vision
1:04:24 - 6 lessons we can learn from children
1:08:36 - Vision and language
1:12:30 - Turing test
1:16:17 - Open problems in computer vision
1:24:49 - AGI
1:35:47 - Pick the right problem

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Jitendra Malik, a professor at Berkeley and 1 of the seminal figures in the field of computer vision, the kind before the deep learning revolution and the kind after.

He has been cited over 180,000 times and has mentored many world-class researchers in computer science.

2 sponsors, 1 new 1, which is BetterHelp and an old goodie, ExpressVPN.

Please consider supporting this podcast by going to betterhelp.com slash Lex and signing up at expressvpn.com slash LexPod.

It really is the best way to support this podcast and the journey I'm on.

If you enjoy this thing, subscribe on YouTube, review it with 5 Stars on Apple Podcast, support it on Patreon, or connect with me on Twitter at Lex Friedman, however the heck you spell that.

As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation.

This show is sponsored by BetterHelp, spelled H-E-L-P, help.

Check it out at betterhelp.com slash Lex.

They figure out what you need and match you with a licensed professional therapist in under 48 hours.

It's not a crisis line, it's not self-help, it's professional counseling done securely online.

I'm a bit from the David Goggins line of creatures, as you may know, and so have some demons to contend with, usually on long runs or all nights working, forever impossibly full of self-doubt.

It may be because I'm Russian, but I think suffering is essential for creation.

But I also think you can suffer beautifully in a way that doesn't destroy you.

For most people, I think a good therapist can help in this.

It's easy, private, affordable, available worldwide.

You can communicate by text any time and schedule weekly audio and video sessions.

I highly recommend that you check them out at betterhelp.com slash Lex.

This show is also sponsored by ExpressVPN.

Get it at expressvpn.com slash LexPod to support this podcast and to get an extra 3 months free on a 1 year package.

I've been using ExpressVPN for many years.

I think ExpressVPN is the best VPN out there.

They told me to say it, but it happens to be true.

It doesn't log your data, it's crazy fast, and it's easy to use.

Literally just 1 big sexy power on button.

Again, for obvious reasons, it's really important that they don't log your data.

It works on Linux and everywhere else too, but really, why use anything else?

Shout out to my favorite flavor of Linux, Ubuntu Mate 2004.

Once again, get it at expressvpn.com slash LexPod to support this podcast and to get an extra 3 months free on a 1 year package.

And now here's my conversation with Jitendra Malik.

In 1966, Seymour Papert at MIT wrote up a proposal called the Summer Vision Project to be given, as far as

So that proposal outlined many of the computer vision tasks we still work on today.

Why do you think we underestimate, and perhaps we did underestimate, and perhaps still underestimate, how hard computer vision is?

Because most of what we do in vision, we do unconsciously or subconsciously.

So that gives us this, that effortlessness gives us the sense that, oh, this must be very easy to implement on a computer.

Now, this is why the early researchers in AI got it so wrong.

However, if you go into neuroscience or psychology of human vision, then the complexity becomes very clear.

The fact is that a very large part of the cerebral cortex is devoted to visual processing.

I mean, and this is true in other primates as well.

So Once we looked at it from a neuroscience or psychology perspective, it becomes quite clear that the problem is very challenging and it will take some time.

You said the high-level parts are the harder parts?

I think vision appears to be easy because most of what visual processing is subconscious or unconscious.

Whereas when you are proving a mathematical theorem or playing chess, the difficulty is much more evident.

So because it is your conscious brain which is processing various aspects of the problem solving behavior.

Whereas in vision, all this is happening, but it's not in your awareness, it's in your, it's operating below that.

Yes, that's true, but it seems strange that as computer vision researchers, for example, the community broadly is time and time again makes the mistake of thinking the problem is easier than it is.

We'll talk a little bit about autonomous driving, for example, how hard of a vision task that is.

Do you think, I mean, is it just human nature or is there something fundamental to the vision problem that we underestimate?

We're still not able to be cognizant of how hard the problem is.

Yeah, I think in the early days, it could have been excused because in the early days, all aspects of AI were regarded as too easy.

But I think today it is much less excusable.

And I think why people fall for this is because of what I call the fallacy of the successful first step.

There are many problems in vision where getting 50% of the solution you can get in 1 minute, getting to 90% can take you a day, getting to 99% may take you 5 years, and

It seems that language, people are not so confident about.

So natural language processing, people are a little bit more cautious about our ability to solve that problem.

I think for language, people intuit that we have to be able to do natural language understanding.

For vision, it seems that we're not cognizant or we don't think about how much understanding is required.

It's probably still an open problem, but in your sense, how much understanding is required to solve vision?

Like this, put another way, how much something called common sense reasoning is required to really be able to interpret even static scenes?

Yeah, so vision operates at all levels and there are parts which can be solved with what we could call maybe peripheral processing.

So in the human vision literature, there used to be these terms sensation, perception, and cognition, which roughly speaking referred to like the front end of processing, middle stages of processing, and higher level of processing.

And I think they made a big deal out of this, and they wanted to just study only perception and then dismiss certain problems as being, quote, cognitive.

But really, I think these are artificial divides.

The problem is continuous at all levels, and there are challenges at all levels.

The techniques that we have today, they work better at the lower and mid levels of the problem.

I think the higher levels of the problem, quote, the cognitive levels of the problem, are there and we, in many real applications, we have to confront them.

Now, how much that is necessary will depend on the application.

So I am, for example, a pessimist on fully autonomous driving in the near future.

And the reason is because I think there will be that

of the cases where quite sophisticated cognitive reasoning is called for.

However, there are tasks where you can, first of all, they are much more, they are robust.

So in the sense that error rates, error is not so much of a problem.

For example, Let's say you're doing image search.

You're trying to get images based on some description, some visual description.

We are very tolerant of errors there, right?

I mean, when Google image search gives you some images back and a few of them are wrong, it's okay.

But making mistakes when you are driving at 60 miles per hour and you could potentially kill somebody is much more important.

So just for the fun of it, since you mentioned, let's go there briefly about autonomous vehicles.

So 1 of the companies in the space, Tesla, is with Andrej Karpathy and Elon Musk are working on a system called Autopilot, which is primarily a vision-based system with 8 cameras and basically a single neural network, a multitask neural network.

They call it HydraNet, multiple heads, so it does multiple tasks, but is forming the same representation at the core.

Do you think driving can be converted in this way to purely a vision problem and then solved with learning?

Or even more specifically in the current approach, what do you think about what Tesla autopilot team is doing?

So the way I think about it is that there are certainly subsets of the visual based driving problem which are quite solvable.

So for example, driving in freeway conditions is quite a solvable problem.

I think there were demonstrations of that going back to the 1980s by someone called Ernst Tickmans in Munich.

In the 90s, there were approaches from Carnegie Mellon, there were approaches from our team at Berkeley.

In the 2000s, there were approaches from Stanford and so on.

So autonomous driving in certain settings is very doable.

The challenge is to have an autopilot work under all kinds of driving conditions.

At that point, it's not just a question of vision or perception, but really also of control and dealing with all the edge cases.

So where do you think most of the difficult cases, to me, even the highway driving is an open problem because it applies the same

rule or the first step, the fallacy of the first step, I forget how you put it, we fall victim to.

I think even highway driving has a lot of elements because to solve autonomous driving, you have to completely relinquish the help of a human being.

So you really going to feel the edge cases.

So I think even highway driving is really difficult.

But in terms of the general driving task, do you think vision is the fundamental problem or is it also your action, the interaction with the environment, the ability to, and then like the middle ground, I don't know if you put that under vision, which is trying to predict the behavior of others, which is a little bit in the world of understanding the scene, but it's also trying to form a model of the actors in the scene and predict their behavior.

Yeah, I include that in vision, because to me, perception blends into cognition and building predictive models of other agents in the world, which could be other agents, could be people, other agents could be other cars, that is part of the task of perception.

Because perception always has to not tell us what is now, but what will happen.

Because what's now is boring, it's done, it's over with.

We care about the future because we act in the future.

in as much as it informs what's gonna happen in the future.

So I think we have to build predictive models of behaviors of people, And those can get quite complicated.

So I mean, I've seen examples of this in actually, I mean, I own a Tesla and it has various safety features built in.

And what I see are these examples where, let's say there is some skateboarder.

I mean, I don't want to be too critical because obviously these systems are always being improved and any specific criticism I have, maybe the system 6 months from now will not have that particular failure mode.

So it had the wrong response and it's because it couldn't predict what this skateboarder was going to do.

And because it really required that higher level cognitive understanding of what skateboarders typically do as opposed to a normal pedestrian.

So what might have been the correct behavior for a pedestrian, a typical behavior for a pedestrian was not the typical behavior for a skateboarder, right?

And so therefore to do a good job there, you need to have enough data where you have pedestrians, you also have skateboarders, you've seen enough skateboarders to see what kinds of patterns of behavior they have.

So it is in principle with enough data that problem could be solved.

But I think our current systems, computer vision systems, they need far, far more data than humans do for learning those same capabilities.

So say that there is going to be a system that solves autonomous driving, do you think it will look similar to what we have today but have a lot more data, perhaps more compute, but the fundamental architecture is involved, like, well, in the case of Tesla Autopilot, is neural networks.

Do you think it will look similar in that regard and it'll just have more data?

That's a scientific hypothesis as to which way is it going to go.

So, and this is my general philosophical position on how these learning systems have been.

What we have found currently very effective in computer vision within the deep learning paradigm is sort of tabula rasa learning and tabula rasa learning in a supervised way with lots and lots of.

Tabula rasa in the sense that blank slate.

We just have the system which is given a series of experiences in this setting and then it learns there.

So at the age of 16 in high school, a teenager goes into driver ed class.

And now at that point they learn, but at the age of 16 they are already visual geniuses because from 0 to 16, they have built a certain repertoire of vision.

In fact, most of it has probably been achieved by age 2.

In this period of age, up to age 2, they know that the world is 3 dimensional.

They know how objects look like from different perspectives.

They know about common dynamics of humans and other bodies.

They have some notion of intuitive physics.

So they built that up from their observations and interactions in early childhood and of course reinforced through their growing up to age 16.

So then At age 16, when they go into driver ed, what are they learning?

They're not learning afresh the visual world.

They're learning how to be smooth about control, about steering and brakes and so forth.

They're learning a sense of typical traffic situations.

Now, that education process can be quite short because they are coming in as visual geniuses.

And of course, in their future, they're going to encounter situations which are very novel, right?

So during my driver ed class, I may not have had to deal with a skateboarder.

I may not have had to deal with a truck driving in front of me, where the back opens up and some junk gets dropped from the truck and I have to deal with it.

But I can deal with this as a driver, even though I did not encounter this in my driver ed class.

And the reason I can deal with it is because I have all this general visual knowledge and expertise.

And do you think the learning mechanisms we have today can do that kind of long-term accumulation of knowledge?

Or do we have to do some kind of, the work that led up to expert systems with knowledge representation, the broader field of artificial intelligence worked on this kind of accumulation of knowledge.

Do you think neural networks can do the same?

I think I don't see any in-principle problem with neural networks doing it, but I think the learning techniques would need to evolve significantly.

So the current learning techniques that we have are supervised learning.

You're given lots of examples, X, Y, pairs, and you learn the functional mapping between them.

I think that human learning is far richer than that.

There is a child explores the world and sees us, For example, a child takes an object and manipulates it in his hand and therefore gets to see the object from different points of view and the child has commanded the movement.

So That's a kind of learning data, but the learning data has been arranged by the child.

The child can do various experiments with the world.

So there are many aspects of sort of human learning and these have been studied in child development by psychologists and what they tell us is that supervised learning is a very small part of it.

There are many different aspects of learning.

And what we would need to do is to develop models of all of these and then train our systems in that, with that kind of protocol.

Some of which might imitate the human brain.

But you also, in your talks, have mentioned sort of the compute side of things, in terms of the difference in the human brain, or referencing Hans Moravec.

The, so, do you think there's something interesting, valuable to consider about the difference in the computational power of the human brain versus the computers of today in terms of instructions per second?

Yes, so if we go back, so this is a point I've been making for 20 years now.

And I think once upon a time, the way I used to argue this was that we just didn't have the computing power of the human brain.

And I mean, there is a well-known trade-off which we know that neurons are slow compared to transistors, but we have a lot of them and they have a very high connectivity.

Whereas in silicon, you have much faster devices, transistors switch at on the order of nanoseconds, but the connectivity is usually smaller.

At this point in time, I mean, we are now talking about 2020, we do have, if you consider the latest GPUs and so on, amazing computing power.

And if we look back at Hans Moravec's type of calculations, which he did in the 1990s, we may be there today in terms of computing power comparable to the brain, but it's not of the same style.

So I mean, for example, the style of computing that we have in our GPUs is far, far more power hungry than the style of computing that is there in the human brain or other biological entities.

Yeah, and that, the efficiency part is, we're gonna have to solve that in order to build actual real world systems of large scale.

Let me ask sort of the high level question.

How would you articulate the general problem of computer vision?

So if you look at the computer vision conferences and the work that's been going on, it's often separated into different little segments, breaking the problem of vision apart into whether segmentation, 3D reconstruction, object detection, I don't know, image capturing, whatever.

But if you were to sort of philosophically say, what is the big problem of computer vision?

So if we have to, So for all intelligence tasks, I always go back to sort of biology or humans.

And if you think about vision or perception in that setting, we realize that perception is always to guide action.

Perception for a biological system does not give any benefits unless it is coupled with action.

So we can go back and think about the first multicellular animals which arose in the Cambrian era,

And these animals could move and they could see in some way.

And the 2 activities helped each other because How does movement help?

Movement helps that because you can get food in different places.

And that's really about perception or seeing.

I mean, vision is perhaps the single most perception sense.

But all the others are equally, are also important.

So, perception and action kind of go together.

So earlier it was in these very simple feedback loops, which were about finding food or avoiding becoming food if there's a predator running, trying to eat you up and so forth.

So we must, at the fundamental level, connect perception to action.

Then as we evolved, perception became more and more sophisticated because it served many more purposes.

And so today we have what seems like a fairly general purpose capability, which can look at the external world and build a model of the external world inside the head.

And psychologists have great fun in pointing out the ways in which the model in your head is not a perfect model of the external world.

They create various illusions to show the ways in which it is imperfect.

But it's amazing how far it has come from a very simple perception action loop that exists in an animal 500 million years ago.

Once we have these very sophisticated visual systems, we can then impose a structure on them.

It's we as scientists who are imposing that structure, where we have chosen to characterize this part of the system as this quote, module of object detection or quote, this module of 3D reconstruction.

What's going on is really all of these processes are running simultaneously, and they are running simultaneously because originally their purpose was, in fact, to help guide action.

So as a guiding general statement of a problem, do you think we can say that the general problem of computer vision, you said in humans it was tied to action.

Do you think we should also say that ultimately the goal, the problem of computer vision is to sense the world in a way that helps you act in the world.

Yes, I think that's the most fundamental, that's the most fundamental purpose.

So we have this visual system which can be used for other things.

For example, judging the aesthetic value of a painting.

Maybe it's guiding action in terms of how much money you will put in your auction bid, but that's a bit stretched.

But the basics are in fact in terms of action.

But we have evolved really this hyper, we have hyper evolved our visual system.

Actually, just to, sorry to interrupt, but perhaps it is fundamentally about action.

You kind of jokingly said about spending, but perhaps the capitalistic drive that drives a lot of the development in this world is about the exchange of money and the fundamental action is money.

If you watch Netflix, if you enjoy watching movies, you're using your perception system to interpret the movie, ultimately your enjoyment of that movie means you'll subscribe to Netflix.

So the action is this extra layer that we've developed in modern society, perhaps is fundamentally tied to the action of spending money.

Well, certainly with respect to interactions with firms.

So in this homo economicus role, when you're interacting with firms, it does become that.

Okay, so to linger on the division between the static and the dynamic, so much of the work in computer vision, so many of the breakthroughs that you've been a part of have been in the static world, in looking at static images.

See all Lex Fridman transcripts on Youtube

Jitendra Malik: Computer Vision | Lex Fridman Podcast #110