The following is a conversation with Tomasso Poggio.

He's a professor at MIT and is a director of the Center for Brains, Minds and Machines.

Cited over 100,000 times, his work has had a profound impact on our understanding of the nature of intelligence in both biological and artificial neural networks.

He has been an advisor to many highly impactful researchers and entrepreneurs in AI, including Demis Hassabis of DeepMind, Amnon Shashua of Mobileye, and Christophe Koch of the Allen Institute for Brain Science.

This conversation is part of the MIT course on artificial general intelligence and the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Tommaso Poggio.

You've mentioned that in your childhood, you've developed a fascination with physics, especially the theory of relativity, and that Einstein was also a childhood hero to you.

What aspect of Einstein's genius, the nature of his genius, do you think was essential for discovering the theory of relativity?

You know, Einstein was a hero to me and I'm sure to many people because he was able to make of course a major major contribution to physics with simplifying a bit just a Gedanken experiment, a thought experiment.

You know, imagining communication with lights between a stationary observer and somebody on a train.

And I thought, you know, the fact that just with the force of his thought, of his thinking, of his mind, he could get to something so deep in terms of physical reality, how time depend on space and speed.

It was the power of intelligence, the power of the mind.

Do you think the ability to imagine, to visualize as he did, as a lot of great physicists do, Do you think that's in all of us, human beings?

I think, you know, all of us can learn and have, in principle, similar breakthroughs.

There are lessons to be learned from Einstein.

He was 1 of 5 PhD students at ETA, the Eidgenossische Technische Hochschule in Zurich in physics.

The only 1 who did not get an academic position when he graduated, when he finished his PhD, and he went to work, as everybody knows, for the patent office.

And so it's not so much that he worked for the patent office, but the fact that obviously he was smart, but he was not the top student.

He was not thinking in the traditional way that probably teachers and the other students were doing.

So there is a lot to be said about trying to be, to do the opposite or something quite different from what other people are doing.

That's certainly true for the stock market.

So you've also mentioned, staying on the theme of physics, that you were excited at a young age by the mysteries of the universe that physics could uncover.

Such, as I saw mentioned, the possibility of time travel.

So the most out of the box question I think I'll get to ask today, do you think time travel is possible?

Well it would be nice if it were possible right now.

But your understanding of the nature of time.

Yeah, it's very likely that it's not possible to travel in time.

We may be able to travel forward in time if we can, for instance, freeze ourselves or go on some spacecraft traveling close to the speed of light.

But in terms of actively traveling, for instance, back in time, I find probably very unlikely.

So do you still hold the underlying dream of the engineering intelligence that will build systems that are able to do such huge leaps like discovering the kind of mechanism that would be required to travel through time.

Do you still hold that dream, or echoes of it from your childhood?

Yeah, I don't think whether, there are certain problems that probably cannot be solved, depending what you believe about the physical reality.

Like, you know, maybe totally impossible to create energy from nothing or to travel back in time.

But about making machines that can think as well as we do or better, or more likely especially in the short and mid term, help us think better, which is in a sense is happening already with the computers we have and it will happen more and more.

Well that I certainly believe and I don't see in principle why computers at some point could not become more intelligent than we are.

Although the word intelligence is a tricky 1 and 1 we should discuss what I mean with that.

And intelligence, consciousness, words like love, is all these are very, need to be disentangled.

So you've mentioned also that you believe the problem of intelligence is the greatest problem in science, greater than the origin of life and the origin of the universe.

You've also, in a talk I've listened to, said that you're open to arguments against you.

So what do you think is the most captivating aspect of this problem of understanding the nature of intelligence?

Well, originally, I think 1 of the motivation that I had as, I guess, a teenager, when I was infatuated with theory of relativity was really that I found that there was the problem of time and space and general relativity, but there were so many other problems of the same level of difficulty and importance that I could, Even if I were Einstein, it was difficult to hope to solve all of them.

So what about solving a problem whose solution allowed me to solve all the problems?

And this was what if we could find the key to an intelligence 10 times better or faster than Einstein.

So that's sort of seeing artificial intelligence as a tool to expand our capabilities.

But is there just an inherent curiosity in you and just understanding what it is in here that makes it all work?

So I started saying this was the motivation when I was a teenager, but soon after, I think the problem of human intelligence became a real focus of my science and my research because I think for me, the most interesting problem is really asking who we are, right?

Is asking not only a question about science, but even about the very tool we are using to do science, which is our brain.

And that in many ways is the ultimate question that underlies this whole effort of science.

So you've made significant contributions in both the science of intelligence and the engineering of intelligence.

In a hypothetical way, let me ask, how far do you think we can get in creating intelligent systems without understanding the biological, the understanding how the human brain creates intelligence?

Put another way, do you think we can build a strong AI system without really getting at the core, the functional, understanding the functional and the issue of the brain?

You know, We did solve problems like flying without really using too much our knowledge about how birds fly.

It was important, I guess, to know that you could have things heavier than air being able to fly like birds.

But beyond that, probably we did not learn very much.

You know, some, The Brothers Wright did learn a lot of observation about birds and designing their aircraft.

But you know, you can argue we did not use much of biology in that particular case.

Now in the case of intelligence I think that it's a bit of a bet right now.

If you ask okay we all agree we'll get at some point, maybe soon, maybe later, to a machine that is indistinguishable from my secretary, say in terms of what I can ask the machine to do.

And now the question is, you can ask people, do you think we'll get there without any knowledge about the human brain or that the best way to get there is to understand better the human brain?

Okay, this is I think an educated bet that different people with different background will decide in different ways.

The recent history of the progress in AI in the last, I would say 5 years or 10 years has been the main breakthroughs, the main recent breakthroughs, are really start from neuroscience.

I can mention reinforcement learning as 1, is 1 of the algorithms at the core of AlphaGo, which is the system that beat the kind of an official world champion of Go, Lee Sedol, and 2, 3 years ago in Seoul.

That's 1, and that started really with the work of Pavlov in

Marvin Minsky in the 60s, and many other neuroscientists later on.

And deep learning started, which is at the core again of AlphaGo and systems like autonomous driving systems for cars, like the systems that Mobil Eye, which is a company started by 1 of my ex-post-docs, Amnon Shashua, that is at the core of those things.

And deep learning, really the initial ideas in terms of the architecture of this layered hierarchical networks started with work of Thorsten Wiesel and David Hubel at Harvard, up the river in the 60s.

So recent history suggests that neuroscience played a big role in these breakthroughs.

My personal bet is that there is a good chance they continue to play a big role, Maybe not in all the future breakthroughs, but in some of them.

At least in inspiration, absolutely, yes.

So you studied both artificial and biological neural networks.

You said these mechanisms that underlie deep learning and reinforcement learning.

But there is nevertheless significant differences between biological and artificial neural networks as they stand now.

So between the 2, What do you find is the most interesting, mysterious, maybe even beautiful difference as it currently stands in our understanding?

I must confess that until recently, I found that the artificial networks, too simplistic relative to real neural networks.

But recently I've been started to think that yes there are very big simplification of what you find in the brain.

But on the other hand there are much closer in terms of the architecture to the brain than other models that we had, that computer science used as model of thinking, which were mathematical logics, you know, Lisp, Prolog, and those kind of things.

So in comparison to those, they're much closer to the brain.

You have networks of neurons, which is what the brain is about.

And the artificial neurons in the models are, as I said, caricature of the biological neurons, but they're still neurons, single units communicating with other units, something that is absent in the traditional computer type models of mathematics, reasoning, and so on.

So what aspect would you like to see in artificial neural networks added over time as we try to figure out ways to improve them?

So 1 of the main differences and problems in terms of deep learning today, and it's not only deep learning, and the brain is the need for deep learning techniques to have a lot of labeled examples.

You know, for instance, for ImageNet, you have like a training set which is 1000000 images, each 1 labeled by some human in terms of which object is there.

And it's clear that in biology, a baby may be able to see a million of images in the first years of life, but will not have a million of labels given to him or her by parents or caretakers.

You know, I think that there is this interesting challenge that today deep learning and related techniques are all about big data.

Big data meaning a lot of examples labeled by humans.

Whereas in nature you have, so this big data is n going to infinity, that's the best, n meaning labeled data.

But I think the biological world is more N going to 1.

You don't need to say, like in ImageNet, this is a car, this is a car, this is not a car, this is not a car, 1000000 times.

So, and of course with AlphaGo or at least the AlphaZero variants, there's because the world of Go is so simplistic that you can actually learn by yourself, through self-play, you can play against each other.

In the real world, I mean, the visual system that you've studied extensively is a lot more complicated than the game of Go.

On the comment about children, which are fascinatingly good at learning new stuff, how much of it do you think is hardware and how much of it is software?

It is in a sense is the old question of nurture and nature, how much is in the gene and how much is in the experience of an individual.

Obviously, it's both that play a role and I believe that the way evolution gives, put prior information, so to speak, hardwired, it's not really hardwired, but That's essentially an hypothesis.

I think what's going on is that evolution has, you know, almost necessarily, if you believe in Darwin, is very opportunistic.

And think about our DNA and the DNA of Drosophila.

Our DNA does not have many more genes than Drosophila.

Now we know that the fruit fly does not learn very much during its individual existence.

It looks like 1 of this machinery that it's really mostly, not

but you know 95% hard-coded by the genes.

But since we don't have many more genes than Drosophila, evolution could encode in us a kind of general learning machinery and then had to give very weak priors.

Like for instance, let me give a specific example which is recent work by a member of our Center for Brains, Minds, and Machines.

We know because of work of other people in our group and other groups that there are cells in a part of our brain, neurons, that are tuned to faces.

They seem to be involved in face recognition.

Now this face area exists, seems to be present in young children and adults.

And 1 question is, is there from the beginning, is hardwired by evolution, or somehow is learned very quickly?

So what's your, by the way, a lot of the questions I'm asking, the answer is we don't really know, but as a person who has contributed some profound ideas in these fields, you're a good person to guess at some of these.

So of course there's a caveat before a lot of the stuff we talk about.

Is the face, the part of the brain that seems to be concentrated on face recognition, are you born with that?

Or you just, it's designed to learn that quickly, like the face of the mother and so

My hunch, my bias was the second 1, learned very quickly.

And it turns out that Marge Livingstone at Harvard has done some amazing experiments in which she raised baby monkeys, depriving them of faces during the first weeks of life.

So they see technicians, but the technicians have a mask.

And so when they looked at the area in the brain of these monkeys that where usually you find faces, they found no face preference.

So my guess is that what evolution does in this case is there is a plastic, an area which is plastic, which is kind of predetermined to be imprinted very easily, but the command from the gene is not a detailed circuitry for a face template.

Could be, but this will require probably a lot of bits.

You have to specify a lot of connection of a lot of neurons.

Instead, the command from the gene is something like imprint, memorize what you see most often in the first 2 weeks of life, especially in connection with food.

And so, and then that area is very plastic at first and then solidifies.

It'd be interesting if a variant of that experiment would show a different kind of pattern associated with food than a face pattern, whether that could stick.

There are indications that during that experiment, what the monkeys saw quite often were the blue gloves of the technicians that were giving to the baby monkeys the milk.

And some of the cells, instead of being face sensitive in that area, are hand sensitive.

Can you talk about what are the different parts of the brain and in your view sort of loosely and how do they contribute to intelligence?

Do you see the brain as a bunch of different modules and they together come in the human brain to create intelligence or is it all 1 mush of the same kind of fundamental architecture?

Yeah, that's an important question and there was a phase in neuroscience back in the 1950 or so in which it was believed for a while that the brain was equipotential, this was the term.

You could cut out a piece and nothing special happened apart a little bit less performance.

There was a surgeon, Lashley, who did a lot of experiments of this type with mice and rats and concluded that every part of the brain was essentially equivalent to any other 1.

It turns out that that's really not true.

It's, there are very specific modules in the brain, as you said, and people may lose the ability to speak if you have a stroke in a certain region or may lose control of their legs in another region.

The brain is also quite flexible and redundant so often it can correct things and kind of take over functions from 1 part of the brain to the other, but really there are specific modules.

So the answer that we know from this old work, which was basically based on lesions, either on animals or very often there were a mine of, well, there was a mine of very interesting data coming from the war, from different types of injuries that soldiers had in the brain.

And more recently, functional MRI, which allow you to check which part of the brain are active when you are doing different tasks, as you know, can replace some of this.

You can see that certain parts of the brain are involved, are active

But sort of taking a step back to that part of the brain that discovers that specializes in the face and how that might be learned, what's your intuition behind, you know, is it possible that sort of from a physicist's perspective when you get lower and lower, that it's all the same stuff, and it just, when you're born, it's plastic and quickly figures out, this part is gonna be about vision, this is gonna be about language, this is about common sense reasoning.

Do you have an intuition that that kind of learning is going on really quickly, or is it really kind of solidified in hardware?

So there are parts of the brain like the cerebellum or the hippocampus that are quite different from each other.

They clearly have different anatomy, different connectivity.

Then there is the cortex, which is the most developed part of the brain in humans.

And in the cortex you have different regions of the cortex that are responsible for vision, for audition, for motor control, for language.

Now 1 of the big puzzles of this is that in the cortex, is the cortex, is the cortex, looks like it is the same in terms of hardware, in terms of type of neurons and connectivity across these different modalities.

So for the cortex, letting aside these other parts of the brain like spinal cord, hippocampus, cerebellum and so on, for the cortex I think your question about hardware and software and learning and so on, it's, I think it's rather open.

And I find very interesting for us to think about an architecture, computer architecture, that is good for vision, and at the same time is good for language.

Seems to be so different problem areas that you have to solve.

But the underlying mechanism might be the same, and that's really instructive for artificial neural networks.

So you've done a lot of great work in vision, in human vision, computer vision, and you mentioned the problem of human vision is really as difficult as the problem of general intelligence.

And maybe that connects to the cortex discussion.

Can you describe the human visual cortex and how the humans begin to understand the world through the raw sensory information.

What's, for folks who are not familiar, especially on the computer vision side, We don't often actually take a step back except saying with a sentence or 2 that 1 is inspired by the other.

What is it that we know about the human visual cortex?

We know quite a bit, at the same time we don't know a lot.

But the bit We know, in a sense we know a lot of the details and many we don't know and we know a lot of the top level, the answer to top level question but we don't know some basic ones, even in terms of general neuroscience, forgetting vision, you know, why do we sleep?

And we really don't have an answer to that.

Do you think, So taking a step back on that, so sleep for example is fascinating.

Do you think that's a neuroscience question?

Or if we talk about abstractions, what do you think is an interesting way to study intelligence or most effective on the levels of abstraction?

Is it chemical, is it biological, is it electrophysical, mathematical, as you've done a lot of excellent work on that side, which psychology, sort of like, which level of abstraction do you think?

Well, in terms of levels of abstraction, I think we need all of them.

It's when, you know, it's like if you ask me, what does it mean to understand a computer, right?

That's much simpler, but in a computer, I could say, well, I understand how to use PowerPoint.

That's my level of understanding a computer.

It's, it has reasonable, you know, it gives me some power to produce slides and beautiful slides.

And now, You can ask somebody else, he says, well I know how the transistor work that are inside the computer.

I can write the equation for, you know, transistor and diodes and circuits, logical circuits.

And I can ask this guy, do you know how to operate PowerPoint?

So do you think if we discovered computers walking amongst us full of these transistors that are also operating under Windows and have PowerPoint, do you think it's digging in a little bit more, how useful is it to understand the transistor in order to be able to understand PowerPoint and these higher level intelligent processes?

So I think in the case of computers, because they were made by engineers by us, this different level of understanding are rather separate on purpose.

They are separate modules so that the engineer that designed the circuit for the chips does not need to know what is inside PowerPoint.

And somebody can write the software translating from 1 to the other.

So, in that case, I don't think understanding the transistor help you understand PowerPoint, or very little.

If you want to understand the computer, this question, I would say you have to understanding at different levels.

But for the brain, I think these levels of understanding, so the algorithms, which kind of computation, you know, the equivalent of PowerPoint, and the circuits, you know, the transistors, I think they are much more intertwined with each other.

There is not, you know, a neatly level of the software separate from the hardware.

And so that's why I think in the case of the brain, the problem is more difficult, more than for computers requires the interaction, the collaboration between different types of expertise.

So it's a big, the brain is a big hierarchical mess.

you can, but it's much more difficult and it's not completely obvious.

And as I said, I think he's 1 of the, personally I think he's the greatest problem in science.

So I think it's fair that it's difficult.

That said, you do talk about compositionality and why it might be useful.

And when you discuss why these neural networks in artificial or biological sense learn anything, you talk about compositionality, there's a sense that nature can be disentangled, well, all aspects of our cognition could be disentangled a little to some degree.

So why do you think, first of all, how do you see compositionality and why do you think it exists at all in nature?

I spoke about, I used the term compositionality when we looked at deep neural networks, multi-layers, and trying to understand when and why they are more powerful than more classical one-layer networks like linear classifier, kernel machines, so-called.

And what we found is that in terms of approximating or learning or representing a function, a mapping from an input to an output, like from an image to the label in the image, If this function has a particular structure, then deep networks are much more powerful than shallow networks to approximate the underlying function.

And the particular structure is a structure of compositionality.

If the function is made up of functions of functions so that you need to look on, when you are interpreting an image, classifying an image, you don't need to look at all pixels at once but you can compute something from small groups of pixels and then you can compute something on the output of this local computation and so on.

It is similar to what you do when you read a sentence.

You don't need to read the first and the last letter, but you can read syllables, combine them in words, combine the words in sentences.

So that's as part of a discussion of why deep neural networks may be more effective than the shallow methods.

And is your sense for most things we can use neural networks for, those problems are going to be compositional in nature, like language, like vision.

So a friend of mine, Max Tegmark, who is a physicist at MIT.

Yeah, we agree on most, but the conclusion is a bit different.

His conclusion is that for images, for instance, the compositional structure of this function that we have to learn or to solve these problems comes from physics, comes from the fact that you have local interactions in physics between atoms and other atoms, between particle of matter and other particles, between planets and other planets, between stars and other, it's all local.

And that's true, but you could push this argument a bit further, not this argument actually, you could argue that, you know, maybe that's part of the truth, but maybe what happens is kind of the opposite, is that our brain is wired up as a deep network.

So it can learn, understand, solve problems that have this compositional structure.

And it cannot do, it cannot solve problems that don't have this compositional structure.

So the problems we are accustomed to, we think about, we test our algorithms on, are this compositional structure because our brain is made up.

And that's in a sense an evolutionary perspective that we've, so the ones that didn't have, that weren't dealing with the compositional nature of reality died off?

Yes, but also could be maybe the reason why we have this local connectivity in the brain, like simple cells in cortex looking only at the small part of the image, each 1 of them, and then other cells looking at the small number of these simple cells and so on.

The reason for this may be purely that it was difficult to grow long range connectivity.

So suppose it's, you know, for biology, it's possible to grow short range connectivity but not long range also because there is a limited number of long range.

And so you have this limitation from the biology.

And this means you build a deep convolutional network.

See all Lex Fridman transcripts on Youtube

Tomaso Poggio: Brains, Minds, and Machines | Lex Fridman Podcast #13