Full episode with John Hopfield (Feb 2020): https://www.youtube.com/watch?v=DKyzcbNr8WE
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

John Hopfield is professor at Princeton, whose life's work weaved beautifully through biology, chemistry, neuroscience, and physics. Most crucially, he saw the messy world of biology through the piercing eyes of a physicist. He is perhaps best known for his work on associate neural networks, now known as Hopfield networks that were one of the early ideas that catalyzed the development of the modern field of deep learning.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

What difference between biological neural networks and artificial neural networks is most captivating and profound to you?

At the higher philosophical level, let's not get technical just yet.

1 of the things that very much intrigues me is the fact that neurons have all kinds of components, properties to them.

In evolutionary biology, if you have some little quirk in how a molecule works, or how a cell works, and it can be made use of, evolution will sharpen it up and make it into a useful feature, rather than a glitch.

And so you expect in neurobiology for evolution to have captured all kinds of possibilities of getting neurons, of how you get neurons to do things for you.

And That aspect has been completely suppressed in artificial neural networks.

Do the glitches become features in the biological neural network?

Look, Let me take 1 of the things that I used to do research on.

If you take things which oscillate, they have rhythms which are sort of close to each other, under some circumstances these things will have a phase transition, and suddenly the rhythm will… everybody will fall into step.

There was a marvelous physical example of that in the Millennium Bridge across the Thames River about… built about

And pedestrians walking across, pedestrians don't walk synchronized, they don't walk in lockstep.

But they all walk at about the same frequency.

And the bridge could sway at that frequency, and the slight sway made pedestrians tend a little bit to lock into step.

And after a while, the bridge was oscillating back and forth and the pedestrians were walking in step to it.

And you could see it in the movies made out of the bridge.

And the engineers made a simple-minded mistake.

They assumed when you walk, it's step, step, step, and it's back and forth motion.

But when you walk, it's also right foot, left foot, side to side motion.

And it's the side to side motion for which the bridge was strong enough, but it wasn't stiff enough.

And as a result, you would feel the motion and you'd fall into step with it.

And people were very uncomfortable with it.

They closed the bridge for 2 years while they built stiffening for it.

Now, nerves, look, nerve cells produce action potentials.

You have a bunch of cells which are loosely coupled together, producing action potentials of the same rate.

There'll be some circumstances under which these things can lock together.

Well, if they fire together, you can be sure that the other cells are going to notice it.

So, you can make a computational feature out of this in an evolving brain.

Most artificial neural networks don't even have action potentials, let alone have the possibility for synchronizing them.

And you mentioned the evolutionary process.

So the evolutionary process that builds on top of biological systems leverages that, the weird mess of it somehow?

So how do you make sense of that ability to leverage all the different kinds of complexities in the biological brain?

Well, look, at the biological molecule level, you have a piece of DNA which encodes for a particular protein.

You could duplicate that piece of DNA, and now 1 part of it can code for that protein, but the other 1 could itself change a little bit and thus start coding for a molecule which is slightly different.

Now, if that molecule was just slightly different, Had a function which helped any old chemical reaction was as important to the cell.

You would go ahead and let that try And evolution would slowly improve that function.

And so you have the possibility of duplicating, and then having things drift apart.

1 of them retain the old function, the other 1 do something new for you.

And there's evolutionary pressure to improve.

Look, there is in computers too, but it's improvement has to do with closing some companies and opening some others.

The evolutionary process looks a little different.

Companies close, yeah, go bankrupt and are born, yeah, shorter, but not much shorter.

Some companies last a century, But yeah, you're right.

I mean, if you think of companies as a single organism that builds and you all know, yeah, it's a fascinating dual correspondence there between biological-

And companies have difficulty having a new product competing with an old product.

And when IBM built its first PC, you probably read the book, they made a little isolated internal unit to make the PC.

And for the first time in IBM's history, they didn't insist that you build it out of IBM components.

But they understood that they could get into this market, which is a very different thing, by completely changing their culture.

And biology finds other markets in a more adaptive way.

So maybe you've already said it, but what to you is the most beautiful aspect or mechanism of the human mind?

Is it the adaptive, the ability to adapt as you've described, or is there some other little quirk that you particularly like?

Adaptation is everything when you get down to it.

But the difference, There are differences between adaptation, where your learning goes on only over generations, and over evolutionary time, where your learning goes on at the time scale of 1 individual who must learn from the environment during that individual's lifetime.

And biology has both kinds of learning in it.

And the thing which makes neurobiology hard is that it's a mathematical system, as it were, built on this other kind of evolutionary system.

Well, when you talk to a computer scientist about neural networks, it's all math.

The fact that biology actually came about from evolution, and the fact that biology is about a system which you can build in 3 dimensions.

If you look at computer chips, computer chips are basically two-dimensional structures, maybe 2.1 dimensions, but they really have difficulty doing three-dimensional wiring.

Biology is, neocortex is actually also sheet-like, and it sits on top of the white matter, which is about 10 times the volume of the gray matter and contains all what you might call the wires.

But there's a huge, the effect of computer structure on what is easy and what is hard is events.

And biology does, it makes some things easy that are very difficult to understand how to do computationally.

On the other hand, you can't do simple floating point arithmetic because it's awfully stupid.

Yeah, and you're saying this kind of 3 dimensional complicated structure makes, it's still math.

The kind of math it's doing enables you to solve problems of a very different kind.

So you mentioned 2 kinds of adaptation, the evolutionary adaptation and the adaptation or learning at the scale of a single human life.

Which do you, which is particularly beautiful to you and interesting from a research and from just a human perspective?

I find things most interesting that I begin to see how to get into the edges of them and tease them apart a little bit and see how they work.

And since I can't see the evolutionary process going on, I'm in awe of it, but I find it just a black hole as far as trying to understand what to do.

And so in a certain sense, I'm in awe of it, but I couldn't be interested in working on it.

The human life's time scale is however thing you can tease apart and study.

Yeah, you can do, there's developmental neurobiology which understands how the connections and how the structure evolves from a combination of what the genetics is like and the real, the fact that you're building a system in 3 dimensions.

In just days and months, those early, early days of a human life are really interesting.

They are and Of course, there are times of immense cell multiplication.

There are also times of the greatest cell death in the brain is during infancy.

So what is not effective, what is not wired well enough to use at the moment, throw it out.

From, let me ask, from what field do you think the biggest breakthroughs in understanding the mind will come in the next decades?

Is it neuroscience, computer science, neurobiology, psychology, physics, maybe math, maybe literature?

Well, of course, I see the world always through a lens of physics.

I grew up in physics, and the way I pick problems is very characteristic of physics and of an intellectual background, which is not psychology, which is not chemistry and so on and so on.

Both of my parents were physicists and The real thing I got out of that was a feeling that the world is an understandable place.

And if you do enough experiments and think about what they mean and structure things, or you can do the mathematics of the relevant to the experiments, you also have to be able to understand how things work.

Did you change your mind at all through many decades of trying to understand the mind, of studying in different kinds of ways, not even the mind, just biological systems.

You still have hope that physics, that you can understand.

There's a question of What do you mean by understand?

When I taught freshman physics, I used to say, I wanted to get physics to understand the subject, to understand Newton's laws.

I didn't want them simply to memorize a set of examples to which they knew the equations to write down to generate the answers.

I had this nebulous idea of understanding.

So that if you looked at a situation, you could say, oh, I expect the ball to make that trajectory, or I expect some intuitive notion of understanding.

And I don't know how to express that very well.

And you run smack up against it When you look at these simple neural nets, feed forward neural nets, which do amazing things, and yet you know contain nothing of the essence of what I would have felt was understanding.

Understanding is more than just an enormous lookup table.

These feed-forward neural networks, do you think they'll ever understand?

I think if you look at real systems, feedback is an essential aspect of how these real systems compute.

On the other hand, if I have a mathematical system with feedback, I know I can unlayer this and do it.

But I have an exponential expansion in the amount of stuff I have to build if I can solve the problem that way.

So we can talk even about recurrent neural

But do you think all the pieces are there to achieve understanding through these simple mechanisms?

Like, back to our original question, what is the fundamental, is there a fundamental difference between artificial neural networks and biological?

Suppose you ask a neurosurgeon, when is somebody dead?

They'll probably go back to saying, well, I can look at the brain rhythms and tell you this is a brain which is never gonna function again, this 1 is, this other 1 is 1 that if we treat it well, is still recoverable.

And then just do that by some electrodes looking at simple electrical patterns which don't look in any detail at all at what individual neurons are doing.

These rhythms are utterly absent from anything which goes on at Google.

So, well that's like comparing, okay, I'll tell you.

It's like you're comparing the greatest classical musician in the world to a child first learning to play.

The question I'm at, but they're still both playing the piano, I'm asking is there, will it ever go on at Google?

Because you're 1 of the seminal figures in both, launching both disciplines, both sides of the river.

I think it's going to go on generation after generation, the way it has, where what you might call the AI computer science community says, let's take the following.

This is our model of neurobiology at the moment.

Let's pretend it's good enough and do everything we can with it.

And it does interesting things, and after a while, it sort of grinds into the sand, and you say, ah, something else is needed for neurobiology, and some other grand thing comes in and enables you to go a lot further.

I think there could be generations of this evolution.

I don't know how many of them, and each 1 is going to get you further into what our brain does.

In some sense, pass the Turing test longer and for more broad aspects.

And how many of these are there are going to have to be before you say, I've made something, I've made a human, I don't know.

And going back to my brainwaves, as it were, from the AI point of view, they would say, ah, maybe these are an heavy phenomenon and not important at all.

The first car I had, a real wreck of a 1936 Dodge, go above 45 miles an hour and the wheels would shimmy.

But in biology, if it were useful to know when are you going more than 45 miles an hour, you just capture that and you wouldn't worry about where it came from.

And it's going to be a long time before that kind of thing, which can take place in large, complex networks of things, is actually used in the computation.

Look, how many transistors are there in your laptop these days?

And all the transistors are somewhat similar.

And most physical systems, with that many parts, all of which are similar, have collective properties.

Sound waves in air, earthquakes, what have you, have collective properties.

There are no collective properties used in artificial neural networks, in AI.

If biology uses them, it's gonna take us to more generations of things for people to actually dig in and see how they are used and what they mean.

We might have to return several times to neurobiology and try to make our transistors more messy.

At the same time, the simple ones will conquer big aspects.

And I think 1 of the most biggest surprises to me was How well learning systems, which are manifestly non-biological, how important they can be actually, and how important and how useful they can be in AI.


Speaker 0: What difference between biological neural networks and artificial neural networks is most captivating and profound to you? At the higher philosophical level, let's not get technical just yet.

Speaker 1: 1 of the things that very much intrigues me is the fact that neurons have all kinds of components, properties to them. In evolutionary biology, if you have some little quirk in how a molecule works, or how a cell works, and it can be made use of, evolution will sharpen it up and make it into a useful feature, rather than a glitch. And so you expect in neurobiology for evolution to have captured all kinds of possibilities of getting neurons, of how you get neurons to do things for you. And That aspect has been completely suppressed in artificial neural networks.

Speaker 0: Do the glitches become features in the biological neural network?

Speaker 1: They can. Look, Let me take 1 of the things that I used to do research on. If you take things which oscillate, they have rhythms which are sort of close to each other, under some circumstances these things will have a phase transition, and suddenly the rhythm will… everybody will fall into step. There was a marvelous physical example of that in the Millennium Bridge across the Thames River about… built about

Speaker 0: 2001.

Speaker 1: And pedestrians walking across, pedestrians don't walk synchronized, they don't walk in lockstep. But they all walk at about the same frequency. And the bridge could sway at that frequency, and the slight sway made pedestrians tend a little bit to lock into step. And after a while, the bridge was oscillating back and forth and the pedestrians were walking in step to it. And you could see it in the movies made out of the bridge.

And the engineers made a simple-minded mistake. They assumed when you walk, it's step, step, step, and it's back and forth motion. But when you walk, it's also right foot, left foot, side to side motion. And it's the side to side motion for which the bridge was strong enough, but it wasn't stiff enough. And as a result, you would feel the motion and you'd fall into step with it.

And people were very uncomfortable with it. They closed the bridge for 2 years while they built stiffening for it. Now, nerves, look, nerve cells produce action potentials. You have a bunch of cells which are loosely coupled together, producing action potentials of the same rate. There'll be some circumstances under which these things can lock together.

Other circumstances in which they won't. Well, if they fire together, you can be sure that the other cells are going to notice it. So, you can make a computational feature out of this in an evolving brain. Most artificial neural networks don't even have action potentials, let alone have the possibility for synchronizing them.

Speaker 0: And you mentioned the evolutionary process. So the evolutionary process that builds on top of biological systems leverages that, the weird mess of it somehow? So how do you make sense of that ability to leverage all the different kinds of complexities in the biological brain?

Speaker 1: Well, look, at the biological molecule level, you have a piece of DNA which encodes for a particular protein. You could duplicate that piece of DNA, and now 1 part of it can code for that protein, but the other 1 could itself change a little bit and thus start coding for a molecule which is slightly different. Now, if that molecule was just slightly different, Had a function which helped any old chemical reaction was as important to the cell. You would go ahead and let that try And evolution would slowly improve that function. And so you have the possibility of duplicating, and then having things drift apart.

1 of them retain the old function, the other 1 do something new for you. And there's evolutionary pressure to improve. Look, there is in computers too, but it's improvement has to do with closing some companies and opening some others. The evolutionary process looks a little different.

Speaker 0: Yeah, similar time scale, perhaps.

Speaker 1: Much shorter in time scale.

Speaker 0: Companies close, yeah, go bankrupt and are born, yeah, shorter, but not much shorter. Some companies last a century, But yeah, you're right. I mean, if you think of companies as a single organism that builds and you all know, yeah, it's a fascinating dual correspondence there between biological-

Speaker 1: And companies have difficulty having a new product competing with an old product. And when IBM built its first PC, you probably read the book, they made a little isolated internal unit to make the PC. And for the first time in IBM's history, they didn't insist that you build it out of IBM components. But they understood that they could get into this market, which is a very different thing, by completely changing their culture. And biology finds other markets in a more adaptive way.

Speaker 0: Yeah, it's better at it. It's better at that kind of integration. So maybe you've already said it, but what to you is the most beautiful aspect or mechanism of the human mind? Is it the adaptive, the ability to adapt as you've described, or is there some other little quirk that you particularly like?

Speaker 1: Adaptation is everything when you get down to it. But the difference, There are differences between adaptation, where your learning goes on only over generations, and over evolutionary time, where your learning goes on at the time scale of 1 individual who must learn from the environment during that individual's lifetime. And biology has both kinds of learning in it. And the thing which makes neurobiology hard is that it's a mathematical system, as it were, built on this other kind of evolutionary system.

Speaker 0: What do you mean by mathematical system? Where is the math in the biology?

Speaker 1: Well, when you talk to a computer scientist about neural networks, it's all math. The fact that biology actually came about from evolution, and the fact that biology is about a system which you can build in 3 dimensions. If you look at computer chips, computer chips are basically two-dimensional structures, maybe 2.1 dimensions, but they really have difficulty doing three-dimensional wiring. Biology is, neocortex is actually also sheet-like, and it sits on top of the white matter, which is about 10 times the volume of the gray matter and contains all what you might call the wires. But there's a huge, the effect of computer structure on what is easy and what is hard is events.

So... And biology does, it makes some things easy that are very difficult to understand how to do computationally. On the other hand, you can't do simple floating point arithmetic because it's awfully stupid.

Speaker 0: Yeah, and you're saying this kind of 3 dimensional complicated structure makes, it's still math. It's still doing math. The kind of math it's doing enables you to solve problems of a very different kind.

Speaker 1: That's right, that's right.

Speaker 0: So you mentioned 2 kinds of adaptation, the evolutionary adaptation and the adaptation or learning at the scale of a single human life. Which do you, which is particularly beautiful to you and interesting from a research and from just a human perspective? And which is more powerful?

Speaker 1: I find things most interesting that I begin to see how to get into the edges of them and tease them apart a little bit and see how they work. And since I can't see the evolutionary process going on, I'm in awe of it, but I find it just a black hole as far as trying to understand what to do. And so in a certain sense, I'm in awe of it, but I couldn't be interested in working on it.

Speaker 0: The human life's time scale is however thing you can tease apart and study.

Speaker 1: Yeah, you can do, there's developmental neurobiology which understands how the connections and how the structure evolves from a combination of what the genetics is like and the real, the fact that you're building a system in 3 dimensions.

Speaker 0: In just days and months, those early, early days of a human life are really interesting.

Speaker 1: They are and Of course, there are times of immense cell multiplication. There are also times of the greatest cell death in the brain is during infancy.

Speaker 0: It's turnover.

Speaker 1: So what is not effective, what is not wired well enough to use at the moment, throw it out.

Speaker 0: It's a mysterious process. From, let me ask, from what field do you think the biggest breakthroughs in understanding the mind will come in the next decades? Is it neuroscience, computer science, neurobiology, psychology, physics, maybe math, maybe literature?

Speaker 1: Well, of course, I see the world always through a lens of physics. I grew up in physics, and the way I pick problems is very characteristic of physics and of an intellectual background, which is not psychology, which is not chemistry and so on and so on.

Speaker 0: Both of your parents were physicists.

Speaker 1: Both of my parents were physicists and The real thing I got out of that was a feeling that the world is an understandable place. And if you do enough experiments and think about what they mean and structure things, or you can do the mathematics of the relevant to the experiments, you also have to be able to understand how things work.

Speaker 0: But that was a few years ago. Did you change your mind at all through many decades of trying to understand the mind, of studying in different kinds of ways, not even the mind, just biological systems. You still have hope that physics, that you can understand.

Speaker 1: There's a question of What do you mean by understand?

Speaker 0: Of course.

Speaker 1: When I taught freshman physics, I used to say, I wanted to get physics to understand the subject, to understand Newton's laws. I didn't want them simply to memorize a set of examples to which they knew the equations to write down to generate the answers. I had this nebulous idea of understanding. So that if you looked at a situation, you could say, oh, I expect the ball to make that trajectory, or I expect some intuitive notion of understanding. And I don't know how to express that very well.

I've never known how to express it well. And you run smack up against it When you look at these simple neural nets, feed forward neural nets, which do amazing things, and yet you know contain nothing of the essence of what I would have felt was understanding. Understanding is more than just an enormous lookup table.

Speaker 0: Let's linger on that. How sure you are of that? What if the table gets really big? So, I mean, ask another way. These feed-forward neural networks, do you think they'll ever understand?

Speaker 1: Could answer that in 2 ways. I think if you look at real systems, feedback is an essential aspect of how these real systems compute. On the other hand, if I have a mathematical system with feedback, I know I can unlayer this and do it. But I have an exponential expansion in the amount of stuff I have to build if I can solve the problem that way.

Speaker 0: So feedback is essential. So we can talk even about recurrent neural

Speaker 1: networks, so

Speaker 0: recurrence. But do you think all the pieces are there to achieve understanding through these simple mechanisms? Like, back to our original question, what is the fundamental, is there a fundamental difference between artificial neural networks and biological? Or is it just a bunch of surface stuff?

Speaker 1: Suppose you ask a neurosurgeon, when is somebody dead?

Speaker 0: Yeah.

Speaker 1: They'll probably go back to saying, well, I can look at the brain rhythms and tell you this is a brain which is never gonna function again, this 1 is, this other 1 is 1 that if we treat it well, is still recoverable. And then just do that by some electrodes looking at simple electrical patterns which don't look in any detail at all at what individual neurons are doing. These rhythms are utterly absent from anything which goes on at Google.

Speaker 0: Yeah, but the rhythms.

Speaker 1: But the rhythms what?

Speaker 0: So, well that's like comparing, okay, I'll tell you. It's like you're comparing the greatest classical musician in the world to a child first learning to play. The question I'm at, but they're still both playing the piano, I'm asking is there, will it ever go on at Google? Do you have a hope? Because you're 1 of the seminal figures in both, launching both disciplines, both sides of the river.

Speaker 1: I think it's going to go on generation after generation, the way it has, where what you might call the AI computer science community says, let's take the following. This is our model of neurobiology at the moment. Let's pretend it's good enough and do everything we can with it. And it does interesting things, and after a while, it sort of grinds into the sand, and you say, ah, something else is needed for neurobiology, and some other grand thing comes in and enables you to go a lot further. We'll go into the sand again.

I think there could be generations of this evolution. I don't know how many of them, and each 1 is going to get you further into what our brain does. In some sense, pass the Turing test longer and for more broad aspects. And how many of these are there are going to have to be before you say, I've made something, I've made a human, I don't know.

Speaker 0: But your sense is it might be a couple. In my

Speaker 1: sense it might be a couple more. More. And going back to my brainwaves, as it were, from the AI point of view, they would say, ah, maybe these are an heavy phenomenon and not important at all. The first car I had, a real wreck of a 1936 Dodge, go above 45 miles an hour and the wheels would shimmy.

Speaker 0: Yeah.

Speaker 1: Good speedometer that. Now, nobody designed the car that way. The car is malfunctioning to have that. But in biology, if it were useful to know when are you going more than 45 miles an hour, you just capture that and you wouldn't worry about where it came from. Yeah.

And it's going to be a long time before that kind of thing, which can take place in large, complex networks of things, is actually used in the computation. Look, how many transistors are there in your laptop these days?

Speaker 0: Actually, I don't know the number.

Speaker 1: It's… On the scale of 10 to the

Speaker 0: 10,

Speaker 1: I can't remember the number either. And all the transistors are somewhat similar. And most physical systems, with that many parts, all of which are similar, have collective properties.

Speaker 0: Yes.

Speaker 1: Sound waves in air, earthquakes, what have you, have collective properties. Weather. There are no collective properties used in artificial neural networks, in AI.

Speaker 0: Yeah, it's very-

Speaker 1: If biology uses them, it's gonna take us to more generations of things for people to actually dig in and see how they are used and what they mean.

Speaker 0: See, you're very right. We might have to return several times to neurobiology and try to make our transistors more messy.

Speaker 1: Yeah, Yeah. At the same time, the simple ones will conquer big aspects. And I think 1 of the most biggest surprises to me was How well learning systems, which are manifestly non-biological, how important they can be actually, and how important and how useful they can be in AI. You you

See all Lex Fridman transcripts on Youtube

Biological versus Artificial Neural Networks (John Hopfield) | AI Podcast Clips