The following is a conversation with Jeff Hawkins.

He's the founder of the Redwood Center for Theoretical Neuroscience in 2002, and NuMenta in 2005.

In his 2004 book titled On Intelligence, and in the research Before and After, he and his team have worked to reverse engineer the neurocortex and propose artificial intelligence architectures, approaches, and ideas that are inspired by the human brain.

These ideas include hierarchical temporal memory, HTM from 2004, and new work, the thousands brains theory of intelligence from 2017, 18 and 19.

Jeff's ideas have been an inspiration to many who have looked for progress beyond the current machine learning approaches, but they have also received criticism for lacking a body of empirical evidence supporting the models.

This is always a challenge when seeking more than small incremental steps forward in AI.

Jeff is a brilliant mind and many of the ideas he has developed and aggregated from neuroscience are worth understanding and thinking about.

There are limits to deep learning as it is currently defined.

Forward progress in AI is shrouded in mystery.

My hope is that conversations like this can help provide an inspiring spark for new ideas.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Jeff Hawkins.

Are you more interested in understanding the human brain or in creating artificial systems that have many of the same qualities but don't necessarily require that you actually understand the underpinning workings of our mind?

So there's a clear answer to that question.

My primary interest is understanding the human brain, No question about it.

But I also firmly believe that we will not be able to create fully intelligent machines until we understand how the human brain works.

So I don't see those as separate problems.

I think there's limits to what can be done with machine intelligence if you don't understand the principles by which the brain works.

And so I actually believe that studying the brain is actually the fastest way to get to machine intelligence.

And within that, let me ask the impossible question, how do you not define, but at least think about what it means to be intelligent?

So I didn't try to answer that question first.

We said, let's just talk about how the brain works and let's figure out how certain parts of the brain, mostly the neocortex, but some other parts too, the parts of the brain most associated with intelligence.

And let's discover the principles by how they work.

Because intelligence isn't just like some mechanism and it's not just some capabilities.

It's like, okay, we don't even know where to begin on this stuff.

And So now that we've made a lot of progress on this, after we've made a lot of progress on how the neocortex works, and we can talk about that, I now have a very good idea what's gonna be required to make intelligent machines.

I can tell you today, some of the things are gonna be necessary, I believe, to create intelligent machines.

We'll get to the neocortex and some of the theories of how the whole thing works.

And you're saying, as we understand more and more about the neocortex, about our own human mind, we'll be able to start to more specifically define what it means to be intelligent, it's not useful to really talk about that until.

Look, there's a long history of AI, as you know.

And there's been different approaches taken to it.

So, the good old fashioned AI, the expert systems, the current convolutional neural networks, they all have their utility.

But I would think almost everyone would agree that none of them are really intelligent in a sort of a deep way that humans are.

And so it's just the question of how do you get from where those systems were or are today to where a lot of people think we're going to go?

And there's a big, big gap there, a huge gap.

And I think the quickest way of bridging that gap is to figure out how the brain does that.

And then we can sit back and look and say, oh, which of these principles that the brain works on are necessary and which ones are not?

Clearly, we don't have to build this in, intelligent machines aren't gonna be built out of, you know, organic living cells.

But there's a lot of stuff that goes on in the brain that's gonna be necessary.

So let me ask maybe, before we get into the fun details, Let me ask maybe a depressing or difficult question.

Do you think it's possible that we will never be able to understand how our brain works?

That maybe there's aspects to the human mind, like we ourselves cannot introspectively get to the core, that there's a wall you eventually hit.

There's not been a single thing humans have ever put their minds to.

There's like, what's the difference between living matter and non-living matter?

Something special that you never understand.

So there's no historical evidence that suggests this is the case.

And I just never even consider that's a possibility.

I would also say, today, we understand so much about the neocortex.

We've made tremendous progress in the last few years that I no longer think of it as an open question.

The answers are very clear to me, and the pieces we don't know are clear to me, but the framework is all there, and it's like, oh, okay, we're gonna be able to do this.

Just takes time and effort, but there's no mystery, big mystery anymore.

So then let's get into it for people like myself who are not very well versed in the human brain, except my own.

Can you describe to me at the highest level what are the different parts of the human brain and then zooming in on the neocortex, the parts of the neocortex and so on, a quick overview.

The human brain, we can divide it roughly into 2 parts.

There's the old parts, lots of pieces, and then there's the new part.

It's new because it didn't exist before mammals.

And in humans, in primates, it's very large.

In the human brain, the neocortex occupies about 70 to

And the old parts of the brain are, there's lots of pieces there.

There's a spinal cord and there's the brain stem and the cerebellum and the different parts of the basal ganglia and so on.

In the old parts of the brain, you have the autonomic regulation, like breathing and heart rate.

So like walking and running are controlled by the old parts of the brain.

All the emotional centers of the brain are in the old part of the brain.

So when you feel anger or hungry, lust or things like that, those are all in the old parts of the brain.

And we associate with the neocortex all the things we think about as sort of high-level perception and cognitive functions, anything from seeing and hearing and touching things to language to mathematics and engineering and science and so on.

Those are all associated with the neocortex and they're certainly correlated.

Our abilities in those regards are correlated with the relative size of our neocortex compared to other mammals.

So that's like the rough division And you obviously can't understand the neocortex completely isolated, but you can understand a lot of it with just a few interfaces, so the old parts of the brain.

The other remarkable thing about the neocortex compared to the old parts of the brain is the neocortex is extremely uniform.

It's not visibly or anatomically or I always like to say it's like the size of a dinner napkin, about 2 and a half millimeters thick.

And it looks remarkably the same everywhere.

Everywhere you look in that 2 and a half millimeters, there's this detailed architecture, and it looks remarkably the same everywhere.

And that's across species, a mouse versus a cat, and a dog, and a human.

Where if you look at the old parts of the brain, there's lots of little pieces do specific things So it's like the old parts of a brain evolved Like this is the part that controls heart rate and this is the part that controls this and this is this kind of thing And that's this kind of thing and these evolve for eons a long long time and they have their specific functions and all sudden mammals come along and they got this thing called the neocortex and it got large by just replicating the same thing over and over and over again.

So all the evidence we have, and this is an idea that was first articulated in a very cogent and beautiful argument by a guy named Vernon Malkasal in 1978, I think it was, that the neocortex all works on the same principle.

So language, hearing, touch, vision, engineering, all these things are basically underlying or all built in the same computational substrate.

So at the low level, the building blocks all look similar.

We're talking about this very complex circuit that exists throughout the neocortex is remarkably similar.

It's like, yes, you see variations of it here and there, more of the cell, less and less, and so on.

But What Mouncaster argued was, he says, if you take a section of the neocortex, why is 1 a visual area and 1 is an auditory area?

And his answer was, it's because 1 is connected to eyes and 1 is connected to ears.

Literally, you mean just it's most closest in terms of number of connections to the sensor.

Literally, if you took the optic nerve and attached it to a different part of the neocortex, that part would become a visual region.

This experiment was actually done by Morganka-Sear in developing, I think it was lemurs, I can't remember what it was, some animal.

If you take a blind person, a person who's born blind at birth, they're born with a visual neocortex.

It may not get any input from the eyes because of some congenital defect or something, and that region does something else.

It's not like, oh, they're all built on neurons.

No, they're all built in this very complex circuit and somehow that circuit underlies everything.

And so this is the, it's called the common cortical algorithm, if you will.

Some scientists just find it hard to believe.

And they just say, I can't believe that's true.

But the evidence is overwhelming in this case.

And so a large part of what it means to figure out how the brain creates intelligence and what is intelligence in the brain is to understand what that circuit does.

If you can figure out what that circuit does, as amazing as it is, then you understand what all these other cognitive functions are.

So if you were to sort of put neurocortex outside of your book on intelligence, you look, If you wrote a giant tome, a textbook on the neocortex, and you look maybe a couple centuries from now, how much of what we know now would still be accurate 2 centuries from now?

So how close are we in terms of understanding it?

I have to speak from my own particular experience here.

There's actually 2 of us and there's a bunch of other people.

We study the inner cortex and we publish our results and so on.

So about 3 years ago, we had a real breakthrough in this field.

We've now published, I think, 3 papers on it.

And so I have a pretty good understanding of all the pieces and what we're missing, I would say that almost all the empirical data we've collected about the brain, which is enormous, if you don't know the neuroscience literature, it's just incredibly big.

And it's, for the most part, all correct.

It's facts and experimental results and measurements and all kinds of stuff.

But none of that has been really assimilated into a theoretical framework.

In the language of Thomas Kuhn, the historian, would be a sort of a pre-paradigm science.

Lots of data, but no way to fit it in together.

And for the most part, there aren't really good cogent theories about how to put it together.

It's not like we have 2 or 3 competing good theories, which ones are right and which ones are wrong.

It's like, nah, people just like scratching their heads, throwing things.

Some people have given up on trying to figure out what the whole thing does.

In fact, there's very, very few labs that we do that focus really on theory and all this unassimilated data and trying to explain it.

So it's really, I would say, pretty early days in terms of understanding the fundamental theories, forces of the way our mind works?

I would have said that's true 5 years ago.

So as I said, we had some really big breakthroughs on this recently and we started publishing papers on this.

I don't think it's, I'm an optimist and from where I sit today, most people would disagree with this, but from where I sit today, from what I know, it's not super early days anymore.

The way these things go is it's not a linear path, right?

You don't just start accumulating and get better and better and better.

No, you've got all this stuff you've collected, none of it makes sense, all these different things are just sort of around.

And then you're going to have some breaking points where all of a sudden, oh my God, now we got it right.

And I personally feel like we passed that little thing about a couple of years ago.

Well, that big thing a couple of years ago.

Time will tell if I'm right, but I feel very confident about it.

That's why I'm willing to stay on tape like this.

So let's, before those few years ago, let's take a step back to HTM, the hierarchical temporal memory theory, which you first proposed on intelligence and went through a few different generations.

Can you describe what it is, how it evolved through the 3 generations since you first put it on paper.

Yeah, so 1 of the things that neuroscientists just sort of missed for many, many years, and especially people who are thinking about theory, was the nature of time in the brain.

The information coming into the brain is constantly changing.

The patterns from my speech right now, if you're listening to it at normal speed, would be changing on your ears about every 10 milliseconds or so, you'd have a change.

When you look at the world, your eyes are moving constantly, 3 to 5 times a second, and the inputs completely.

If I were to touch something like a coffee cup, as I move my fingers, the input changes.

So this idea that the brain works on time-changing patterns is almost completely, or was almost completely missing from a lot of the basic theories like fears of vision and so on.

It's like, oh no, we're going to put this image in front of you and flash it and say, what is it?

Convolutional neural networks work that way today, right?

Vision is this sort of crazy time-based pattern that's going all over the place, and so is touch and so is hearing.

So the first part of hierarchical temporal memory was the temporal part.

It's to say, you won't understand the brain, nor will you understand intelligent machines unless you're dealing with time-based patterns.

The second thing was the memory component of it was, is to say that we aren't just processing input, we learn a model of the world.

The point of the brain, the part of the neocortex, it learns a model of the world.

We have to store things, our experiences, in a form that leads to a model of the world.

So we can move around the world, we can pick things up and do things, navigate, know how it's going on.

And many people just, they were thinking about like certain processes without memory at all.

And then finally, the hierarchical component was a reflection to that the neocortex, although it's this uniform sheet of cells, different parts of it project to other parts, which project to other parts.

And there is a sort of a rough hierarchy in terms of that.

So the hierarchical temporal memory is just saying, look, we should be thinking about the brain as time-based, you know, model memory-based and hierarchical processing.

And that was a placeholder for a bunch of components that we would then plug into that.

We still believe all those things I just said, but we now know so much more that I'm stopping to use the word hierarchical temporal memory because it's insufficient to capture the stuff we know.

So again, it's not incorrect, but it's, I now know more and I would rather describe it more accurately.

Yeah, so you're basically, we can think of HTM as emphasizing that there's 3 aspects of intelligence that are important to think about whatever the eventual theory converges to.

So in terms of time, How do you think of nature of time across different time scales?

So you mentioned things changing, sensory inputs changing every 10, 20 minutes.

What about every few minutes, every few months and years?

Well, if you think about a neuroscience problem, the brain problem, Neurons themselves can stay active for certain periods of time.

They're parts of the brain where they stay active for minutes.

So you could hold a certain perception or an activity for a certain period of time, but most of them don't last that long.

And so if you think about your thoughts are the activity neurons, if you're going to want to involve something that happened a long time ago, even just this morning, for example, the neurons haven't been active throughout that time.

So if I ask you, what did you have for breakfast today?

That is, you've built that into your model of the world now.

You remember that, and that memory is in the synapses, is basically in the formation of synapses.

And so, you're sliding into what, you know, you said different time scales.

There's time scales of which we are like understanding my language and moving about and seeing things rapidly over time.

That's the timescales of activities of neurons.

But if you wanna get longer timescales, then it's more memory.

And we have to invoke those memories to say, oh yes, well now I can remember what I had for breakfast because I stored that someplace.

I may forget it tomorrow, but I'd store it for now.

So does memory also need to have, so the hierarchical aspect of reality is not just about concepts, it's also about time?

It's like you really can't separate it out.

If I ask you, how does the brain learn a model of this coffee cup here?

I say, well, time is not an inherent property of the model I have of this cup, whether it's a visual model or tactile model.

I can sense it through time, but the model itself doesn't really have much time.

If I asked you, if I said, well, what is the model of my cell phone?

My brain has learned to model the cell phones.

If you have a smartphone like this, and I said, well, this has time aspects to it.

I have expectations when I turn it on, what's going to happen, what, what are, how long it's going to take to do certain things.

And so I have, and it's like melodies in the world, you know, melody has a sense of time.

So many things in the world move and act, and there's a sense of time related to them.

So it's sort of infused throughout the models of the world.

You build a model of the world, you're learning the structure of the objects in the world, and you're also learning how those things change through time.

Okay, so it really is just a fourth dimension that's infused deeply, and you have to make sure that your models of intelligence incorporate it.

So, like you mentioned, the state of neuroscience is deeply empirical.

A lot of data collection, it's, you know, that's where it is.

And then you're proposing a theory of intelligence, and which is really the next step, the really important step to take, but why is HTM, or what we'll talk about soon, the right theory?

Is it kind of closer to where string theory is in physics where there's mathematical components which show that, you know what, it seems that this, it fits together too well for it not to be true, which is where string theory is.

It's a mixture of all those things, although definitely where we are right now, it's definitely much more on the empirical side than let's say string theory.

The way this goes about, we're theorists, right?

So we look at all this data and we're trying to come up with some sort of model that explains it basically.

And there's, unlike string theory, there's vast more amounts of empirical data here that I think than most physicists deal with.

And so our challenge is to sort through that and figure out what kind of constructs would explain this.

And when we have an idea, you come up with a theory of some sort, you have lots of ways of testing it.

First of all, there are a hundred years of assimilated, unassimilated empirical data from neuroscience.

So we go back and read papers and we say, oh, did someone find this already?

We can predict X, Y, and Z, and maybe no one's even talked about it since 1972 or something, but we go back and find that and we say, oh, either it can support the theory or it can invalidate the theory.

We say, okay, we have to start over again.

So the way I view it, when we do our work, we look at all this empirical data, And it's what I call it is a set of constraints.

We're not interested in something that's biologically inspired.

We're trying to figure out how the actual brain works.

So every piece of empirical data is a constraint on a theory.

If you have the correct theory, it needs to explain every pin, right?

So we have this huge number of constraints on the problem, which initially makes it very, very difficult.

If you don't have any constraints, you can make up stuff all the day.

See all Lex Fridman transcripts on Youtube

Jeff Hawkins: Thousand Brains Theory of Intelligence | Lex Fridman Podcast #25