Jay McClelland is a cognitive scientist at Stanford. Please support this podcast by checking out our sponsors:
- Paperspace: https://gradient.run/lex to get $15 credit
- Skiff: https://skiff.org/lex to get early access
- Uprising Food: https://uprisingfood.com/lex to get $10 off 1st starter bundle
- Four Sigmatic: https://foursigmatic.com/lex and use code LexPod to get up to 60% off
- Onnit: https://lexfridman.com/onnit to get up to 10% off

EPISODE LINKS:
Jay's Website: https://stanford.edu/~jlmcc/

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
0:43 - Beauty in neural networks
5:02 - Darwin and evolution
10:47 - The origin of intelligence
17:29 - Explorations in cognition
23:33 - Learning representations by back-propagating errors
29:58 - Dave Rumelhart and cognitive modeling
43:01 - Connectionism
1:05:54 - Geoffrey Hinton
1:07:49 - Learning in a neural network
1:24:42 - Mathematics & reality
1:31:50 - Modeling intelligence
1:42:28 - Noam Chomsky and linguistic cognition
1:56:49 - Advice for young people
2:07:56 - Psychiatry and exploring the mind
2:20:35 - Legacy
2:26:24 - Meaning of life

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Jay McClelland, a cognitive scientist at Stanford and 1 of the seminal figures in the history of artificial intelligence and specifically neural networks.

Having written the parallel distributed processing book with David Rommelhart, who co-authored the back propagation paper with Jeff Hinton.

In their collaborations, they've paved the way for many of the ideas at the center of the neural network based machine learning revolution of the past 15 years.

To support this podcast, please check out our sponsors in the description.

This is the Lex Friedman Podcast, and here is my conversation with Jay McClelland.

You are 1 of the seminal figures in the history of neural networks at the intersection of cognitive psychology and computer science.

What to you has, over the decades, emerged as the most beautiful aspect about neural networks, both artificial and biological?

The fundamental thing I think about with neural networks is how they allow us to link

biology with the mysteries of thought and You know in the when I was first entering the field myself in the late 60s, early 70s, cognitive psychology had just become a field.

There was a book published in 67 called Cognitive Psychology.

And the author said that the study of the nervous system was only of peripheral interest.

It wasn't going to tell us anything about the mind.

I always felt, oh look, I'm a physical being.

From dust to dust, ashes to ashes, and somehow I emerged from that.

So there was a sense with cognitive psychology that in understanding the sort of neuronal structure of things, you're not going to be able to understand the mind.

And then your sense is if we study these neural networks, we might be able to get at least very close to understanding the fundamentals of the human mind.

I used to think, or I used to talk about the idea of awakening from the Cartesian dream.

So Descartes thought about these things, right?

He was walking in the gardens of Versailles 1 day and he stepped on a stone and a statue moved.

And he walked a little further, he stepped on another stone and another statue moved.

And he like, why did the statue move when I stepped on the stone?

And he went and talked to the gardeners and he found out that they had a hydraulic system that allowed the physical contact with the stone to cause water to flow in various directions, which caused water to flow into the statue and move the statue.

And he used this as the beginnings of a theory about how animals act.

And he had this notion that these little fibers that people had identified that weren't carrying the blood, were these little hydraulic tubes that if you touch something, there would be pressure and it would send a signal of pressure to the other parts of the system and that would cause action.

So he had a mechanistic theory of animal behavior.

And he thought that the human had this animal body, but that some divine something else had to have come down and been placed in him to give him the ability to think.

Right, so the physical world includes the body in action but it doesn't include thought according to Descartes, right?

And so the study of physiology at that time was the study of sensory systems and motor systems and things that you could directly measure when you stimulated neurons and stuff like that.

And the study of cognition was something that was tied in with abstract computer algorithms and things like that.

But when I was an undergraduate, I learned about the physiological mechanisms.

And so when I'm studying cognitive psychology as a first year PhD student, I'm saying, wait a minute, the whole thing is biological, right?

Isn't that magical though that from just the little bit of biology can emerge the full beauty of the human experience?

Well, obvious and not obvious at the same time.

And I think about Darwin in this context too, because Darwin knew very early on that none of the ideas that anybody had ever offered gave him a sense of understanding how evolution could have worked.

But he wanted to figure out how it could have worked.

And he spent a lot of time working on this idea and coming, you know, reading about things that gave him hints and thinking they were interesting but not knowing why, drawing more and more pictures of different birds that differ slightly from each other and so on, you know, and then then he figured it out.

But after he figured it out, he had nightmares about it.

He would dream about the complexity of the eye and the arguments that people had given about how ridiculous it was to imagine that that could have ever emerged from some sort of unguided process, that it hadn't been the product of design.

And So he didn't publish for a long time, in part because he was scared of his own ideas.

He didn't think they could possibly be true.

But then by the time the a century rolls around, we all, you know, we understand that evolution, or many people understand or believe that evolution produced, you know, the entire range of animals that there are.

And, you know, Descartes' idea starts to seem a little wonky after a while, right?

There's the apes and the chimpanzees and the bonobos and they're pretty smart in some ways.

So what, oh, somebody comes, oh, there's a certain part of the brain that's still different.

They don't, there's no hippocampus in the monkey brain.

Huxley had to do a surgery in front of many, many people in the late 19th century to show to them there's actually a hippocampus in the chimpanzee's brain.

So their continuity of the species is another element that contributes to this idea that we are ourselves a total product of nature.

And that to me is the magic and the mystery how nature could actually

give rise to organisms that have the capabilities that we have.

So it's interesting because even the idea of evolution is hard for me to keep all together in my mind.

So because we think of a human time scale, it's hard to imagine, the development of the human eye would give me nightmares too.

Because you have to think across many, many, many generations.

And it's very tempting to think about kind of a growth of a complicated object.

And it's like, how is it possible for that such, such a thing to be built?

Because also, me from a robotics engineering perspective, it's very hard to build these systems.

How can, through an undirected process, can a complex thing be designed?

And a slightly different career path that would have been equally interesting to me would have been to actually study the process of embryological development flowing on into brain development and the exquisite sort of laying down of pathways and so on that occurs in the brain.

And I know the slightest bit about that is not my field, but there are fascinating aspects to this process that eventually result in the complexity of various brains.

At least 1 thing we're, in the field I think people have felt for a long time, in the study of vision, the continuity between humans and non-human animals has been second nature for a lot longer.

I had this conversation with somebody who's a vision scientist and you're saying, oh, we don't have any problem with this.

You know, the monkey's visual system and the human visual system, extremely similar up to certain levels, of course, they diverge after a while.

But the first, the visual pathway from the eye to the brain and the first few layers of cortex, or cortical areas, I guess 1 would say, are extremely similar.

Yeah, so on the cognition side is where the leap seems to happen with humans.

And that's a really interesting question when thinking about alien life or if there's other intelligent alien civilizations out there, is how special is this leap?

So 1 special thing seems to be the origin of life itself.

However you define that, there's a gray area.

And the other leap, this is very biased perspective of a human, is the origin of intelligence.

And again, from an engineering perspective, it's a difficult question to ask, an important 1, is how difficult does that leap?

Did aliens bring down a monolith and some apes had to touch a monolith

Exactly, but it just seems 1 heck of a leap to get to this level of intelligence.

Yeah, and so Chomsky argued that some genetic fluke occurred 100,000 years ago, And just happened that some human, some hominin predecessor of current humans had this 1 genetic tweak that resulted in language.

And language then provided this special thing that separates us from all other animals.

I think there's a lot of truth to the value and importance of language, but I think it comes along with the evolution of a lot of other related things related to sociality and mutual engagement with others and establishment of, I don't know, rich mechanisms for organizing and understanding of the world, which language then plugs into.

Right, so language is a tool that allows you to do this kind of collective intelligence.

And whatever is at the core of the thing that allows for this collective intelligence is the main thing.

And it's interesting to think about that 1 fluke, 1 mutation could lead to the first crack opening of the door to human intelligence.

Like evolution just kind of opens the door a little bit and then time and selection takes care of the rest.

You know, There's so many fascinating aspects to these kinds of things.

So we think of evolution as continuous, right?

We think, oh, yes, okay, over 500 million years, there could have been this relatively continuous changes.

But that's not what anthropologists, evolutionary biologists found from the fossil record.

They found hundreds of millions of years of stasis.

Well, suddenly on that scale is a million years or something, or even 10 million years.

But the concept of punctuated equilibrium was a very important concept in evolutionary biology.

And that also feels somehow right about the stages of our mental abilities.

We seem to have a certain kind of mindset at a certain age.

And then at another age, we like look at that four-year-old and say, oh my God, how could they have thought that way?

Piaget was known for this stage theory of child development.

You look at it closely and suddenly those stages are so discrete, the transitions, but the difference between the four-year-old and the seven-year-old is profound.

And that's another thing that's always interested me is how we, something happens over the course of several years of experience where at some point we reach the point where something like an insight or a transition or a new stage of development occurs.

And these kinds of things can be understood in complex systems research.

And so evolutionary biology, developmental biology, cognitive development are all things that have been approached in this kind

Yeah, just like you said, I find both fascinating, those early years of human life, but also the early like minutes, days of, from the embryonic development to like how, from embryos you get like the brain, that development, again, from an engineer perspective, it's fascinating.

So it's not, so the early, when you deploy the brain to the human world and it gets to explore that world and learn, that's fascinating.

But just like the assembly of the mechanism that is capable of learning.

The stuff they're doing with like brain organoids where you can build many brains and study that self-assembly of a mechanism from like the DNA material.

You have literally like biological programs that just generate a system, this mushy thing that's able to be robust and learn in a very unpredictable world and learn seemingly arbitrary things or like a very large number of things that enable survival.

Yeah, Ultimately, that is a very important part of the whole process of understanding this sort of emergence of mind from brain kind of thing.

And the whole thing seems to be pretty continuous.

So Let me step back to neural networks for another brief minute.

You wrote parallel distributed processing books that explored ideas of neural networks in the 1980s, together with a few folks.

But the books you wrote with David Rommelhart, who is the first author on the back propagation paper, which you have hinted.

So these are just some figures at the time that were thinking about these big ideas.

What are some memorable moments of discovery and beautiful ideas from those early days.

I'm gonna start sort of with my own process in the mid-70s and then into the late 70s when I met Jeff Henson and he came to San Diego and we were all together.

In my time in graduate schools, I've already described to you, I had this sort of feeling of, okay, I'm really interested in human cognition, but this disembodied sort of way of thinking about it that I'm getting from the current mode of thought about it isn't working fully for me.

And when I got my assistant professorship, I went to UCSD and that was in

Dave Rommelhart had written a book together with another man named Don Norman, and the book was called Explorations in Cognition.

And it was a series of chapters exploring interesting questions about cognition, but in a completely sort of abstract, non-biological kind of way.

I'm coming to this community where people can get together and feel like they've collectively exploring ideas.

And it was a book that had a lot of, I don't know, lightness to it.

And Don Norman, who was the more senior figure to Rumelhart at that time, who led that project, always created this spirit of playful exploration of ideas.

But I was also still trying to get from the neurons to the cognition.

And I realized at 1 point, I got this opportunity to go to a conference where I heard a talk by a man named James Anderson, who was an engineer, but by then a professor in a psychology department who had used linear algebra to create neural network models of perception and categorization and memory.

And I, just blew me out of the water that 1 could create a model that was simulating neurons, not just engaged in a stepwise algorithmic process that was construed abstractly, but it was simulating remembering and recalling and recognizing the prior occurrence of a stimulus or something like that.

For me, this was a bridge between the mind and the brain.

I remember I was walking across campus 1 day in 1977, and I almost felt like St.

I said to myself, you know, if I think about the mind in terms of a neural network, it will help me answer the questions about the mind that I'm trying to answer.

So I think that a lot of people were becoming excited about that.

And 1 of those people was Jim Anderson, who I had mentioned.

Another 1 was Steve Grossberg, who had been writing about neural networks since the 60s.

And his PhD dissertation showed up in an applicant pool to a postdoctoral training program that Dave and Don, the 2 men I mentioned before, Romelhart and Norman, were administering and Romelhart got really excited about Hinton's PhD dissertation.

And so Hinton was 1 of the first people who came and joined this group of postdoctoral scholars that was funded by this wonderful grant that they got.

Another 1 who is also well-known in neural network circles is Paul Smolensky.

Anyway, Jeff and Jim Anderson organized a conference at UCSD where we were, and it was called Parallel Models of Associative Memory and it brought all the people together who had been thinking about these kinds of ideas in 1979 or 1980.

And this began to kind of really resonate with some of Rommelhart's own thinking, some of his reasons for wanting something other than the kinds of computation he'd been doing so far.

So let me talk about Rommelhart now for a minute, okay, with that context.

Well, let me also just pause, because he said so many interesting things before we go to Rommel-Hart.

So first of all, for people who are not familiar, neural networks are at the core of the machine learning, deep learning revolution of today.

Jeffrey Hidden, that we mentioned, is 1 of the figures that were important in the history like yourself in the development of these neural networks, artificial neural networks that are then used for the machine learning application.

Like I mentioned, the back propagation paper is 1 of the optimization mechanisms by which these networks can learn.

And the word parallel is really interesting.

So it's almost like synonymous from a computational perspective how you thought at the time about neural networks, that it's parallel computation.

Well, yeah, the word parallel in this comes from the idea that each neuron is an independent computational unit, right?

It gathers data from other neurons, it integrates it in a certain way, and then it produces a result.

And it's a very simple little computational unit, But it's autonomous in the sense that it does its thing, right?

It's in a biological medium where it's getting nutrients and various chemicals from that medium.

But you can think of it as almost like a little computer in and of itself.

The idea is that each, our brains have, oh, look, a hundred or hundreds, almost a billion of these little neurons,

And they're all capable of doing their work at the same time, so it's like, instead of just a single central processor that's engaged in chug, chug, 1 step after another.

We have a billion of these little computational units working at the same time.

So at the time, I don't know, maybe you can comment, it seems to me, even still to me, quite a revolutionary way to think about computation relative to the development of theoretical computer science alongside of that, where it's very much like sequential computer.

You're analyzing algorithms that are running on a single computer.

You're saying, wait a minute, why don't we take a really dumb, very simple computer and just have a lot of them interconnected together and they're all operating in their own little world and they're communicating with each other and thinking of computation that way.

And from that kind of computation, trying to understand how things like certain characteristics of the human mind can emerge.

That's quite a revolutionary way of thinking, I would say.

Not sort of knowing how we kind of get all the way there, I think.

And this very much remains at the core of the questions that everybody's asking about the capabilities of deep learning and all these kinds of things.

But if I could just play this out a little bit, a convolutional neural network or a CNN, which many people may have heard of, is a set of, you could think of it biologically as a set of collections of neurons.

Each 1 had, each collection has maybe 10,000 neurons in it but there's many layers, right?

Some of these things are hundreds or even a thousand layers deep, but others are closer to the biological brain and maybe they're like 20 layers deep or something like that.

So we have, within each layer, we have thousands of neurons or tens of thousands maybe.

Well, in the brain, we probably have millions in each layer, but we're getting sort of similar in a certain way, right?

And then we think, okay, at the bottom level, there's an array of things that are like the photoreceptors.

In the eye, they respond to the amount of light of a certain wavelength at a certain location on the pixel array.

And then there's several further stages going up, layers of these neuron-like units.

And you go from that raw input, array of pixels, to the classification, you've actually built a system that could do the same kind of thing that you and I do when we open our eyes and we look around and we see there's a cup, there's a cell phone, there's a water bottle, and these systems are doing that now, right?

So they are, in terms of the parallel idea that we were talking about before, They are doing this massively parallel computation in the sense that each of the neurons in each of those layers is thought of as computing its little bit of something about the input simultaneously with all the other ones in the same layer.

We get to the point of abstracting that away and thinking, oh, it's just 1 whole vector that's being computed, 1 activation pattern that's computed in a single step, and that abstraction is useful, but it's still that parallel and distributed processing.

Each 1 of these guys is just contributing a tiny bit to that whole thing.

And that's the excitement that you felt that from these simple things, you can emerge when you add these level of abstractions on it.

You can start getting all the beautiful things that we think about as cognition.

And so, okay, so you have this conference, I forgot the name already, but it's Parallel and Something Associated with Memory and so on.

Very exciting, technical and exciting title.

And you started talking about Dave Romahart.

So who is this person that was so, You've spoken very highly of him.

Can you tell me about him, his ideas, his mind, who he was as a human being, as a scientist?

So Dave came from a little tiny town in Western South Dakota.

And his mother was the librarian and his father was the editor of the newspaper.

And I know 1 of his brothers pretty well.

They grew up, there were 4 brothers, and they grew up together, and their father encouraged them to compete with each other a lot.

They competed in sports, And they competed in mind games.

I don't know, things like Sudoku and chess and various things like that.

He went at a younger age than most people do to college at the University of South Dakota and majored in mathematics.

And I don't know how he got interested in psychology, but he applied to the mathematical psychology program at Stanford and was accepted as a PhD student to study mathematical psychology at Stanford.

So mathematical psychology is the use of mathematics to model mental processes.

So something that I think these days might be called cognitive modeling, that whole space.

Yeah, it's mathematical in the sense that you say if this is true and that is true, then I can derive that this should follow.

And so you say, these are my stipulations about the fundamental principles, and this is my prediction about behavior, And it's all done with equations.

It's not done with a computer simulation.

So you solve the equation, and that tells you what the probability that the subject will be correct on the seventh trial of the experiment is or something like that.

So it's a use of mathematics to descriptively characterize aspects of behavior.

And Stanford at that time was the place where there were several really, really strong mathematical thinkers who were also connected with 3 or 4 others around the country, who brought a lot of really exciting ideas onto the table.

And it was a very, very prestigious part of the field of psychology at that time.

So Rumelhart comes into this, he was a very strong student within that program, and he got this job at this brand new university in San Diego in 1967, where he's 1 of the first assistant professors in the Department of Psychology at UCSD.

So I got there in 74, 7 years later, and Runghart at that time was still doing mathematical modeling, But he had gotten interested in cognition.

And understanding, I think, remains… what does it mean to understand anyway?

It's an interesting sort of curious, like how would we know if we really understood something?

But he was interested in building machines that would hear a couple of sentences and have an insight about what was going on.

So for example, 1 of his favorite things at that time was, Margie was sitting on the front step when she heard the familiar jingle of the Good Humor Man.

She remembered her birthday money and ran into the house.

Well, there's a couple of ideas you could have, but the most natural 1 is that the good humor man brings ice cream.

She knows she needs money to buy ice cream, so she's going to run into the house and get her money so she can buy herself an ice cream.

It's a huge amount of inference that has to happen to get those things to link up with each other.

He was interested in how the hell that could happen.

He was trying to build good old-fashioned AI style models of representation of language and content of things like has money.

So like formal logic and knowledge bases, like that kind of stuff.

So he was integrating that with his thinking about cognition.

The mechanisms cognition, how can they mechanistically be applied to build these knowledge, like to actually build something that looks like a web of knowledge and thereby from there emerges something like understanding.

grappling, This was something that they grappled with at the end of that book that I was describing, Explorations in Cognition.

See all Lex Fridman transcripts on Youtube

Jay McClelland: Neural Networks and the Emergence of Cognition | Lex Fridman Podcast #222