François Chollet is an AI researcher at Google and creator of Keras. Support this podcast by supporting our sponsors (and get discount):
- Babbel: https://babbel.com and use code LEX
- MasterClass: https://masterclass.com/lex
- Cash App: download app & use code "LexPodcast"

EPISODE LINKS:
Francois's Twitter: https://twitter.com/fchollet
Francois's Website: https://fchollet.com/
On the Measure of Intelligence (paper): https://arxiv.org/abs/1911.01547

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
5:04 - Early influence
6:23 - Language
12:50 - Thinking with mind maps
23:42 - Definition of intelligence
42:24 - GPT-3
53:07 - Semantic web
57:22 - Autonomous driving
1:09:30 - Tests of intelligence
1:13:59 - Tests of human intelligence
1:27:18 - IQ tests
1:35:59 - ARC Challenge
1:59:11 - Generalization
2:09:50 - Turing Test
2:20:44 - Hutter prize
2:27:44 - Meaning of life

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Francois Chollet, his second time on the podcast.

He's both a world-class engineer and a philosopher in the realm of deep learning and artificial intelligence.

This time, we talk a lot about his paper titled On the Measure of Intelligence that discusses how we might define and measure general intelligence in our computing machinery.

Quick summary of the sponsors, Babbel, Masterclass, and Cash App.

Click the sponsor links in the description to get a discount and to support this podcast.

As a side note, let me say that the serious, rigorous, scientific study of artificial general intelligence is a rare thing.

The mainstream machine learning community works on very narrow AI with very narrow benchmarks.

This is very good for incremental and sometimes big incremental progress.

On the other hand, the outside the mainstream renegade, you could say, AGI community works on approaches that verge on the philosophical and even the literary without big public benchmarks.

Walking the line between the 2 worlds is a rare breed, but it doesn't have to be.

I ran the AGI series at MIT as an attempt to inspire more people to walk this line.

Deep mind and open AI for a time, and still on occasion walk this line.

It's a beautiful dream to work towards and to make real 1 day.

If you enjoy this thing, subscribe on YouTube, review it with 5 stars on Apple Podcasts, follow us on Spotify, support on Patreon, or connect with me on Twitter at Lex Friedman.

As usual, I'll do a few minutes of ads now and no ads in the middle.

I try to make these interesting, but I give you timestamps so you can skip.

But still, please do check out the sponsors by clicking the links in the description.

It's the best way to support this podcast.

This show is sponsored by Babbel, an app and website that gets you speaking in a new language within weeks.

Go to babbel.com and use code Lex to get 3 months free.

They offer 14 languages, including Spanish, French, Italian, German, and yes, Russian.

Daily lessons are 10 to 15 minutes, super easy, effective, designed by over 100 language experts.

Let me read a few lines from the Russian poem Noch, ulitsa, fanary, apteka by Alexander Blok that you'll start to understand if you sign up to Babbel.

Noch, ulitsa, fanar, apteka, besmyslenny I tusklyy svet.

Now, I say that you'll start to understand this poem because Russian starts with a language and ends with vodka.

Now, the latter part is definitely not endorsed or provided by Babbel.

It will probably lose me this sponsorship, although it hasn't yet.

But once you graduate with Babbel, you can enroll in my advanced course of late night Russian conversation over vodka.

So get started by visiting babbel.com and use code Lex to get 3 months free.

This show is also sponsored by Masterclass.

Sign up at masterclass.com slash lux to get a discount and to support this podcast.

When I first heard about Masterclass, I thought it was too good to be true.

For $180 a year, you get an all-access pass to watch courses from, to list some of my favorites.

Chris Hadfield on space exploration, hope to have him in this podcast 1 day.

Neil deGrasse Tyson on scientific thinking and communication, Neil too.

Will Wright, creator of SimCity and Sims and Game Design, Carlos Santana on guitar, Gary Kasparov on chess, Daniel Negrano on poker, and many more.

Chris Hadfield explaining how rockets work and the experience of being launched into space alone is worth the money.

By the way, you can watch it on basically any device Once again sign up at masterclass.com slash Lex to get a discount and to support this podcast This show finally is presented by cash app the number 1 finance app in the App Store.

Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1.

Since Cash App allows you to send and receive money digitally, let me mention a surprising fact related to physical money.

Of all the currency in the world, roughly 8% of it is actually physical money.

The other 92% of the money only exists digitally.

So again, if you get Cash App from the App Store or Google Play and use code LEXPODCAST, you get 10 bucks and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world.

And now here's my conversation with Francois Chollet.

What philosophers, thinkers, or ideas had a big impact on you growing

So 1 author that had a big impact on me when I read his books as a teenager with Jean Piaget, who is a Swiss psychologist, is considered to be the father of developmental psychology.

And he has a large body of work about basically how intelligence develops in children.

And so it's very old work, like most of it is from the 1930s, 1940s.

It's actually superseded by many newer developments in developmental psychology.

But to me it was very interesting, very striking and actually shaped the early ways in which I started to think about the mind and the development of intelligence as a teenager.

His actual ideas or the way he thought about it or just the fact that you could think about the developing mind at all?

Jean-Pierre is the author that reintroduced me to the notion that intelligence and the mind is something that you construct throughout your life and that children construct it in stages.

And I thought that was a very interesting idea, which is, you know, of course, very relevant to AI, to building artificial minds.

Another book that I read around the same time that had a big impact on me, and there was actually a little bit of overlap with Jean-Pierre as well, and I read it around the same time, is Jeff Hawkins on intelligence,

And he has this vision of the mind as a multi-scale hierarchy of temporal prediction modules.

And these ideas really resonated with me, like the notion of modular hierarchy, of, you know, potentially of compression functions or prediction functions.

I thought it was really, really interesting.

And it really shaped the way I started thinking about how to build minds.

Also, he's a neuroscientist, So he was thinking actual, he was basically talking about how our mind works.

Yeah, the notion that cognition is prediction was an idea that was kind of new to me at the time and that I really loved at the time.

And yeah, and the notion that there are multiple scales of processing in the brain.

These ideas of hierarchies in AI have been around for a long time, even before on intelligence, I mean, they've been around since the 1980s.

And yeah, that was before deep learning, but of course I think these ideas really found their practical implementation in deep learning.

I think he was talking about knowledge representation.

1 way you can think of neural networks as a kind of memory, you're memorizing things, but it doesn't seem to be the kind of memory that's in our brains, or it doesn't have the same rich complexity, long-term nature that's in our brains?

of a sparse access memory so that you can actually retrieve very precisely like bits of your experience.

The retrieval aspect, you can like introspect, you can ask yourself questions,

Yes, you can program your own memory and language is actually the tool you use to do that.

I think language is a kind of operating system for the mind and use language.

Well, 1 of the uses of language is as a query that you run over your own memory, use words as keys to retrieve specific experiences, specific concepts, specific thoughts, like language is a way you store thoughts, not just in writing in the, in the physical world, but also in your own mind.

Like imagine if you didn't have language, then you would have to, you would not really have a self internally triggered way of retrieving past thoughts.

You would have to rely on external experiences.

For instance, you see a specific site, you smell a specific smell, and that brings up memories, but you would not really have a way to deliberately, deliberately access these memories without language.

Well, the interesting thing you mentioned is you can also program the memory.

You can change it probably with language.

Well, let me ask you a Chomsky question, which is like, First of all, do you think language is like fundamental, like there's turtles, what's at the bottom of the turtles?

They don't go, it can't be turtles all the way down.

Is language at the bottom of cognition of everything?

Is like language the fundamental aspect of like what it means to be a thinking thing?

I think language is a layer on top of cognition.

So it is fundamental to cognition in the sense that to use a computing metaphor, I see language as the operating system of the brain, of the human mind.

And the operating system, you know, is a layer on top of the computer.

The computer exists before the operating system, but the operating system is how you make it truly useful.

And the operating system is most likely Windows, not Linux.

And it's pretty difficult to inspect it, introspect it.

Like we use actually sort of human interpretable language, But is there something like a deeper, that's closer to like, like logical type of statements?

Like, yeah, what is the nature of language?

Because there's something deeper than like the syntactic rules we construct?

Is there something that doesn't require utterances or writing or so on?

Are you asking about the possibility that there could exist languages for thinking that are not made of words.

And language is almost like the outermost, the uppermost layer.

But Before we think in words, I think we think in terms of emotion in space and we think in terms of physical actions.

And I think babies in particular, probably expresses thoughts in terms of the actions that they've seen of that, or that they can perform.

And in terms of the, in terms of motions of objects in their environment, before they start thinking in terms of words.

It's amazing to think about that as the building blocks of language.

So like the kind of actions and ways the baby see the world as like more fundamental than the beautiful Shakespearean language you construct on top of it.

And we probably don't have any idea what that looks like, right?

Like what, because it's important for them trying to engineer it into AI systems.

I think visual analogies and motion is a fundamental building block of the mind.

And you, you actually see it reflected in language.

Like language is full of special metaphors.

And when you think about things, I consider myself very much as a visual thinker.

You, you often express your thoughts by using things like visualizing concepts in a 2D space or like you solve problems by imagining yourself navigating a concept space.

I don't know if you have this sort of experience.

So like, so I certainly think about, I certainly visualize mathematical concepts, but you mean like in concept space?

Visually, you're embedding ideas into a three-dimensional space you can explore with your mind, essentially?

You're showing me more like 2D, but yeah.

I always have to, before I jump from concept to concept, I have to put it back down.

I can only travel on 2D paper, not inside my mind.

But even if you're writing like a paper, for instance, don't you have like a spatial representation of your paper?

Like you visualize where ideas lie topologically in relationship to other ideas, kind of like a subway map of the ideas in your paper.

I mean, there is, in papers, I don't know about you, but it feels like there's a destination.

There's a key idea that you want to arrive at and a lot of it is in the fog and you're trying to kind of, it's almost like, what's that called when you do a path planning search from both directions, from the start and from the end?

And then you find, you do like shortest path, but like, you know, in game playing, you do this with like A star from both sides.

Yeah, so you kind of do, at least for me, I think like, first of all, just exploring from the start, from like first principles, what do I know?

What can I start proving from that, right?

And then from the destination, if I use their backtracking, like if I want to show some kind of sets of ideas, what would it take to show them?

But yeah, I don't think I'm doing all that in my mind though.

Do you use mind maps to organize your ideas?

Let's get into this, because I've been so jealous of people.

I've been jealous of people that seem to, like, they get like this fire of passion in their eyes because everything starts making sense.

It's like Tom Cruise in the movie was like moving stuff around.

Some of the most brilliant people I know use mind maps.

Can you explain what the hell a mind map is?

I guess a mind map is a way to make kind of like the mess inside your mind to just put it on paper so that you gain more control over it.

And as kind of like a consequence for organizing things on paper, it started being more organized inside your own mind.

Like, what's the first thing you write on paper?

I mean, typically you draw a mind map to organize the way you think about the topics.

So you would start by writing down like the, the key concept about that topic, like you would write intelligence or something, and then you would start adding, associative connections.

Like, what do you think about when you think about intelligence?

What do you think are the key elements of intelligence?

So maybe you would have language, for instance, and you'd have motion.

And so you would start drawing notes with these things.

And then you would see, what do you think about when you think about motion?

And so on, and you would go like that, like a tree.

It's a tree or a tree mostly, or is it a graph too?

And it's not limited to just, you know, writing down words.

You can also draw things and it's not, it's not supposed to be purely hierarchical.

Like you can, the point is that you can start, once you start writing it down, you can start reorganizing it so that it makes more sense so that it's connected in a more effective way.

See, but I'm so OCD that you just mentioned intelligence and the language in motion.

I would start becoming paranoid that the categorization is imperfect.

That I would become paralyzed with the mind map.

Even though you're just doing associative kind of connections, there's an implied hierarchy that's emerging, and I will start becoming paranoid that it's not the proper hierarchy.

So you're not just, 1 way to see mind maps is you're putting thoughts on paper, it's like a stream of consciousness, but then you can also start getting paranoid, well, is this the right hierarchy?

Sure, but it's a mind map, it's your mind map.

You're free to draw any connection you want.

And you can just make a different mind map if you think the central node is not the right node.

Yeah, I suppose there's a fear of being wrong.

If you want to organize your ideas by writing down what you think, which I think is, is, is very effective.

Like how do you know what you think about something if you, if you don't write it down, right?

If you do that, the thing is that it imposes a much more, syntactic structure over your ideas, which is not required with a mind map.

So mind map is kind of like a lower level, more freehand way of organizing your thoughts.

And once you've drawn it, then you can start actually voicing your thoughts in terms of, you know, paragraphs.

And sometimes- It's a two-dimensional aspect of layout too, right?

And it's a kind of flower, I guess, you start.

There's usually, you want to start with a central concept?

Typically, it ends up more like a subway map.

So it ends up more like a graph, a topological graph.

subway map, there are some nodes that are more connected than others.

And there are some nodes that are more important than others.

So there are destinations, but it's not gonna be purely like a tree, for instance.

It's fascinating to think that if there's something to that about our, about the way our mind thinks, by the way, I just kind of remembered obvious thing that I have probably thousands of documents in Google Doc at this point that are bullet point lists, which is you can probably map a mind map to a bullet point list.

It's the same, It's a, no, it's not, it's a tree.

So I create trees, but also they don't have the visual element.

Like I guess I'm comfortable with the structure.

It feels like, the narrowness, the constraints feel more comforting.

If you have thousands of documents with your own thoughts in Google Docs, why don't you write some kind of search engine, like maybe a mind map, a piece of software, a mind mapping software, where you write down a concept and then it gives you sentences or paragraphs from your SARS and Google Docs document that match this concept.

The problem is it's so deeply, unlike mind maps, It's so deeply rooted in natural language.

So it's not semantically searchable, I would say, because the categories are very, you kind of mentioned intelligence, language, and motion.

It feels like the mind map forces you to be semantically clear and specific.

The bullet points list I have are sparse, desperate thoughts that poetically represent a category like motion as opposed to saying motion.

So unfortunately, that's the same problem with the internet.

That's why the idea of semantic web is difficult to get.

Most language on the internet is a giant mess of natural language that's hard to interpret.

Which, so do you think there's something to mind maps as you actually originally brought it up as we were talking about kind of cognition and language.

Do you think there's something to mind maps about how our brain actually deals, like think reasons about things?

I think it's reasonable to assume that there is some level of topological processing in the brain, that the brain is very associative in nature.

And I also believe that a topological space is a better medium to encode thoughts than a geometric space.

What's the difference in a topological and a geometric space?

Well, if you're talking about topologies, then points are either connected or not.

So the topology is more like a subway map.

And geometry is when you're interested in the distance between things.

And in subway maps, you don't really have the concept of distance.

You only have the concept of whether there is a train going from station A to station B.

And what we do in deep learning is that we're actually dealing with geometric spaces.

We are dealing with concept vectors, word vectors that have a distance between the distance in terms of that product.

We are not, we are not really building topological models.

Like distance is a fundamental importance in deep learning.

I mean, it's, it's the continuous aspect of it.

Yes, because everything is a vector and everything has to be a vector because everything has to be differentiable.

If your space is discrete, it's no longer differentiable.

You cannot do deep learning in it anymore.

Well, you could, but you can only do it by embedding it in a bigger continuous space.

So if you do topology in the context of deep learning, you have to do it by embedding your topology in a geometry,

Let's get into your paper on the measure of intelligence.

You could travel, you could actually go outside and see friends.

I think there's some non-zero probability there'll be a textbook 1 day, like 200 years from now, on artificial intelligence, or it'll be called just intelligence because humans will already be gone.

This is 1 of the early biological systems would consider the nature of intelligence and there'll be like a definition of how they thought about intelligence.

Which is 1 of the things you do in your paper on measuring intelligence is to ask like, well, what is intelligence and how to test for intelligence and so on.

So is there a spiffy quote about what is intelligence?

See all Lex Fridman transcripts on Youtube

François Chollet: Measures of Intelligence | Lex Fridman Podcast #120