As I explain in the introduction to the full conversation with Noam, due to an unfortunate rare mishap, this conversation is audio-only (and so is the clip). Hope you still enjoy it and find it interesting. Full episode with Noam Chomsky (Nov 2019): https://www.youtube.com/watch?v=cMscNuSUy0I
(more links below)

Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman



Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Noam Chomsky is one of the greatest minds of our time and is one of the most cited scholars in history. He is a linguist, philosopher, cognitive scientist, historian, social critic, and political activist. He has spent over 60 years at MIT and recently also joined the University of Arizona.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

Let me ask you about a field of machine learning, deep learning.

There's been a lot of progress in neural networks based neural network based machine learning in the recent decade.

Of course, neural network research goes back many decades.

What do you think are the limits of deep learning, of neural network based machine learning?

Well, to give a real answer to that, you'd have to understand the exact processes that are taking place.

So it's pretty hard to prove a theorem about what can be done and what can't be done.

I mean, putting technicalities aside, what deep learning is doing is taking huge numbers of examples and finding some patterns.

In some areas it is, but we have to ask here a certain question.

Engineering in the sense of just trying to build something that's useful, or science in the sense that it's trying to understand something about elements of the world.

So on engineering grounds, it's kind of worth having, like a bulldozer.

Does it tell you anything about human language?

It's from the very beginning, it's just totally remote from science.

It's taking an enormous text, let's say the Wall Street Journal corpus, and asking how close can we come to getting the right description of every sentence in the corpus?

Well, every sentence in the corpus is essentially an experiment.

Each sentence that you produce is an experiment which is, am I a grammatical sentence?

So most of the stuff in the corpus is grammatical sentences.

But now ask yourself, is there any science which takes random experiments which are carried out for no reason whatsoever and tries to find out something from them?

Like if you're, say, a chemistry PhD student, you wanna get a thesis, can you say, well, I'm just gonna do a lot of, mix a lot of things together, no purpose, just maybe I'll find something.

Science tries to find critical experiments, ones that answer some theoretical question.

Doesn't care about coverage of millions of experiments.

So it just begins by being very remote from science, and it continues like that.

So the usual question that's asked about, say, a Google parser is how well does it do, or some parser, how well does it do on a corpus?

But there's another question that's never asked.

How well does it do on something that violates all the rules of language?

So for example, take the structure dependence case that I mentioned.

Suppose there was a language in which you used linear proximity as the mode of interpretation.

These deep learning had worked very easily on that.

In fact, much more easily than an actual language.

From a scientific point of view, it's a failure.

It shows that we're not discovering the nature of the system at all, because it does just as well or even better on things that violate the structure of the system.

So yes, so neural networks are kind of approximators that look, there's echoes of the behavioral debates, right, behavioralism.

the people in deep learning say they've vindicated.

Terry Sanyoski, for example, in his recent books, says this vindicates Skinnerian behaviors.

Yes, but I think there's something actually fundamentally different when the data set is huge, but your point is extremely well taken.

But do you think we can learn, approximate, that interesting complex structure of language with neural networks that will somehow help us understand the science?

I mean, you find patterns that you hadn't noticed, let's say, could be.

In fact, it's very much like a kind of linguistics that's done, what's called corpus linguistics.

When you, suppose you have some language where all the speakers have died out, but you have records.

So you just look at the records and see what you can figure out from that.

It's much better to have actual speakers where you can do critical experiments.

But if they're all dead, you can't do them.

So you have to try to see what you can find out from just looking at the data that's around.

Actually, paleoanthropology is very much like that.

You can't do a critical experiment on what happened 2000000 years ago, so you're kind of forced just to take what data's around and see what


Speaker 0: Let me ask you about a field of machine learning, deep learning. There's been a lot of progress in neural networks based neural network based machine learning in the recent decade. Of course, neural network research goes back many decades. What do you think are the limits of deep learning, of neural network based machine learning?

Speaker 1: Well, to give a real answer to that, you'd have to understand the exact processes that are taking place. And those are pretty opaque. So it's pretty hard to prove a theorem about what can be done and what can't be done. But I think it's reasonably clear. I mean, putting technicalities aside, what deep learning is doing is taking huge numbers of examples and finding some patterns.

Okay, that could be interesting. In some areas it is, but we have to ask here a certain question. Is it engineering or is it science? Engineering in the sense of just trying to build something that's useful, or science in the sense that it's trying to understand something about elements of the world. So it takes a Google parser.

We can ask that question. Is it useful? Yeah, it's pretty useful. You know, I use a Google translator. So on engineering grounds, it's kind of worth having, like a bulldozer.

Does it tell you anything about human language? 0. Nothing. And in fact, it's very striking. It's from the very beginning, it's just totally remote from science.

So what is a Google parser doing? It's taking an enormous text, let's say the Wall Street Journal corpus, and asking how close can we come to getting the right description of every sentence in the corpus? Well, every sentence in the corpus is essentially an experiment. Each sentence that you produce is an experiment which is, am I a grammatical sentence? The answer is usually yes.

So most of the stuff in the corpus is grammatical sentences. But now ask yourself, is there any science which takes random experiments which are carried out for no reason whatsoever and tries to find out something from them? Like if you're, say, a chemistry PhD student, you wanna get a thesis, can you say, well, I'm just gonna do a lot of, mix a lot of things together, no purpose, just maybe I'll find something. You'd be laughed out of the department. Science tries to find critical experiments, ones that answer some theoretical question.

Doesn't care about coverage of millions of experiments. So it just begins by being very remote from science, and it continues like that. So the usual question that's asked about, say, a Google parser is how well does it do, or some parser, how well does it do on a corpus? But there's another question that's never asked. How well does it do on something that violates all the rules of language?

So for example, take the structure dependence case that I mentioned. Suppose there was a language in which you used linear proximity as the mode of interpretation. These deep learning had worked very easily on that. In fact, much more easily than an actual language. Is that a success?

No, that's a failure. From a scientific point of view, it's a failure. It shows that we're not discovering the nature of the system at all, because it does just as well or even better on things that violate the structure of the system. And it goes on from there. It's not an argument against doing it.

It is useful to have devices like this.

Speaker 0: So yes, so neural networks are kind of approximators that look, there's echoes of the behavioral debates, right, behavioralism. More than echoes. Many of

Speaker 1: the people in deep learning say they've vindicated. Terry Sanyoski, for example, in his recent books, says this vindicates Skinnerian behaviors. It doesn't have anything to do with it.

Speaker 0: Yes, but I think there's something actually fundamentally different when the data set is huge, but your point is extremely well taken. But do you think we can learn, approximate, that interesting complex structure of language with neural networks that will somehow help us understand the science?

Speaker 1: It's possible. I mean, you find patterns that you hadn't noticed, let's say, could be. In fact, it's very much like a kind of linguistics that's done, what's called corpus linguistics. When you, suppose you have some language where all the speakers have died out, but you have records. So you just look at the records and see what you can figure out from that.

It's much better to have actual speakers where you can do critical experiments. But if they're all dead, you can't do them. So you have to try to see what you can find out from just looking at the data that's around. You can learn things. Actually, paleoanthropology is very much like that.

You can't do a critical experiment on what happened 2000000 years ago, so you're kind of forced just to take what data's around and see what

See all Lex Fridman transcripts on Youtube

Noam Chomsky: Deep Learning is Useful but It Doesn't Tell You Anything about Human Language