This is a clip from a conversation with Jeremy Howard on the Artificial Intelligence podcast. You can watch the full conversation here: http://bit.ly/2NG4qwr If you enjoy these, consider subscribing, sharing, and commenting below.

Full episode: http://bit.ly/2NG4qwr
Full episodes playlist: http://bit.ly/2EcbaKf
Clips playlist: http://bit.ly/2JYkbfZ
Podcast website: https://lexfridman.com/ai

Jeremy Howard is the founder of fast.ai, a research institute dedicated to make deep learning more accessible. He is also a Distinguished Research Scientist at the University of San Francisco, a former president of Kaggle as well a top-ranking competitor there, and in general, he's a successful entrepreneur, educator, research, and an inspiring personality in the AI community.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

So much of fast AI students and researchers and the things you teach are pragmatically minded, practically minded, figuring out ways how to solve real problems and fast.

So from your experience, what's the difference between theory and practice of deep learning?

Well, Most of the research in the deep learning world is a total waste of time.

Yeah, it's a problem in science in general.

Scientists need to be published, Which means they need to work on things that their peers are extremely familiar with and can recognize and advance in that area.

So that means that they all need to work on the same thing.

And so it really, and the thing they work on, there's nothing to encourage them to work on things that are practically useful.

So you get just a whole lot of research which is minor advances and stuff that's been very highly studied and has no significant practical impact.

Whereas the things that really make a difference, like I mentioned transfer learning.

Like if we can do better at transfer learning, then it's this like world-changing thing where suddenly like lots more people can do world-class work with less resources and less data and, but almost nobody works on that.

Or another example, active learning, which is the study of like, how do we get more out of the human beings

So active learning is great, but it's almost Nobody working on it because it's just not a trendy thing right now.

You know what somebody started to interrupt.

He was saying that nobody is publishing on active learning.

But there's people inside companies, anybody who actually has to solve a problem, they're going to innovate on active learning.

Yeah, everybody kind of reinvents active learning when they actually have to work in practice because they start labeling things and they think, gosh, this is taking a long time and it's very expensive.

And then they start thinking, well, why am I labeling everything?

I'm only, the machine's only making mistakes on those 2 classes.

Maybe I'll just start labeling those 2 classes.

And then you start thinking, well, why did I do that manually?

Why can't I just get the system to tell me which things are going to be hardest?

It's an obvious thing to do, but yeah, it's just like transfer learning.

It's understudied and the academic world just has no reason to care about practical results.

The funny thing is, like I've only really ever written 1 paper.

It was my colleague Sebastian Ruda who actually wrote it I just did the research for it, but it was basically introducing transfer learning successful transfer learning to NLP for the first time and the algorithm is called ULM fit and It actually I actually wrote it for the course, for the fast.ai course.

I wanted to teach people NLP and I thought I only want to teach people practical stuff and I think the only practical stuff is transfer learning.

And I couldn't find any examples of transfer learning in NLP So I just did it and I was shocked to find that as soon as I did it was you know The basic prototype took a couple of days Smash the state of the art on 1 of the most important data sets in a field that I knew nothing about and I just thought well, this is ridiculous and so I spoke to Sebastian about it and he kindly offered to write it up, the results.

And so it ended up being published in, ACL, which is the top link with a computational linguistics conference.

So like people do actually care once you do it, but I guess it's difficult for maybe like junior researchers or like, like I don't care whether I get citations on papers or whatever.

I don't, There's nothing in my life that makes that important, which is why I've never actually bothered to write a paper myself.

But for people who do, I guess they have to pick the kind of safe option, which is like, yeah, make a slight improvement on something that everybody's already working on.


Speaker 0: So much of fast AI students and researchers and the things you teach are pragmatically minded, practically minded, figuring out ways how to solve real problems and fast. So from your experience, what's the difference between theory and practice of deep learning?

Speaker 1: Well, Most of the research in the deep learning world is a total waste of time.

Speaker 0: Right, that's what I was getting at.

Speaker 1: Yeah, it's a problem in science in general. Scientists need to be published, Which means they need to work on things that their peers are extremely familiar with and can recognize and advance in that area. So that means that they all need to work on the same thing. And so it really, and the thing they work on, there's nothing to encourage them to work on things that are practically useful. So you get just a whole lot of research which is minor advances and stuff that's been very highly studied and has no significant practical impact.

Whereas the things that really make a difference, like I mentioned transfer learning. Like if we can do better at transfer learning, then it's this like world-changing thing where suddenly like lots more people can do world-class work with less resources and less data and, but almost nobody works on that. Or another example, active learning, which is the study of like, how do we get more out of the human beings

Speaker 0: in the loop? That's my favorite topic.

Speaker 1: Yeah. So active learning is great, but it's almost Nobody working on it because it's just not a trendy thing right now.

Speaker 0: You know what somebody started to interrupt. He was saying that nobody is publishing on active learning. But there's people inside companies, anybody who actually has to solve a problem, they're going to innovate on active learning.

Speaker 1: Yeah, everybody kind of reinvents active learning when they actually have to work in practice because they start labeling things and they think, gosh, this is taking a long time and it's very expensive. And then they start thinking, well, why am I labeling everything? I'm only, the machine's only making mistakes on those 2 classes. They're the hard ones. Maybe I'll just start labeling those 2 classes.

And then you start thinking, well, why did I do that manually? Why can't I just get the system to tell me which things are going to be hardest? It's an obvious thing to do, but yeah, it's just like transfer learning. It's understudied and the academic world just has no reason to care about practical results. The funny thing is, like I've only really ever written 1 paper.

I hate writing papers. And I didn't even write it. It was my colleague Sebastian Ruda who actually wrote it I just did the research for it, but it was basically introducing transfer learning successful transfer learning to NLP for the first time and the algorithm is called ULM fit and It actually I actually wrote it for the course, for the fast.ai course. I wanted to teach people NLP and I thought I only want to teach people practical stuff and I think the only practical stuff is transfer learning. And I couldn't find any examples of transfer learning in NLP So I just did it and I was shocked to find that as soon as I did it was you know The basic prototype took a couple of days Smash the state of the art on 1 of the most important data sets in a field that I knew nothing about and I just thought well, this is ridiculous and so I spoke to Sebastian about it and he kindly offered to write it up, the results.

And so it ended up being published in, ACL, which is the top link with a computational linguistics conference. So like people do actually care once you do it, but I guess it's difficult for maybe like junior researchers or like, like I don't care whether I get citations on papers or whatever. I don't, There's nothing in my life that makes that important, which is why I've never actually bothered to write a paper myself. But for people who do, I guess they have to pick the kind of safe option, which is like, yeah, make a slight improvement on something that everybody's already working on.

See all Lex Fridman transcripts on Youtube

Most Research in Deep Learning is a Total Waste of Time - Jeremy Howard | AI Podcast Clips