This is a clip from a conversation with Jeremy Howard from Aug 2019. New full episodes every Mon & Thu and 1-2 new clips or a new non-podcast video on all other days. You can watch the full conversation here: https://www.youtube.com/watch?v=J6XcP4JOHmk
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on iTunes:
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Note: I select clips with insights from these much longer conversation with the hope of helping make these ideas more accessible and discoverable. Ultimately, this podcast is a small side hobby for me with the goal of sharing and discussing ideas. For now, I post a few clips every Tue & Fri. I did a poll and 92% of people either liked or loved the posting of daily clips, 2% were indifferent, and 6% hated it, some suggesting that I post them on a separate YouTube channel. I hear the 6% and partially agree, so am torn about the whole thing. I tried creating a separate clips channel but the YouTube algorithm makes it very difficult for that channel to grow unless the main channel is already very popular. So for a little while, I'll keep posting clips on the main channel. I ask for your patience and to see these clips as supporting the dissemination of knowledge contained in nuanced discussion. If you enjoy it, consider subscribing, sharing, and commenting.

Jeremy Howard is the founder of fast.ai, a research institute dedicated to make deep learning more accessible. He is also a Distinguished Research Scientist at the University of San Francisco, a former president of Kaggle as well a top-ranking competitor there, and in general, he's a successful entrepreneur, educator, research, and an inspiring personality in the AI community.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

There's some magic on learning rate that you played around with.

So this is all work that came from a guy called Leslie Smith.

Leslie's a researcher who, like us, cares a lot about just the practicalities of training neural networks quickly and accurately, which you would think is what everybody should care about, but almost nobody does.

And he discovered something very interesting, which he calls superconvergence, which is there are certain networks that with certain settings of high parameters could suddenly be trained 10 times faster by using a 10 times higher learning rate.

Now no 1 published that paper Because it's not an area of kind of active research in the academic world.

No academics recognize this is important.

And also, deep learning in academia is not considered an experimental science.

So unlike in physics, where you could say, like, I just saw a subatomic particle do something which the theory doesn't explain, you could publish that without an explanation.

And then in the next 60 years, people can try to work out how to explain it.

We don't allow this in the deep learning world.

So it's literally impossible for Leslie to publish a paper that says, I've just seen something amazing happen.

This thing trained 10 times faster than it should have.

And so the reviewers were like, well, you can't publish that because you don't know why.

That's important to pause on because there's so many discoveries that would need to start like that

every every other scientific field I know of works of that way.

I don't know why ours is uniquely disinterested in Publishing Unexplained experimental results, but there it is.

Having said that, I read a lot more unpublished papers than published papers, because that's where you find the interesting insights.

So I absolutely read this paper, and I was just like, this is astonishingly mind-blowing and weird and awesome and like, why isn't everybody only talking about this?

Because like, if you can train these things 10 times faster, they also generalize better because you're, you're doing less epochs, which means you look at the data less, you get better accuracy.

So I've been kind of studying that ever since.

And eventually Leslie kind of figured out a lot of how to get this done, and we added minor tweaks.

And a big part of the trick is starting at a very low learning rate Very gradually increasing it So as you're training your model you would take very small steps at the start and you gradually make them bigger and bigger Until eventually you're taking much bigger steps than anybody thought was possible.

There's a few other little tricks to make it work, but basically we can reliably get superconvergence and so for the dawn bench thing we were using just much higher learning rates than people expected to work.

What do you think the future of, I mean, it makes so much sense for that to be a critical hyperparameter, learning rate that you vary.

What do you think the future of learning rate magic looks like?

Well, there's been a lot of great work in the last 12 months in this area.

And people are increasingly realizing that optimizes, like we just have no idea really how optimizers work.

And the combination of weight decay, which is how we regularize optimizers, and the learning rate, and then other things like the epsilon we use in the atom optimizer, they all work together in weird ways.

And different parts of the model, this is another thing we've done a lot of work on, is research into how different parts of the model should be trained at different rates in different ways.

So we do something we call discriminative learning rates, which is really important, particularly for transfer learning.

So really, I think in the last 12 months, a lot of people have realized that this, all this stuff is important.

There's been a lot of great work coming out And we're starting to see algorithms appear which have very, very few dials, if any, that you have to touch.

So like, I think what's going to happen is the idea of a learning rate will, it almost already has disappeared in the latest research.


Speaker 0: There's some magic on learning rate that you played around with. Yeah. It's quite interesting. Yeah.

Speaker 1: So this is all work that came from a guy called Leslie Smith. Leslie's a researcher who, like us, cares a lot about just the practicalities of training neural networks quickly and accurately, which you would think is what everybody should care about, but almost nobody does. And he discovered something very interesting, which he calls superconvergence, which is there are certain networks that with certain settings of high parameters could suddenly be trained 10 times faster by using a 10 times higher learning rate. Now no 1 published that paper Because it's not an area of kind of active research in the academic world. No academics recognize this is important.

And also, deep learning in academia is not considered an experimental science. So unlike in physics, where you could say, like, I just saw a subatomic particle do something which the theory doesn't explain, you could publish that without an explanation. And then in the next 60 years, people can try to work out how to explain it. We don't allow this in the deep learning world. So it's literally impossible for Leslie to publish a paper that says, I've just seen something amazing happen.

This thing trained 10 times faster than it should have. I don't know why. And so the reviewers were like, well, you can't publish that because you don't know why. So anyway.

Speaker 0: That's important to pause on because there's so many discoveries that would need to start like that

Speaker 1: every every other scientific field I know of works of that way. I don't know why ours is uniquely disinterested in Publishing Unexplained experimental results, but there it is. So it wasn't published. Having said that, I read a lot more unpublished papers than published papers, because that's where you find the interesting insights. So I absolutely read this paper, and I was just like, this is astonishingly mind-blowing and weird and awesome and like, why isn't everybody only talking about this?

Because like, if you can train these things 10 times faster, they also generalize better because you're, you're doing less epochs, which means you look at the data less, you get better accuracy. So I've been kind of studying that ever since. And eventually Leslie kind of figured out a lot of how to get this done, and we added minor tweaks. And a big part of the trick is starting at a very low learning rate Very gradually increasing it So as you're training your model you would take very small steps at the start and you gradually make them bigger and bigger Until eventually you're taking much bigger steps than anybody thought was possible. There's a few other little tricks to make it work, but basically we can reliably get superconvergence and so for the dawn bench thing we were using just much higher learning rates than people expected to work.

Speaker 0: What do you think the future of, I mean, it makes so much sense for that to be a critical hyperparameter, learning rate that you vary. What do you think the future of learning rate magic looks like?

Speaker 1: Well, there's been a lot of great work in the last 12 months in this area. And people are increasingly realizing that optimizes, like we just have no idea really how optimizers work. And the combination of weight decay, which is how we regularize optimizers, and the learning rate, and then other things like the epsilon we use in the atom optimizer, they all work together in weird ways. And different parts of the model, this is another thing we've done a lot of work on, is research into how different parts of the model should be trained at different rates in different ways. So we do something we call discriminative learning rates, which is really important, particularly for transfer learning.

So really, I think in the last 12 months, a lot of people have realized that this, all this stuff is important. There's been a lot of great work coming out And we're starting to see algorithms appear which have very, very few dials, if any, that you have to touch. So like, I think what's going to happen is the idea of a learning rate will, it almost already has disappeared in the latest research. And instead It's just like, you you

See all Lex Fridman transcripts on Youtube

Jeremy Howard: Very Fast Training of Neural Networks | AI Podcast Clips