The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

CONNECT:
- If you enjoyed this video, please subscribe to this channel.
- AI Podcast: https://lexfridman.com/ai/
- Show your support: https://www.patreon.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

So when we're organizing this workshop, my co-organizers initially asked me, hey Andrew, end of the first day, go give a visionary talk.

So until several hours ago, my talk was advertised as visionary talk.

But I was preparing for this presentation over the last several days.

I've tried to think what would be the most useful information to you.

And what are the things that you could take back to work on Monday and do something different at your job next Monday.

And I thought that we need to set context right now, as Peter mentioned, I lead Baidu's AI team.

So it's a team of about 1,000 people working on vision, speech, NLP, lots of applications and machine learning.

And so what I thought I'd do instead is, instead of taking the shiniest pieces of deep learning that I know, I want to take the lessons that I saw at Baidu that are common to so many different academic areas as well as applications, autonomous cars, augmented reality, advertising, web search, medical diagnosis.

We take what I saw, the common lessons, the simple, powerful ideas that I've seen help drive a lot of machine learning progress at Baidu.

And I thought I will share those ideas with you because the patterns I see across a lot of projects I thought might be the patterns that would be most useful to you as well, whatever you are working on in the next several weeks or months.

So 1 common theme that will appear in this presentation today is that the workflow of organizing machine learning projects feels like parts of it are changing in the era of deep learning.

So for example, 1 of the ideas I'll talk about is bias-variance.

And then, many of you, maybe all of you have heard of bias-invariance.

But in the era of deep learning, I feel like there have been some changes to the way we think about bias-invariance.

So I wanna talk about some of these ideas, which maybe aren't even deep learning per se, but have been slowly shifting as we apply deep learning to more and more of our applications, okay?

And instead of holding all your questions until the end, if you have a question in the middle, feel free to raise your hand as well.

I'm very happy to take questions in the middle, since this is a more maybe informal whiteboard talk, right?

And also, say hi to our home viewers, hi.

So 1 question that I still get asked sometimes is, and I think Andre alluded to this earlier, a lot of the basic ideas of deep learning have been around for decades, so why are they taking off just now, right?

Why is it that deep learning, these neural networks have all known for maybe decades, why are they working so well now?

So I think that the 1 biggest trend in deep learning is scale, that scale drives deep learning progress.

And I think Andre mentioned scale of data and scale of computation.

And I just draw a picture that illustrates that concept maybe a little bit more.

So if I plot a figure, where on the horizontal axis I plot, the amount of data we have for a problem.

And on the vertical axis we plot, you know, performance.

So x-axis is the amount of spam data you've collected, y-axis is how accurately can you classify spam.

Then if you apply, you know, traditional learning algorithms, right?

What we found was that the performance often looks like it starts to plateau after a while.

It was as if the older generations of learning algorithms, including, you know, support vector, logistic regression, SVMs, was as if they didn't know what to do with all the data that we finally had.

And what happened kind of over the last 20 years, last 10 years, was with the rise of the Internet, rise of mobile, rise of IoT, where's the society sort of marched to the right of this curve, right?

And so, with all the buzz and all the hype about deep learning, in my opinion, the number 1 reason, that deep learning algorithms work so well is that if you train, let me call a small neural net, maybe you get slightly better performance.

If you train a medium-sized neural net, right, maybe you get even better performance and it's only if you train a large neural net, that you could train a model with the capacity to absorb all this data that we don't have access to, that allows you to get the best possible performance.

And so I feel like this is a trend that we're seeing in many verticals, in many application areas.

1 is that this, actually when I draw this picture, some people ask me, well, does this mean a small neural net always dominates a traditional learning algorithm?

Technically, if you look at the small data regime, if you look at the left end of this plot, right?

The relative ordering of these algorithms is not that well defined.

It depends on who's more motivated to engineer the features better, right?

So If the SVM guy is more motivated to spend more time engineering features, they might beat out the neural network application.

But because when you don't have much data, a lot of the knowledge of the algorithm comes from hand engineering.

But this trend is much more evident in a regime of big data where you just can't hand engineer enough features.

And the large neural net combined with a lot of data tends to outperform.

The implication of this figure is that in order to get the best performance, in order to hit that target, you need 2 things, right?

1 is you need to train a very large neural network, or a reasonably large neural network, and you need a large amount of data.

And so this in turn has caused pressure to train large nets, right?

Build large nets as well as get huge amounts of data.

So 1 of the other interesting trends I've seen is that increasingly, I'm finding that it makes sense to build an AI team as well as build a computer systems team and have the 2 teams kind of sit next to each other.

And the reason I say that is, I guess, so let's see.

So when we started Baidu Research, we set our team that way.

I think Peter mentioned to me that OpenAI also has a systems team and a machine learning team.

And the reason we're starting to organize our teams that way, I think, is that some of the computer systems work we do, right?

So we have an HPCT team, high performance team, a super computing team at Baidu.

Some of the extremely specialized knowledge in HPC is just incredibly difficult for an AI researcher to learn, right?

Some people are super smart, maybe Jeff Dean's smart enough to learn everything.

But it's just difficult for any 1 human to be sufficiently expert in HPC and sufficiently expert in machine learning.

And so we've been finding, and then Shubo actually, 1 of the co-organizers on our HPC team.

We've been finding that bringing talent, knowledge from these multiple sources, multiple communities allows us to get our best performance.

You've heard a lot of fantastic presentations today.

I want to draw 1 other picture, which is, in my mind, this is how I mentally bucket, you know, work in, in, in deep learning.

So this might be a useful categorization, right?

When you look at the talk, you can mentally put each talk into 1 of these buckets I'm about to draw.

But I feel like there's a lot of work on, I'm gonna call, you know, general DL or general models and this would be basically what, the type of model that Hugo Larochelle talked about this morning, where you have, you know, really densely connected layers, right?

I guess FC, right, was, was the- So there's a huge bucket of models there.

And then I think a second bucket is, sequence models.

So 1D sequences, and this is where I would bucket a lot of the work on RNNs, you know, LSTMs, right, GRUs, some of the attention models, which I guess probably Yoshua Bengio is talking about tomorrow, or maybe, maybe, maybe others, maybe Quokka, so I'm not sure, right?

But so the 1D sequence models is another huge bucket.

This is really 2D and maybe sometimes 3D, but this is where I would tend to bucket all the work of CNNs, convolutional nets.

And then in my mental bucket, then there's a fourth 1 which is the other, right?

And this includes unsupervised learning, you know, the reinforcement learning, right, as well as lots of other creative ideas, being explored in the training.

You know, like, what's- I, I still find slow feature analysis, sparse coding, you know, ICA, various models kind of in the other category, super exciting.

So it turns out that if you look across industry today, almost all the value today is driven by these 3 buckets, right?

So what I mean is those 3 buckets of algorithms are driving, causing us to have much better products, right, or monetizing very well.

It's just incredibly useful for lots of things.

In some ways, I think this bucket might be the future of AI, right?

So I find unsupervised learning especially super exciting.

So I'm actually super excited about this as well.

Although I think that if on Monday you have a job and you're trying to build a product or whatever, the chance of you using something from 1 of these 3 buckets will be highest.

But I definitely encourage you to contribute to research here as well.

So I said that trend 1, the major trends 1 of deep learning is scale.

This is what I would say is maybe major trend 2 of 2 of 2 trends.

Is I feel major trend 2 is the rise of end-to-end deep learning.

And so end-to-end deep learning, I'll say a little bit more in a second exactly what I mean by that, but the examples I'm gonna talk about are all from 1 of these 3 buckets, right, general DL, sequence models, image 2D, 3D models.

But let's see, it's best illustrated with a few examples.

Until recently, a lot of machine learning used to output just row numbers, you know, so I guess in Richard's, example, you have a movie review, right?

And then, actually, I, but I prepared totally different examples.

I was editing my examples earlier to- to- to be more coherent with the speakers before me.

We have a movie review and then output the sentiment, you know, is this a positive or a negative movie review?

And then you want to do image net object recognition, you know, so this would be a 0, 1 output.

But so until recently, a lot of machine learning was about outputting a single number, maybe a real number, maybe an integer.

And I think the, the, the number 2 major trend that I'm really excited about is, entering deep learning algorithms that can output much more complex things than numbers.

And so 1 example that you've seen is, image captioning where instead of taking an image and saying this is a cat, you can now take an image and output, you know, an entire string of texts using RNN to generate that sequence.

So I guess what Andre who spoke just now, I think Orel Vindal's, Shu Wei at Baidu, right?

A whole bunch of people have worked on this problem.

1 of the things that I guess my collaborator Adam Coates will talk about tomorrow, maybe Kwok as well, not sure, is speech recognition, where you take as input audio and you directly output the text transcript.

Right, and so when we first proposed using this kind of end-to-end architecture to do speech recognition, this is very controversial.

We're building the work of Alex Graves, but the idea of actually putting this in the production speech system was very, very controversial.

When we first said we wanted to do this, but I think the whole community's coming around to this point of view more recently.

Or machine translation, say, go from English to French.

So, Ia Saskiava, Kwok Leur, others working on this, a lot of teams now.

Or, you know, given the parameters, synthesize a brand new image, right?

And you saw some examples of image synthesis.

So I feel like the second major trend of deep learning that I find very exciting.

And I mean, this allowing us to build transformative things that we just couldn't build 3 or 4 years ago.

As in this trend to what not just learning algorithms and output not just a number, but it can output very complicated things like a sentence or a caption or French sentence or image or like the recent WaveNet paper, output audio, right?

So I think this is maybe the second major trend.

So despite all the excitement about end-to-end deep learning, I think that end-to-end deep learning, sadly, is not the solution to everything.

I wanna give you some rules of thumb for deciding when to use, what is exactly end-to-end deep learning, and when to use it and when not to use it.

So I'm moving to the second bullet and we'll go through these.

So the trend towards end-to-end deep learning has been, this idea that instead of engineering a lot of intermediate representations, maybe you can go directly from your raw input to whatever you wanna predict.

So for example, actually, what to take is- I'm gonna use speech as a recurring example.

So for speech recognition, previously, 1 used to go from the audio to, you know, hand-engineered features like MFCCs or something and then maybe extract phonemes, right, and then eventually you try to generate the transcript.

For those of you that aren't sure what a phoneme is, so if you look at the word, listen to the word cat and the word kick, the k sound, right, is the same sound.

And so phonemes are these basic units of sound, such as k as a phoneme, and is hypothesized by linguists to be the basic units of sound.

So, ke, ah, te would be the, maybe the 3 phonemes that make up the word cat, right?

So, traditional speech systems used to, used to do this and I think 2011, Lee Dang and Jeff Hinton made a lot of progress in speech recognition by saying, we can use deep learning to do this first step.

But the end to end approach to this would be to say, let's forget about phonemes.

Right, Input the audio and output the transcript.

So it turns out that in some problems, there's end-to-end approach.

So 1 end is the input, the other end is the output.

So the phrase end-to-end deep learning refers to, just having a neural net or, you know, like a learning algorithm directly go from input to output.

This end-to-end formula, I think it makes for great PR and it's actually very simple, but it only works sometimes.

Actually, let me just tell you an interesting story.

This end-to-end story really upset a lot of people.

When we were doing this work, I used to go around and say, I think phonemes are a fantasy of linguists and we should do away with them.

And I still remember there was a meeting at Stanford, and some of you know who it was, there was a linguist kind of yelling at me in public, for saying that.

So maybe, maybe I should not, but we turned out to be right, you know, so.

But the, the, the Achilles heel of a lot of deep learning is that you need tons of label data, right?

So if this is your x And that's your y, then for end to end deep learning to work, you need a ton of label input output data, x, y.

So to take an example where 1 may or may not consider end to end deep learning.

This is a problem I learned about just last week from Curtis Langlis and Darwin who's in the audience, I think.

Imagine you want to use x-ray pictures of your hand in order to predict a child's age, right?

Doctors actually care to look at an x-ray of a child's hand in order to predict the age of the child.

So, boy, let me draw an X-ray image, right?

So that's a hand and, and, and you see the bones.

And so more traditional algorithm might input an image, and then first, you know, extract the bones.

So first figure out, oh, there's a bone here, there's a bone here, there's a bone here, and then maybe measure the length of these bones, right?

So really I'm gonna say bone lengths, and then maybe have some formula like some linear regression average, some simple thing to go from the bone length to estimate the age of the child, right?

So this is a non-end-to-end approach to try and solve this problem.

An end-to-end approach would be to take an image and then, you know, run a ConvNet or whatever and just try to output the age of the child.

And I think this is 1 example of a problem where it's very challenging to get end-to-end deep learning to work because you just don't have enough data.

You just don't have enough x-rays of children's hands annotated with their ages.

And Instead, where we see deep learning coming in is in this step, right?

To go from image to figure out where the bones are, use deep learning for that.

But the advantage of this non-end-to-end architecture is it allows you to hand engineer in more information about the system, such as how bone lengths map to age, right?

There are a lot of examples like this, and I think 1 of the unfortunate things about deep learning is that, let's see.

You can, for suitably sexy values of x and y, you could almost always train a model and publish a paper, but that doesn't always mean that, you know, it's actually a good idea.

So maybe it's the definition of end-to-end, but you could have principle people are in the front part knowing what's coming later.

So Peter's pointing out that in practice you could, if this is a fixed function f, right?

You could backprop all the way from the age, all the way back to the image.

Who was it just now who said you better do it quickly?

Let me give a couple other examples, that where, where it might be harder to backprop all the way through, right?

You know, most teams are using an architecture where you input an image, you know, what's in front of the car let's say, and then you, you know, detect other cars, right?

And then, and, and maybe use the image to detect pedestrians, right?

Self-driving cars are obviously more complex than this, right?

But then now that you know where the other cars and where the pedestrians are relative to your car, you then have a planning algorithm, to then, you know, come up with a trajectory, right?

And then now that you know, what's the trajectory that you want your car to drive through, you could then, you know, compute the steering direction, right?

And so, this is actually the architecture that most self-driving car teams are using.

And you know, there have been interesting approaches to- to say, well, I'm gonna input an image and I'll put a steering direction.

And I think this is an example of where at least with today's state of technology, I'd be very cautious about the second approach.

And I think if you have enough data, the second approach will work.

And you can even prove a theorem, showing that it will work, I think.

But I don't know that anyone today has enough data to make the second approach really, really work well, right?

And I think Peter made a great comment just now.

And I think some of these components will be incredibly complicated, like this could be a power plan of an explicit search.

And you could actually design a really complicated power plan and generate the trajectory.

And your ability to hand code that still has a lot of value, right?

I have seen project teams say, I can get x, I can get y, I'm going to train deep learning.

But unless you actually have the data, some of these things make for great demos if you cherry pick the examples, but it can be challenging to get to work at scale.

I should say, for self-driving cars, this debate is still open.

I don't think this will necessarily fail.

I just think the data needed to do this will be really immense.

So I'd be very cautious about it right now.

But it might work if you have enough data.

So 1 of the themes that comes up in machine learning, really, If you're working on a machine learning project, 1 thing that will often come up is you will develop a learning system, train it, maybe it doesn't work as well as you're hoping yet.

And the question is, what do you do next?

This is a very common part of a machine learning researcher or machine learning engineer's life.

Which is you train a model, doesn't do what you want it to yet, so what do you do next, right?

You could collect more data, maybe you want to train it longer, Maybe you want a different neural network architecture.

Maybe you want to try regularization, maybe a bigger model, run some more GPUs.

And I think that a lot of the skill of a machine learning researcher or machine learning engineer is knowing how to make these decisions, right?

And the difference in performance and whether you do train a bigger model or do you try regularization, your skill at picking these decisions will have a huge impact on how rapidly you can make progress on actual machine learning problem.

So I want to talk a little about bias and variance, since that's 1 of the most basic, you know, concepts in machine learning, and I feel like it's evolving slightly in the era of, of, of deep learning.

So to use a- as a motivating example, let's say the goal is to build a human level, right?

So, what we would typically do, especially in academia, is we'll get a data set.

Here's my data set with a lot of examples.

And then we shuffle it and we randomly split it into 70-30 trained tests.

And some people use the term validation set, but I'm just gonna use the dev set, or it stands for development set, means the same thing as validation set.

And so what I encourage you to do, if you aren't already, is to measure the following things.

So let- let- actually let me illustrate an example.

Let's say that on your dev- let's say that, on your dev set, you know, human level error is, 1% error.

Let's say that your training set error is, let me use

And let's say that your dev set error, Really, right?

Dev set is a proxy for test set, except you tune to the dev set, right?

So this is 1 of the most basic- this, this is really a, a step in developing a learning algorithm that I encourage you to do if you aren't already, to figure out what are these 3 numbers.

Because these 3 numbers, really helps in terms of telling you what to do next.

So in this example, you see that you're doing much worse than human level performance.

And so you see that there's a huge gap here from 1% to 5%.

And I'm gonna call this, you know, right, the bias of your learning algorithm.

And for the statisticians in the room, I'm using the terms bias and variance informally and doesn't correspond exactly to the way they're defined in textbooks.

But I find these useful concepts for deciding how to make progress on your problem.

And so I would say that in this example, you have a high bias classifier, try training a bigger model, maybe try training longer.

See all Lex Fridman transcripts on Youtube

Nuts and Bolts of Applying Deep Learning (Andrew Ng)