The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

So I'll tell you about some very high level stuff today and no new algorithm.

Some of you already know about the book that Ian Goodfellow, Aaron Corville, and I have written.

I think you can find it on Amazon or something.

And the paper, the actual shipping is going to be in December, hopefully for NIPS.

So, we've already heard that story, at least, well, from several people here, at least from Andrew, I think.

But it's good to ponder a little bit some of these ingredients that seem to be important for deep learning to succeed, but in general for machine learning to succeed, to learn really complicated tasks of the kind we want to reach human level performance.

So if a machine is gonna be intelligent, it's going to need to acquire a lot of information about the world.

And the big success of machine learning for AI has been to show that we can provide that information through data, through examples.

But really think about it, that machine will need to know a huge amount of information about the world around us.

This is not how we are doing it now because we're not able to train such big models, but it will come 1 day.

And so we'll need models that are much bigger than the ones we currently have.

Of course, that means machine learning algorithms that can represent complicated functions.

But there are many other machine learning approaches that allow you, in principle, to represent very flexible forms like non-parametric methods, classical non-parametric methods, or SVMs.

But they're gonna be missing 0.4 and potentially 0.5 depending on the methods.

0.3 of course, you need enough computing power to train and use these big models.

And 0.5 just says that it's not enough to be able to train the model.

You have to be able to use it in a reasonably efficient way from a computational perspective.

This is not always the case with some probabilistic models where inference, in other words, answering questions, having the computer do something, can be intractable and then you need to do some approximations, which could be efficient or not.

Now, the point I really want to talk about is the fourth 1, how do we defeat the curse of dimensionality?

In other words, if you don't assume much about the world, it's actually impossible to learn about it.

And so I'm gonna tell you a bit about the assumptions that are behind a lot of deep learning algorithms which make it possible to work as well as we are seeing in practice in the last few years.

So how do we bypass the curse of dimensionality?

The curse of dimensionality is about the exponentially large number of configurations of the space variables that we want to model.

The number of values that all of the variables that we observe can take is going to be exponentially large in general, because there's a compositional nature.

If each pixel can take 2 values, and you got a million pixels, then you got 2 to 1000000 number of possible images.

So the only way to beat an exponential is to use another exponential.

So, we need to make our models compositional.

We need to build our models in such a way that they can represent functions that look very complicated, but yet these models need to have a reasonably small number of parameters, reasonably small in the sense that compared to the number of configurations of the variables, the number of parameters should be small.

And we can achieve that by composing little pieces together, composing layers together, composing units on the same layer together, And that's essentially what's happening with deep learning.

So you actually have 2 kinds of compositions.

There's the composition is happening on the same layer.

This is the idea of distributed representations, which I'm going to try to explain a bit more.

This is what you get when you learn embeddings for words or for images, representations in general.

And then there's the idea of having multiple levels of representation.

And there, there is another kind of composition that takes place, whereas the first 1 is a kind of parallel composition.

I can choose the values of my different units separately, and then they together represent an exponentially large number of possible configurations.

In the second case, there's a sequential composition where I take the output of 1 level and I combine them in new ways to build features for the next level and so on and so on.

So the reason deep learning is working is because the world around us is better modeled by making these assumptions.

It's not necessarily true that deep learning is gonna work for any machine learning problem.

In fact, if we consider the set of all possible distributions that we would like to work from, deep learning is no better than any other.

And that's basically what the no-free-lunch theorem is saying.

It's because we are incredibly lucky that we live in this world, which can be described by using composition, that these algorithms are working so well.

This is important to really understand this.

So before I go a bit more into distributed representations, let me say a few words about non-distributed representations.

So if you're thinking about things like clustering, n-grams for language modeling, classical nearest neighbors, SVMs with Gaussian kernels, classical non-parametric models with local kernels, and decision trees, all these things.

The way these algorithms really work is actually pretty straightforward.

If you cut the crap and hide the math and try to understand what is going on, they look at the data in the data space and they break that space into regions and they're going to use different free parameters for each of those regions to figure out what the right answer should be.

The right answer, it doesn't have to be supervised learning.

Even in unsupervised learning, there's a right answer.

It might be the density or something like that.

Okay, and you might think that that's the only way of solving a problem.

We consider all of the cases and we have an answer for each of the cases.

And we can maybe interpolate between those cases that we've seen.

The problem with this is somebody comes up with a new example which isn't in between 2 of the examples we've seen, something that requires us to extrapolate something that's a non-trivial generalization, and these algorithms just fail.

They don't really have a recipe for saying something meaningful away from the training examples.

There's another interesting thing to note here, which I would like you to keep in mind before I show the next slide, which is in red here, which is we can do a kind of simple counting to relate the number of parameters, the number of free parameters that can be learning, and the number of regions in the data space that we can distinguish.

So here, we basically have a linear relationship between these 2 things.

So for each region, I'm going to need at least something like some kind of center for the region.

And maybe if I need to output something, I'll need an extra set of parameters to tell me what the answer should be in that area.

So the number of parameters grows linearly with the number of regions that I'm gonna be able to distinguish.

The good news is I can have any kind of function, right?

So I can break up the space in any way I want, and then for each of those regions, I can have any kind of output that I need.

So for decision trees, the regions would be splitting across axes and so on, and this is more like for nearest neighbor or something like that.

So, here's the point of view of distributed representations for solving the same general machine learning problem.

We have a data space and we want to break it down.

But we're going to break it down in a way that's not general.

We're going to break it down in a way that makes assumptions about the data.

But it's going to be compositional and it's going to allow us to be exponentially more efficient.

So in the picture on the right, what you see is a way to break the input space by the intersection of half planes.

And this is the kind of thing you would have with what happens on the first layer of a neural net.

So here, imagine the input is 2 dimensional, so I can plot it here, and I have 3 binary hidden units, C1, C2, C3.

So because they're binary, you can think of them as little binary classifiers.

And because it's only a one-layer net, you can think of what they're doing as a linear classification.

And so those colored hyperplanes here are the decision surfaces for each of them.

Now, these 3 bits, they can take 8 values, right, corresponding to whether each of them is on or off, and those different configurations of those bits correspond to actually 7 regions here, because there's 1 of the 8 regions which is not feasible.

So now you see that we are defining a number of regions which is corresponding to all of the possible intersections of the corresponding half planes.

And now we can play the game of how many regions do we get for how many parameters.

And what we see is that if we played the game of growing the number of dimensions, of features, and also of inputs, we can get an exponentially large number of regions, which are all of these intersections, right?

There's an exponential number of these intersections, corresponding to different binary configurations.

Yet, the number of parameters grows linearly with the number of units.

So it looks like we're able to express a function, and then on top of that, I could imagine you have a linear classifier, right?

So The number of parameters grows just linearly with the number of features, but the number of regions that the network can really provide a different answer to grows exponentially.

And the reason it's very cool is that it allows those neural nets to generalize.

Because while we're learning about each of those features, we can generalize to regions we've never seen, because we've learned enough about each of those features separately.

I'm going to give you an example of this in a couple of slides.

Let's say the input is an image of a person.

And think of those features as things like, I have a detector that says that the person wears glasses.

And I have another unit that's detecting that the person is a female or male.

And I have another unit that detects that the person is a child or not.

And you can imagine hundreds or thousands of these things, of course.

So the good news is you could imagine learning about each of these feature detectors, these little classifiers, separately.

You could share intermediate layers between the input and those features.

But Let's take even the worst case and imagine we were to train those separately, which is the case in the linear model that I showed before.

We have a separate set of parameters for each of these detectors.

So if I have n features, each of them, say, needs order of k parameters, then I need order of nk parameters, and I need order of nk examples.

And 1 thing you should know from machine learning theory is that if you have order of p parameters, you need order of p examples to do a reasonable job of generalizing.

You can get around that by regularizing and effectively having less degrees of freedom.

But To keep things simple, you need about the same number of examples, or maybe say 100 times more, or 10 times more, as the number of really free parameters.

So now the relationship between the number of regions that I can represent and the number of examples I need is quite nice because the number of regions is gonna be 2 to the number of features of these binary features.

So a person could wear glasses or not, be a female or a male, a child or not, and I could have 100 of these things, and I could probably recognize reasonably well all of these 2 to the 100 configurations of people, even though I've obviously not seen all of those 2 to 100 configurations.

I'm able to do that because the models can learn about each of these binary features kind of independently in the sense that I don't need to see every possible configuration of the other features to know about wearing glasses.

Like I can learn about wearing glasses even though I've never seen somebody who was a female and a child and chubby and had yellow shoes.

And I have seen enough examples of people wearing glasses, I can learn about wearing glasses in general.

I don't need to see all of the configurations of the other features to learn about 1 feature.

And So this is really why this thing works, is because we're making assumptions about the data that those features are meaningful by themselves.

And you don't need to actually have data for each of the regions, the exponential number of regions, in order to learn the proper way of detecting or of discovering these intermediate features.

There were some experiments recently actually showing that this kind of thing is really happening.

Because the features I was talking about, not only I'm assuming that they exist, but the optimization methods, the training procedures discover them.

And this is an experiment that's been done in Antonio Toralba's lab at MIT, where they trained a usual ConvNet to recognize places.

So the outputs of the net are just the types of places, like is this a beach scene or an office scene or a street scene and so on.

But then the thing they've done is they asked people to analyze the hidden units to try to figure out what each hidden unit was doing.

And they found that there's a large proportion of units that humans can find a pretty obvious interpretation for what those units like.

So they see a bunch of units which like people, or different kinds of people, or animals, or buildings, or seatings, or tables, lighting, and so on.

So it's like if indeed those neural nets are discovering semantic features.

They're semantic because actually people give them names as the intermediate features in order to reach the final goal of here, classifying scenes.

And The reason they're generalizing is because now you can combine those features in an exponentially large number of ways, right?

You could have a scene that has a table, a different kind of lighting, some people, you know, maybe a pet, and you can say something meaningful about the combinations of these things, because the network is able to learn all of these features without having to see all of the possible configurations of them.

So I don't know if my explanation makes sense to you, but now is a chance to ask me a question.

So with 1 decision piece, you can kind of focus the same as well, right?

So the question is, can't we do the same thing with a set of decision trees?

Yeah, in fact, this is 1 of the reasons why forests work better, or bagged trees work better than single trees.

Forests are actually, or bagged trees, are like 1 layer, 1 level deeper than a single trees.

But they still don't have as much of a sort of distributed aspect as neural nets.

I mean, boosted trees are, to some extent, in a greedy way.

Do you find cases that are non-compositional?

I mean, I don't understand what you mean.

I don't think that there are examples of neural nets that really work well where the data doesn't have some kind of compositional structure in it.

But if you come up with an example, I'd like to hear about it.

So in the language of graph models, do you mean that we're facing a model on this graph and in the real world, we're trying to look for some independent, but we cannot get independent, but it's not somewhere with a very small square.

To think about This issue in graphical model terms can be done, but you have to think about not feature detection, like I've been doing here, but about generating an image or something like that.

If you think about how I could generate an image, if you think about underlying factors, like which objects, where they are, what's their identity, what's their size, These are all independent factors, which you compose together in funny ways.

If you were to do a graphics engine, you can see exactly what those ways are.

And it's much, much easier to represent that generative distribution using this compositional structure than if you're trying to work directly in pixel space, which is normally what you would do with a classical non-parametric method.

But if you look at our best deep generative models now for images, for example, like GANs or VAEs, they're really, you know, we're not there yet, but they're amazingly better than anything that people could dream of just a few years ago in machine learning.

Okay, Let me move on, because I have other things to talk about.

So this is all kind of hand wavy, but some people have done some math around these ideas.

And so for example, there's 1 result from 2 years ago, right here, where we studied the single layer case, and we consider a network with rectifiers, relus, and we find that the network, of course, computes a piecewise linear function.

And so, 1 way to quantify the richness of the function that it can compute.

I was talking about regions here, but well, you can do the same thing here.

You can count how many pieces does this network have in its input to output function.

And it turns out that it's exponential in the number of inputs.

Well, it's number of units to the power number of inputs.

So that's for distributed representation.

So what you need to know about depth is that there's a lot of earlier theory that says that a single layer is sufficient to represent any function.

However, that theory doesn't specify how many units you might need.

And in fact, you might need an exponentially large number of units.

So what several results show is that there are functions that can be represented very efficiently with few units, so few parameters, if you allow the network to be deep enough.

So out of all the functions, Again, it's a luckiness thing, right?

Out of all the functions that exist, there's a very, very small fraction, which happened to be very easy to represent with a deep network.

And if you try to represent these functions with a shallow network, you're screwed.

You're gonna need an exponential number of parameters.

And so you're gonna need an exponential number of examples to learn these things.

But again, we're incredibly lucky that the function we wanna learn have this property.

I mean, we use this kind of compositionality and depth everywhere.

When we write a computer program, we just don't have like a single main.

And we were able to show similar things as what I was telling you about for the single layer case, that as you increase depth for these deep ReLU networks, the number of pieces in the piecewise linear function grows exponentially with the depth.

So it's already exponentially large with a single layer, but it gets exponentially even more with a deeper net.

Okay, so This was a topic of representation of functions, why deep learn, deep architectures can be very powerful if we're lucky, and we seem to be lucky.

The other, another topic I wanna mention that's kind of very much in the foundations is how is it that we're able to train these neural nets in the first place?

In the 90s many people decided to not do any more research on neural nets, because there were theoretical results showing that there are really an exponentially large number of local minima in the training objective of a neural net.

So in other words, The function we want to learn has many of these holes, and if we start at a random place, well, what's the chance we're going to find the best 1, the 1 that corresponds to a good cost?

And that was 1 of the motivations for people who flocked into a very large area of research in machine learning in the 90s and 2000s, based on algorithms that require only convex optimization to train.

Cuz of course, if we can do convex optimization, we eliminate this problem.

If the objective function is convex in the parameters, then we know there's a single global minimum.

You get a sense of, If you look on the right-hand top, if you draw a random function in 1D or 2D or 3D, like here is a kind of a random smooth function in 2D, you see that it's going to have many ups and downs.

But the good news is that in high dimension, it's a totally different story.

We're talking about the parameters of the model, and the vertical axis is the cost that we're trying to minimize.

And what happens in high dimension is that instead of having a huge number of local minima on our way when we're trying to optimize, what we encounter instead is a huge number of saddle points.

So saddle point is like the thing on the bottom right in 2D.

So you have 2 parameters and the y-axis is the cost you want to minimize.

And so what you see in a saddle point is you have dimensions or directions where the objective function draws a minimum, so there's a curve that curves up, and in other directions it curves down.

So a saddle point has both a minimum in some direction and a maximum in other directions.

So this is interesting because even though these points, like saddle points and minima, are places where you could get stuck.

In principle, if you're exactly at the saddle point, you don't move, but if you move a little bit away from it, you will go down the saddle, right?

So what our work and other work from NYU, Chormanska and collaborators of Jan Lekin, showed is that actually, In very high dimension, not only, it's the issue is more saddle points than local minima, but the local minima are good.

So let me try to explain what I mean by this.

So let me show you actually first an experiment from the NYU guys.

So they did an experiment where they gradually changed the size of the neural net and they look at what looks like local minima, but they could be saddle points that are the lowest that they could obtain by training.

And what you're looking at is a distribution of errors they get from different initializations of their training.

And so what happens is that when the network is small, like the pink here on the right, there's a widespread distribution of costs that you can get depending on where you start and you're pretty high.

And if you increase the size of the network, it's like all of the local minima that you find concentrate around a particular cost.

So you don't get any of these bad local minima that you would get with a small network.

And if you increase even more the size of network, this is like a single hidden layer network, you know, not very complicated.

In other words, they all kind of converge to the same kind of costs.

So let me try to explain what's going on.

So if we go back to the picture of the saddle point, but instead of being in 2D, imagine you are in a million D.

And in fact, people have billion D networks these days.

I'm sure Andrew has even bigger ones, I'm sure.

But, so what happens in this very high dimensional space of parameters is that If things are not really bad for you, so if you imagine a little bit of randomness in the way the problem is set up, and it seems to be the case, in order to have a true local minimum, you need to have the curvature going up like this in all the billion directions.

So if there is a certain probability of this event happening that all, that this particular direction is curving up and this 1 is curving up, the probability that all of them curve up becomes exponentially small.

What you see in the bottom left is a curve that shows the training error as a function of what's called the index of the critical point, which is just the fraction of the directions which are curving down.

100% would be it's a local maximum, and anything in between is a saddle point.

So what we find is that as training progresses, we're going close to a bunch of saddle points, and none of them are local minima, otherwise we would be stuck.

And in fact, we never encounter local minima until we reach the lowest possible cost that we are able to get.

In addition, there is a theory suggesting that the local minima will actually be close in cost to the global minimum.

They will be above, and they will concentrate in a little band above the global minimum.

But that band of local minima will be close to the global minimum.

And the larger the dimension, the more this is going to be true.

So to go back to my analogy, at some point, of course, you will get local minima, even though it's unlikely when you're in the middle, when you get close to the bottom, well, you can't go lower.

So it has to rise up in all the directions.

See all Lex Fridman transcripts on Youtube

Foundations and Challenges of Deep Learning (Yoshua Bengio)