The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

So yeah, so I was asked to give this presentation on the foundations of deep learning, which is mostly going over basic feedforward neural networks and motivating a little bit deep learning and some of the more recent developments and some of the topics that you'll see across the next 2 days.

So as Andrew mentioned, I have just an hour, so I'm gonna go fairly quickly on a lot of these things, which I think will mostly be fine if you're familiar enough with some machine learning and a little bit about neural nets.

But if you'd like to go into some of the more specific details, you can go check out my online lectures on YouTube.

It's now taught by a much younger version of myself.

And so just search for Hugo Larochelle, and I am not the guy doing a bunch of skateboarding, I'm the geek teaching about neural nets, so go check those out if you want more details.

But so what I'll cover today is, I'll start with just describing and laying out the notation on feed-for-all neural networks, that is, models that take an input vector x, that might be an image or some text, and produces an output f of x.

So I'll just describe for propagation and the different types of units and the type of functions we can represent with those.

And then I'll talk about how we actually train neural nets, describing things like loss functions, back propagation, that allows us to get a gradient for training with stochastic gradient descent, and mention a few tricks of the trade, so some of the things we do in practice to successfully train neural nets.

And then I'll end by talking about some developments that are specifically useful in the context of deep learning, that is neural networks with several hidden layers, that came out at the very after the beginning of deep learning, say, in 2006, that is things like dropout, batch normalization, and if I have some time, unsupervised pre-training.

So let's get started and talk about assuming we have some neural network, how do they actually function, how do they make predictions?

So a multilayer feed-forward neural network is a model that takes as input some vector X, which I'm representing here with a different node for each of the dimensions in my input vector, so each dimension is essentially a unit in that neural network, and then it eventually produces at its output layer an output, and we'll focus on classification mostly, so you will have multiple units here, And each unit would correspond to 1 of the potential classes in which we would want to classify our input.

So if we're identifying digits in handwritten character images, and say we're focusing on digits, you'd have 10 digits.

So you would have a sort of 0 from 0 to 9.

And to produce an output, the neural net will go through a series of hidden layers, and those will be essentially the components that introduce non-linearity that allows us to capture and perform very sophisticated types of classification functions.

So if we have L hidden layers, the way we compute all of the layers in our neural net is as follows.

We first start by computing what I'm going to call a pre-activation.

I'm going to note that a, and I'm going to index the layers by k.

So ak is just the pre-activation at layer k.

And that is only simply going to be a linear transformation of the previous layer.

So I'm going to note hk as the activation in the layer, and by default, I will assume that layer 0 is going to be the input.

And so using that notation, the preactivation at layer k is going to correspond to taking the activation at the previous layer, k minus 1, multiplying it by a matrix, wk, those are the parameters of the layer, those essentially correspond to the connections between the units, between adjacent layers, and I'm going to add a bias vector that's another parameter in my layer.

So that gives me the pre-activation, and then next I'm going to get a hidden layer activation by applying an activation function.

This will introduce some non-linearity in the model.

So I'm going to call that function G, and we'll go over a few choices that we have, common choices for the activation function.

And so I do this from layer 1 to layer L, and when it comes to the output layer, I'll also compute a pre-activation by performing a linear transformation.

But then I'll usually apply a different activation function depending on the problem I'm trying to solve.

So having said that, let's go to some of the choices for the activation function.

So, some of the activation functions you will see, 1 common 1 is the sigmoid activation function, it is this function here, it is just 1 divided by 1 plus the exponential of minus the pre-activation.

The shape of this function, you can focus on that, is this here.

It takes the pre-activation, which can vary from minus infinite to plus infinite, and it squashes this between 0 and 1.

So it's bounded by below and above, below by 0, and above by 1.

So it's a function that saturates if you have very large or very large magnitude positive or negative preactivations.

Another common choice is the hyperbolic tangent or tangent activation function, this picture here, so it squashes everything, but instead of being between 0 and 1, it is between minus 1 and 1.

And 1 that has become quite popular in neural nets is what is known as the rectified linear activation function, or in papers, you will see the relu unit that refers to the use of this activation function.

So this 1 is different from the others in that it is not bounded above, but it is bounded below.

And it is actually, it will output zeros exactly if the preactivation is negative.

So those are the choices of activation functions for the hidden layers.

And for the output layer, if we're performing classification, as I said, in our output layer, we will have as many units as there are classes in which an input could belong.

And what we'd like is potentially, and what we often do is interpret each unit's activation as the probability, according to the neural network, that the input belongs to the corresponding class, that its label, y, is the corresponding class, c.

So c would be like the index of that unit in the output layer.

So we need an activation function that produces probabilities, produces a multinomial distribution over all the different classes, and the activation function we use for that is known as the soft max activation function.

It is simply as follows, you take your preactivations and you exponentiate them, so that's going to give us positive numbers, and then we divide each of the exponentiated preactivations by the sum of all the exponentiated preactivations.

So because I'm normalizing this way, it means that all my values in my output layer are going to sum to 1, and they are positive because I took the exponential, so I can interpret that as a multinomial distribution over the choice of all the C different classes.

So that's what I will use as the activation function at the output layer.

And now, beyond the math in terms of conceptually and also in the way we are going to program neural networks, often what we will do is all of these different operations, the linear transformations, the different types of activation functions, we will implement all of them as an object, and object that take arguments, and the arguments would essentially be what are the things that are being combined to produce the next value.

So for instance, we would have an object that might correspond to the computation of preactivation, which would take as argument what is the weight matrix and the bias vector for that layer and take some layer to transform, and that would, this object would sort of compute its value by applying the linear transformation.

And then we might have objects that correspond to specific activation functions, so like a sigmoid object, or a tanh object, or a ReLU object.

And we just combine these objects together, chain them into what ends up being a graph, which I refer to as a flow graph, that represents the computation done when you do a forward pass in your neural network, up until you reach the output layer.

So I mention it now because you'll see the different softwares that we presented over the weekend will essentially sort of exploit some of that representation of the computation in neural nets, and it will also be handy for computing gradients, which I'll talk about in a few minutes.

And so that's how we perform predictions in neural networks, so we get an input, we eventually reach an output layer that gives us a distribution over classes if we are performing classification.

If I want to actually classify, I would just assign the class corresponding to the unit that has the highest activation, that would correspond to classifying to the class that has the highest probability according to their neural net.

But then you might ask the question, OK, what kind of problems can we solve with neural networks?

Or more Technically, what kind of functions can we represent mapping from some input x into some arbitrary output?

And so, if you look at my videos, I try to give more intuition as to why we have this result here, but essentially, if we have a single hidden layer neural network, it has been shown that with a linear output, we can approximate any continuous function arbitrarily well, as long as we have enough hidden units.

So, that is, there is a value for these biases and these weights such that any continuous function, I can actually represent it as well as I want.

So this result applies if you use activation functions, nonlinear activation functions like sigmoid and tanh.

So as I said in my video, if you want a bit more intuition as to why that would be, you can go check that out.

But that's a really nice result, it means that by focusing on this family of machine learning models that are neural networks, I can pretty much potentially represent any kind of classification function.

However, this does not tell us how to find the weights and the bias values such that I can represent a given function, it doesn't tell us

network, and that is what we will discuss next.

So let's talk about that, How do we actually,

a neural network to perform good classification for that problem?

So, what we will typically do is use a framework that is very generic in machine learning, known as empirical risk minimization or structural risk minimization if you are using regularization.

So this framework essentially transforms a problem of learning as a problem of optimizing.

So what we'll do is that we'll first choose a loss function that I'm noting as L, and the loss function, it compares the output of my model, so the output layer of my neural network, with the actual target.

So I'm indexing with an exponent here with T to essentially have the index over all my different examples in my training set.

And so my loss function will tell me, is this output good or bad, given that the label is actually y?

And what I'll do, I'll also define a regularizer.

So theta here is, you can think of it as the concatenation of all of my biases and weights in my neural net.

Those are all the parameters of my neural network.

And the regularizer will penalize certain values of these weights.

So I will talk more specifically later on, for instance, you might want to have your weights not be too far from 0.

That is a frequent intuition that we implement with regularizer.

And So the optimization problem that we will try to solve when learning is to minimize the average loss of my neural network over my training examples, so summing over all training examples, I have capital T examples, plus some weight here that is known as the weight decay, some hyperparameter lambda, times my regularizer.

So in other words, I'm going to try to have my loss on my training set as small as possible over all the training example, and also try to satisfy my regularizer as much as possible.

And so now we have this optimization problem, and learning will just correspond to trying to solve this problem.

So finding this argument here over my weights and my biases.

And if I want to do this, I can just invoke some optimization procedure from the optimization community, and the 1 algorithm that you will see constantly in deep learning is stochastic gradient descent, this is the optimization algorithm that we will often use for training neural networks.

So SGD, stochastic gradient descent, functions as follows.

You first initialize all of your parameters, that is finding initial values for all of my weight matrices and all of my biases.

And then for a certain number of epochs, so an epoch will be a full pass over all my examples.

So for a certain number of full iterations over my training set, I'll draw each training example, so I pair x, input x, target y, and then I will compute what is the gradient of my loss with respect to my parameters, all of my parameters, all of my weights and all of my biases.

This is what this notation here, so the gradient of the loss function, and here I'm indexing with respect to which parameter I want the gradient.

So I'm going to compute what is the gradient of my loss function with respect to my parameters, and plus lambda times the

gradient of my regularizer as well, and I'm going to get a direction which is going

to be the direction with respect to my parameters, and plus lambda times the gradient of my regularizer as well.

And I'm going to get a direction in which I should move my parameters, since the gradient tells me how to increase the loss, I want to go in the opposite direction and decrease it, so my direction will be the opposite, so that's why I have a minus here.

And so this delta is going to be the direction in which I'll move my parameters by taking a step, and the step is just a step size alpha, which is often referred to as a learning rate, times my direction, which I just add to my current values of my parameters, my biases and my weights.

And that's going to give me my new value for all of my parameters, and I iterate like that, going over all pairs, x, y, computing my gradient, taking a step side in the opposite direction, and then doing that several times.

So that's how stochastic gradient descent works, and that's essentially the learning procedure, it's represented by this procedure.

So in this algorithm, there are a few things we need to specify to be able to implement it and execute it.

We need a loss function, a choice for the loss function.

We need a procedure that's efficient for computing the gradient of the loss with respect to my parameters.

We need to choose a regularizer if we want 1, and we need a way of initializing my parameters.

So next what I'll do is I'll go through each of these 4 different things we need to choose before actually being able to execute stochastic gradient descent.

So as I said, we will interpret the output layer as assigning probabilities to each potential class in which I can classify my input x.

Well, in this case, something that would be natural is to try to maximize the probability of the correct class, the actual class in which my example x, t belongs to.

I'd like to increase the value of the probability assigned by computed by my neural network.

And so because we set up the problem in which we have a loss that we minimize, instead of maximizing the probability, what we'll actually do is minimize the negative and the actual log probability, so the log likelihood of assigning X to the correct class Y.

So this is represented here, so given my output layer and the true label y, my loss will be minus the log of the probability of y for my neural, according to my neural net, and that would be, well, take my output layer and look at the unit, so index the unit corresponding to the correct class.

We take the log because numerically it turns out to be more stable.

And sometimes in certain softwares, you'll see instead of talking about the

negative log likelihood or log probability, you will see it referred as the cross-entropy.

And that's because you can think of this as performing a sum over all possible classes, and then for each class, checking, well, is this potential class the target class?

So I have an indicator function that is 1

if y is equal to c, so if my iterator class c is actually equal to the real class, I'm going to multiply that by the log of the probability actually assigned to that class C.

And this function here, so this expression here is like a cross-entropy between the empirical distribution, which assigns 0 probability to all the other classes, but a probability of 1 to the correct class, and the actual distribution over classes that my neural net is computing, which is f of x.

Here, I only mention it because in certain libraries, it's actually mentioned as the cross-entropy loss.

Then we need also a procedure for computing what is the gradient of my loss with respect to all of my parameters in my neural net, so the biases and the weights.

You can go look at my videos if you want the

of the details for all of these different expressions.

I don't have time for that, so all I will do, and presumably a lot of you have seen these derivations, if you haven't, just go check out the videos.

In any case, I'm going to go through what the algorithm is, and I'm going to highlight

points that will come up later in understanding our back propagation functions.

So the basic idea is that we will compute gradients by exploiting the chain rule, and we'll go from the top layer all the way to the bottom, computing gradients for layers that are closer and closer to the input as we go, and exploiting the chain rule to exploit or reuse previous computations we've made at upper layers to compute the gradients at the layers below.

So we usually start by computing what is the gradient at the output layer, and then I compute the gradient at the upper layers to compute

the output layer, so what is the gradient of my loss with respect to my output layer, and actually it is more convenient to compute the loss with respect to the pre-activation, it is a simple expression, so that's why I have the gradient of this vector, al plus 1, that's the pre-activation at the very last layer of the loss function, which is minus the log f of x, y.

this gradient is super simple, it is minus E of y, so that is the one-hot vector for class y.

So what this means is E of y is just a vector filled with a bunch of zeros, and

So if y was the fourth class, then in this case, it would be this vector, where I have a 1 at the fourth dimension.

So E of y is a vector, we call it the one-hot vector, full of zeros, and the single 1 at the position corresponding to the correct class.

So what this part of the gradient is essentially saying is that I'm going to increase, I want to increase the probability of the correct class.

I want to increase the pre-activation, which will increase the probability of the correct class.

And I'm going to subtract what is the current probabilities assigned by my neural net to all of the classes.

So f of x, that's my output layer, and that's the current beliefs of the neural net as to in which class, what's the probability of assigning the input to each class.

So what this is doing is essentially trying to decrease the probability of everything and specifically decrease it as much as the neural net currently believes that the input belongs to it.

And so if you think about the subtraction of these 2 things, Well, for the class that's the correct class, I'm going to have 1 minus some number between 0 and 1, because it's a probability.

So I'm going to increase the probability of the correct class.

And for everything else, it's going to be 0 minus a positive number.

So I'm actually going to decrease the probability of everything else.

So intuitively, it makes sense, this gradient has the right behavior.

And I'm going to take that preactivation gradient, I'm going to propagate it from the top to the bottom, and essentially iterating from the last layer, which is the output layer, L plus 1, all the way down to the first layer.

And as I'm going down, I'm going to compute the gradients with respect to my parameters, and then compute what's the gradient for the pre-activation at the layer below, and then iterate like that.

So at each iteration of that loop, I take what is the current gradient of the loss function with respect to the pre-activation at the current layer, and I can compute the gradient of the loss function with respect to my weight matrix.

So not doing the derivation here, it is actually simply this vector, so in my notation, I assume that all the vectors are column vectors, so this pre-activation gradient vector, and I multiply it by the transpose of the activations, so the value of the layer right below, the layer k minus 1.

So because I take the transpose, that's a multiplication like this, and you can see if I do the outer product, essentially, between these 2 vectors, I'm going to get a matrix of the same size as my weight matrix.

Turns out that the gradient of the loss with respect to the bias is exactly the gradient of the loss with respect to the pre-activation.

So that gives me now my gradients for my parameters, and now I need to compute, okay, what is going to be the gradient of the pre-activations at the layer below.

Well, first I'm going to get the gradient of the loss function with respect to the activation at the layer below.

But that's just taking my pre-activation gradient vector and multiplying it by, for some reason, does it show here?

And multiply by the transpose of my weight matrix.

Super simple operation, just a linear transformation of my gradients at layer k, linear transform to get my gradients of the activation at the layer k minus 1.

And then to get the gradients of the pre-activation, so before the activation function, I'm going to take this gradient here, which is the gradient of the activation function, of the layer k minus 1, and then I apply the gradient corresponding to the partial derivative of my nonlinear activation function.

So this here, this refers to an element-wise product.

So I'm taking these 2 vectors, this vector here and this vector here.

I'm going to do an element-wise product between the 2.

And this vector here is just the partial derivative of the activation function for each unit individually that I've put together into a vector.

Now the key things to notice is first that this pass, computing all the gradients and doing all these iterations is actually fairly cheap.

Complexity is essentially the same as the 1 that's doing a forward pass.

So all I'm doing are linear transformations multiplying by matrices, in this case, the transpose of my weight matrix.

And then I'm also doing this sort of nonlinear operation where I'm multiplying by the gradient of the activation function.

And the second thing to notice is that here I'm doing this element-wise product.

So if any of these terms here for a unit is very close to 0, then the pre-activation gradient is going to be 0 for the next layer.

And I highlight this point because, essentially, whenever that's something to think about a lot when you're training neural nets.

Whenever this gradient here, these partial derivatives, come close to 0, then it means the gradient will not propagate well to the next layer, which means that you are not going to get a good gradient to update your parameters.

When will you see these terms here being close to 0?

Well, that's going to be when the partial derivatives of these nonlinear activation functions are close to 0 or 0.

So we can look at the partial derivatives, say, of the sigmoid function.

It's just the sigmoid itself times 1 minus the sigmoid itself.

So that means that whenever the activation of the unit for a sigmoid unit is close to 1 or close to 0, I essentially get a partial derivative that's close to 0.

You can kind of see it here, the slope here is essentially flat, and the slope here is flat, that's the value of the partial derivative.

So in other words, if my preactivations are very negative or very positive, so if my unit is very saturated, then gradients will have a hard time propagating to the next layer.

So it turns out the partial derivative is also easy to compute.

You just take the tanh value, square it, and you're going to subtract it to 1.

And indeed, if it's close to minus 1 or close to 1, you can see that the slope is flat.

So, again, if the unit is saturating, gradients will have a hard time propagating to the next layers.

And for the ReLU, the rectified linear activation function, the gradient is even simpler.

You just check whether the preactivation is greater than 0, if it is, the partial derivative is 1, if it is not, it is 0.

You are going to multiply by 1 or 0, you get a binary mask when you are performing the propagation through the ReLU.

And you can see the slope here is flat, and otherwise you have a linear function.

So actually here, the shrinking of the gradient towards 0 is even harder, it is exactly multiplying by 0 if you have a unit that is saturating below.

And beyond all the math, in terms of actually using those in practice, during the weekend, you will see 3 different libraries that allow you to compute these gradients for you.

You don't write down back prop, you use all of these modules that you have implemented, and it turns out there is a way of automatically differentiating your loss function and getting gradients for free in terms of effort, in terms of programming effort, with respect to your parameters.

So conceptually, the way you do this, and you'll see essentially 3 different libraries doing it in slightly different ways.

What you do is you augment your flow graph by adding, at the very end, the computation of your loss function.

And then each of these boxes, which are conceptually objects that are taking arguments and computing a value, you're going to augment them to also have a method that's a back prop or a B prop method, you will often see this expression being used, B prop.

And what this method should do is take as input, what is the gradient of the loss with respect to the loss, the gradient, so the things that it's parents in the flow graph, the things it takes to compute its own value, it's going to propagate them using the chain rule, what is their gradients with respect to the loss.

So what this means is that you would sort of start the process by initializing, well, the gradient of the loss with respect to itself is 1, and then you pass the B prop method here 1, and then it's going to propagate to its argument, what is, by using the chain rule, what is the gradient of the loss with respect to f of x.

And then you're going to call bprop on this object here, and it's going to compute, well, I have the gradient of the loss with respect to myself, f of x.

From this, I can compute what's the gradient of my argument, which is the pre-activation at layer 2, with respect to the loss.

So I'm going to reuse the computation I just got and update it using my, what is essentially the Jacobian.

And then I'm going to take the pre-activation here, which now knows what is the gradient of the loss with respect

I just got and update it using my, what is essentially the Jacobian.

the gradient of the loss, the preactivation, and it's going to

biases and the layer below, update them with informing them of what is the gradient of the loss with respect to themselves.

And you continue like this, essentially going through the flow graph, but in the opposite direction.

So the library torch, the basic library torch, essentially functions like this quite explicitly.

You construct, you chain these elements together, and then when you are performing back propagation, you are going in the reverse order of these chained elements.

And then you have libraries like Torch autograd, and Tieno, and TensorFlow, which you will learn about, which are doing things slightly more sophisticated there.

OK, so that's a discussion of how you actually compute gradients of the loss with respect to the parameters.

So that's another component we need in stochastic gradient descent.

choose a regularizer, 1 that is often used is the L2 regularization, so that is just the sum of the squared of all the weights, and

it is a super simple gradient to compute.

We don't regularize the biases, there is no particularly important reason for that, there are much fewer biases, so it seems less important.

And often, this L2 regularization is often referred to as weight decay.

So if you hear about weight decay, that often refers to L2 regularization.

And then finally, and this is also a very important point, you have to initialize the parameters before you actually start doing backprop.

And there are a few tricky cases you need to make sure that you don't fall into.

So the biases, often we initialize them to 0, there are certain exceptions, but for the most part we initialize them

weights, there are a few things we can't do.

See all Lex Fridman transcripts on Youtube

Foundations of Deep Learning (Hugo Larochelle, Twitter)