The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

So I wanted to talk to you about unsupervised learning, and that's the area where there's been a lot of research, but compared to supervised learning that you've heard about today, like convolutional networks, unsupervised learning is not there yet.

Parts of the talk are going to be a little bit more mathematical.

I apologize for that, but I'll try to give you a gist of the foundations, the math behind these models, as well as try to highlight some of the application areas.

Well, the motivation is that the space of data that we have today is just growing.

If you look at the space of images, speech, If you look at social network data, if you look at scientific data, I would argue that most of the data that we see today is unlabeled.

So how can we develop statistical models, models that can discover interesting kind of structure in unsupervised way or semi-supervised way.

And that's what I'm interested in, as well as how can we sort of apply these models across multiple different domains.

And 1 particular framework of doing that is the framework of deep learning, where you're trying to learn hierarchical representations of data.

And, again, as I go through the talk, I'm

You can take a simple bag-of-words representation of an article or a newspaper.

You can use something that's called an autoencoder, just multiple levels, you extract some latent code, and then you get some representation out of it.

kind of structure that the model is discovering, it could be useful for visualization, for example, to see what kind of structure you see in your data.

I've tried to kind of cluster together lots of different unsupervised learning techniques, and I'll touch on some of them.

It's a little bit, you know, it's not a full set, But the way that I typically think about these models is that there's a class of

what I would call non-probabilistic models, models like sparse coding, autoencoders, clustering-based methods.

are all very, very powerful techniques, and I'll cover some of them in that talk as well.

And then there is a space of probabilistic models, and within probabilistic models you have tractable models, things like fully observed belief networks.

There's a beautiful class of models called neural autoregressive density estimators.

More recently, we've seen some successes of so-called pixel recurrent neural network models.

There is a class of so-called intractable models where you are looking at models like Boltzmann machines and models like variation autoencoder, something that's been quite, there's been a lot of development in our community, in deep learning community in that space.

Helmholtz machines, I'll tell you a little bit about what these models are and a whole bunch of others as well.

1 particular structure within these models is that when you're building these generative models of data, you typically have to specify what the distributions you're looking at.

So you have to specify what the probability of the data and generally doing some kind of approximate maximum likelihood estimation.

And then more recently, we've seen some very exciting models coming out.

These are generative adversarial networks, moment matching networks.

And this is sort of a slightly different class of models where you don't really have to specify what the density is.

And I'm going to show you some examples of that.

So my talk is going to be sort of structured.

I'd like to introduce you to the basic building blocks, models like sparse coding models, because I think that these are very important classes of models, particularly for folks who are working in industry and looking for simpler models.

Autoencoder is a beautiful class of models.

And then the second part of the talk, I'll focus more on generative models.

I'll give you an introduction into restricted-bolt machines and deep-bolt machines.

And I'll spend some time showing you some examples, some recent developments in our community, specifically in

the case of variational autoencoders, which I view them as a subclass of Helmholtz machines.

And I'll finish off by giving you an intuition about a slightly different class of models, which would be these generative adversarial networks.

But before I do that, let me just sort of give you a little bit

I know Andre's done a great job, and Richard sort of alluded to that as well.

But the idea is, if I'm trying to classify a particular image, right?

And if I say, if I'm looking at specific pixel representation, it might be difficult for me to classify what I'm seeing.

On the other hand, if I can find the right representations, the right representations for these images, and then I get the right features, so get the right structure from the data, then it might be easier for me to see what's going on with my data.

And this is sort of 1 of traditional approaches that we've seen for a long time is that you have a data, you're creating some features, and then you're running your learning algorithm.

And for the longest time, in object recognition or in audio classification, you typically use some kind of hand-designed features, and

then you start classifying what you have.

in the space of vision, there's been a lot of different

features, designs of what's the right structure we should see in the data.

In the space of audio, same thing is happening.

How can you find these right representations for your data?

And the idea behind representation learning, in particular, in deep learning, is can we actually learn these representations automatically, right?

And more importantly, can we actually learn these representations in unsupervised way, right?

By just seeing lots and lots of unlabeled data, can we achieve that?

So I wanted to sort of lower your expectations as I show you some of the results.

I think that everybody should know what it is.

It was actually, first has its roots in 96, and it was originally developed to explain early visual processing in the brain, sort of, I think of it as an edge detector.

set of data points, x1 up to xn, you'd want to learn a dictionary of basis, phi 1 up to phi k, so that every single data point can be written as a linear combination of the basis.

There is 1 constraint in that you'd want your coefficients to be sparse.

So every data point is represented as a sparse linear combination of bases, right?

So if you apply sparse coding to natural images, right?

And this was originally, has been a lot of work developed at Stanford with Andrew Ng's group.

So if you apply sparse coding to, take little patches of images and learn these bases, these dictionaries, this is how they

And they look really nice in terms of finding edge-like

So if given a new example, I can say, well, this new example can be written as a linear combination of

And taking that representation, it turns out that particular representation, the sparse representation, is quite useful as a feature representation of your data.

And in general, how do we fit these models?

Well, if I give you a whole bunch of image patches, but these don't necessarily have to be image patches, this could be little speech signals or any kind of data you're working with, you'd want to learn a dictionary of basis, you have to form, you have to solve this optimization problem, right?

So the first term here, you can think of it as a reconstruction error, which is to say, well, I take a linear combination of my basis, I want them to match my data.

And then there's a second term, which is you can think of it as a sparse penalty term, which essentially says, try to penalize my coefficients so that most of them are 0.

That way every single data point can be written as just a linear combination, sparse linear combination of the basis.

If you fix your dictionary of basis, right, 5, 1 up to 5k, and you solve for the activations, that becomes a standard lasso problem.

There's a lot of solvers for solving that particular problem.

it's a lasso problem which is fairly easy to optimize.

and you optimize for dictionary basis, then it's a well-known quadratic programming problem.

Each problem is convex, so you can sort of alternate between finding coefficients, finding basis, and so forth, so you can optimize this function.

And there's been a lot of recent work in the last 10 years of doing these things online and doing it more efficiently and so forth.

or a new image patch, and given a set of learned basis, once you have your dictionary,

you can then just solve a lasso problem to find

So in this case, given a test sample or a test patch, you can find, well, it's written as a linear combination of a subset of the basis.

And it turns out, again, that that particular representation is very useful, particularly if you're interested in classifying what you see in images.

There is no specific supervisory signal that's here.

So back in 2006, there was work done again at Stanford

that basically showed a very interesting result.

So if I give you an input like this, and these are my learned basis, remember these little edges, what happens is that you just convolve these bases.

You can get these different feature maps, much like the feature maps that we've seen in convolutional neural networks.

And then you take these feature maps and you can just do a classification.

This was done on 1 of the older datasets, the Caltech 101, which is a dataset that predates ImageNet.

And if you look at Some of the competing algorithms, if you do a simple logistic regression versus if you do PCA and then do logistic regression versus finding these features using sparse coding, you can get substantial improvements.

So that's, again, that's, and you see sparse coding popping up

in a lot of different areas, not just in deep learning, but folks who are using, looking at the medical imaging domain, in neuroscience, these are very popular models.

Because they're easy, they're easy to fit, they're easy to deal with.

So what's the interpretation of the sparse coding?

And we can think of sparse coding as finding an overcomplete representation of your data.

Now the encoding function, we can think of this encoding function, which is, well, I give you an input, find me the features or sparse coefficients or bases that make up my image.

We can think of encoding as an implicit and a very nonlinear function of x, but it's an implicit function.

And the decoder, or the reconstruction, is just a simple linear function, and it's very explicit.

You just take your coefficients and then find the right basis and get back the image or the data.

And that sort of flows naturally into the ideas of autoencoders.

The autoencoder is a general framework where if I give you an input data Let's say it's an input image you encode it you get some representation some feature representation And then you have a decoder given that representation you decoding it back into the image So you

bottom-up pass, much like in a convolutional neural network, given the image, you're doing a forward pass.

And then there is also feedback and generative or top-down pass.

Given features, you're reconstructing back the input image.

And the details of what's going inside the encoder-decoder, they matter a lot.

And obviously you need some form of constraints.

You need some of constraints to avoid learning an identity.

Because if you don't put these constraints, what you could do is just take your input, copy it to your features, and then reconstruct back.

So we need to introduce some additional constraints.

If you're dealing with binary features, if you want to extract binary features, for example, I'm going to show you later why you'd want to

your encoder through sigmoid non-linearity, much like in the neural network.

And then you have a linear decoder that reconstructs back the input.

And the way we optimize these little building blocks or these little blocks is we can just have an encoder, right, which takes your input, takes a linear combination, passes it through some nonlinearity, the sigmoid nonlinearity, it could be rectified linear units, it could be 10H nonlinearity, and then there is a decoder where you reconstruct back your original input.

So this is nothing more than a neural network with 1 hidden layer.

And typically, that hidden layer would have a smaller dimensionality than the input, so we can think of it as a bottleneck layer.

And we can determine the network parameters,

the parameters of the encoder and the parameters of the decoder by writing down the reconstruction error, and that's what the reconstruction error would look like.

Given the input, encode, decode, and make sure whatever you're decoding is as close as possible to the original input.

All right, then we can use backpropagation algorithm to train.

There is an interesting sort of relationship between autoencoders and principal component analysis.

Many of you have probably heard about PCA.

As a practitioner, if you're dealing with large data and you want to see what's going on, PCA is the first thing to use, much like logistic regression.

And the idea here is that if the parameters of encoder and decoder are shared and you actually have The hidden layer, which is a linear layer, so you don't use any nonlinearities Then it turns out that the space The latent space

is going to be the same space as the space discovered by PCA.

It effectively will collapse the principal component analysis, right?

connection because it basically says that autoencoders, you can think of them as nonlinear extensions of PCA.

So you can learn a little richer features if you are using autoencoders.

If you're dealing with binary input, Sometimes we're dealing with like MNIST, for example.

Again, your encoder and decoder could use sigmoid nonlinearities.

So given an input, you extract some binary features, give binary features, you reconstruct back the binary input.

And that actually relates to a model called the restricted Boltz machine, something that I'm going to tell you about later in the talk.

There's also other classes of models where you can say, well, I can also introduce some sparsity, much like in sparse coding, to say that I need to constrain my latent features, so my latent space to be sparse.

And that actually allows you to learn quite reasonable features, nice features.

Here's 1 particular model called predictive sparse decomposition, where you effectively, if you look at the first part of the equation here, the decoder part, that pretty much looks like a sparse coding model.

But in addition, you have an encoding part that essentially says, train an encoder such that it actually approximates what my latent code should be.

So effectively, you can think of this model as there is an encoder, there

is a decoder, but then you put the sparsity constraint on your latent representation.

other thing that we've been doing in the last,

you know, 7, 8, and 10 years is, well, what you can do is you can actually

So you can learn low-level features, try to learn high-level features, and so forth.

And perhaps at the top level, if you're trying to solve a classification problem, you can do that.

And this is sometimes known as a greedy layer wise learning.

And this is sometimes useful whenever you have lots and lots of unlabeled data, and when you have a little labeled data, right, a small sample of labeled data.

Typically these models help you find meaningful representations such that you don't need a lot of labeled data to solve a particular task that you're trying to solve.

And this is, again, you can remove the decoding part, and then you end up with a standard or a convolutional architecture.

Again, your encoder and decoder could be convolutional, And it depends on what problem you're tackling.

And typically, you can stack these things together and optimize for a particular task that you're trying to solve.

Here's an example of, I just wanted to show you Some examples, some early examples.

Back in 2006, this was a way of trying to build these nonlinear autoencoders.

And you can sort of pre-train these models using restricted-bolt machines or autoencoders generally, and then You can stitch them together into this deep autoencoder and back propagate through reconstruction loss.

1 thing I want to point out is that here's 1 particular example.

The second row, you're seeing faces reconstructed from a bottleneck of 30-dimensional real-valued bottlenecks.

You can think of it as just a compression mechanism.

Given the data, high dimensional data, you're compressing it down to 30-dimensional code, and then from that 30-dimensional code, you're reconstructing back the original data.

So if you look at the first row, this is the data.

The second row shows you reconstructed data.

1 thing I want to point out is that the solution here, you have a much sharper representation, which means that it's capturing a little bit more structure in the data.

It's also kind of interesting to see that sometimes these models tend to, how should I say it, they tend to regularize your data, right?

Like for example, if you see this person with glasses, removes the glasses, And that generally has to do with the fact that there is only 1 person with glasses.

So the model just basically says, that's noise.

Or it sort of gets rid of mustaches, right?

Like if you see in this place, there's no mustache, right?

And then again, that has to do with the fact that there's enough capacity.

So the model might think that that's just a noise.

And if you're dealing with text type of data, this was done using a Reuters data set.

You take bag of words representation, something very simple, you can press it down to two-dimensional space and then you see what that space looks like.

And I always like to joke that the model basically discovers that European community economic policies are just next to disasters and accidents.

This is done, this was back in, I think the

I think today it's probably going to become closer

But again, this is just the way Typically auto encode is

a way of compression Or trying to do dimensionality reduction, but we'll see later that they don't have to be

Okay, there's another class of algorithm called semantic hashing which is to say well What if you take your data and compress it down to binary representation?

Because if you have binary representation you can search in the binary space very efficiently.

In fact, if you can compress your data down to 20-dimensional binary code, 2 to the 20 is about 4 gigabytes, So you can just store everything in memory and you can look at the, you know, just do memory lookups without actually doing any search at

So this sort of representation sometimes have been used successfully in computer vision where you take your images and then you learn these binary representations, you know, 30 dimensional codes, 200 dimensional codes, and it turns out it's very efficient to search through large volumes of data using binary representation.

So you can, you know, it takes a fraction of a millisecond to retrieve images from a set of millions and millions of images.

And again, this is also an active area of research right now because people are trying to figure out, we have these large databases, How can you search through them efficiently?

And learning a semantic hashing function that maps your data to the binary presentation turns out to be quite useful.

Okay, now let's step back a little bit and say let's now look at generative models.

Let's look at probabilistic models and how different they are.

you some examples of where they're applicable.

trying to learn a distribution of these handwritten characters.

So we have Sanskrit, we have Arabic, we have Cyrillic.

And now we can build a model that says, well, can you actually generate me what a Sanskrit should look like?

The flickering you see at the top, these are neurons.

And what you're seeing at the bottom is you're seeing what the model generates, what it believes Sanskrit should look like.

Right, so in some sense, when you think about generative models, you think about models that can generate or they can sample the distribution or they can sample the data.

We have about 25,000 characters coming from 50 different alphabets around the world.

This is 1 of the older models, but this is what the model believes Sanskrit should look like.

And I think that I've asked a couple of people to say that, does

See all Lex Fridman transcripts on Youtube

Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU)