The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

Now, the final module is a very simple GRU and softmax to produce the final answers.

The main difference here is that instead of just having the current, the previous hidden state, 18 minus 1 as input, we will also include the question at every time, and we will include the answer that was generated at the previous time step.

But other than that, it is our standard softmax, we use standard cross-entropy errors to minimize it.

And the beautiful thing of this model is that it is end-to-end trainable.

These 4 modules will all train based on the cross-entropy error of that final softmax.

All these different modules communicate with vectors, and we'll just have delta messages and backpropagation to train them.

Now, there's been a lot of work in the last 2 years on models like this.

In fact, Kwak will cover a lot of these really interesting models tomorrow, different types of memory, structures, and so on, and The dynamic memory network is, in some sense, 1 of those models.

1 particular model is a proper comparison because there are a lot of similarities, namely memory networks from Jason Weston.

Those basically also have inputs and scoring and attention and response mechanisms.

The main difference is that they use different kinds of basic Lego blocks for these different kinds of mechanisms.

For input, they use bag of words representations, or non-linear and linear embeddings.

For the attention and responses, they have different kinds of iteratively run functions.

The main interesting sort of difference to the DMN is that the DMN really uses recurrent neural network type sequence models for all of these different modules and capabilities.

And in some sense that helps us to have a broader range of applications that include things like sequence tagging.

And so let me go over a couple of results and experiments of this model.

So the first 1 is on this Bobby dataset that Facebook published.

It basically has a lot of these kinds of simple, logical reasoning type questions.

In fact, all these, like where's the football, those were examples from the Facebook Bobby data set.

And it also includes things like yes, no questions, simple counting, negation, some indefinite knowledge where the answer might be maybe.

Basic coreference where you have to realize who does she refer to or he.

Reasoning over time, if this happened before that, and so on.

And basically this dynamic memory network, I think, is currently the state of the art on this data set of the simple logical reasoning.

Now, the problem with this data set is that it's a synthetic data set.

And so it had only a certain set of generating human-defined generative functions that created certain patterns.

And in that sense, it's only a necessary and not a sufficient condition of solving it with sometimes 100% accuracy to real question answering.

The main interesting bit to point out here is that there are different numbers of training examples for each of these different subtasks.

And so you have basically 1,000 examples of simple negation, for instance.

And it's always a similar kind of pattern, and hence you're able to classify it very well.

Now, real language, you will never have that many examples for each type of pattern you want to learn, and so it's still, general question answering is still an open problem and non-trivial.

Now what's cool is this same architecture of allowing the model to go over inputs multiple times also got state of the art and sentiment analysis, very different kind of task.

And we actually analyzed whether it's really helpful to have multiple passes over the input, and it turns out it is.

So there are certain things like reasoning over 3 facts or counting, where you really have to have this dynamic, this episodic memory module and it goes over the input maybe 5 times.

For sentiment, it actually turns out it hurts after going over the input more than 2 times.

And that's actually 1 of the things we're now working on, is can we find a single model that does the same thing for every single input with the same weights to try to learn these different tasks?

We can actually look at a couple of fun examples of this model and what happens with tough sentiment sentences.

Generally, to be honest, sentiment, you can probably get to like 75% accuracy with some very simple models that just basically find like great words like great and wonderful and awesome.

And you'll get to something that's roughly right.

Here are some of the examples that those are the kinds of examples that you now need to get right to really try to push the state of the art further in sentiment analysis.

So here, the sentence is, in its ragged, cheap, and unassuming way the movie works.

So this sentence is incorrect, even if you allow the DMN but have this whole architecture but only allow 1 pass over the input.

Once you have 2 passes over the input, it actually learns to pay attention not just to these very strong adjectives, but in the end, actually to the movie working.

So here, these fields are essentially the gating function G that we defined, that pays attention to specific words.

And the darker it is, the larger that gate is, and the more open it is, and the more that word affects the hidden state in the episodic memory module.

So it goes over the input the first time, pays attention to cheap and unassuming and way and a little bit of works too.

But the second time, it basically figured out.

It agglomerated the facts of that sentence and then learned to pay attention more to specific words that seem more important.

My response to the film is best described as lukewarm.

So in general, a sentiment analysis, when you look at sort of Unigram scores, like the word best is basically 1 of the most positive words you could possibly use in a sentence.

And the first time the model passes over the sentence, it also pays most attention to this incredibly positive word, namely best.

But then, once it agglomerated the context, it actually realizes, well, best actually here is not used in its adjective way, but it's actually an adverb that best describes something.

And what it describes is actually lukewarm, and hence, it's actually a negative sentence.

So those are the kinds of examples that you need to get to now to appreciate improvements in sentiment analysis, where we basically also went from, on this particular data set, these are all neural network type models that started in 82.

Until then, that same data set existed for around 8 years, and none of the standard NLP models had reached above 80% accuracy.

And those are the kinds of improvements that you see across a variety of different NLP tasks now that deep learning has come and deep learning techniques are being used in NLP.

And now the last task in NLP that this model turned out to also work incredibly well on is part of speech tagging.

Now part of speech tagging is less exciting of a task, it's more of an intermediate task.

But it's still fascinating to see that after this data set has been around for over 20 years, you can still improve the state of the art with the same kind of architecture that also did well on fuzzy reasoning of sentiment and discrete logical reasoning for question answering.

Now, we had a new person join the group, Tsai Ming.

But he was more of a computer vision researcher.

And so he thought, well, could I use this great question answering module now to do visual question answering.

So combine some stuff that was going on in the group in NLP and apply it to computer vision.

And he did not have to know all of the different aspects of the code.

All he had to do was change the input module from 1 that gives you hidden states at each word over a long sequence of words and sentences to an input module that would give him vectors for sequences of regions in an image.

And he literally did not touch some of the other parts of the code.

I did have to look carefully at this input module, where, again, here, our basic Lego block that Andre introduced really well of our convolutional neural network.

And then each of the convolutional neural networks will essentially give us a 14 by 14 many vectors, 1 for each in 1 of its top states, 1 representing each region of an image, and then what we'll do is basically take those vectors and now replace the word vectors we used to have with CNN vectors and then plug them into a GRU.

Now, again, the GRU, We know as our basic Lego block, we already defined it.

1 addition here is that it would actually be a bidirectional GRU.

We'll go once from left to right in this snake-like fashion, and another 1 goes from right to left backwards.

Now, both of these will basically have a hidden state, and you can just concatenate the hidden states of both of these to compute the final hidden state for each block of the image.

And that model, too, actually achieved state-of-the-art results.

This data set has been only released last year, so everybody now works on deep learning techniques to try to solve it.

It was just too good to be true that this model we developed for NLP would work so well.

So we really dug into looking at the attention.

So what I showed you here, these g values, again, that we computed with this equation, Now, instead of paying attention to words, it paid attention to different regions in the image.

And we started basically analyzing, going through a bunch of those on the dev set, and analyzing what is it actually paying attention to.

Again, it's being trained only with the image, the question, and the final answer.

You do not get this sort of latent representation of where you should actually pay attention to in the image, in order to answer that question correctly.

So when the question was, what is the main color on the bus, it learned to actually pay attention here to that bus.

I'm like, well, OK, maybe that's not that impressive.

It's just the main object in the center of the image.

And what are the types, the type of trees are in the background?

Well, maybe it just connects tree with anything that's green and pays attention to that.

So it was neat, but not super impressive yet.

So is this in the wild kind of more interesting and actually pays attention to a man-made structure in the background?

Now to be honest, I don't think the model actually knows that there are 2 people, tries to match them, and so on.

It just finds the main person or main object in this scene.

The main object is a little baby girl, so it says girl.

The answer is night, because it's a very dark picture, at least in the sky.

This 1 is getting a little more interesting.

And it actually does pay attention to both of the arms, and then what's just below that arm.

So that's a little more interesting kind of attention visualization.

And then, for a while, we're also worried, well, what if in the data set, it just learns really well from language alone?

Yes, it pays attention to things, but maybe it will just say things that it often sees in the text.

So if I ask you what color are the bananas, you don't really have to look at an image.

In 95% of the cases, you're right just saying yellow without seeing an image.

So this 1 I was kind of excited about, because it actually paid attention to the bananas in the middle, and then did say green, and kind of overruled the prior that it would get from language alone.

What is the pattern on the cat's fur on its tail?

Pays attention mostly to the tail and says stripes.

The answer is yes, though I have to say that we later had a journalist want to do his own question.

He asked John Marker from New York Times, and we just put together this demo the night before.

And He's like, well, I want to ask my own question.

And it wasn't made for production, so it was kind of slow.

I'm like, well, you know, like, I was trying to come up with excuses.

It's kind of black background and a black hat, and it might be kind of hard to see.

And unfortunately, it got it right and said yes.

And then after the interview, I said, well, maybe let's look and see if I would just ask it myself, less stressful situation, a bunch of questions on my own.

And these are all the questions, like the first 8 questions that I could come up with.

And somewhat to my surprise, it actually got them all right.

Then I was like, well, OK, let's try to break it by asking just what's the color of some of the smallest object.

Also kind of interesting, when you ask the model what she's wearing, shorts, but when you ask about the skirt, and it still sort of is capturing that you might call this different things.

And then I was like, Well, what if I ask, is the girl about to hit the tennis ball?

And then, did the girl just hit the tennis ball?

So then I finally found a way to break it.

So it doesn't have enough of the core current statistics to understand, and again, I'm scared I could understand sort of which angle does the arm have to be in order to assume that the ball was just hit or was about to hit.

But what it basically does show us is that once it saw a lot of examples on a specific domain, it really can capture quite a lot of different things.

And now, let's see if we can get the demo up.

The best way to hope for any chance of enjoying this film is by lowering your expectations.

Again, 1 of those kinds of sentences that you have to now get correct in order to get improved performance on sentiment, and actually correctly says that this is negative.

Now, we can also actually ask that question in Chinese.

This is 1 of the beautiful things of the DMN and in general, really, of most deep learning techniques.

We don't have to be experts in a domain or even in a language to create a very, very accurate model for that language or that domain.

I'm not going to make a fool of myself trying to read that 1 out loud, but that's an interesting example.

You can also, this is the, what parts of speech are there.

You can have other things like, you know, named entities and other sequence problems.

You can also ask, what are the men wearing on the head?

And then maybe a slightly more interesting question, why are the men wearing helmets?

So, especially we're close to the circle of death here at Stanford, where a lot of bikes crash, and it's a good answer.

With that, I want to leave a couple of minutes for questions.

So basically the summary is, word vectors and recurrent neural networks are super useful building blocks.

Once you really appreciate and understand those 2 building blocks, you're kind of ready to have some fun and build more complex models.

Really in the end, this DMN is a way to combine that in a variety of new ways to a larger, more complex model.

And that's also where the state, I think, of deep learning is for natural language processing.

We've tackled a lot of these smaller sub-problems, intermediate tasks, and now we can work on more interesting, complex problems like dialogue and question answering, machine translation, and things like that.

In the dynamic memory network, you have the RNN, and you also mentioned that if you have better assumption of the input, right?

So You also worked on the tree LSTM, right?

So if you change the RNN into a tree structure, would that help?

I did my whole PhD about tree structures.

And somewhat surprising in the last couple of weeks, there are actually some new results on SNLI, the Stanford Natural Language Inference data set, where tree structures are again the state of the art.

Though I have to say that I think the dynamic memory network, by having this ability in the episodic memory to keep track of different sub-phrases and pay attention to those and then combine them over multiple passes, I think you can kind of get away with not having a tree structure.

So yes, you might have a slight improvement representing sentences as trees in your input module.

But I think they're only going to be slight.

And I think the episodic memory module that has this capability to go over the input multiple times, pay attention to certain subphrases, will capture a lot of the kinds of complexities that you might want to capture in tree structures.

So my short answer is I don't think you necessarily need it.

So if we want to apply question answering to some specific domains like health care, but we don't really have the data, we don't have question answer pairs, then what should we do?

What do you do if you want to do question answering on a complex domain and you don't have the data?

I think, and this feels maybe like a cop out, but I think it's very true both in practice and in theory.

Like, If you cannot possibly create more than 1,000 examples of anything, then maybe automating that process is not that important.

So clearly, you should be able to create some data.

And in many cases, that is the best use of your time, is just to sit down or ask the domain expert to create a lot of questions and then have people find the answers.

And then measure how they actually get to those answers.

Try to have them in a constrained environment and so on.

I think most companies, for instance, when you try to do automated email replies, which is in some ways a little bit similar to question answering.

Well, there is a nice domain because everybody had already emailed.

Now, if you had a search engine where people asked a lot of questions, then you can also use that to bootstrap and see where did they actually fail, and then take all those really tough queries where they failed, have some humans sit there and collect the data.

Now, the other answer is let's work together for the next many years on research for smaller training data set sizes and complex reasoning.

The fact of the matter for that line of research will still be, if a system has never seen a certain type of reasoning, it'll be hard for the system to pick up that type of reasoning.

I think we're going to get, with these kinds of architectures, to the space where at least if it has seen this type of reasoning, a specific type of transitive reasoning, or temporal reasoning, or cause and effect type reasoning, at least a couple hundred times, then you should be able to train a system with these kinds of models to do it.

Are these QA systems currently robust to false input or questions?

For the woman playing tennis, if you asked, what's the man holding, would it reply, there's no man?

And largely because at training time, you never try to mess with it like that.

I'm pretty sure if you added a lot of training examples where you had those, it would probably eventually pick it up.

Those would be important for real-world implementations.

So real-world implementations of this insecurity are actually kind of tricky.

I think whenever you train a system, we know we can, for instance, both steal certain classifiers by using them a lot.

We know we can fool them into classifying certain images, for instance, as others.

We have folks in the audience who worked on that exact line of work.

So I would be careful using it in security environments right now.

There was a slide where you had the input module and then there were a bunch of sentences.

Because, you know, sequence is basically made up of those individual words and say glove, you know, representations.

So were those, you know, also RNNs that were, you know, stitched together or?

So the answer there is a little complex because we have 2 papers with the DMN and the answer is different for each.

In the simplest form of that, it is actually a single GRU that goes from the first word through all the sentences as if they're 1 gigantic sequence.

But it has access to each sentence period at the end to pay special attention to the end of sentences.

And so, yes, in the simplest form, it is just a GRU that goes over all the words.

So is this a normal process to basically just concatenate all the sentences into 1 gigantic, you know?

So the answer there, and this is kind of why I split the talk into 3 different ones from words, single sentences, and then multiple sentences.

I think if you just had a single GRU that goes over everything, and now you try to reason over that entire sequence, it would not work very well.

You really need to have an additional structure, such as an attention mechanism or a pointer mechanism, that has the ability to pay attention to specific parts of your input to do that very accurately.

But yeah, in general, that's fine, as long as you have this additional mechanism.

So in recurrent neural nets, you're using sigmoids.

In visual recognition, I guess, rectified linear units were the more popular non-linearity.

Now when you look at the GRU equations here, you have these reset gates.

And so these reset gates here, you want them to essentially be between 0 and 1 so that it can either ignore this input entirely or you have it normally be part of the computation of h tildes.

So in some cases, you really do want to have sigmoids there.

But other ones, for instance, some simpler things where you actually don't have that much recurrence, such as going from 1 memory state to another, In the second iteration of this model, actually, ReLUs were good activation functions too.

Did you guys try to, after training this network, try to take these weights for the images and do object detection again?

So these weights would be augmented with text vectors.

That is a very cool idea that we did not explore.

So those attention models are pretty powerful when you have enough training data and then you can learn to make good use of the data.

But even though some of the tasks are pretty, I guess, trivial to humans, but it's hard for a model to learn.

So what do you think of, I guess, right now, even right now we have a lot of knowledge base on the web, like Wikipedia.

But what do you think about incorporate those knowledge base into those models?

I actually love that line of research too, and that was kind of what we started out with.

This semantic memory module in the simplest form is just word vectors.

I think 1 next iteration would actually be to have knowledge bases also influence the reasoning.

There's very little work on combining text and knowledge bases to do overall complex question answering that requires reasoning.

I think it's a phenomenally interesting area of research.

So were any night hints or any starting point about how

See all Lex Fridman transcripts on Youtube

Deep Learning for Natural Language Processing (Richard Socher, Salesforce)