- The speaker discusses sequence to sequence learning and its application in automatically replying to emails with yes or no answers.
- They explain the steps of tokenization and normalization, as well as feature representation, to convert the emails into input-output pairs.
- The speaker describes the logistic regression approach to finding a vector that can approximate the output.
- They discuss the use of stochastic gradient descent to update the parameters and make predictions.
- The speaker introduces the problem of information loss in the representation and suggests using recurrent networks to preserve ordering information.
- They explain the computation of hidden units in recurrent networks and the process of making predictions.

- Updating matrices and computing partial derivatives is complicated, but toolkits like TensorFlow and Torch can help
- The matrix U is shared for all vertical matrices and its size is determined ahead of time
- The number of word vectors is the size of the vocabulary, typically around 20,000
- The big picture is using back-of-words representations and auto differentiation to train neural networks
- Training time depends on the size of the training set and computer power, but RNN with a hidden state of 100 should take a few hours
- The problem requires mapping between variable size input and output, with applications in auto-reply, translation, image captioning, summarization, speech transcription, conversation, and Q&A
- Recurrent networks can be configured in different ways, but the solution in the red box allows for flexibility in input and output sizes
- The solution involves predicting tokens one by one, using the previous prediction as input for the next step
- The model knows when to stop when it produces the "end" token

- The process involves sampling an email and reply, updating decoder parameters, and computing partial derivatives
- Auto differentiation in TensorFlow or Torch is recommended for computing partial derivatives
- The number of nodes can change, but the number of parameters remains fixed
- The number of steps can vary for different examples
- The length of the RNN is usually stopped at 400 steps, but it can go longer
- Prediction can be done using greedy decoding or by predicting multiple candidates
- Out-of-vocabulary words are mapped to an unknown token
- The cost function involves increasing the probability of predicting the correct word
- Personalization can be done by embedding a user as a vector
- Beam search can be truncated to reduce the number of paths
- Padding may be used for batch processing, but only with special tokens
- The user embedding can be inserted at the beginning of the model for personalization.

- The speaker addresses questions about the accuracy of predictions and the potential for the model to produce incorrect or garbage output
- They introduce an algorithm called scheduled sampling, which involves feeding the model's own generated output as input to help it recover from bad predictions
- The size of the decoder is discussed, with the suggestion to make it as large as possible but acknowledging that it may slow down the process
- The issue of learning synonyms is raised, with the speaker explaining that they have tried training the model using Word2Vec but it did not work well
- The speaker mentions that the algorithm they have shown has been used in production for answering emails in the smart reply feature of Inbox
- The diversity of the generated answers is addressed, with a heuristic algorithm used to make the beams more diverse
- The question of how to ensure the beam terminates is raised, and the speaker explains that there are no guarantees but training the model well can usually prevent it from going on indefinitely
- The speaker acknowledges that the model may generate replies that seem to imply multiple people are being invited, but this is due to the model learning from statistics and not specific email context
- The grammar of the generated replies does not require post-processing correction in this algorithm
- The speaker mentions that the model runs on a server and not on a phone
- The issue of determining which emails are suitable for smart reply is addressed, with two algorithms used to decide whether to reply and to set a threshold for relevance
- The speaker introduces the use of attention in the model to address the issue of using fixed length representations for variable length inputs

- The attention model involves computing coefficients and a softmax function
- The model is useful for translation as it focuses on relevant words
- The model can handle different word orders and punctuation through learning
- Deep networks can be used to improve results
- Vocabulary issues can be addressed through word segmentation or character splitting
- Gradient explosion can be managed by clipping the gradient
- Long short-term memory (LSTM) can be used in the model
- The model is implemented in TensorFlow
- There are various applications and variations of the model.

- The speaker discusses various applications of a specific algorithm
- Applications include image captioning, speech recognition, transcription, and Q and A
- In speech recognition, the algorithm can be used to crop input into windows and process it with an RNN
- The algorithm has an implicit language model in the output, but decoding must be done offline
- The algorithm can be used for translation, but it is not as effective as other methods for speech recognition
- The speaker answers questions about attention in translation, using the algorithm for multilingual email, and improving the model with reinforcement learning

- The model's output is preferred by humans, even if it doesn't optimize the metrics used in translation
- Adding a GAN loss is a good idea
- User input can influence the encoder and improve the output
- The model used several WMT corpora with tens of millions of sentence pairs
- The model's performance compared to Google Search auto-completion is unknown
- Sequence to sequence learning is a general framework that can be applied to many NLP tasks
- When there is limited data, dividing the problem into smaller components and merging them can be effective
- Adding new data can be done by increasing the learning rate and continuing training
- Automatic Q&A is an active area of research, with various approaches such as memory networks and dynamic memory networks
- Augmented memory can be used to improve the attention mechanism in the model
- RNN with augmented memory allows for continuous reading and writing from a memory bank
- The reading process can look at the entire memory bank or predict where to look

- Sequence to sequence learning is a framework used for automatically replying to emails with yes or no answers.
- Tokenization, normalization, and feature representation are used to convert emails into input-output pairs.
- Logistic regression is used to approximate the output vector.
- Stochastic gradient descent is used to update parameters and make predictions.
- Recurrent networks are used to preserve ordering information and make predictions.
- Toolkits like TensorFlow and Torch can assist with updating matrices and computing partial derivatives.
- The number of word vectors is determined by the vocabulary size.
- Back-of-words representations and auto differentiation are used to train neural networks.
- Training time depends on the size of the training set and computer power.
- The problem has applications in auto-reply, translation, image captioning, summarization, speech transcription, conversation, and Q&A.
- Recurrent networks can be configured in different ways, but flexibility in input and output sizes is important.
- Prediction is done one token at a time using the previous prediction as input.
- The model stops when it produces the "end" token.
- Auto differentiation in TensorFlow or Torch is recommended for computing partial derivatives.
- Personalization can be achieved by embedding a user as a vector.
- Beam search can be truncated to reduce the number of paths.
- The model has been used in production for answering emails in the smart reply feature of Inbox.
- The diversity of generated answers can be improved using a heuristic algorithm.
- Training the model well can prevent it from generating replies indefinitely.
- The model may generate replies that imply multiple people are being invited due to learning from statistics.
- The grammar of generated replies does not require post-processing correction.
- The model runs on a server, not on a phone.
- Two algorithms are used to determine which emails are suitable for smart reply and set a relevance threshold.
- Attention is used in the model to address the issue of variable length inputs.
- Deep networks can be used to improve results.
- Gradient explosion can be managed by clipping the gradient.
- Long short-term memory (LSTM) can be used in the model.
- The model is implemented in TensorFlow.
- The algorithm has various applications in image captioning, speech recognition, transcription, and Q&A.
- Adding new data can be done by increasing the learning rate and continuing training.
- Automatic Q&A is an active area of research with different approaches.
- RNN with augmented memory allows for continuous reading and writing from a memory bank.

- The paper "Reinforcement Learning Neuroturing Machines" deals with discrete actions
- Augmenting neural networks with functions like addition and subtraction can improve their ability to answer questions
- The principle of neuroprogrammer interpreters involves using functions like addition and subtraction in neural networks
- Sequence-to-sequence learning is a trend in natural language processing that requires a lot of supervised data
- Memory operation augmentation is an exciting area of research, but still a work in progress
- Context resolution is a challenge in Q&A networks, but personalization and additional knowledge can help
- SkipThoughts and autoencoders are approaches to generalize word to Vec to more than one word
- Common sense reasoning is a difficult problem, but combining modalities or ensuring unsupervised learning works may help
- Representing rules in neural networks can enable logical reasoning
- There is no specific rule of thumb for the number of sequence pairs needed to train a successful model

- The bigger the data set, the better for training a successful model
- The translation corpus for English to German is relatively small with about 3,500,000 pairs of sentences
- Despite the small data set, people have achieved state-of-the-art results
- Tips to reduce the number of sequence pairs needed include pre-training word vectors with language models like word2vec, reusing the softmax, using dropout in input embedding, and dropping random words in the input sentence
- The next session will be Yosha Bengio's closing keynote at 06:00.

- "Reinforcement Learning Neuroturing Machines" focuses on discrete actions in neural networks
- Adding and subtracting functions can improve neural networks' ability to answer questions
- Neuroprogrammer interpreters use functions like addition and subtraction in neural networks
- Sequence-to-sequence learning in natural language processing requires a lot of supervised data
- Memory operation augmentation is a promising area of research but still in progress
- Context resolution is a challenge in Q&A networks, but personalization and additional knowledge can help
- SkipThoughts and autoencoders are methods to generalize word to Vec to multiple words
- Combining modalities or using unsupervised learning may help with common sense reasoning
- Representing rules in neural networks enables logical reasoning
- There is no specific rule for the number of sequence pairs needed to train a successful model
- A larger data set is better for training a successful model
- The English to German translation corpus has about 3,500,000 sentence pairs
- Despite its small size, state-of-the-art results have been achieved with the translation corpus
- Tips to reduce the number of sequence pairs needed include pre-training word vectors, reusing softmax, using dropout, and dropping random words in input sentences
- The next session will be Yosha Bengio's closing keynote at 06:00.

The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

So number 1, I will work with you and develop the sequence to sequence learning.

And then the second part, I will play sequence to sequence in a broader context on a lot of exciting work in this area.

Now, so let's motivate this by an example.

So a week ago I came back from vacation and in my inbox I have 508 emails, un-replied emails.

And a lot of emails basically just require just yes and no answer.

So let's try to see whether we can build a system that can automatically reply these emails to say yes and no.

And for example, some of the email would be from my friend, Ann.

And she said, are you visiting Vietnam for the New Year quark?

So for now, let's ignore the author of the email and the subject, but let's focus on the content.

So let's suppose that you gather some email and some input would be something like, are you visiting Vietnam for the New Year quark?

And then another email would be, are you hanging out with us tonight?

The answer is no, because I'm quite busy.

So the third email would be, did you read the coolness paper on ResNet?

Now let's do a little bit of processing, where basically In the previous slide, we have year and comma, and then quark, and then question mark, and so on.

So let's do a little bit of processing, and then put the comma, a space between year and comma, and then quark and question mark, and so on.

So this step, a lot of people call tokenization and normalization.

Now, so and Then the second step I would do would be to do feature representation.

So in this step, what I'm going to do is the following.

I'm going to construct a 2,000 dimensional vector.

2,000 represent the size of English vocabulary.

I'm going to count how many times a particular word occurs in my email.

For example, the word R occurs once in my email, so I increase the counter.

And then U occurs once, so I increase another counter.

And then I will reserve at the end a token to just count all the words that are just out of vocabulary.

And then Now, if you do this process, you're going to convert all of your email from input to output pairs, where the input would be a fixed-length representation of 20,000 dimensional vector, and output would be either 0 or 1.

OK, so I will get, so as you said, somebody in the audience said the order of the words don't matter.

So I'm going to get back to that issue later.

Now, so that x and y, and now, your job, my job now is to try to find some w such that w times x can approximate y, y is the output, right?

So because of this problem, it has 2 categories, you can think of it as a logistic regression problem.

Now, if anybody follow the great CS229 class by Andrew, probably can formulate this very quickly.

But in very short, the algorithm comes as follows.

You try to come up with a vector for every email.

The first column will find the probability whether the email have to be answered as yes.

And then, you basically take the dot product between the first column and the stochastic gradient descent.

So you run for iteration 1 to like a million, you run for a long, long time.

You sample a random email x, and there's some reply.

And then if the reply is yes, then you want to update your w1 and w2 such that you increase the probability that the answer is yes.

no, then you are going to update W1 and W2, so that you can increase the probability of the email to be answered as no.

Now, so, because to update, I said to update the increase of probability, what does that mean?

What that means is that you find the gradient of the partial gradient

of the objective function with respect to some parameter.

So now you have to pick some alpha, which is the learning rate.

And then you say w1 is equal to w1 plus some alpha, the partial derivative of log of p1 with respect to d of w1.

Now, I cheated a little bit here because I used the log function.

It turns out because the log function is a monotonic increasing function.

So increasing p1 is equivalent to increasing the log of P1.

And usually, with this formulation, stochastic gradient descent works better.

And then you can also update W2 if the email to be replied is yes, and you can have different way to update if the reply is no.

So, and then if you have a new email coming in, then you take x and then you convert

then you compute the first probability, okay, w1 time x divided by exponential w1 time x plus exponential w2 time x, And if that probability is larger than 0.5, then you say yes.

And if that probability is less than 0.5, then you say no.

So that's how you do prediction with this.

Now, there's a problem with this representation, is that there's some information loss.

So somebody in the audience just said that the order of the words don't matter, and that's true.

Now let's fix this problem by using something called the Recurrent Network.

And I think Richard Solcher already talked about Recurrent Networks and some part of it yesterday, and Andre as well.

Now, the idea of a Recurrent Network is basically you also have fixed length representation for your input, but it actually preserves some sort of ordering information.

And the way that you compute the hidden units are the following.

So the function H of 0 is basically a hyperbolic tangent of some matrix U times the word vector for the word R.

So Richard also talked about word vectors yesterday.

So you can take word vectors coming out of word2vec, or you can just actually randomly initialize them if you want to.

Now, H of 1 would be a function of H0 and the vector for u, which is A times H of 0 plus u times v of vector u.

So everybody familiar with recurrent nets, huh?

So to make prediction with this, you tag on the label at the last step, and then you say, try to predict Y for me.

Now, here, basically, you went the way you did before.

And basically, you make update on the W matrix, which is the classifier at the top, like what I said earlier.

But you also have to update all the relevant matrices, which is the matrix U, the matrix A, and some work vectors.

So this is basically, you have to compute the partial derivative of the loss function with respect to those parameters.

Now, that's going to be very complicated.

And usually, when I do that myself, I get that wrong.

But there's a lot of toolkits out there that you can use.

You can use auto differentiation in TensorFlow, or you can call torch, or you can call Tiano to actually compute the derivatives.

And once you have the derivatives, you can just make the updates.

So the matrix U, I share for all vertical matrices.

And the size, you have to determine ahead of time.

For example, the number of column would be the size of the word vectors, but the number of rows must be like a thousand, if you want, or maybe 255, if you want.

And it depends on whether you're underfitting or overfitting to choose a bigger model or a smaller model.

And you'll compute power so that you can train a larger model or a smaller model.

So do you have the number of words in the dictionary?

So the word vectors, The number of word vectors that you use are the size of vocabulary.

So you're going to tend to end up with 20,000 word vectors.

But the size of, So that means you have 20,000 rows in matrix U, but the number of column you can, sorry, the number of column is 20,000, but the number of row would be, you have to determine that yourself.

So the big picture is I started with back-of-words representations, And then I talk about I and N as a new way to represent variable size input that can capture some ordering information.

And I talk about auto differentiation, so that you can compute the partial derivatives, and you can find auto in TensorFlow or Tiano or Torch.

Now, then I talk about stochastic gradient descent as a way to train the neural networks.

Oh, that also depends on how big your training set and how big is your computer and so on, right?

But usually, if you use RNN and if you use a hidden state of 100, it should take a couple of hours.

But it largely depends on the size of training data, because you want to iterate for a lot of, you sample a lot of emails, right?

And you want your algorithm to see as many emails as possible.

So, OK, so if you use such algorithm to just say yes, no, and yes, no, then you might end up losing a lot of friends.

Because when you say, for example, my friend asked me, are you visiting Vietnam for the New Year walk, then maybe the better answer would be, yes, see you soon.

That's a better, nicer way to approach this.

And then if my friends ask me, Are you hanging out with us tonight?

So instead of just say, no, I would say, no, I'm too busy.

So let's see how we're going to fix this.

So before I'm going to tell you the solution, I would say this problem basically requires you to map between variable size input and some variable size output.

And if you can do something like this, then there's a lot of applications.

Because you can do auto-reply, which is what we've been working on so far.

But we can also work on user to do translation, translate between English, French.

So input would be a fixed length vector, representation coming from conf net, and then output would be the cat sat on the mat.

You can do summarization, the input would be a document and output would be some summary of it.

Or you can do speech transcription, where you can have input would be speech frames and output would be words.

So basically, the input would be the conversation so far and the output would be my reply.

So let's check out what Andrej Karpathy has to say about recurrent networks.

So Andre says there's more than 1 way that you can configure neural networks to do things.

So you can use neural network to map recurrent networks to map one-to-one, right?

So at the bottom, that's the input, the green would be the hidden state, and the output would be the, what you want to predict.

Because we have many to many, so it's probably more like the last 2 to the right.

But we arrived at the solution that I said in the red box, and the reason why it's a better solution is because the size of the input and the size of output can vary a lot.

Sometimes you have smaller input, but larger output.

But sometimes you have larger input and smaller output.

So if you do the 1 in the red circle, you can be very flexible.

If you do the 1 to the extreme right, then maybe the output has to be smaller, or at least the same with the input.

So let's construct a solution that look like that.

So the input would be something like, hi, how are you, right?

And then let's put a special token, let's say the token is end.

And then you are going to predict the first token, which is am, and then you predict the second token, fine, and then you predict the third token, thanks, and then you keep going on until you predict the word end.

Now, I want to mention that in the previous set of slides, I was just talking about yes and no.

And in yes and no, you have only 2 choices.

And you can actually use the algorithm that are the logistic regression, and you can extend it to cover more than 2 choices.

And then the algorithm will just follow the same way.

Now, so this is my first solution, when I worked with 626, but it didn't work very well.

And the reason why it didn't work very well is the model never know what it actually predicted in the last step.

So it keep going and it keep synthesizing output, but it didn't know what it said.

It didn't know what decision it committed in the previous step.

So a better solution would look like this.

A better solution is you basically you feed what the model predicts in the previous step as input to the next step.

So for example, in this case, I'm going to take amp, I'm going to feed it in to the next step so that I'm going to predict the second word, which is fine, and et cetera.

So a lot of people call this concept autoregressive.

So you eat your own output and make it as your input.

Oh, whenever it produce end, then you stop.

Now, OK, so relevant architecture here would be the encode.

People also call the encoder as the recorder network in the input, and the decoder would be the recorder network in the output.

So again, so you basically, you run for a million steps.

For each iteration, you sample an email X, and a reply Y, Y would be, you know, I'm fine, thanks.

And then you sample random YT in y, and then you

parameters so that you can increase the probability that y of t is correct, given all what you've seen before.

Which is your yt minus 1, yt minus 2, et cetera, and also all the x's, right?

And then you have to compute the partial derivatives, which is your

And then you have to compute the partial derivatives to make it work.

So the computing part of the partial derivative is very difficult.

So again, I recommend you use something like auto differentiation in TensorFlow or Torch or Tiana.

So the question in the audience is that if the I and n are different for different example, and the answer is yes.

How do you evaluate that it's doing the right thing

Because it's not going to guess what you would say.

Yeah, I'm going to get to that in the next slide.

So the question is, In practice, how long would I go for the RNN?

I would say you usually stop at 400 steps or something like that, because outside of that it's going to be too long to make the update.

And compute, it's very expensive to compute.

What if you get the, you know, first of the y is wrong, then all the 60.1s

So I'm going to talk about the prediction next.

So let me go to the prediction, and then you can ask questions.

So the first algorithm that you can do is called greedy decoding.

And greedy decoding is for any incoming email x, I'm going to find, I'm going to predict the first word.

And then you find the most likely word, and then you feed back in.

And then you find the next most likely word, and then you feed back in, and et cetera.

You keep going until you see the word end, and then stop.

Or if it exceeds a certain length, you stop.

So it turns out that, so given x, you can predict more than 1 candidate.

So let's say you can predict k candidates.

So if you take 3 candidates, And then for each candidate, you're going to feed in the next step, and then you arrive at 3.

So the next step, you're going to have 9 candidates.

And then you're going to end up going that way.

So given input x, I'm going to predict the first token.

And given every first token like this, I'm going to feed back into the network, and the network will produce another 3, and et cetera.

So you're going to end up with a lot of candidates.

Well, you can traverse each beam, and then you compute the joint probability at each step.

And then you find the sequence that have the highest probability to be the sequence of choice.

This is the most complicated slide in my talk.

So there will be no out-of-vocabulary words in your algorithm, right?

So the question is, what do you do with our vocabulary words?

Now, it turns out in this algorithm, what you do is that for any word that is out of vocabulary, you create a token called unknown, and you map everything to unknown, or anything that is out of vocabulary to be unknown.

So it doesn't seem a very nice thing, but usually it works well.

There's a bunch of algorithms to address these issues.

For example, they break it into characters and things like that.

function is that you sample a random word, yt.

Let's suppose that this is my input so far, input, and I'm sample yt, let's say t is equal to 2, so which means the word fine, okay?

I want to increase the probability of the model to predict world fine.

So every time, the model will make a lot of predictions.

You have probability for the word and the probability and et cetera, and then probability for ZZZZZ.

You want the probability for the word fine to be as high as possible.

So when I'm at fine, my input would be hi, how are you, and and I'm.

And if I'm at the word thanks, my input will be hi, how are you, and I'm fine.

And I've got to get my thanks probability right.

So how do you personalize the stream and all the following features?

So the question is, how do you personalize?

So well, 1 way to do it is basically embed a user as a vector.

So let's suppose that you have a lot of users and you embed a user as a vector.

So basically, if all the sequences were the same length, the number of paths on the tree is k to the n, right?

See all Lex Fridman transcripts on Youtube

Sequence to Sequence Deep Learning (Quoc Le, Google)