Basic recurrent neural networks are great, because they can handle different amounts of sequential data, but even relatively small sequences of data can make them difficult to train. This is where Long Short-Term Memory (LSTM) saves the day. Long Short-Term Memory is a type of recurrent neural network that can handle much larger sequences of data without those pesky exploding/vanishing gradient problems that plague basic recurrent neural networks.

Spanish
Este video ha sido doblado al español con voz artificial con https://aloud.area120.google.com para aumentar la accesibilidad. Puede cambiar el idioma de la pista de audio en el menú Configuración.

Portuguese
Este vídeo foi dublado para o português usando uma voz artificial via https://aloud.area120.google.com para melhorar sua acessibilidade. Você pode alterar o idioma do áudio no menu Configurações.

For a complete index of all the StatQuest videos, check out...
https://app.learney.me/maps/StatQuest
...or...
https://statquest.org/video-index/

If you'd like to support StatQuest, please consider...
Patreon: https://www.patreon.com/statquest
...or...
YouTube Membership: https://www.youtube.com/channel/UCtYLUTtgS3k1Fg4y5tAhLbw/join

...a cool StatQuest t-shirt or sweatshirt: 
https://shop.spreadshirt.com/statquest-with-josh-starmer/

...buying one or two of my songs (or go large and get a whole album!)
https://joshuastarmer.bandcamp.com/

...or just donating to StatQuest!
https://www.paypal.me/statquest

Lastly, if you want to keep up with me as I research and create new StatQuests, follow me on twitter:
https://twitter.com/joshuastarmer

0:00 Awesome song, introduction and main ideas
4:19 The sigmoid and tanh activation functions
5:58 LSTM Stage 1: The percent to remember
9:25 LSTM Stage 2: Update the long-term memory
12:42 LSTM Stage 3:Update the short-term memory
14:33 LSTM in action with real data

#StatQuest #LSTM #Dubbedwithaloud

Long, short term memories, I've got them both and so does this network, hooray,

Hello I'm Josh Starmer and welcome to StatQuest.

Today we're going to talk about Long Short Term Memory, LSTM, and it's going to be clearly explained.

This StatQuest is also brought to you by the letters A, B, and C.

Note, this stat quest assumes you are already familiar with recurrent neural networks and the vanishing exploding gradient problem.

Also note, although long short-term memory is totally awesome, it is also a stepping stone to learning about transformers, which we will talk about in future stat quests.

In other words, today we're taking the second step in our quest.

Now, in the stat quest on basic vanilla recurrent neural networks, we saw how we can use a feedback loop to unroll a network that works well with different amounts of sequential data.

However, we also saw that when we plug in the numbers, when the weight on the feedback loop is greater than 1, and in this example the weight is 2, then when we do the math we end up multiplying the input by the weight, which in this case is 2, raised to the number of times we unrolled the network.

And thus if we had 50 sequential data points, like 50 days of stock market data, which isn't really that much, then we would raise 2 by

And 2 to the 50th power is a huge number.

And this huge number would cause the gradient which we need for gradient descent to explode.

Alternatively, we saw that if the weight on the feedback loop was less than 1 and now we have it set to

then we'll end up multiplying the input value by 0.5 raised to the 50th power.

And 0.5 raised to the 50th power is a number super close to 0.

And this number super close to 0 would cause the gradient, which we need for gradient descent, to vanish.

In summary, basic vanilla recurrent neural networks are hard to train because the gradients can explode, kaboom, or vanish.

The good news is that it doesn't take much to extend the basic vanilla recurrent neural network so that we can avoid this problem.

So today we're going to talk about long short-term memory, LSTM, which is a type of recurrent neural network that is designed to avoid the exploding slash vanishing gradient problem.

The main idea behind how long short-term memory works is that instead of using the same feedback loop connection for events that happened long ago and events that just happened yesterday to make a prediction about tomorrow, long Short-term memory uses 2 separate paths to make predictions about tomorrow 1 path is for long-term memories and 1 is for short-term memories BAM!

Now that we understand the main idea behind long short-term memory that it uses different paths for long and short-term memories, let's talk about the details.

The bad news is that compared to a basic vanilla recurrent neural network which unrolls from a relatively simple unit, long short-term memory is based on a much more complicated unit.

Holy smokes, this looks really complicated!

Don't worry, Squatch, we'll go through this 1 step at a time so that you can easily understand each part.

Note, unlike the networks we've used before in this series, long short-term memory uses sigmoid activation functions and tanh activation functions.

So let's quickly talk about sigmoid and tanh activation functions.

In a nutshell, the sigmoid activation function takes any x-axis coordinate and turns it into a y-axis coordinate between 0 and 1.

For example, when we plug in this x-axis coordinate, 10, into the equation for the sigmoid activation function, we get

And if we plug in this x-axis coordinate, negative 5, then we get 0.01 as the y-axis coordinate.

In contrast, the tanh, or hyperbolic tangent activation function, takes any x-axis coordinate and turns it into a y-axis coordinate between negative 1 and 1.

For example, if we plug this x-axis coordinate 2 into the equation for the tanh activation function, we get 0.96 as the y-axis coordinate.

And if we plug in this x-axis coordinate, negative 5, we get negative 1 as the y-axis coordinate.

So now that we know that the sigmoid activation function turns any input into a number between 0 and 1, and the tanh activation function turns any input into a number between negative 1 and 1, let's talk about how the long, short-term memory unit works.

First, this green line that runs all the way across the top of the unit is called the cell state and represents the long-term memory.

Although the long-term memory can be modified by this multiplication and then later by this addition, you'll notice that there are no weights and biases that can modify it directly.

This lack of weights allows the long-term memories to flow through a series of unrolled units without causing the gradient to explode or vanish.

Now, this pink line, called the hidden state, represents the short-term memories.

And, as we can see, the short-term memories are directly connected to weights that can modify them.

To understand how the long and short-term memories interact and result in predictions, let's run some numbers through this unit.

First, for the sake of making the math interesting, let's just assume that the previous long-term memory is 2, and the previous short-term memory is 1 and let's set the input value to 1.

Now that we have plugged in some numbers let's do the math to see what happens in the first stage of the long short-term memory unit.

We'll start with the short-term memory 1 times its weight

Then we multiply the input 1 by its weight

an x-axis coordinate for the sigmoid activation function.

Now we plug the x-axis coordinate into the equation for the sigmoid activation function and we get the y-axis coordinate

Lastly, we multiply the long-term memory, 2, by the y-axis coordinate

So this first stage of the long short-term memory unit reduced the long-term memory by a little bit.

In contrast, if the input to the LSTM was a relatively large negative number like negative 10, then after calculating the x-axis coordinate the output from the sigmoid activation function will be 0.

And that means the long-term memory would be completely forgotten because anything multiplied by

Thus, because the sigmoid activation function turns any input into a number between 0 and 1, the output determines what percentage of the long-term memory is remembered.

To summarize, the first stage in a long short-term memory unit determines what percentage of the long-term memory is remembered.

Even though this part of the long short-term memory unit determines what percentage of the long-term memory will be remembered, it is usually called the forget gate.

Now that we know what the first part of the LSTM unit does, it determines what percentage of the long-term memory will be remembered.

Let's go back to when the input was 1 and talk about what the second stage does.

In a nutshell, the block on the right combines the short-term memory and the input to create a potential long-term memory.

And the block on the left determines what percentage of that potential memory to add to the long-term memory.

So let's plug the numbers in and do the math to see how a potential memory is created and how much of it is added to the long-term memory.

Starting with the block furthest to the right, we multiply the short-term memory and the input by their respective weights.

Then we add those values together and add a bias term to get

the input value for the tanh activation function.

Now we plug 2.03 into the equation for the tanh activation function, and we get the y-axis coordinate

Remember, the tanh activation function turns any input into a number between negative 1 and 1.

And, in this case, when the input to the LSTM is 1, then, after calculating the x-axis coordinate, the tanh activation function gives us an output close to 1.

In contrast, if the input to the LSTM was negative 10, then, after calculating the x-axis coordinate, the output from the tanh activation function would be negative 1.

Going back to when the input to the LSTM was 1, we have a potential memory,

based on the short-term memory and the input.

Now the LSTM has to decide how much of this potential memory to save.

And this is done using the exact same method we used earlier when we determined what percentage of the long-term memory to remember.

In other words, after multiplying the short-term memory and the input by weights, and adding those products together, and adding a bias, we get

the x-axis input value for the sigmoid activation function.

And that means the entire potential long-term memory is retained because multiplying it by 1 doesn't change it.

Note if the original input value was negative 10 then the percentage of the potential memory to remember would be 0.

So we would not add anything to the long term memory.

Now going back to when the original input value was 1, we add 0.97 to the existing long term memory, and we get a new long term memory

Even though this part of the long short-term memory unit determines how we should update the long-term memory, it's usually called the input gate.

Now that we have a new long-term memory, we're ready to talk about the final stage in the LSTM.

This final stage updates the short-term memory.

We start with the new long-term memory and use it as input to the TANH activation function.

into the TANH activation function, we get

represents a potential short-term memory.

Now the LSTM has to decide how much of this potential short-term memory to pass on.

And this is done using the exact same method we used 2 times earlier.

When we determined what percentage of the original long-term memory to remember, and when we determined what percentage of the potential long-term memory to remember.

In all 3 cases, we use a sigmoid activation function to determine what percent the LSTM remembers.

In this case, when we do the math, we get

And we create the new short-term memory by multiplying 0.99 by

This new short-term memory, 0.98, is also the output from this entire LSTM unit.

Oh no, it's the dreaded terminology alert again.

Because the new short-term memory is the output from this entire LSTM unit, this stage is called the output gate.

And, at long last, the common terminology seems reasonable to me.

Now that we understand how all 3 stages in a single LSTM unit work, let's see them in action with real data.

Here, we have stock prices for 2 companies, Company A and Company B.

On the Y axis, we have the stock value, and on the X axis, we have the day the value was recorded.

Note, if we overlap the data from the 2 companies, we see that the only differences occur on day 1 and on day 5.

On day 1, company A is at 0 and Company B is at 1.

And on Day 5, Company A returns to 0 and Company B returns to 1.

On all of the other days, Days 2, 3, and 4, both companies have the exact same values.

Given this sequential data, we want the LSTM to remember what happened on day

so it can correctly predict what will happen on day 5.

In other words, we're going to sequentially run the data from days 1 through 4 through an unrolled LSTM and see if it can correctly predict the values for day 5 for both companies.

So let's go back to the LSTM and initialize the long and short-term memories to 0.

Now, because this single LSTM unit is taking up the whole screen, let's shrink it down to this smaller simplified diagram.

Now, if we want to sequentially run company A's values from days 1 through 4 through this LSTM, then we'll start by plugging the value for day 1, which is 0, into the input.

And after doing the math, we see that the new or updated long-term memory is negative

And the new, updated short-term memory is negative

So we plug in negative 0.2 for the updated long-term memory and negative 0.1 for the updated short-term memory.

Now we unroll the LSTM using the updated memories and plug the value from day

Then the LSTM does its math using the exact same weights and biases as before.

And we end up with these updated long and short-term memories.

Note, if you can't remember the StatQuest on recurrent neural networks very well, the reason the LSTM reuses the exact same weights and biases is so that it can handle data sequences of different lengths.

Anyway, we unroll the LSTM again and plug in the value for Day 3.

Then the LSTM does the math, again using the exact same weights and biases, and gives us these updated memories.

Then we unroll the LSTM 1 last time and plug in the value for day 4.

And the LSTM does the math, again using the exact same weights and biases, and gives us the final memories.

And the final short-term memory, 0.0, is the output from the unrolled LSTM.

And that means the output from the LSTM correctly predicts Company A's value for Day 5.

Now that we have shown that the LSTM can correctly predict the value on day 5 for company A, let's show how the same LSTM with the same weights and biases can correctly predict the value on day 5 for company B.

Note, remember on days 1 through 4, the only difference between the companies occurs on day 1.

And that means the LSTM has to remember what happened on day 1 in order to correctly predict the different output values on day 5.

So let's start by initializing the long and short term memories to

Now let's plug in the value for day 1 from company B, 1.

And the LSTM does the math, just like before, using the exact same weights and biases.

After doing the math, we see that the updated long-term memory is 0.5 and the updated short-term memory is

So we plug in 0.5 for the updated long-term memory and

Now we unroll the LSTM and do the math with the remaining input values.

And the final short-term memory, 1.0, is the output from the unrolled LSTM.

And that means the output from the LSTM correctly predicts Company B's value for Day 5.

In summary, using separate paths for long-term memories and short-term memories, long short-term memory networks avoid the exploding vanishing gradient problem.

And that means we can unroll them more times to accommodate longer sequences of input data than a vanilla recurrent neural network.

At first I was scared of how complicated the LSTM was, but now I understand.

Now it's time for some shameless self-promotion.

If you want to review statistics and machine learning offline, check out the StatQuest PDF study guides and my book, the StatQuest Illustrated Guide to Machine Learning at StatQuest.org.

We've made it to the end of another exciting StatQuest!

If you like this StatQuest and want to see more, please subscribe.

And if you want to support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying 1 or 2 of my original songs or a t-shirt or a hoodie, or just donate.


Speaker 0: Long, short term memories, I've got them both and so does this network, hooray,

Speaker 1: StatQuest. Hello I'm Josh Starmer and welcome to StatQuest. Today we're going to talk about Long Short Term Memory, LSTM, and it's going to be clearly explained. This StatQuest is also brought to you by the letters A, B, and C. A, always.

BBC, curious. Always be curious. Note, this stat quest assumes you are already familiar with recurrent neural networks and the vanishing exploding gradient problem. If not, check out the quest. Also note, although long short-term memory is totally awesome, it is also a stepping stone to learning about transformers, which we will talk about in future stat quests.

In other words, today we're taking the second step in our quest. Now, in the stat quest on basic vanilla recurrent neural networks, we saw how we can use a feedback loop to unroll a network that works well with different amounts of sequential data. However, we also saw that when we plug in the numbers, when the weight on the feedback loop is greater than 1, and in this example the weight is 2, then when we do the math we end up multiplying the input by the weight, which in this case is 2, raised to the number of times we unrolled the network. And thus if we had 50 sequential data points, like 50 days of stock market data, which isn't really that much, then we would raise 2 by

Speaker 0: 50.

Speaker 1: And 2 to the 50th power is a huge number. And this huge number would cause the gradient which we need for gradient descent to explode. Kaboom! Alternatively, we saw that if the weight on the feedback loop was less than 1 and now we have it set to

Speaker 0: 0.5,

Speaker 1: then we'll end up multiplying the input value by 0.5 raised to the 50th power. And 0.5 raised to the 50th power is a number super close to 0. And this number super close to 0 would cause the gradient, which we need for gradient descent, to vanish. Poof. In summary, basic vanilla recurrent neural networks are hard to train because the gradients can explode, kaboom, or vanish.

Poof. The good news is that it doesn't take much to extend the basic vanilla recurrent neural network so that we can avoid this problem. So today we're going to talk about long short-term memory, LSTM, which is a type of recurrent neural network that is designed to avoid the exploding slash vanishing gradient problem. Hooray! The main idea behind how long short-term memory works is that instead of using the same feedback loop connection for events that happened long ago and events that just happened yesterday to make a prediction about tomorrow, long Short-term memory uses 2 separate paths to make predictions about tomorrow 1 path is for long-term memories and 1 is for short-term memories BAM!

Now that we understand the main idea behind long short-term memory that it uses different paths for long and short-term memories, let's talk about the details. The bad news is that compared to a basic vanilla recurrent neural network which unrolls from a relatively simple unit, long short-term memory is based on a much more complicated unit. Holy smokes, this looks really complicated! Don't worry, Squatch, we'll go through this 1 step at a time so that you can easily understand each part. BAM!

Note, unlike the networks we've used before in this series, long short-term memory uses sigmoid activation functions and tanh activation functions. So let's quickly talk about sigmoid and tanh activation functions. In a nutshell, the sigmoid activation function takes any x-axis coordinate and turns it into a y-axis coordinate between 0 and 1. For example, when we plug in this x-axis coordinate, 10, into the equation for the sigmoid activation function, we get

Speaker 0: 0.99995

Speaker 1: as the y-axis coordinate. And if we plug in this x-axis coordinate, negative 5, then we get 0.01 as the y-axis coordinate. In contrast, the tanh, or hyperbolic tangent activation function, takes any x-axis coordinate and turns it into a y-axis coordinate between negative 1 and 1. For example, if we plug this x-axis coordinate 2 into the equation for the tanh activation function, we get 0.96 as the y-axis coordinate. And if we plug in this x-axis coordinate, negative 5, we get negative 1 as the y-axis coordinate.

So now that we know that the sigmoid activation function turns any input into a number between 0 and 1, and the tanh activation function turns any input into a number between negative 1 and 1, let's talk about how the long, short-term memory unit works. First, this green line that runs all the way across the top of the unit is called the cell state and represents the long-term memory. Although the long-term memory can be modified by this multiplication and then later by this addition, you'll notice that there are no weights and biases that can modify it directly. This lack of weights allows the long-term memories to flow through a series of unrolled units without causing the gradient to explode or vanish. Now, this pink line, called the hidden state, represents the short-term memories.

And, as we can see, the short-term memories are directly connected to weights that can modify them. To understand how the long and short-term memories interact and result in predictions, let's run some numbers through this unit. First, for the sake of making the math interesting, let's just assume that the previous long-term memory is 2, and the previous short-term memory is 1 and let's set the input value to 1. Now that we have plugged in some numbers let's do the math to see what happens in the first stage of the long short-term memory unit. We'll start with the short-term memory 1 times its weight

Speaker 0: 2.7.

Speaker 1: Then we multiply the input 1 by its weight

Speaker 0: 1.63

Speaker 1: and then we add those 2 terms together. And lastly we add this bias

Speaker 0: 1.62

Speaker 1: to get

Speaker 0: 5.95

Speaker 1: an x-axis coordinate for the sigmoid activation function. Now we plug the x-axis coordinate into the equation for the sigmoid activation function and we get the y-axis coordinate

Speaker 0: 0.997.

Speaker 1: Lastly, we multiply the long-term memory, 2, by the y-axis coordinate

Speaker 0: 0.997,

Speaker 1: and the result is

Speaker 0: 1.99.

Speaker 1: So this first stage of the long short-term memory unit reduced the long-term memory by a little bit. In contrast, if the input to the LSTM was a relatively large negative number like negative 10, then after calculating the x-axis coordinate the output from the sigmoid activation function will be 0. And that means the long-term memory would be completely forgotten because anything multiplied by

Speaker 0: 0

Speaker 1: is 0. Thus, because the sigmoid activation function turns any input into a number between 0 and 1, the output determines what percentage of the long-term memory is remembered. To summarize, the first stage in a long short-term memory unit determines what percentage of the long-term memory is remembered. BAM! Oh no!

It's the dreaded terminology alert! Even though this part of the long short-term memory unit determines what percentage of the long-term memory will be remembered, it is usually called the forget gate. Small BAM. Now that we know what the first part of the LSTM unit does, it determines what percentage of the long-term memory will be remembered. Let's go back to when the input was 1 and talk about what the second stage does.

In a nutshell, the block on the right combines the short-term memory and the input to create a potential long-term memory. And the block on the left determines what percentage of that potential memory to add to the long-term memory. So let's plug the numbers in and do the math to see how a potential memory is created and how much of it is added to the long-term memory. Starting with the block furthest to the right, we multiply the short-term memory and the input by their respective weights. Then we add those values together and add a bias term to get

Speaker 0: 2.03,

Speaker 1: the input value for the tanh activation function. Now we plug 2.03 into the equation for the tanh activation function, and we get the y-axis coordinate

Speaker 0: 0.97.

Speaker 1: Remember, the tanh activation function turns any input into a number between negative 1 and 1. And, in this case, when the input to the LSTM is 1, then, after calculating the x-axis coordinate, the tanh activation function gives us an output close to 1. In contrast, if the input to the LSTM was negative 10, then, after calculating the x-axis coordinate, the output from the tanh activation function would be negative 1. Going back to when the input to the LSTM was 1, we have a potential memory,

Speaker 0: 0.97,

Speaker 1: based on the short-term memory and the input. Now the LSTM has to decide how much of this potential memory to save. And this is done using the exact same method we used earlier when we determined what percentage of the long-term memory to remember. In other words, after multiplying the short-term memory and the input by weights, and adding those products together, and adding a bias, we get

Speaker 0: 4.27,

Speaker 1: the x-axis input value for the sigmoid activation function. Now we plug the x-axis coordinate into the equation for the sigmoid activation function and we get the y-axis coordinate

Speaker 0: 1.0.

Speaker 1: And that means the entire potential long-term memory is retained because multiplying it by 1 doesn't change it. Note if the original input value was negative 10 then the percentage of the potential memory to remember would be 0. So we would not add anything to the long term memory. Now going back to when the original input value was 1, we add 0.97 to the existing long term memory, and we get a new long term memory

Speaker 0: 2.96.

Speaker 1: Double Bam! Oh no! It's the dreaded terminology alert! Even though this part of the long short-term memory unit determines how we should update the long-term memory, it's usually called the input gate. Tiny BAM.

Now that we have a new long-term memory, we're ready to talk about the final stage in the LSTM. This final stage updates the short-term memory. We start with the new long-term memory and use it as input to the TANH activation function. And after plugging

Speaker 0: 2.96

Speaker 1: into the TANH activation function, we get

Speaker 0: 0.99. 0.99

Speaker 1: represents a potential short-term memory. Now the LSTM has to decide how much of this potential short-term memory to pass on. And this is done using the exact same method we used 2 times earlier. When we determined what percentage of the original long-term memory to remember, and when we determined what percentage of the potential long-term memory to remember. In all 3 cases, we use a sigmoid activation function to determine what percent the LSTM remembers.

In this case, when we do the math, we get

Speaker 0: 0.99

Speaker 1: And we create the new short-term memory by multiplying 0.99 by

Speaker 0: 0.99

Speaker 1: to get

Speaker 0: 0.98.

Speaker 1: This new short-term memory, 0.98, is also the output from this entire LSTM unit. Oh no, it's the dreaded terminology alert again. Because the new short-term memory is the output from this entire LSTM unit, this stage is called the output gate. And, at long last, the common terminology seems reasonable to me. TRIPLE BAM!

Now that we understand how all 3 stages in a single LSTM unit work, let's see them in action with real data. Here, we have stock prices for 2 companies, Company A and Company B. On the Y axis, we have the stock value, and on the X axis, we have the day the value was recorded. Note, if we overlap the data from the 2 companies, we see that the only differences occur on day 1 and on day 5. On day 1, company A is at 0 and Company B is at 1.

And on Day 5, Company A returns to 0 and Company B returns to 1. On all of the other days, Days 2, 3, and 4, both companies have the exact same values. Given this sequential data, we want the LSTM to remember what happened on day

Speaker 0: 1,

Speaker 1: so it can correctly predict what will happen on day 5. In other words, we're going to sequentially run the data from days 1 through 4 through an unrolled LSTM and see if it can correctly predict the values for day 5 for both companies. So let's go back to the LSTM and initialize the long and short-term memories to 0. Now, because this single LSTM unit is taking up the whole screen, let's shrink it down to this smaller simplified diagram. Now, if we want to sequentially run company A's values from days 1 through 4 through this LSTM, then we'll start by plugging the value for day 1, which is 0, into the input.

Now, just like before, we do the math. And after doing the math, we see that the new or updated long-term memory is negative

Speaker 0: 0.20.

Speaker 1: And the new, updated short-term memory is negative

Speaker 0: 0.13.

Speaker 1: So we plug in negative 0.2 for the updated long-term memory and negative 0.1 for the updated short-term memory. Now we unroll the LSTM using the updated memories and plug the value from day

Speaker 0: 2, 0.5,

Speaker 1: into the input. Then the LSTM does its math using the exact same weights and biases as before. And we end up with these updated long and short-term memories. Note, if you can't remember the StatQuest on recurrent neural networks very well, the reason the LSTM reuses the exact same weights and biases is so that it can handle data sequences of different lengths. Small bam.

Anyway, we unroll the LSTM again and plug in the value for Day 3. Then the LSTM does the math, again using the exact same weights and biases, and gives us these updated memories. Then we unroll the LSTM 1 last time and plug in the value for day 4. And the LSTM does the math, again using the exact same weights and biases, and gives us the final memories. And the final short-term memory, 0.0, is the output from the unrolled LSTM.

And that means the output from the LSTM correctly predicts Company A's value for Day 5. Bam! Now that we have shown that the LSTM can correctly predict the value on day 5 for company A, let's show how the same LSTM with the same weights and biases can correctly predict the value on day 5 for company B. Note, remember on days 1 through 4, the only difference between the companies occurs on day 1. And that means the LSTM has to remember what happened on day 1 in order to correctly predict the different output values on day 5.

So let's start by initializing the long and short term memories to

Speaker 0: 0.

Speaker 1: Now let's plug in the value for day 1 from company B, 1. And the LSTM does the math, just like before, using the exact same weights and biases. Beep boop! After doing the math, we see that the updated long-term memory is 0.5 and the updated short-term memory is

Speaker 0: 0.28.

Speaker 1: So we plug in 0.5 for the updated long-term memory and

Speaker 0: 0.3

Speaker 1: for the updated short-term memory. Now we unroll the LSTM and do the math with the remaining input values. And the final short-term memory, 1.0, is the output from the unrolled LSTM. And that means the output from the LSTM correctly predicts Company B's value for Day 5. Double BAM!

In summary, using separate paths for long-term memories and short-term memories, long short-term memory networks avoid the exploding vanishing gradient problem. And that means we can unroll them more times to accommodate longer sequences of input data than a vanilla recurrent neural network. At first I was scared of how complicated the LSTM was, but now I understand. Triple BAM! Now it's time for some shameless self-promotion.

If you want to review statistics and machine learning offline, check out the StatQuest PDF study guides and my book, the StatQuest Illustrated Guide to Machine Learning at StatQuest.org. There's something for everyone! Hooray! We've made it to the end of another exciting StatQuest! If you like this StatQuest and want to see more, please subscribe.

And if you want to support StatQuest, consider contributing to my Patreon campaign, becoming a channel member, buying 1 or 2 of my original songs or a t-shirt or a hoodie, or just donate. The links are in the description below. Alright, until next time, quest on!

See all StatQuest with Josh Starmer transcripts on Youtube

Long Short-Term Memory (LSTM), Clearly Explained