The talks at the Deep Learning School on September 24/25, 2016 were amazing. I clipped out individual talks  from the full live streams and provided links to each below in case that's useful for people who want to watch specific talks several times (like I do). Please check out the official website (http://www.bayareadlschool.org) and full live streams below.

Having read, watched, and presented deep learning material over the past few years, I have to say that this is one of the best collection of introductory deep learning talks I've yet encountered. Here are links to the individual talks and the full live streams for the two days:

1. Foundations of Deep Learning (Hugo Larochelle, Twitter) - https://youtu.be/zij_FTbJHsk
2. Deep Learning for Computer Vision (Andrej Karpathy, OpenAI) - https://youtu.be/u6aEYuemt0M
3. Deep Learning for Natural Language Processing (Richard Socher, Salesforce) - https://youtu.be/oGk1v1jQITw
4. TensorFlow Tutorial (Sherry Moore, Google Brain) - https://youtu.be/Ejec3ID_h0w
5. Foundations of Unsupervised Deep Learning (Ruslan Salakhutdinov, CMU) - https://youtu.be/rK6bchqeaN8
6. Nuts and Bolts of Applying Deep Learning (Andrew Ng) - https://youtu.be/F1ka6a13S9I
7. Deep Reinforcement Learning (John Schulman, OpenAI) - https://youtu.be/PtAIh9KSnjo
8. Theano Tutorial (Pascal Lamblin, MILA) - https://youtu.be/OU8I1oJ9HhI
9. Deep Learning for Speech Recognition (Adam Coates, Baidu) - https://youtu.be/g-sndkf7mCs
10. Torch Tutorial (Alex Wiltschko, Twitter) - https://youtu.be/L1sHcj3qDNc
11. Sequence to Sequence Deep Learning (Quoc Le, Google) - https://youtu.be/G5RY_SUJih4
12. Foundations and Challenges of Deep Learning (Yoshua Bengio) - https://youtu.be/11rsu_WwZTc

Full Day Live Streams:
Day 1: https://youtu.be/eyovmAtoUx0
Day 2: https://youtu.be/9dXiAecyJrY

Go to http://www.bayareadlschool.org for more information on the event, speaker bios, slides, etc. Huge thanks to the organizers (Shubho Sengupta et al) for making this event happen.

So I'm going to talk about some of the core methods in deep reinforcement learning.

First I'll do a brief introduction to what Deep RL is and whether it might make sense to apply it

I'll talk about some of the core techniques.

So on the 1 hand, we have the policy gradient methods.

Q function, including Q-learning and SARSA.

And I'll talk a little at the end about what are the pros and cons of these different methods.

So first, what is reinforcement learning?

It's a branch of machine learning concerned with taking sequences of actions.

So often it's described in terms of an agent interacting with a previously unknown environment.

And it's trying to maximize some kind of reward function that we've defined, accumulated over time.

pretty much any kind of task where you have some kind

of goal that you want to achieve can be stated in these terms.

So this is an extremely general formulation.

It's just reinforcement learning where you're using neural networks as function approximators.

So the interesting thing about reinforcement learning in contrast to supervised learning is it's actually not totally obvious what you should use your neural network to approximate in reinforcement learning.

And there are different kinds of algorithms that approximate different things.

So 1 choice is to use the neural network to approximate your policy, which is how the agent chooses its actions.

Another choice is to approximate the value functions, which measure how good or bad different states are or actions.

And last, you can try to learn a model of the system, a dynamics model, which will make predictions about next states and rewards.

Okay, so I'll now give a few examples of different places where you might apply reinforcement learning and what the observations and actions would be.

So here you could imagine a robot where the observations are the camera images and the joint angles of the robot.

The actions are the joint torques you're applying.

And the reward is gonna depend on what you want the robot to do.

So this is something we, as the algorithm designer, get to define.

So the rewards could be to stay balanced, to navigate to some target location, or something more abstract like serve and protect humans.

So reinforcement learning has also been used in a lot of

Well, applications that have been practical in the past.

But for example, 1 area is inventory management.

So this is just 1 example of how you could use reinforcement learning for a decision making problem.

So you have to decide how much to stock up on of every item.

And your observations would be your current inventory levels.

Actions would be how much of each item you're gonna purchase, and reward is your profit.

So people in operations research, this is a subfield, study this kind of problem a lot.

Okay, there are also a lot of machine learning problems where people have started to apply reinforcement learning techniques.

So the idea in attention is you don't want to look at the whole input at once.

So 1 example of this is with a large image, you might want to just crop out part of it and use that, and just do detection on that part of the image.

So here, your observation would be your current image window.

Action is where to look or where to crop your image.

And reward is whether you make a classification error or not.

to, here you have to try to choose the right area of

the image to look at so you'll do the correct classification.

Reinforcement learning has also been used in structured prediction problems,

which in the past often weren't considered to be reinforcement learning problems.

But it turns out that to actually properly solve them, it actually is a reinforcement learning problem.

So machine translation, for example, So you get a sentence in the source language and you have to admit a sentence in the target language.

And you can, here your observations are the sentence in the source language.

You're admitting 1 word at a time in the target language.

And you have some reward function that looks at the whole sentence and tells you how good your translation was.

So since this is non-differentiable, and it's, yeah, you can't just differentiate through the whole thing and do gradient descent.

So it turns out you have to do, you can use a policy gradient method to optimize your translation system.

But I just wanna say a little bit about how reinforcement learning fits into the picture of all the other types of machine learning problems.

So previous courses in this series have talked about supervised learning and unsupervised learning.

So how does reinforcement learning relate to them?

So Let's just first compare it to, let's look at supervised learning.

So in supervised learning, first the environment samples an input-output pair from some distribution row.

The agent makes a prediction, y hat, using its function f.

And it receives some loss which tells it if it made the right prediction or the wrong prediction.

So the interpretation is environment asks the agent a question and then tells her the right answer.

So contextual bandits make this problem a little harder in that they give the learning agent a little bit less information.

So now the environment samples an input, but notice that there's not a correct output associated with it.

And the agent receives some cost, which is from some probability distribution.

We're sampling it from some probability distribution.

And the agent doesn't know what this probability distribution is, so that's what makes the problem hard.

So environment asks the agent a question, the agent answers, and the environment gives her a noisy score on the answer.

So this is applied, this actually has a lot of applications.

So personalized recommendations is 1 big 1 along with advertising.

a customer and you know what they liked in the past, so you have to make a prediction about what they're gonna like in the future, so you show them appropriate ads or links.

Like what book you wanna try to advertise to them or what video you want to show them.

and the supervised learning setting is you don't have access to the loss function you're trying to optimize.

So in particular, you can't differentiate through it.

We don't know the process that generates C, so we can't compute the gradient of the loss function and use that to tune the agent's parameters.

So that makes the problem a bit harder, or you have to use a different kind of algorithm.

Lastly, reinforcement learning is almost the same as the contextual bandit setting, except now the environment is stateful.

So now, instead of sampling the initial state from scratch every time step from the same distribution.

So you have some transition probability distribution called P here, where the state x sub t is conditioned on the previous state and the previous action.

a bit harder because now, well for a number of reasons.

For 1 thing, the inputs you're getting depend on what actions you're taking.

So now, that makes it harder to develop a stable, reliable algorithm, because now, as the agent starts to learn, it gets different inputs.

So that can lead to all sorts of out-of-control behavior.

And it also means you have delayed effects, because since the system is stateful, you might need to take a lot of actions to get into the right state.

So you might need to, you can't just act greedily every time step, you have to think ahead effectively.

Okay, so just to summarize these differences, there

The first 1 is you don't have full analytic access to the function you're trying to optimize, you have to query it through interaction.

Second, you're interacting with a stateful world, which means that the input you get is going

And if you just take the first of those differences between supervised learning and reinforcement learning, you get the contextual bandit setting.

are multiple, this audience probably has people with different interests.

Some people are doing research and wanna know about what's the latest in the research world, and some people want to apply these machine learning techniques to practical applications.

So this slide is for the latter group of people.

So if you're wondering, if you have some problem where you think reinforcement learning might be relevant and you're wondering if you should apply reinforcement learning.

So first, I should say that the answer might be no, it might be overkill, especially deep reinforcement learning.

So this is a set of fairly new techniques where it's not going to work out of the box very well.

And these techniques aren't that well established, so they have a lot of knobs to be tuned.

And yeah, these techniques aren't that well established at the moment.

So it might be worth investigating some other methods first.

So if your problem has a small number of parameters you're trying to optimize over.

And you have a simulator that you can just do lots of experiments on.

Then derivative-free optimization methods are likely to be better than reinforcement learning, or they're likely to be easier to get working.

So these methods just look at, you give them a black box function where you put in a parameter vector and it'll give you a noisy estimate of the score.

And these algorithms will just optimize over the parameters of that black box, I mean, that are being put into that black box.

So, yeah, there's a variety of different methods for derivative-free optimization.

But these are easier to understand than reinforcement learning, And they do kind of work out of the box.

Okay, a lot of problems are actually can be seen as contextual bandit problems and the statefulness of

For example, in advertising, this is where people look at advertising as

a contextual bandit problem most of the time, because you decide what ad to present the user with, and then they either click on it

of stateful because if you show them a terrible ad, they might just go and download ad block.

So there is, like your actions do have some repercussions.

But often you can just approximate it as being a contextual bandit problem where there is no state.

So there's a better theoretical understanding of contextual bandit problems and methods that have some guarantees.

So in that case, so if it is a contextual bandit problem, you might wanna use those kind of algorithms instead.

And lastly, the operations research field has been using

a set of methods which are just pretty much the basic algorithms, policy iteration and value iteration, but

they're well-developed ways of doing feature engineering for these problems that end up working pretty decently.

These techniques are also worth considering instead of trying to throw a big neural network at it.

Okay, so now that I've talked about why not to use deep

reinforcement learning or what it's not good for.

I'll just talk about some recent success stories in deep reinforcement learning, which are achievements that probably wouldn't have been possible using these other techniques.

So A few years ago, there was a pretty influential result by Manny et al from DeepMind where they used a deep Q-learning algorithm to play Atari games using the screen images as input.

And that's hard because you have these games, you're trying to do different things in

all these games, and some of them are kind of complicated.

So it's pretty remarkable that a simple algorithm can solve them all.

So since then, people have also solved this domain using policy gradients and another algorithm called Dagger.

So another big groundbreaking result was beating

supervised learning from expert games, plus policy gradients to fine tune the supervised learning policy.

plus value functions to make the search work better.

So a combination of techniques and reinforcement learning.

Robotic, so some of my colleagues at Berkeley had some very nice results learning in real time how to do manipulation tasks using an algorithm called Guided Policy Search using the PR2 robot.

And some of my colleagues and I have been working on robotic locomotion using policy gradient methods.

And people have been working on locomotion for a while and have been able to achieve pretty good results using very, like highly engineered domain specific methods.

But previously there hadn't been much success using general methods to solve it.

And last, there have been some recent results playing 3D games using policy gradients.

In fact, there is even a contest I heard about a couple days ago with this new VisDoom task, which is pretty nice.

Okay, so that's it for the high level overview part of this.

Now I'm gonna start getting into the actual formalism and the technical details.

Okay, so the basic object in the field of reinforcement learning is the Markov decision process.

So the Markov decision process is defined by the following components.

This is all the different states of the system.

The action space, these are all the actions the agent can take.

And you have this probability distribution,

which determines the probability of next state and reward.

So R is the reward, S prime is the next state, S and A are the actions.

So it's a conditional probability distribution.

Sometimes people split this out into a separate reward function, but that's basically an equivalent formulation.

Okay, and sometimes there's some extra objects to find.

We'll consider an initial state distribution.

So this is the world starts out in a certain state.

And the typical optimization problem you want to solve given this MDP is to maximize expected cumulative reward.

Though there are various ways of defining that more precisely, which I'll go into later.

Okay, so there are various different settings of reinforcement learning, where you define a slightly different optimization problem.

The 1 we'll be most concerned with is called the episodic setting.

So here, the agent's experience is split up into a series of episodes, which have finite length.

So in each episode, we first sample the initial state of the world from some probability distribution mu, and then the agent keeps on acting until the world ends up in some terminal state.

So just to give an example of what terminal states might be like and how an episodic reinforcement learning problem might look.

So 1 example is when termination is good and you want to terminate the episode as fast as possible.

So if we imagine setting up a task with some kind of taxi robot that should get to the destination as fast as possible, then the episode would be like 1 trip.

And it's trying to terminate the episode as fast as possible.

Another example is a waiter robot where you have a fixed length shift, but the waiter has to accumulate, it has to do as well as possible during that shift.

The waiter has to say maximize tips or customer happiness.

And then you could imagine another kind of task where termination is bad and you want the episode to last as long as possible.

So you can view life as an example of that.

But also you could imagine having a walking robot where

you want it to walk as far as possible before it falls over.

And in this setting, it's pretty easy to define what the goal is.

We just wanna maximize the expectation of

Okay, and The last object we're gonna introduce here is a policy.

So the policy is just the function that the agent uses to choose its actions.

So we have deterministic policies, which are just, the policy is denoted by pi.

So we have the action as just some function of the state.

And we also have stochastic policies where the policy is a conditional probability distribution.

So here we're just gonna make a little bit more precise the setting of the episodic MDP.

So first we sample the initial state from this distribution, mu.

Then we sample the first action from the policy, a0 from the policy.

Then we sample next state and reward from the transition probability distribution and so on until we reach a terminal state s sub t.

And then the quantity we care about is the sum of all these rewards, r0 plus r1 dot dot dot plus r sub t minus 1.

And we want to maximize, yeah, so eta of pi is just defined as the expected total reward of the policy pi.

Here's the picture that illustrates exactly the same thing.

So you can look at it as a graphical model.

Okay, and lastly, in the policy gradient section in particular, we're gonna be interested in parameterized policies.

So here we have a parameter vector, theta, which specifies exactly what the policy is.

So for example, the family of policies could be just a neural net, you have a certain neural network architecture, and theta specifies all the weights of this neural network.

So we could have a deterministic policy, of course, or a stochastic policy.

And if you're wondering, like, concretely, what would a policy look like?

I mean, how do you use a neural network to represent your policy?

It's actually exactly, you do exactly the same thing you would do if this were a classification or a regression problem.

So S here, the state here is your input and the action is your output.

So if you have a discrete action space, a discrete set of actions, then you would use a network that outputs a vector of probabilities.

The probabilities of the different actions.

And if you have a continuous action space, you would have your neural network output the mean and the diagonal of a covariance matrix of a Gaussian distribution.

So this is just like you're doing regression.

So you can use the same kind of architectures you'd use in supervised learning.

Okay, so that's it for the formalism of MDPs.

So now I'm going to go into policy gradient methods, which are 1 broad and general class of reinforcement learning methods which are quite effective.

So to give a brief overview of this, here's the intuition of what policy grading methods are gonna do.

So here, capital R means the sum of rewards of the whole episode.

So our optimization problem is, we wanna maximize the expectation of the total reward given our parameterized policy pi sub theta.

And the intuition of how our algorithm is gonna work is We're gonna collect a bunch of trajectories.

I mean, this is just run a bunch of episodes using our policy.

And then we wanna make the good trajectories more probable.

So, I mean, some of the trajectories were lucky and they were really good.

Some of them, the agent was unlucky and they were bad.

And the ones that were good, meaning there was high reward, that means the agent probably took good actions there.

See all Lex Fridman transcripts on Youtube

Deep Reinforcement Learning (John Schulman, OpenAI)