This is a talk by Ilya Sutskever for course 6.S099: Artificial General Intelligence. He is the Co-Founder of OpenAI. This class is free and open to everyone. Our goal is to take an engineering approach to exploring possible paths toward building human-level intelligence for a better world.

OUTLINE:
0:00 - Introduction
0:55 - Talk
43:04 - Q&A

INFO:
Course website: https://agi.mit.edu
AI podcast: https://lexfridman.com/ai

CONNECT:
- AI Podcast: https://lexfridman.com/ai/
- Subscribe to this YouTube channel
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman

Welcome back to CIGSA 099, Artificial General Intelligence.

Today we have Ilya Tsitskever, co-founder and research director of OpenAI.

He started in the ML group in Toronto with Jeffrey Hinton, then at Stanford with Andrew Ang, co-founded DNN Research, for 3 years was a research scientist at Google Brain, and finally co-founded OpenAI.

Citations aren't everything, but they do indicate impact.

And his work, recent work, in the past 5 years has been cited over 46,000 times.

He has been the key creative intellect and driver behind some of the biggest breakthrough ideas in deep learning and artificial intelligence ever.

All right, thanks for the introduction, Lex.

I will tell you about some work we've done over the past year on meta-learning and self-play at OpenAI, and before I dive into some of the more technical details of the work, I want to spend a little bit of time talking about deep learning and why it works at all in the first place, which I think it's actually not a self-evident thing that they should work.

1 fact, it's actually a fact, it's a mathematical theorem that you can prove, is that if you could find the shortest program that does very well on your data, then you will achieve the best generalization possible.

With a little bit of modification, you can turn it into a precise theorem.

And on a very intuitive level, it's easy to see why it should be the case.

If you have some data, and you're able to find a shorter program which generates this data Then you've essentially extracted all conceivable regularity from this data into your program And then you can use this object to make the best predictions possible like if If you have data, which is so complex, but there is no way to express it as a shorter program, then it means that your data is totally random.

There is no way to extract any regularity from it whatsoever.

Now there is little known mathematical theory behind this, and the proofs of these statements are actually not even that hard.

But the 1 minor slight disappointment is that it's actually not possible, at least given today's tools and understanding, to find the best short program that explains or generates or solves your problem given your data.

This problem is computationally intractable.

The space of all programs is a very nasty space.

Small changes to your program result in massive changes to the behavior of the program, as it should be.

You have a loop, you change the inside of the loop, of course you get something totally different.

So the space of programs is so hard, at least given what we know today, search there seems to be completely off the table.

Well, if we give up on short programs, what about small circuits?

It turns out that when it comes to small circuits, you can just find the best small circuit that solves your problem using backpropagation.

And this is the miraculous fact on which the rest of AI stands.

It is the fact that when you have a circuit and you impose constraints on your circuits using data, you can find a way to satisfy these constraints using backprop, by iteratively making small changes to the weights of your neural network until its predictions satisfy the data What this means is that the computational problem that's solved by backpropagation is extremely profound It is circuit search.

Now, we know that you can't solve it always, but you can solve it sometimes.

You can solve it at those times where we have a practical data set.

It is easy to design artificial data sets for which you cannot find the best neural network but in practice that seems to be not a problem.

You can think of training a neural network as solving a neural equation in many cases where you have a large number of equation terms like this, f of xi theta equals yi.

So you've got your parameters, and they represent all your degrees of freedom.

And you use gradient descent to push the information from these equations into the parameters to satisfy them all.

And you can see that the neural network, let's say 1 with 50 layers, is basically a parallel computer that is given 50 time steps to run.

And you can do quite a lot with 50 time steps of a very, very powerful, massively parallel computer.

So for example, I think it is not widely known that you can learn to sort n n-bit numbers using a modestly sized neural network with just 2 hidden layers Which is not bad It's not self-evident Especially since we've been taught that sorting requires log n parallel steps, with a neural network you can sort successfully using only 2 parallel steps.

So there's something slightly non-obvious going on.

Now these are parallel steps of threshold neurons, so they're doing a little bit more work.

But if you've got 50 such layers, you can do quite a bit of logic, quite a bit of reasoning, all inside the neural network.

Given the data, we are able to find the best neural network.

And because the neural network is deep, because it can run computation inside of its layers, the best neural network is worth finding.

You need something, you need a model class which is worth optimizing.

And deep neural networks satisfy both of these constraints.

This is the basis on which everything else resides.

Now I want to talk a little bit about reinforcement learning.

So reinforcement learning is a framework.

It's a framework of evaluating agents in their ability to achieve goals in complicated stochastic environments.

You got an agent which is plugged into an environment as shown in the figure right here.

And for any given agent, you can simply run it many times and compute its average reward.

Now, the thing that's interesting about the reinforcement learning framework is that there exist interesting, useful reinforcement learning algorithms.

It became interesting once we realized that good algorithms exist.

Now these are not perfect algorithms, but they are good enough to do interesting things.

And all you want, the mathematical problem, is 1 where you need to maximize the expected reward.

Now, 1 important way in which the reinforcement learning framework is not quite complete is that it assumes that the reward is given by the environment.

The agent sends an action while the reward sends it an observation and both the observation and the reward backwards.

That's what the environment communicates back.

The way in which this is not the case in the real world is that we figure out what the reward is from the observation.

We are not told, the environment doesn't say, hey, here's some negative reward.

It's our interpretation of our senses that lets us determine what the reward is.

And there is only 1 real true reward in life, and this is existence or non-existence.

And everything else is a corollary of that.

It should be a neural network, because whenever you want to do something, the answer is going to be a neural network.

And you want the agent to map observations to actions.

So you let it be parameterized with a neural net, and you apply learning algorithm.

So I want to explain to you how reinforcement learning works.

This is model-free reinforcement learning.

The reinforcement learning has actually been used in practice everywhere.

But it's also deeply, it's very robust, it's very simple.

This is literally the 1 sentence description of what happens.

And compare the result to your expectation.

If the result surprises you, if you find that the result exceeded your expectation, then change your parameters to take those actions in the future.

This is the full idea of reinforcement learning.

Try it out, see if you like it, and if you do, do more of that in the future.

Now it turns out it's not difficult to formalize mathematically, but this is really what's going on.

If in a neural network, in a regular neural network, like this, you might say, okay, what's the goal?

You run the neural network, you get an answer.

And whatever difference you have between those 2, you send it back to change the neural network.

In reinforcement learning, you run a neural network, you add a bit of randomness to your action, and then if you like the result, your randomness turns into the desired target in effect.

Without explaining what these equations mean, the point is not really to derive them, but just to show that they exist.

There are 2 classes of reinforcement learning algorithms.

1 of them is the policy gradient, Where basically what you do is that you take this expression right there, the sum of rewards, and you just crunch through the derivatives, you expand the terms, you run, you do some algebra, and you get a derivative.

And miraculously, the derivative has exactly the form that I told you, which is try some actions, and if you like them, increase the log probability of the actions.

It's very nice when the intuitive explanation has a one-to-one correspondence to what you get in the equation.

Even though you'll have to take my word for it if you're not familiar with it.

Then there is a different class of reinforcement learning algorithms, which is a little bit more difficult to explain.

It's called the Q-learning based algorithms.

They are a bit less stable, a bit more sample efficient, and it has the property that it can learn not only from the data generated by the actor, but from any other data as well.

So it has a different robustness profile, which would be a little bit important, but it's only gonna be a technicality.

So yeah, this is the on-policy, off-policy distinction, but it's a little bit technical, So if you find this hard to understand, don't worry about it.

If you already know it, then you already know it.

So now what's the potential reinforcement learning?

The reinforcement learning algorithms of today are already useful and interesting and especially if you have a really good simulation of your world you could train agents to do lots of interesting things.

But what's really exciting is if you can build a super amazing sample efficient reinforcement learning algorithm.

We just give it a tiny amount of data and the algorithm just crunches through it and extracts every bit of entropy out of it in order to learn in the fastest way possible.

Now today our algorithms are not particularly data efficient.

But as our field keeps making progress, this will change.

Next, I want to dive into the topic of meta-learning.

So, meta-learning is a beautiful idea that doesn't really work, but it kind of works.

Perhaps we could use those learning algorithms in order to learn to learn.

That'd be nice if we could learn to learn.

You would take a system which You train it not on 1 task, but on many tasks, and you ask it that it learns to solve these tasks quickly.

Here's how most traditional meta-learning looks like.

You have a model which is a big neural network.

But what you do is that you treat every, instead of training cases, you have training tasks.

And instead of test cases, you have test tasks.

So your input may be, instead of just your current test case, it would be all the information about the test tasks plus the test case.

And you'll try to output the prediction or action for that test case.

So Basically you say, yeah, I'm gonna give you your 10 examples as part of your input to your model.

Figure out how to make the best use of them.

You turn the neural network into the learning algorithm by turning a training task into a training case.

And so there've been several success stories which I think are very interesting.

1 of the success stories of meta-learning is learning to recognize characters quickly.

So there's been a data set produced by MIT by Lake et al.

And this is a data set, We have a large number of different handwritten characters.

And people have been able to train extremely strong meta-learning system for this task.

Another very successful example of meta-learning is that of neural architecture search by Zop and Li from Google where they found a neural architecture that solved 1 problem well, a small problem and then it would successfully solve large problems as well.

So this is kind of the small number of bits meta-learning.

It's like when you learn the architecture or maybe even learn a program, a small program or learning algorithm, which you apply to new tasks.

So this is the other way of doing meta-learning.

So anyway, but the point is, what's happening, What's really happening in meta-learning in most cases is that you turn a training task into a training case and pretend that this is totally normal deep learning.

So now that I've finished the introduction section, I want to start discussing different work by different people from OpenAI.

And I wanna start by talking about hindsight experience replay.

There's been a large effort by Andrijovic-Cheral to develop a learning algorithm for reinforcement learning that doesn't solve just 1 task but it solves many tasks and it learns to make use of its experience in a much more efficient way.

And I wanna discuss 1 problem in reinforcement learning.

It's actually, I guess, a set of problems which are related to each other.

But 1 really important thing you need to learn to do is to explore.

You start out in an environment, you don't know what to do.

So 1 very important thing that has to happen is that you must get rewards from time to time.

If you try something and you don't get rewards, then how can you learn?

So that's the kind of the crux of the problem.

And relatedly, Is there any way to meaningfully benefit from the experience, from your attempts, from your failures?

If you try to achieve a goal and you fail, can you still learn from it?

Instead of asking your algorithm to achieve a single goal, you want to learn a policy that can achieve a very large family of goals.

For example, instead of reaching 1 state, you want to learn a policy that reaches every state of your system.

Anytime you do something, you achieve some state.

So let's suppose you say, I want to achieve state A.

I try my best and I end up achieving state B.

I can either conclude, well that was disappointing, I haven't learned almost anything.

I still have no idea how to achieve state A.

But alternatively I can say, well wait a second, I've just reached a perfectly good state, which is B.

Can I learn how to achieve state B from my attempt to achieve state A?

And I just wanna point out, this is the 1 case, there's a small subtlety here, which may be interesting to those of you who are very familiar with the distinction between on policy and off policy.

When you try to achieve A, you are doing on-policy learning for reaching the state A, but you're doing off-policy learning for reaching the state B, because you would take different actions if you would actually try to reach state B.

So that's why It's very important that the algorithm you use here can support off-policy learning.

The crux of the idea is you make the problem easier by ostensibly making it harder.

By training a system which aspires to reach, to learn to reach every state, to learn to achieve every goal, to learn to master its environment in general, you build a system which always learns something.

It learns from success as well as from failure.

Because if it tries to do 1 thing and it does something else, it now has training data for how to achieve that something else.

I wanna show you a video of how this thing works in practice.

So 1 challenge in reinforcement learning systems is the need to shape the reward.

It means that at the beginning of the system, at the start of learning, when the system doesn't know much, it will probably not achieve your goal.

And so it's important that you design your reward function to give it gradual increments to make it smooth and continuous so that even when the system is not very good it achieves the goal.

Now if you give your state your system a very sparse reward where the reward is achieved only when you reach a final state then it becomes very hard for normal reinforcement learning algorithms to solve a problem because naturally you never get the reward, so you never learn.

But here, because you learn from failure as well as from success, this problem simply doesn't occur.

I think, you know, let's look at the videos a little bit more.

Like, it's nice how this, it confidently and energetically moves the little green puck to its target.

It works if you do it on a physical robot as well, but we can skip it.

So I think the point is that the hindsight experience replay algorithm is directionally correct because you want to make use of all your data and not only a small fraction of it.

Now 1 huge question is where do you get the high-level states?

Where do the high-level states come from?

Because in the work I've shown you so far the system is asked to achieve low-level states.

So I think 1 thing that will become very important for these kind of approaches is representation learning and unsupervised learning.

Figure out what are the right states, what's the state space of goals that's worth achieving.

Now I want to go through some real meta-learning results, And I'll show you a very simple way of doing Sim2Real from simulation to the physical robot with meta-learning.

So, I think we can agree that in the domain of robotics, it would be nice if you could train your policy in simulation, and then somehow this knowledge would carry over to the physical robot.

Now, we can build simulators that are okay, but they can never perfectly match the real world unless you want to have an insanely slow simulator.

And the reason for that is that it turns out that simulating contacts is super hard.

And I heard somewhere, correct me if I'm wrong, that simulating friction is NP-complete.

So your simulation is just not going to match reality.

There will be some resemblance, but that's it.

So let's say 1 thing that would be nice is that if you could learn a policy that would quickly adapt itself to the real world.

Well if you want to learn a policy that can quickly adapt we need to make sure that it has opportunities to adapt during training time.

Instead of solving our problem in just 1 simulator We add a huge amount of variability to the simulator.

We say we will randomize the frictions, we will randomize the masses, the length of the different objects and their, I guess, dimensions.

So you try to randomize physics, the simulator, in lots of different ways.

And then importantly, you don't tell the policy how you randomized it.

You take your policy and you put it in an environment and it says, well, this is really tough.

I don't know what the masses are, and I don't know what the frictions are.

I need to try things out and figure out what the friction is as I get responses from the environment.

So you learn a certain degree of adaptability into the policy.

This is what happens when you just train a policy in simulation and deploy it on the physical robot.

And here the goal is to bring the hockey puck towards the red dot.

And the reason it struggles is because of the systematic differences between the simulator and the real physical robot.

So like even the basic movement is difficult for the policy because the assumptions are violated so much.

So if you do the training as I discussed, we train a recurrent neural network policy which learns to quickly infer properties of the simulator in order to accomplish the task.

You can then give it the real thing, the real physics, and it will do much better.

So now this is not a perfect technique, but it's definitely very promising.

It's promising whenever you are able to sufficiently randomize the simulator.

So it's definitely very nice to see the closed loop nature of the policy.

You can see that it would push the hockey puck and it would correct it very, very gently to bring it to the goal.

So, that was a cool application of meta-learning.

I want to discuss 1 more application of meta-learning, which is learning a hierarchy of actions.

Actually, Kevin Franz, the ancient who did it, was in high school when he wrote this paper.

So 1 thing that would be nice is if reinforcement learning was hierarchical.

If instead of simply taking micro actions, you had some kind of little subroutines that you could deploy.

Maybe the term subroutine is a little bit too crude, but if you had some idea of which action primitives are worth starting with.

Now, no 1 has been able to get actually a real value add from hierarchical reinforcement learning yet.

So far all the really cool results, all the really convincing results of reinforcement learning do not use it.

That's because we haven't quite figured out what's the right way for reinforcement learning, for hierarchical reinforcement learning.

And I just want to show you 1 very simple approach where you use meta-learning to learn a hierarchy of actions.

You have, in this specific work, you have a certain, let's say you have a certain number of low-level primitives.

And your goal is to learn low-level primitives such that when they're used inside a very brief run of some reinforcement learning algorithm, you will make as much progress as possible.

So the idea is you want to get the greatest amount of progress, you want to learn policies that result in the great, sorry, you want to learn primitives that result in the greatest amount of progress possible when used inside learning.

See all Lex Fridman transcripts on Youtube

Ilya Sutskever: OpenAI Meta-Learning and Self-Play | MIT Artificial General Intelligence (AGI)