The following is a conversation with Peter Abbeel.

He's a professor at UC Berkeley and the director of the Berkeley Robotics Learning Lab.

He's 1 of the top researchers in the world working on how we make robots understand and interact with the world around them, especially using imitation and deep reinforcement learning.

This conversation is part of the MIT course on artificial general intelligence and the Artificial Intelligence Podcast.

If you enjoy it, please subscribe on YouTube, iTunes, or your podcast provider of choice, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Peter Abbeel.

You've mentioned that if there was 1 person you could meet, it would be Roger Federer.

So let me ask, when do you think we'll have a robot that fully autonomously can beat Roger Federer at tennis?

Huh, well, first, if you can make it happen for me to meet Roger, let me know.

In terms of getting a robot to beat him at tennis, it's kind of an interesting question because for a lot of the challenges we think about in AI, the software is really the missing piece, but for something like this, the hardware is nowhere near either.

To really have a robot that can physically run around, the Boston Dynamics robots are starting to get there, but still not really human level ability to run around and then swing a racket.

I don't think it's a hardware problem only.

I think it's a hardware and a software problem.

And I think they'll have independent progress.

So I'd say the hardware maybe in 10-15 years.

Well the clay, I'm not sure what's harder, grass or clay.

The clay involves sliding, which might be harder to master actually, yeah.

we can build a machine, it's a whole different question of course.

If we can say, okay, this robot can be on wheels, it can move around on wheels and can be designed differently, then I think that can be done sooner probably than a full humanoid type of setup.

What do you think a swing a racket, so you've worked at basic manipulation, how hard do you think is the task of swinging a racket with a, be able to hit a nice backhand or a forehand?

Let's say, let's say, let's say we just set up stationary, a nice robot arm, let's say, you know, a standard industrial arm and it can watch the ball come and then swing the racket.

I'm not sure it would be super hard to do.

I mean I'm sure it would require a lot, if we do it with reinforced millennium, it would require a lot of trial and error.

It's not gonna swing it right the first time around, but yeah, I don't see why I couldn't swing it the right way.

I think if you set up a ball machine, let's say on 1 side, and then a robot with a tennis racket on the other side, I think it's learnable and maybe a little bit of pre-training and simulation.

I think the swing the racket is feasible.

It'd be very interesting to see how much precision it can get.

Because I mean, that's where, I mean, some of the human players can hit it on the lines, which is very high precision.

The spin is an interesting, whether RL can learn to put a spin on the ball.

Your answer is basically, okay, for this problem it sounds fascinating, but for the general problem of a tennis player, we might be a little bit farther away.

What's the most impressive thing you've seen a robot do in the physical world?

So physically for me it's the Boston Dynamics videos always just bring home and I'm just super impressed.

Recently the robot running up the stairs, doing the parkour type thing.

I mean, yes, we don't know what's underneath.

They don't really write a lot of detail, but even if it's hard-coded underneath, which it might or might not be, just the physical abilities of doing that parkour, that's a very impressive robot right there.

Have you met Spot Mini or any of those robots in person?

Met Spot Mini last year in April at the Mars event that Jeff Bezos organizes.

They brought it out there and It was nicely following around Jeff.

When Jeff left the room, they had to follow him along, which is pretty impressive.

So I think there's some confidence to know that there's no learning going on in those robots.

The psychology of it, so while knowing that, while knowing there's not, if there's any learning going on, it's very limited.

I met Spot Mini earlier this year, and knowing everything that's going on, having one-on-one interaction, so I get to spend some time alone.

And There's immediately a deep connection on the psychological level.

Even though you know the fundamentals, how it works, there's something magical.

So do you think about the psychology of interacting with robots in the physical world?

Even you just showed me the PR2, the robot, and there was a little bit something like a face, you had a little bit something like a face, there's something that immediately draws you to it.

Do you think about that aspect of the robotics problem?

Berkeley robot for the elimination of tedious tasks.

It's very hard to not think of the robot as a person.

And it seems like everybody calls him a he for whatever reason, but that also makes it more a person than if it was a it.

And it's, it seems pretty natural to think of it that way.

I've seen Pepper many times on, on videos, but then I was at an event organized by, this was by Fidelity, and they had scripted Pepper to help moderate some sessions, and they had scripted Pepper to help moderate some sessions.

And they had scripted Pepper to have the personality of a child a little bit.

And it was very hard to not think of it as its own person in some sense, because it was just gonna jump, it would just jump into conversation, making it very interactive.

Moderate would be saying, Pepper would just jump in, hold on, how about me?

And you're just like, okay, this is like, like a person.

And even then it was hard not to have that sense of somehow there is something there.

So as we have robots interact in this physical world, is that a signal that could be used in reinforcement learning?

You've worked a little bit in this direction, But do you think that psychology can be somehow pulled in?

Yes, that's a question I would say a lot of people ask.

And I think part of why they ask it is they're thinking about how unique are we really still as people?

Like after they see some results, they see a computer play Go, they see a computer do this, that, they're like okay, but can it really have emotion?

Can it really interact with us in that way?

And then once you're around robots, you already start feeling it, And I think that kind of maybe methodologically, the way that I think of it is, if you run something like reinforcement learning, it's about optimizing some objective.

And there's no reason that the objective couldn't be tied into how much does a person like interacting with this system, and why could not the reinforcement learning system optimize for the robot being fun to be around, and why wouldn't it then naturally become more and more attractive, and more and more maybe like a person or like a pet, I don't know what it would exactly be, but more and more have those features and acquire them automatically.

As long as you can formalize an objective of what it means to like something, how you exhibit, what's the ground truth, How do you get the reward from human?

Because you have to somehow collect that information within you, human.

But you're saying if you can formulate it as an objective, it can be learned.

There's no reason it couldn't emerge through learning and maybe 1 way to formulate as an objective, you wouldn't have to necessarily score it explicitly.

So standard rewards are numbers, and numbers are hard to come by.

But much easier is for a person to say, okay, what you did the last 5 minutes was much nicer than we did the previous 5 minutes.

And in fact, there have been some results on that.

For example, Paul Cristiano and collaborators at OpenAI had the hopper, Mojoco hopper, a one-legged robot, learn to do backflips.

Purely from feedback, I like this better than that.

And after a bunch of interactions, it figured out what it was the person was asking for, namely a backflip.

It was just getting a score from, the comparison score from the person based on.

Person having in mind, in their own mind, I wanted to do a back flip, but the robot didn't know what it was supposed to be doing.

It just knew that sometimes the person said this is better, this is worse, and then the robot figured out what the person was actually after was a backflip.

And I'd imagine the same would be true for things like more interactive robots that the robot would figure out over time, oh, this kind

of thing apparently is appreciated more than this other kind of thing.

So when I first picked up Sutton's, Richard Sutton's reinforcement learning book, before sort of this deep learning, before the reemergence of neural networks as a powerful mechanism for machine learning, RL seemed to me like magic.

So that seemed like what intelligence is, RL, reinforcement learning.

So how do you think we can possibly learn anything about the world when the reward for the actions is delayed, is so sparse?

Like where is, why do you think RL works?

Why do you think you can learn anything under such sparse rewards, whether it's regular reinforcement learning or deep reinforcement learning?

The counterpart of that is, why is RL, why does it need so many samples, so many experiences to learn from.

Because really what's happening is when you have a sparse reward, you do something, maybe for like, I don't know, you take a hundred actions and then you get a reward.

And I'm like, okay, 3, not sure what that means.

And now you know that that sequence of 100 actions that you did the second time around somehow was worse than the sequence of 100 actions you did the first time around.

But that's tough to now know which 1 of those were better or worse.

Some might have been good and bad in either 1.

And so that's why you need so many experiences.

But once you have enough experiences, effectively RL is teasing that apart.

It's trying to say, okay, what is consistently there when you get a higher reward?

And what's consistently there when you get a lower reward?

And then kind of the magic of, in some sense, the policy gradient update is to say, now let's update the neural network to make the actions that were kind of present when things are good, more likely, and make the actions that are present when things are not as good, less likely.

So that is the counterpoint, but it seems like you would need to run it a lot more than you do, even though right now, people could say that RL is very inefficient, but it seems to be way more efficient than 1 would imagine on paper.

That the simple updates to the policy, the policy gradient, that somehow you can learn, exactly as you said, what are the common actions that seem to produce some good results, that that somehow can learn anything, it seems counterintuitive at least.

Yeah, so I think there's a few ways to think about this.

The way I tend to think about it mostly, originally when we started working on deep reinforcement learning here at Berkeley, which was maybe 2011, 12, 13, around that time.

John Shulman was a PhD student, initially kind of driving it forward here.

And the way we thought about it at the time was, if you think about rectified linear units, or kind of rectifier type neural networks, what do you get?

You get something that's piecewise linear feedback control.

And if you look at the literature, linear feedback control is extremely successful, can solve many, many problems surprisingly well.

I remember, for example, when we did helicopter flight, if you're in a stationary flight regime, not a non-stationary but a stationary flight regime like hover, you can use linear feedback control to stabilize a helicopter, a very complex dynamical system, but the controller is relatively simple.

And so I think that's a big part of it is that if you do feedback control, even though the system you control can be very, very complex, often relatively simple control architectures can already do a lot, But then also just linear is not good enough.

And so 1 way you can think of these neural networks is that in some sense they tile the space, which people were already trying to do more by hand or with finite state machines.

Say this linear controller here, this linear controller here.

Neural network learns to tile the space and say linear controller here, another linear controller here, but it's more subtle than that.

And so it's benefiting from this linear control aspect, it's benefiting from the tiling, but it's somehow tiling it 1 dimension at a time.

Because if let's say you have a 2 layer network, if in that hidden layer, you make a transition from active to inactive or the other way around.

That is essentially 1 axis, but not axis aligned, but 1 direction that you change.

And so you have this kind of very gradual tiling of the space where you have a lot of sharing between the linear controllers that tile the space.

And that was always my intuition as to why to expect that this might work pretty well.

It's essentially leveraging the fact that linear feedback control is so good, but of course not enough.

And this is a gradual tiling of the space with linear feedback controls that share a lot of expertise across them.

But do you think that scales to the more and more general problems of when you start going up the number of dimensions, when you start going down in terms of how often you get a clean reward signal, does that intuition carry forward to those crazier, weirder worlds that we think of as the real world?

So, I think where things get really tricky in the real world compared to the things we've looked at so far with great success in reinforcement learning is the time scales, which takes us to an extreme.

So, when you think about the real world, I mean, I don't know, maybe some student decided to do a PhD here, right?

Okay, that's a decision, that's a very high-level decision, But if you think about their lives, I mean any person's life, it's a sequence of muscle fiber contractions and relaxations and that's how you interact with the world.

And that's a very high frequency control thing, but it's ultimately what you do and how you affect the world until I guess we have brain readings, and you can maybe do it slightly differently, but typically that's how you affect the world.

And the decision of doing a PhD is like so abstract relative to what you're actually doing in the world.

And I think that's where credit assignment becomes just completely beyond what any current RL algorithm can do and we need hierarchical reasoning at a level that is just not available at all yet.

Where do you think we can pick up hierarchical reasoning?

Yeah so maybe let me highlight what I think the limitations are of what already was done 20, 30 years ago.

In fact, you'll find reasoning systems that reason over relatively long horizons, but the problem is that they were not grounded in the real world.

So people would have to hand design some kind of logical, dynamical descriptions of the world and that didn't tie into perception.

And so it didn't tie into real objects and so forth.

Now with deep learning, we start having the ability to really see with sensors, process that and understand what's in the world, And so it's a good time to try to bring these things together.

1 way to get there would be to say, deep learning can get bolted on somehow to some of these more traditional approaches.

Now, bolted on would probably mean you need to do some kind of end-to-end training where you say, my deep learning processing somehow leads to a representation that in turn uses some kind of traditional underlying dynamical systems that can be used for planning.

And that's for example the direction Aviv Tamar and Thanar Kuretach here have been pushing with causal info again, and of course other people too.

Can we somehow force it into the form factor that is amenable to reasoning?

Another direction we've been thinking about for a long time and didn't make any progress on was more information theoretic approaches.

So the idea there was that what it means to take high-level action is to take and choose a latent variable now that tells you a lot about what's going to be the case in the future, because that's what it means to take a high-level action.

I say, okay, I decide I'm going to navigate to the gas station because I need to get gas for my car.

Well, that'll now take 5 minutes to get there.

But the fact that I get there, I could already tell that from the high-level action I took much earlier.

That, we had a very hard time getting success with.

Not saying it's a dead end necessarily, but we had a lot of trouble getting that to work.

And then we start revisiting the notion of what are we really trying to achieve?

What we're trying to achieve is not necessarily hierarchy per se, but you can think about what does hierarchy give us?

We hope it would give us this better credit assignment.

Kind of what is better credit assignment is giving us?

And so faster learning is ultimately maybe what we're after.

And So that's what we ended up with the RL squared paper on learning to reinforcement learn, which at a time Rocky Duan led.

And that's exactly the meta learning approach where you say, okay, we don't know how to design hierarchy.

Let's just enter and optimize for what we want to get from it and see if it might emerge.

The maze navigation had consistent motion down hallways, which is what you want.

A hierarchical control should say, I want to go down this hallway.

And then when there is an option to take a turn, I can decide whether to take a turn or not and repeat.

Even had the notion of where have you been before or not to not revisit places you've been before.

It still didn't scale yet to the real world kind of scenarios I think you had in mind, but it was some sign of life that maybe you can meta-learn these hierarchical concepts.

I mean, it seems like through these meta-learning concepts, get at what I think is 1 of the hardest and most important problems of AI, which is transfer learning.

How far along this journey towards building general systems are we?

So there's some signs that you can generalize a little bit, but do you think we're on the right path or is totally different breakthroughs are needed to be able to transfer knowledge between different learned models?

Yeah, I'm pretty torn on this in that I think there are some very

there's just some very impressive results already, right?

I mean, I would say when, even with the initial kind of big breakthrough in 2012 with AlexNet, right?

The initial thing is, okay, great, this does better on ImageNet, hence image recognition.

But then immediately thereafter, there was of course the notion that, wow, what was learned on ImageNet and you now want to solve a new task, you can fine tune AlexNet for new tasks.

And that was often found to be the even bigger deal, that you learn something that was reusable, which was not often the case before.

Usually machine learning, you learn something for 1 scenario and that was it.

That's probably the biggest success of transfer learning to date, in terms of scope and impact.

And then recently, I feel like similar, kind of, by scaling things up, it seems like this has been expanded upon.

Like people training even bigger networks, they might transfer even better.

If you looked at, for example, some of the OpenAI results on language models and some of the recent Google results on language models, they are learned for just prediction and then they get reused for other tasks.

And so I think there is something there where somehow if you train a big enough model on enough things, it seems to transfer some deep mind results that I thought were very impressive, the Unreal results, where it was learning to navigate mazes in ways where it wasn't just doing reinforcement learning, but it had other objectives, was optimizing for.

So I think there's a lot of interesting results already.

I think maybe where it's hard to wrap my head around is to which extent or when do we call something generalization, right?

Or the levels of generalization involved in these different tasks.

You draw this, by the way, just to frame things.

I've heard you say somewhere, it's the difference between learning to master versus learning to generalize.

It's a nice line to think about, and I guess you're saying it's a gray area of what learning to master and learning to generalize, where 1 starts and

I think I might have heard this, I might have heard it somewhere else, and I think it might have been 1 of your interviews, and maybe the 1 with Yoshua Benjamin, I'm not 100% sure, but I like the example, I'm not sure who it was, but the example was essentially if you use current deep learning techniques, what we're doing to predict, let's say, the relative motion of our planets, it would do pretty well.

But then now if a massive new mass enters our solar system, it would probably not predict what will happen.

And that's a different kind of generalization.

That's a generalization that relies on the ultimate simplest, simplest explanation that we have available today to explain the motion of planets, whereas just pattern recognition could predict our current solar system motion pretty well, no problem.

And so I think that's an example of a kind of generalization that is a little different from what we've achieved so far.

And it's not clear if just regularizing more and forcing it to come up with a simpler, simpler, simpler explanation, say look, this is not simple, but that's what physics researchers do, right?

What's the simplest equation that can explain everything, right?

The master equation for the entire dynamics of the universe.

We haven't really pushed that direction as hard in deep learning, I would say.

Not sure if it should be pushed, but it seems a kind of generalization you get from that that you don't get in our current methods so far.

So I just talked to Vladimir Vapnik, for example, who's a statistician, statistical learning, and he kind of dreams of creating the E equals MC squared for learning, right, the general theory of learning.

Do you think that's a fruitless pursuit in near term, within the next several decades?

I think that's a really interesting pursuit and in the following sense, in that there is a lot of evidence that the brain is pretty modular And so I wouldn't maybe think of it as the theory maybe, the underlying theory, but more kind of the principle where there have been findings where people who are blind will use the part of the brain usually used for vision for other functions.

And even after some kind of, if people get rewired in some way, they might be able to reuse parts of their brain for other functions.

And so what that suggests is some kind of modularity.

And I think it is a pretty natural thing to strive for to see can we find that modularity?

Of course, every part of the brain is not exactly the same.

Not everything can be rewired arbitrarily.

But if you think of things like the neocortex, which is a pretty big part of the brain, that seems fairly modular from what the findings so far.

Can you design something equally modular and if you can just grow it, it becomes more capable probably.

I think that would be the kind of interesting underlying principle to shoot for that is not unrealistic.

Do you think you prefer math or empirical trial and error for the discovery of the essence of what it means to do something intelligent.

So reinforcement learning embodies both groups, right?

To prove that something converges, prove the bounds, and then at the same time, a lot of those successes are, well, let's try this and see if it works.

How do you think of those 2 parts of your brain?

maybe I would prefer we could make the progress with mathematics.

And the reason maybe I would prefer that is because often if you have something you can mathematically formalize, you can leapfrog a lot of experimentation.

See all Lex Fridman transcripts on Youtube

Pieter Abbeel: Deep Reinforcement Learning | Lex Fridman Podcast #10