The following is a conversation with Leslie Kaelbling.

She's recognized for her work in reinforcement learning, planning, robot navigation, and several other topics in AI.

She won the IJCAI Computers and Thought Award and was the editor-in-chief of the prestigious Journal of Machine Learning Research.

This conversation is part of the Artificial Intelligence Podcast at MIT and beyond.

If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D.

And now, here's my conversation with Leslie Kaelbling.

What made me get excited about AI, I can say that, is I read Gödel, Escher, Bach when I was in high school.

That was pretty formative for me because it exposed the interestingness of primitives and combination and how you can make complex things out of simple parts and ideas of AI and what kinds of programs might generate intelligent behavior.

So you first fell in love with AI reasoning logic versus robots?

Yeah, the robots came because My first job, so I finished an undergraduate degree in philosophy at Stanford and was about to finish master's in computer science and I got hired at SRI in their AI lab and they were building a robot.

It was a kind of a follow on to Shakey but all the Shakey people were not there anymore.

And so my job was to try to get this robot to do stuff, and that's really kind of what got me interested in robots.

So maybe taking a small step back, your bachelor's in Stanford in philosophy, did a master's and PhD in computer science, but the bachelor's in philosophy.

What elements of philosophy do you think you bring to your work in computer science?

So the part of the reason that I didn't do a computer science undergraduate degree was that there wasn't 1 at Stanford at the time.

But that there's part of philosophy and in fact Stanford has a special sub-major in something called now Symbolic Systems, which is logic, model, theory, formal, semantics of natural language.

And so That's actually a perfect preparation for working in AI and computer science.

So if you were interested in artificial intelligence, what kind of majors were people even thinking about taking?

So besides philosophies, what were you supposed to do if you were fascinated by the idea of creating intelligence?

There weren't enough people who did that for that even to be a conversation.

I mean, it's interesting, in my class, my graduating class of undergraduate philosophers, probably maybe slightly less than half went on in computer science, slightly less than half went on in law, and like 1 or 2 went on in philosophy.

Do you think AI researchers have a role, be part-time philosophers, or should they stick to the solid science and engineering without sort of taking the philosophizing tangents?

I mean, you work with robots, you think about what it takes to create intelligent beings.

Aren't you the perfect person to think about the big picture philosophy of it all?

The parts of philosophy that are closest to AI, I think, or at least the closest to AI that I think about are stuff like belief and knowledge and denotation and that kind of stuff.

And that's, you know, it's quite formal and it's like just 1 step away from the kinds of computer science work that we do kind of routinely.

I think that there are important questions still about what you can do with a machine and what you can't and so on.

Although at least my personal view is that I'm completely a materialist and I don't think that there's any reason why we can't make a robot be behaviorally indistinguishable from a human.

And the question of whether it's distinguishable internally, whether it's a zombie or not in philosophy terms, I actually don't, I don't know, and I don't know if I care too much about that.

Right, but there is a philosophical notions, they're mathematical and philosophical because we don't know so much of how difficult that is, how difficult is the perception problem, how difficult is the planning problem, how difficult is it to operate in this world successfully?

Because our robots are not currently as successful as human beings in many tasks.

The question about the gap between current robots and human beings borders a little bit on philosophy.

You know, the expanse of knowledge that's required to operate in this world, the ability to form common sense knowledge, the ability to reason about uncertainty, much of the work you've been doing, There's open questions there that, I don't know, required to activate a certain big picture view.

To me, that doesn't seem like a philosophical gap at all.

But I don't see any reason why it's more than a technical gap.

So when you mentioned AI, you mentioned SRI, and maybe can you describe to me when you first fell in love with robotics, the robots were inspired, which, so you mentioned Flaky or Shaky, Shaky Flaky, and what was the robot that first captured your imagination, what's possible?

Right, well, so the first robot I worked with was Flaky.

Shaky was a robot that the SRI people had built.

But by the time, I think, when I arrived, it was sitting in a corner of somebody's office dripping hydraulic fluid into a pan.

But it's iconic, and really, everybody should read the Shakey Tech Report because it has so many good ideas in it.

I mean, they invented A-star search and symbolic planning and learning macro operators.

They had low-level kind of configuration space planning for their robot.

They had all, this, the basic ideas of a ton of things.

Can you take a step back, does Shakey have arms, what was the job, what was the goal?

Yeah, good, Shakey was a mobile robot, but it could push objects, and so it would move things around.

And they had painted the baseboards black.

So it used vision to localize itself in a map.

It could detect objects that were surprising to it, it would plan and replan based on what it saw.

It reasoned about whether to look and take pictures.

I mean, it really had the basics of so many of the things that we think about now.

How did it represent the space around it?

So it had representations at a bunch of different levels of abstraction.

So it had, I think, a kind of an occupancy grid of some sort at the lowest level.

At the high level, it was abstract, symbolic kind of rooms and connectivity.

So I showed up at SRI and we were building a brand new robot.

As I said, none of the people from the previous project were kind of there or involved anymore.

So we were kind of starting from scratch.

He ended up being my thesis advisor and he was motivated by this idea of situated computation or situated automata.

And the idea was that the tools of logical reasoning were important, but possibly only for the engineers or designers to use in the analysis of a system, but not necessarily to be manipulated in the head of the system itself, right?

So I might use logic to prove a theorem about the behavior of my robot, even if the robot's not using logic in its head to prove theorems, right?

And so the idea was to kind of use those principles to make a robot do stuff.

But a lot of the basic things we had to kind of learn for ourselves, because I had 0 background in robotics, I didn't know anything about control, I didn't know anything about sensors.

So we reinvented a lot of wheels on the way to getting that robot to do stuff.

Do you think that was an advantage or a hindrance?

I mean, I'm big in favor of wheel reinvention, actually.

I mean, I think you learn a lot by doing it.

It's important though to eventually have the pointers so that you can see what's really going on.

But I think you can appreciate much better the good solutions once you've messed around a little bit on your own and found a bad 1.

Yeah, I think you mentioned reinventing reinforcement learning and referring to rewards as pleasures, a pleasure, I think, which I think is a nice name for it.

Do you think you could tell the history of AI, machine learning, reinforcement learning, how you think about it from the 50s to now?

So things become fashionable and then they go out and then something else becomes cool and then it goes out and so on.

And I think there's, so there's some interesting sociological process that actually drives a lot of what's going on.

Early days was kind of cybernetics and control, right?

People who made these robots that could, I don't know, try to plug into the wall when they needed power and then come loose and roll around and do stuff.

And then I think over time, the thought, well, That was inspiring, but people said, no, no, no, we want to get maybe closer to what feels like real intelligence or human intelligence.

And then maybe the expert systems people tried to do that, but maybe a little too superficially, right?

So, oh, we get the surface understanding of what intelligence is like because I understand how a steel mill works and I can try to explain it to you and you can write it down in logic and then we can make a computer infer that.

But what's interesting, I think is when a thing starts to not be working very well, it's not only do we change methods, we change problems, right?

So it's not like we have better ways of doing the problem that the expert systems people were trying to do.

We have no ways of trying to do that problem.

But we kind of give up on that problem and we switch to a different problem.

And we work that for a while and we make progress.

And there's a lot of people who would argue you don't give up on the problem, it's just you decrease the number of people work on it.

You almost kind of like put it on the shelf, say we'll come back to this 20 years later.

Like you might say, it's wrong to just try to make something that does superficial symbolic reasoning behave like a doctor.

You can't do that until you've had the sensory motor experience of being a doctor or something, right?

So there's arguments that say that that problem was not well formed, or it could be that it is well formed, but we just weren't approaching it well.

So you mentioned that your favorite part of logic and symbolic systems is that they give short names for large sets.

So there is some use to this, the use, to symbolic reasoning.

So looking at expert systems and symbolic computing, what do you think are the roadblocks that were hit in the 80s and 90s.

So right, so the fact that I'm not a fan of expert systems doesn't mean that I'm not a fan of some kinds of symbolic reasoning, right?

Well, the main roadblock, I think, was that the idea that humans could articulate their knowledge effectively into, you know, some kind of logical statements.

So it's not just the cost, the effort, but really just the capability of doing it.

Because we're all experts in vision, right?

But totally don't have introspective access into how we do that, right?

And it's true that, I mean, I think the idea was, well, of course, even people then would know, of course I wouldn't ask you to please write down the rules that you use for recognizing a water bottle.

And everyone understood that, but we might ask you to please write down the rules you use for deciding, I don't know, what tie to put on or how to set up a microphone or something like that.

But even those things, I think people maybe, I think what they found, I'm not sure about this, but I think what they found was that the so-called experts could give explanations that sort of post hoc explanations for how and why they did things, but they weren't necessarily very good.

And then they depended on maybe some kinds of perceptual things, which Again, they couldn't really define very well.

So I think fundamentally, I think that the underlying problem with that was the assumption that people could articulate how and why they make their decisions.

Right, so it's almost encoding the knowledge, converting from expert to something that a machine can understand and reason with.

No, no, no, not even just encoding, but getting it out of you.

Not writing it, I mean, yes, hard also to write it down for the computer.

But I don't think that people can produce it.

You can tell me a story about why you do stuff, but I'm not so sure that's the why.

Great, so there are still on the hierarchical planning side places where symbolic reasoning is very useful.

So saying that humans can't provide a description of their reasoning processes, that's okay, fine, but that doesn't mean that it's not good to do reasoning of various styles inside a computer.

So then the question is, what kind of reasoning should you do inside a computer?

And the answer is, I think you need to do all different kinds of reasoning inside a computer, depending on what kinds of problems you face.

I guess the question is, what kind of things can you encode symbolically so you can reason about?

I think the idea about, and even symbolic, I don't even like that terminology because I don't know what it means technically and formally.

You cannot reason at completely fine grain about everything in your life, right?

You can't make a plan at the level of images and torques for getting a PhD.

So you have to reduce the size of the state space and you have to reduce the horizon if you're going to reason about getting a PhD or even buying the ingredients to make dinner.

And so how can you reduce the spaces and the horizon of the reasoning you have to do?

Spatial abstraction, temporal abstraction.

I think abstraction along the lines of goals is also interesting.

Like you might, or well, abstraction and decomposition.

Goals is maybe more of a decomposition thing.

So I think that's where these kinds of, if you want to call it symbolic or discrete models come in.

You talk about a room of your house instead of your pose.

You talk about, you know, doing something during the afternoon instead of at 2.54.

And you do that because it makes your reasoning problem easier, and also because you don't have enough information to reason in high fidelity about your pose of your elbow at 2.35 this afternoon anyway.

Except for at that moment, at that moment, you do have to reason about the pose of your elbow, maybe.

But then maybe you do that in some continuous joint space kind of model.

So again, my biggest point about all of this is that there should be, that dogma is not the thing.

It shouldn't be that I am in favor against symbolic reasoning and you're in favor against neural networks.

It should be that just computer science tells us what the right answer to all these questions is if we were smart enough to figure it out.

Well, yeah, when you try to actually solve the problem with computers, the right answer comes out.

I mean, neural networks form abstractions, or rather, there's automated ways to form abstractions, and there's expert-driven ways to form abstractions, and expert human-driven ways, And humans just seem to be way better at forming abstractions currently and certain problems.

Versus afternoon, how do we construct that taxonomy?

Is there any room for automated construction of such abstractions?

I mean, I think when we get to be better and machine learning engineers, we'll build algorithms that build awesome abstractions.

That are useful in this kind of way that you're describing.

So let's then step from the abstraction discussion and let's talk about POM MDPs, partially observable Markov decision processes.

So first, what are Markov decision processes?

And maybe how much of our world can be models and MDPs?

How much, when you wake up in the morning, you're making breakfast, do you think of yourself as an MDP?

So how do you think about MDPs and how they relate to our world?

Well, so there's a stance question, right?

So a stance is a position that I take with respect to a problem.

So I, as a researcher or a person who designs systems, can decide to make a model of the world around me in some terms.

So I take this messy world and I say, I'm gonna treat it as if it were a problem of this formal kind, and then I can apply solution concepts or algorithms or whatever to solve that formal thing, right?

So of course the world is not anything, it's not an MDP or a POMDP, I don't know what it is, but I can model aspects of it in some way or some other way.

And when I model some aspect of it in a certain way, that gives me some set of algorithms I can use.

You can model the world in all kinds of ways.

Some have, some are more accepting of uncertainty, more easily modeling uncertainty of the world.

Some really force the world to be deterministic.

And so certainly MDPs model the uncertainty of the world.

They model not present state uncertainty, but they model uncertainty in the way the future will unfold.

It's a kind of a model that you could make that says, I, I know completely the current state of my system And what it means to be a state is that I, that all the, I have all the information right now that will let me make predictions about the future as well as I can.

So that remembering anything about my history wouldn't make my predictions any better.

And, but, but then it also says that then I can take some actions that might change the state of the world and that I don't have a deterministic model of those changes.

I have a probabilistic model of how the world might change.

It's a, It's a useful model for some kinds of systems.

I think it's a, I mean, it's certainly not a good model for most problems.

I think because for most problems, you don't actually know the state.

For most problems, it's partially observed.

So, okay, that's where the POMDPs, the partially observed Markov decision processes step in.

So how do they address the fact that you can't observe most, you have incomplete information about most of the world around you.

Right, so now the idea is, we still kind of postulate that there exists a state.

We think that there is some information about the world out there, such that if we knew that we could make good predictions, but we don't know the state.

And so then we have to think about how, but we do get observations.

Maybe I get images or I hear things or I feel things and those might be local or noisy and so therefore they don't tell me everything about what's going on.

And then I have to reason about, given the history of actions I've taken and observations I've gotten, what do I think is going on in the world?

And then given my own kind of uncertainty about what's going on in the world, I can decide what actions to take.

And so difficult is this problem of planning under uncertainty in your view, in your long experience of modeling the world, trying to deal with this uncertainty in, especially in real world systems.

Optimal planning for even discrete POMDPs can be undecidable depending on how you set it up.

And so lots of people say, I don't use POMDPs because they are intractable.

And I think that that's a kind of a very funny thing to say because the problem you have to solve is the problem you have to solve.

So if the problem you have to solve is intractable, that's what makes us AI people, right?

So we solve, we understand that the problem we're solving is, is completely wildly intractable that we can't, we will never be able to solve it optimally.

So later we can come back to an idea about bounded optimality and something.

But anyway, we can't come up with optimal solutions to these problems.

Approximations in modeling, approximations in solution algorithms and so on.

And so I don't have a problem with saying, yeah, my problem actually, it is POMDP in continuous space with continuous observations and it's so computationally complex.

I can't even think about it's, you know, big O, whatever, But that doesn't prevent me from, it helps me, gives me some clarity to think about it that way.

And to then take steps to make approximation after approximation to get down to something that's like computable in some reasonable time.

When you think about optimality, the community broadly has shifted on that, I think a little bit, in how much they value the idea of optimality, of chasing an optimal solution.

How has your views of chasing an optimal solution changed over the years when you work with robots?

I think we have a little bit of a methodological crisis actually from the theoretical side.

I mean, I do think that theory is important and that right now we're not doing much of it.

So there's lots of empirical hacking around and training this and doing that and reporting numbers.

And if you look at like computer science theory, so people talked for a while, everyone was about solving problems optimally or completely.

And then there were interesting relaxations.

So people look at, oh, can I, are there regret bounds or can I do some kind of, you know, approximation, Can I prove something that I can approximately solve this problem or that I get closer to the solution as I spend more time and so on?

What's interesting I think is that we don't have good approximate solution concepts for very difficult problems.

I like to, you know, I like to say that I, I'm interested in doing a very bad job of very big problems.

I like to do that, but I would, I wish I could say something.

I wish I had a, I don't know, some kind of a formal solution concept that I could use to say, oh, this algorithm actually, it gives me something.

I can do something other than just run it and get out

That notion is still somewhere deeply compelling to you.

The notion that you can say, you can drop thing on the table says this, you can expect this algorithm will give me some good results?

I hope there's, I hope science will, I mean, there's engineering and there's science.

I think that they're not exactly the same.

And I think right now we're making huge engineering, like leaps and bounds, so that engineering is running way ahead of the science, which is cool, and often how it goes, right?

So we're making things and nobody knows how and why they work, roughly.

There's some form, it's, yeah, there's some room for formalizing.

I mean, for a while people built bridges by trying, but now we can often predict whether it's going to work or not without building it.

Can we do that for learning systems or for robots?

So your hope is from a materialistic perspective that intelligence, artificial intelligence systems, robots, I would just fancy your bridges.

See all Lex Fridman transcripts on Youtube

Leslie Kaelbling: Reinforcement Learning, Planning, and Robotics | Lex Fridman Podcast #15