The following is a conversation with Vijay Kumar.

He's 1 of the top roboticists in the world, a professor at the University of Pennsylvania, a Dean of Penn Engineering, former director of Grasp Lab, or the General Robotics, Automation Sensing, and Perception Laboratory at Penn that was established back in 1979, that's 40 years ago.

Vijay is perhaps best known for his work in multi-robot systems, robot swarms, and micro aerial vehicles, robots that elegantly cooperate in flight under all the uncertainty and challenges that the real world conditions present.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, give it 5 stars on iTunes, support it on Patreon, or simply connect with me on Twitter at Lex Friedman, spelled F-R-I-D-M-A-N.

And now, here's my conversation with Vijay Kumar.

What is the first robot you've ever built or were a part of building?

Way back when I was in graduate school, I was part of a fairly big project that involved building a very large hexapod.

And it was powered by hydraulic actuation or it was actuated by hydraulics with 18 motors, hydraulic motors, each controlled by an Intel 8085 processor and an 8086 coprocessor.

And so imagine this huge monster that had 18 joints, each controlled by an independent computer, and there was a 19th computer that actually did the coordination between these 18 joints.

So I was part of this project, and my thesis work was how do you coordinate the 18 legs?

And in particular, the pressures in the hydraulic cylinders to get efficient locomotion.

How difficult is it to make all the motors communicate?

Presumably you have to send signals hundreds of times a second.

So this was not my work, but the folks who worked on this wrote what I believe to be the first multiprocessor operating system.

And you had to make sure that obviously messages got across from 1 joint to another.

You have to remember the clock speeds on those computers were about half a megahertz.

So not to romanticize the notion, but how did it make you feel to see that robot move?

In hindsight, it looks like, well, we built this thing which really should have been much smaller.

And of course, today's robots are much smaller.

You look at Boston Dynamics or Ghost Robotics, a spin-off from Penn.

But back then, you were stuck with the substrate you had, the compute you had, so things were unnecessarily big.

But at the same time, and this is just human psychology, somehow bigger means grander.

You know, people never have the same appreciation for nanotechnology or nanodevices as they do for the space shuttle or the Boeing 747.

Yeah, you've actually done quite a good job at illustrating that small is beautiful in terms of robotics.

So what is on that topic is the most beautiful or elegant robot in motion that you've ever seen?

Not to pick favorites or whatever, but something that just inspires you that you remember.

Well, I think the thing that I'm most proud of that my students have done is really think about small UAVs that can maneuver in constrained spaces and in particular, their ability to coordinate with each other and form three-dimensional patterns.

So once you can do that, you can essentially create 3D objects in the sky and you can deform these objects on the fly.

So in some sense, your toolbox of what you can create is suddenly got enhanced.

And before that, we did the two-dimensional version of this.

So we had ground robots forming patterns and so on.

But if you do it in 3D, suspended in midair, and you've got to go back to 2011 when we did this.

Now it's actually pretty standard to do these things 8 years later.

But back then it was a big accomplishment.

So the distributed cooperation is where beauty emerges in your eyes.

Well, I think beauty to an engineer is very different from beauty to, you know, someone who's looking at robots from the outside, if you will.

But what I meant there, so before we said that grand is associated with size, and another way of thinking about this is just the physical shape, and the idea that you can get physical shapes in midair and have them deform, that's beautiful.

But the individual components, the agility is beautiful too, right?

So then how quickly can you actually manipulate these 3 dimensional shapes and the individual components?

By the way, you said UAV, Unmanned Aerial Vehicle.

What's a good term for drones, UAVs, quadcopters?

Is there a term that's being standardized?

Everybody wants to use the word drones, and I've often said this, drones to me is a pejorative word.

It signifies something that's dumb, it's pre-programmed, it does 1 little thing, and robots are anything but drones.

So I actually don't like that word, but that's what everybody uses.

But even unpiloted could be radio controlled, could be remotely controlled in many different ways.

And I think the right word is, thinking about it as an aerial robot.

You also say agile, autonomous, aerial robot, right?

Yeah, so agility is an attribute, but they don't have to be.

So what biological system, because you've also drawn a lot of inspiration with those, I've seen bees and ants that you've talked about.

What living creatures have you found to be most inspiring as an engineer, instructive in your work in robotics?

To me, so ants are really quite incredible creatures, right?

So you, I mean, the Individuals arguably are very simple in how they're built, and yet they're incredibly resilient as a population.

And as individuals, they're incredibly robust.

So if you take an ant, it's 6 legs, you remove 1 leg, it still works just fine.

And it moves along and I don't know that he even realizes it's lost a leg.

So that's the robustness at the individual ant level.

But then you look about this instinct for self-preservation of the colonies and they adapt in so many amazing ways.

You know, transcending gaps by just chaining themselves together when you have a flood, being able to recruit other teammates to carry big morsels of food, and then going out in different directions looking for food, and then being able to demonstrate consensus, even though they don't communicate directly with each other the way we communicate with each other.

In some sense, they also know how to do democracy probably better than what we do.

Yeah, somehow it's that even democracy is emergent.

It seems like all of the phenomena that we see is all emergent.

It seems like there's no centralized communicator.

There is, so I think a lot is made about that word, emergent, and it means lots of things to different people, but you're absolutely right.

I think as an engineer, you think about what element, elemental behaviors, what primitives you could synthesize so that the whole looks incredibly powerful, incredibly synergistic, the whole definitely being greater than the sum of the parts, and ants are living proof of that.

So when you see these beautiful swarms where there's biological systems of robots, do you sometimes think of them as a single individual living intelligent organism?

So it's the same as thinking of our human civilization as 1 organism, or do you still, as an engineer, think about the individual components and all the engineering that went into the individual components?

So again, philosophically, as engineers, what we want to do is to go beyond the individual components, the individual units, and think about it as a unit, as a cohesive unit, without worrying about the individual components.

If you start obsessing about the individual building blocks and what they do, you inevitably will find it hard to scale up.

Just mathematically, just think about individual things you want to model.

And if you want to have 10 of those, then you essentially are taking Cartesian products of 10 things.

Then to do any kind of synthesis or design in that high-dimensional space is really hard.

So the right way to do this is to think about the individuals in a clever way so that at the higher level, when you look at lots and lots of them, abstractly, you can think of them in some low dimensional space.

For the individual, Do you have to try to make the way they see the world as local as possible?

And the other thing, do you just have to make them robust to collisions?

Like you said with the ants, if something fails, the whole swarm doesn't fail.

I mean, you think about we build planes or we build iPhones.

And we know that by taking individual components, well-engineered components with well-specified interfaces that behave in a predictable way, you can build complex systems.

So that's ingrained, I would claim, in most engineers' thinking.

And it's true for computer scientists as well.

I think what's different here is that you want the individuals to be robust in some sense, as we do in these other settings, but you also want some degree of resiliency for the population.

And so you really want them to be able to reestablish communication with their neighbors.

You want them to rethink their strategy for group behavior.

And that's where I think a lot of the challenges lie.

So just at a high level, what does it take for a bunch of, what should we call them, flying robots to create a formation?

Just for people who are not familiar with robotics in general, how much information is needed?

How do you even make it happen without a centralized controller?

So, I mean, there are a couple of different ways of looking at this.

If you are a purist, you think of it as a way of recreating what nature does.

So nature forms groups for several reasons, but mostly it's because of this instinct that organisms have of preserving their colonies, their population.

You need shelter, you need food, you need to procreate, and That's basically it.

So the kinds of interactions you see are all organic, they're all local.

And the only information that they share, and mostly it's indirectly, is to, again, preserve the herd or the flock or the swarm, and either by looking for new sources of food or looking for new shelters, right?

As engineers, when we build swarms, we have a mission.

And When you think of a mission and it involves mobility, most often it's described in some kind of a global coordinate system.

As a human, as an operator, as a commander, or as a collaborator, I have my coordinate system and I want the robots to be consistent with that.

So I might think of it slightly differently.

I might want the robots to recognize that coordinate system, which means not only do they have to think locally in terms of who their immediate neighbors are, but they have to be cognizant of what the global environment looks like.

So if I say surround this building and protect this from intruders, well, they're immediately in a building-centered coordinate system, and I have to tell them where the building is.

And they're globally collaborating on the map of that building.

They're maintaining some kind of global, not just in the frame of the building, but there's information that's ultimately being built up explicitly as opposed to kind of implicitly like nature might.

So in some sense nature is very, very sophisticated, but the tasks that nature solves or needs to solve are very different from the kind of engineered tasks, artificial tasks that we are forced to address.

And again, there's nothing preventing us from solving these other problems, But ultimately it's about impact.

You want these swarms to do something useful.

And so you're kind of driven into this very unnatural, if you will, unnatural meaning not like how nature does, setting.

And it's probably a little bit more expensive to do it the way nature does, because nature is less sensitive to the loss of the individual, and cost-wise in robotics, I think you're more sensitive to losing individuals.

I think that's true, although if you look at the price to performance ratio of robotic components, it's coming down dramatically.

So I think we're asymptotically approaching the point where we would get, yeah, the cost of individuals would really become insignificant.

So let's step back at a high level of view, the impossible question of what kind of, as an overview, what kind of autonomous flying vehicles are there in general?

I think the ones that receive a lot of notoriety are obviously the military vehicles.

Military vehicles are controlled by a base station but have a lot of human supervision but have limited autonomy, which is the ability to go from point A to point B, and even the more sophisticated now, sophisticated vehicles can do autonomous takeoff and landing.

And those usually have wings and they're heavy.

Usually they're wings, but there's nothing preventing us from doing this for helicopters as well.

I mean, there are many military organizations that have autonomous helicopters in the same vein.

And by the way, you look at autopilots and airplanes And it's actually very similar.

In fact, 1 interesting question we can ask is, if you look at all the air safety violations, all the crashes that occurred, would they have happened if the plane were truly autonomous?

And I think you'll find that in many of the cases, because of pilot error, we make silly decisions.

And so in some sense, even in air traffic, commercial air traffic, there's a lot of applications, although we only see autonomy being enabled at very high altitudes when the plane is on autopilot.

There's still a role for the human and that kind of autonomy is, you're kind of implying, I don't know what the right word is, but it's a little dumber than it could be.

Right, so in the lab, of course, we can afford to be a lot more aggressive.

And the question we try to ask is, can we make robots that will be able to make decisions without any kind of external infrastructure?

So the most common piece of infrastructure that airplanes use today is GPS.

GPS is also the most brittle form of information.

If you have driven in a city, try to use GPS navigation, you know, in tall buildings, you immediately lose GPS.

And so that's not a very sophisticated way of building autonomy.

I think the second piece of infrastructure they rely on is communications.

Again, it's very easy to jam communications.

In fact, if you use Wi-Fi, you know that Wi-Fi signals drop out, cell signals drop out.

So to rely on something like that is not good.

The third form of infrastructure we use, and I hate to call it infrastructure, but it is that in the sense of robots, it's people.

So you can rely on somebody to pilot you.

And so the question you want to ask is if there are no pilots, if there's no communications with any base station, if there's no knowledge of position.

And if there's no a priori map, a priori knowledge of what the environment looks like, a priori model of what might happen in the future, can robots navigate?

And we're talking about, you mentioned like military application of drones.

You talk about agile, autonomous flying robots, aerial robots, so that's a different kind of, it's not winged, it's not big, at least it's small.

So I use the word agility mostly, or at least we're motivated to do agile robots, mostly because robots can operate and should be operating in constrained environments.

And if you want to operate the way a global hawk operates, I mean, the kinds of conditions in which you operate are very, very restrictive.

If you wanna go inside a building, for example, for search and rescue, or to locate an active shooter, or you want to navigate under the canopy in an orchard to look at health of plants, or to look for, to count fruits, to measure the tree trunks.

Yeah, some cool agriculture stuff you've shown in the past.

Right, so in those kinds of settings, you do need that agility.

Agility does not necessarily mean you break records for the 100 meters dash.

What it really means is you see the unexpected and you're able to maneuver in a safe way and in a way that gets you the most information about the thing you're trying to do.

By the way, you may be the only person who in a TED Talk has used a math equation, which is amazing.

People should go see 1 of your TED Talks.

Actually, it's very interesting because the TED curator, Chris Anderson, told me you can't show math.

And I thought about it, but that's who I am.

And so I felt compelled to give the audience a taste for at least some math.

So on that point, simply, what does it take to make a thing with 4 motors fly, a quadcopter, 1 of these little flying robots?

What's, how do you convert those motors into actual movement?

It is a commentary on the sensors that were available back then, the computers that were available back then.

A number of things happened between 2000 and 2007.

1 is the advances in computing, which is We all know about Moore's law, but I think 2007 was a tipping point, the year of the iPhone, the year of the cloud.

But going back even further, inertial measurement units as a sensor really matured.

Certainly, there's a lot of federal funding, particularly DARPA in the US, but they didn't anticipate this boom in IMUs.

But if you look, subsequently what happened is that every car manufacturer had to put an airbag in, which meant you had to have an accelerometer on board.

And so that drove down the price to performance ratio.

That's very interesting, the connection there.

And that's why research is very, It's very hard to predict the outcomes.

And again, the federal government spent a ton of money on things that they thought were useful for resonators, but it ended up enabling these small UAVs, which is great because I could have never raised that much money and sold this project, hey, we want to build these small UAVs.

Can you actually fund the development of low-cost IMUs?

But so in 2007, 2008, we were able to build these and then the question you're asking was a good 1, how do you coordinate the motors to develop this?

But over the last 10 years, everything is commoditized.

A high school kid today can pick up a Raspberry Pi kit and build this.

All the low-level functionality is all automated.

But basically at some level, you have to drive the motors at the right RPMs, the right velocity, in order to generate the right amount of thrust, in order to position it and orient it in a way that you need to in order to fly.

The feedback that you get is from onboard sensors and the IMU is an important part of it.

The IMU tells you what the acceleration is, as well as what the angular velocity is.

And those are important pieces of information.

In addition to that, you need some kind of local position or velocity information.

For example, when we walk, we implicitly have this information because we kind of know what our stride length is.

We also are looking at images fly past our retina, if you will, and so we can estimate velocity.

We also have accelerometers in our head, and we're able to integrate all these pieces of information to determine where we are as we walk.

And so robots have to do something very similar.

You need an IMU, you need some kind of a camera or other sensor that's measuring velocity.

And then you need some kind of a global reference frame if you really want to think about doing something in a world coordinate system.

And so how do you estimate your position with respect to that global reference frame?

So coordinating the RPMs of the 4 motors is what allows you to first of all fly and hover, and then you can change the orientation and the velocity and so on.

So it's a bunch of degrees of freedom that you're

There's 6 degrees of freedom, but you only have 4 inputs, the 4 motors.

And it turns out to be a remarkably versatile configuration.

You think at first, well, I only have 4 motors, how do I go sideways?

But it's not too hard to say, well, if I tilt myself, I can go sideways.

And then you have 4 motors pointing up, how do I rotate in place about a vertical axis?

Well, you rotate them at different speeds and that generates reaction moments and that allows you to turn.

So it's actually a pretty, it's an optimal configuration from an engineer standpoint.

It's very simple, very cleverly done and very versatile.

So if you could step back to a time, so I've always known flying robots as, to me it was natural that the quadcopter should fly.

But when you first started working with it, how surprised are you that you can make, do so much with the 4 motors?

How surprising is it you can make this thing fly, first of all, you can make it hover, then you can add control to it.

Firstly, this is not, the 4 motor configuration is not ours.

You can, it has at least a hundred year history.

various people, various people try to get quadrotors to fly without much success.

As I said, we've been working on this since 2000.

Our first designs were, well, this is way too complicated.

Why not we try to get an omnidirectional flying robot?

And so these 8 rotors were arranged uniformly on a sphere, if you will.

So you can imagine a symmetric configuration.

And so you should be able to fly anywhere.

But the real challenge we had is the strength to weight ratio is not enough.

And of course we didn't have the sensors and so on.

See all Lex Fridman transcripts on Youtube

Vijay Kumar: Flying Robots | Lex Fridman Podcast #37