This is a talk by Richard Moyes for MIT course 6.S099: Artificial General Intelligence. He is the Co-Founder and Managing Director of Article 36, which is a UK-based not-for-profit organization working to prevent the unintended, unnecessary or unacceptable harm caused by certain weapons. This class is free and open to everyone. Our goal is to take an engineering approach to exploring possible paths toward building human-level intelligence for a better world.

OUTLINE:
0:00 - Introduction
47:28 - Q&A

INFO:
Course website: https://agi.mit.edu
Contact: agi@mit.edu
Playlist: http://bit.ly/2EcbaKf

CONNECT:
- AI Podcast: https://lexfridman.com/ai/
- Subscribe to this YouTube channel
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Twitter: https://twitter.com/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Slack: https://deep-mit-slack.herokuapp.com

Welcome back to 6S099 Artificial General Intelligence Today we have Richard Moyes He's the founder and managing director of Article 36 A UK-based not-for-profit organization working to prevent the unintended, unnecessary and unacceptable harm caused by certain weapons, including autonomous weapons and nuclear weapons.

He will talk with us today about autonomous weapon systems in the context of AI safety.

This is an extremely important topic for engineers, humanitarians, legal minds, policy makers and everybody involved in paving the path for safe positive future for AI in our society, which I hope is what this course is about.

Richard flew all the way from the UK to visit us today in snowy Massachusetts, so please give him a You're welcome.

Thanks very much, Lex, and thank you all for coming out.

As Lex said, I work for a not-for-profit organization based in the UK.

We specialize in thinking about policy and legal frameworks around weapon technologies particularly, and generally about how to establish more constraining policy and legal frameworks around weapons.

And I guess I'm mainly gonna talk today about these issues of to what extent we should enable machines to kill people, to make decisions to kill people.

It's, I think, a conceptually very interesting topic, quite challenging in lots of ways.

There's lots of unstable terminology and lots of sort of blurry boundaries.

My own background, as I said, we work on weapons policy issues and I've worked on the development of 2 legal international legal treaties prohibiting certain types of weapons.

I worked on the development of a 2008 convention on cluster munitions, which prohibits cluster bombs, and worked on, our organization pioneered the idea of a treaty prohibition on nuclear weapons, which was agreed last year in the UN, and we're part of the steering group of ICANN, the International Campaign to Abolish Nuclear Weapons, which won the Nobel Peace Prize last year, so that was a good year for us.

The issue of autonomous weapons, killer robots, which I'm gonna talk about today, We're also part of an NGO, non-governmental organization coalition on this issue called the Campaign to Stop Killer Robots.

However, I think when we get into some of the details of the issue, we'll find that perhaps the snappiness of the name in a way masks some of the complexity that lies underneath this.

But this is a live issue in international policy and legal discussions.

At the United Nations for the last several years, 3 or 4 years now, there have been groups of governments coming together to discuss autonomous weapons and whether or not there should be some new legal instrument that tackles this issue.

So it's a live political issue that is being debated in policy legal circles.

And really my comments today are gonna be speaking to that context.

I guess I'm gonna try and give us a bit of a briefing about what the issues are in this international debate, how different actors are orientating to these issues, some of the conceptual models that we use in that.

So I'm not really gonna give you a particular sales pitch as to what you should think about this issue, though my own biases are probably gonna be fairly evident during the process, but really to try and lay out a bit of a sense of how these questions are debated in the international political scene, and maybe in a way that's useful for reflecting on sort of wider questions of how AI technologies might be orientated to an approach by policymakers and the legal framework.

So in terms of the structure of my comments, I'm gonna talk a bit about some of the pros and cons that are put forward around autonomous weapons or movements towards greater autonomy in weapon systems.

I'm going to talk a bit more about the political legal framework within which these discussions are taking place.

And then I'm going to try to sort of lay out some of the models, the conceptual models that we as an organization have developed and are sort of using in relation to these issues.

And perhaps to reflect a bit on where I see the political conversation, the legal conversation on this going at an international level.

And maybe just finally to try to reflect on or just draw out some more general thoughts that I think occur to me about what some of this says about thinking about AI functions in different social roles.

But before getting into that sort of pros and cons type stuff, I just wanted to start by suggesting a bit of a sort of conceptual timeline, because 1 of the things, this could be the present, 1 of the things you find when you say to somebody, well, we work on this issue of autonomous weapons, They tend to orientate to it in 2 fairly distinct ways.

Some people will say, oh, you mean armed drones, and we know what armed drones are.

They're being used in the world today, and That's kind of an issue here in the present, right?

But other people, most of the media, and certainly pretty much every media photo editor, thinks you're talking about the Terminator over here.

Maybe a bit of Skynet thrown in so this is a sort of Advanced futuristic sci-fi orientation to the to the issues My thinking about this I come from a background of working on the impact of weapons in the present.

My thinking and my anxieties or my concerns around this issue don't come from this area.

I drew this line a bit wiggly here because I also don't want to suggest there's any kind of teleological certainty going on here.

But I think it's important just in terms of situating where I'm coming from in the debate that I'm definitely not starting at that end.

And yet in the political discussion amongst governments and states, well you have people coming in at all sorts of different positions along here, imagining that autonomous weapons may exist at somewhere along this sort of spectrum.

So I'm gonna think more about stuff that's going on around here and how some of our conceptual models really build around some of this thinking, not so much actually armed drones, but some other systems.

But my background before I started working on policy and law around weapons was setting up and managing landmine clearance operations overseas.

And well, they've been around for quite a long time, landmines.

And I think it's interesting just to start with, just to reflect on the basic anti-personnel landmine.

It's simple but it gives us I think some sort of useful entry points into thinking about what an autonomous weapon system might be in its most simple form.

If we think about a landmine, well essentially we have a person, there's an input into the landmine, and there's a function that goes on here.

There's a basic mechanical algorithm goes on and you get an output explosion that goes back against the person who trod on the landmine.

So it's a fairly simple system of a signal, a sensor taking a signal from the outside world, the land mine is viewing the outside world through its sensor, it's a basic pressure plate, and according to a certain calculus here, you get an output and it's directed back at this person and it's a loop.

And that's 1 of the things that I think is fundamental essentially to understanding the idea of autonomous weapons and in a way this is where the autonomy comes in.

That there's no other person intervening in this process at any point.

There's just a sort of straightforward relationship from the person or object that has initiated the system back into the effects that are being applied.

So in some ways we'll come back to this later and think about how some of the basic building blocks of this may be there in our thinking about other weapon systems and weapon technologies as they're developing.

And maybe thinking about landmines and thinking about these the processes of technological change, We see a number of different dynamics at play in this sort of imaginary timeline.

Anti-personnel landmines of course are static, they just sit in the ground where you've left them, but we get more and more mobility perhaps as we go through this system.

Certainly drones and other systems that I'll talk about, you start to see more mobility in the weapon system.

Perhaps greater sophistication of sensors, I mean a basic pressure plate just gauging weight, that's a very simple sensor structure for interrogating the world.

We have much more sophisticated sensor systems in weapons now.

So we have weapon systems now that are looking at radar signatures, they're looking at the heat shapes of objects, And we'll come back and talk about that, but more sophistication of sensors and more sophistication of the computer algorithms that are basically interrogating those sensor inputs.

Perhaps a little bit as well of a movement in this sort of trajectory from physically very unified objects.

I always sort of wrestle slightly whether this is the word I want, but it's a sort of self-contained entity, the landmine.

Whereas as we move in this direction, maybe we see more dispersal of functions through different systems.

And I think that's another dynamic that when we think about the development of autonomy and weapon systems, it might not all live in 1 place physically, moving around in 1 place.

It can be an array of different systems functioning in different places.

And perhaps for people with a sort of AI type of mindset, maybe there's some sort of movement from more specific types of AI functioning here, use of different specific AI functions here to something more general going in this direction.

I'm wary of necessarily buying straightforward into that but maybe you could see some movement in that sort of direction.

So I just want to sort of put this to 1 side for now, but we'll come back to it and think about some systems that are existing here that I think sort of raise issues for us and around which we could expand some models.

But I just want us to have this in mind when thinking about this, that we're not necessarily, we're definitely not, for me, thinking about humanoid robots walking around, fighting like a soldier.

Rather, we're thinking about developments and trajectories we can see coming out of established military systems now.

So I was gonna talk now a bit about the political and the legal context.

Obviously, there's a lot of complexity in the world of politics and legal structures, so I don't want to get too bogged down in it, but I think in terms of understanding the basics of this debate on the international landscape, we have to have a bit of background in that area.

Essentially there's, I think, 3 main types of international law that we're concerned with here, and Again, concerned with international law rather than domestic legislation which any individual state can put in place whatever domestic legislation they want.

We're looking at the international legal landscape.

Basically you have international human rights law which applies in pretty much all circumstances and it involves the right to life and the right to dignity and various other legal protections for people.

And then particularly prominent in this debate if you have what's called international humanitarian law, which is the rules that govern behavior during armed conflict and provide obligations on militaries engaged in armed conflict for how they have to conduct themselves.

This isn't the legal framework that decides whether it's okay to have a war or not.

This is a legal framework that once you have in the war, this is the obligations that you've got to follow.

And it basically includes rules that say, you're not allowed to directly kill civilians, you've got to aim your military efforts at the forces of the enemy, at enemy combatants.

You're not allowed to kill civilians directly or deliberately, but you are allowed to kill some civilians as long as you don't kill too many of them for the military advantage that you're trying to achieve.

So there's a sort of balancing acts like this, this is called proportionality.

Nobody ever really knows where the balance lies, but it's a sort of principle of the law that whilst you can kill civilians, you mustn't kill an excessive number of civilians.

These apply pretty much to all states in armed conflict situations.

And then you have treaties on specific weapon types.

And this is really where you have weapons that are considered to be particularly problematic in some way, and it's decided, a group of states decides to develop and put in place a treaty that applies specifically to those weapons.

I think it's important to recognize that these legal treaties are all developed and agreed by states.

They're agreed by international governments talking together, negotiating what they think the law should say, and they generally only bind on states if they choose to adopt that legal instrument.

So I guess what I'm emphasizing there is a sense that these are sort of social products in a way, they're political products.

It isn't a sort of magical law that's come down from on high perfectly written to match the needs of humanity.

It's a negotiated outcome developed by a complicated set of actors who may or may not agree with each other on all sorts of things.

And what that means is there's quite a lot of wiggle room in these legal frameworks and quite a lot of uncertainty within them.

Lawyers of international humanitarian law will tell you that's not true, but that's because they're particularly keen on that legal framework.

But in reality, there's a lot of fuzziness to what some of the legal provisions say.

And it also means that the extent to which this law binds on people and bears on people is also requires some social enactment.

There's not a sort of world police who can follow up on all of these legal frameworks, it requires a sort of social function from states and from other actors to keep articulating their sense of the importance of these legal rules and keep trying to put pressure on other actors to accord with them.

So the issue of autonomous weapons is in discussion at the United Nations under a framework called the UN Convention on Conventional Weapons.

And this is a body that has the capacity to agree new protocols, new treaties essentially on specific weapon systems.

And that means that diplomats from lots of countries, diplomats from the US, from the UK, from Russia and Brazil and China and all the countries of the world will be sitting around in a conference room putting forward their perspectives on this issue and trying to find common ground or trying not to find common ground depending on what sort of outcome they're working towards.

So, the UN isn't a completely separate entity of its own, it's just the community of states in the world sitting together talking about things.

So, main focus of concern in those discussions when it comes to autonomy is not some sort of generalized autonomy or not autonomy of all of its forms that may be pertinent in the military space.

It's rather much more these questions of how the targets of an attack are selected, identified, decided upon, and how is the decision to apply force to those targets made.

And it's really, these are sort of the critical functions of weapon systems, where the movement towards greater autonomy is considered a source of anxiety essentially, that we may see machines making decisions on what is a target for an attack and choosing when and where forces apply to that specific to that specific target.

So obviously in this context, not everybody's like-minded on this.

There are potential advantages to increasing autonomy in weapon systems and there's potential disadvantages and problems associated with it.

And within this international discussion, we see different perspectives laid out and some states of course will be able to see some advantages and some disadvantages.

It's not a black and white sort of discussion.

In terms of the possible advantages for autonomy, I mean, 1 of the key ones ultimately is framed in terms of military advantage, that we want to have more autonomy in weapon systems because it will maintain or give us military advantage over possible adversaries.

Because in the end, military stuff is about winning wars, right?

So you want to maintain military advantage.

And military advantage, number of factors really within that, Speed is 1 of them, speed of decision making.

Can computerized autonomous systems make decisions about where to apply force faster than a human would be capable of doing and therefore this is advantageous for us.

Also speed allows for coordination of numbers.

So if you wanna have swarms of systems, swarms of small drones or some such, you need quite a lot of probably autonomy in decision making and communication between those systems because again, the level of complexity and the speed involved is greater than a human would be able to sort of manually engineer.

So speed both in terms of responding to external effects, but also perhaps coordinating your own forces.

Reach, potential for autonomous systems to be able to operate in communication-denied environments where if you're relying on an electronic communications link to say a current armed drone, maybe in a future battle space where the enemy is denying communications in some way.

You could use an autonomous system to still fulfill a mission without needing to rely on that communications infrastructure.

General force multiplication, there's a bit of a sense that there's gonna be more and more teaming of machines with humans, so machines operating alongside humans in the battle space.

And then there's importantly, as it's presented at least, a sense that these are systems which could allow you to reduce the risk to your own forces.

That maybe if we can put some sort of autonomous robotic system at work in a specific environment then we don't need to put 1 of our own soldiers in that position and as a result we're less likely to have casualties coming home which of course politically is problematic for maintaining any sort of conflict posture.

Set against all that stuff, there's a sense that I think most fundamentally, there's perhaps a moral hazard that we come across at some point, that there's some sort of boundary where seeing or conceptualizing a situation where machines are deciding who to kill in a certain context is just somehow wrong.

And while that's not a very easy argument to just start articulate in a sort of rationalized sense but there's some sort of moral revulsion that perhaps comes about in the sense that machines are now deciding who should be killed in a particular environment.

Can these systems be used in accordance with the existing legal obligations?

And I'm gonna come on a little bit later to our orientation and the legal side, which is also about how they may stretch the fabric of the law and the structure of the law as we see it.

There's some concerns in this sort of legal arguments for me that we sometimes slip into a language of talking about machines making legal decisions?

Will a machine be able to apply the rule of proportionality properly?

I know what it means but at the same time the law is addressed to humans, the law isn't addressed to machines, so it's humans who have the obligation to enact the legal obligation.

A machine may do a function that is sort of analogous to that legal decision, but ultimately, to my mind, it's still a human who has to be making the legal determination based on some prediction of what that machine will do.

And I think this is a very dangerous slippage because even senior legal academics can slip into this mindset, which is a little bit like handing over the legal framework to machines before you've even got onto arguing about what we should or shouldn't have.

And it's a little bit to do with, for me, continuing to treat these technologies as machines rather than into treating them as agents in some way of a sort of equal or equivalent or similar moral standing to humans.

And then we have a whole set of wider concerns that are raised.

So we've got moral anxieties, legal concerns, and then a set of other concerns around risks that could be unpredictable risks.

There's a sort of normal accidents theory.

There's a bit of that sort of language in the debate about complicated systems and not being able to, you know, not be able to avoid accidents in some respects.

Some anxieties about maybe this will reduce the barriers to engaging in military action.

Maybe being able to use autonomous weapons will make it easier to go to war.

And some anxieties about sort of international security and balance of power and arms races and the like.

I don't tend to think much in this area partly because they involve quite a lot of speculation about what may or may not be in the future and they're quite difficult to populate with sort of more grounded arguments, I find.

But that doesn't mean that they aren't significant in themselves, but I find them less straightforward as an entry point.

So, in all of these different issues, there's lots of unstable terminology, lots of arguments coming in different directions, and our job as an NGO in a way, is we're trying to find ways of building a constructive conversation in this environment, which can move towards states adopting a more constraining orientation to this movement towards autonomy.

And the main tool we've used to work towards that so far has been to perhaps stop focusing on the technology per se and the idea of what is autonomy and how much autonomy is a problem.

And to bring the focus back a bit onto what is the human element that we want to preserve in all of this.

Because it seems like most of the anxieties that come from a sense of a problem with autonomous weapons are about some sort of absence of a human element that we want to preserve.

But unless we can in some way define what this human element is that we want to preserve, I'm not sure we can expect to define its absence very straightforwardly.

So I kind of feel like we want to pull the discussion onto a focus on the human element.

And the tool we've used for this so far has been basically a terminology about the need for meaningful human control.

And This is just a form of words that we've sort of introduced into the debate and we've promoted it in discussions with diplomats and with different actors.

And we've built up the idea of this terminology as being a sort of tool.

You create the terms and then you use that to sort of structure the discussion in a productive way.

1 of the reasons I like it is it works partly because the word meaningful doesn't mean anything particular or at least it means whatever you might want it to mean.

And I find an enjoyable sort of tension in that.

But the term meaningful human control has been quite well picked up in the literature on this issue and in the diplomatic discourse.

And it's helping to structure us towards what we think are the key questions.

Basic arguments for the idea of meaningful human control from my perspective are quite simple and we tended to use basically a sort of absurdist sort of logic, if there is such a thing.

First of all really to recognize that no governments are in favor of an autonomous weapon system that has no human control whatsoever, right?

Nobody is arguing that it would be a good idea for us to have some sort of autonomous weapon that just flies around the world, deciding to kill people.

We don't know who it's gonna kill or why.

It doesn't have to report back to us, but we're in favor.

So there needs to be some form of human control because we can rule out that sort of, you know, ridiculous extension of the argument.

And on the other hand, if you just have a person in a dark room with a red light that comes on every now and again, and they don't know anything else about what's going on, but they're the human who's controlling this autonomous weapon, and when the red light comes on, they push the fire button to launch a rocket or something.

We know that that isn't sufficient human control either.

There's a person doing something, there's a person engaged in the process, but clearly it's just some sort of mechanistic pro forma human engagement.

So between these 2 kind of ridiculous extremes, I think we get the idea that there's some sort of fuzzy line that must exist in there somewhere and that everybody can in some way agree to the idea that such a line should exist.

So The question then for us is how to move the conversation in the international community towards a productive sort of discussion of where the parameters of this line might be conceptualized.

So That's brought us on to thinking about a more substantive set of questions about what are the key elements of meaningful human control.

So I might get rid of that fuzzy line because it's a bit useless anyway, isn't it?

And then I can put my key elements on here.

Well, 1 of them is predictable, reliable, transparent technology.

This is kind of before you get into exactly what the system's gonna do.

We want the technology itself to be sort of well made and it's gonna basically do what it says it's gonna do, whatever that is, and we wanna be able to understand it to some extent.

Obviously this becomes a bit of a challenge in some of the AI type functions where you start to have machine learning issues and all like these issues of transparency, perhaps they start to come up a little bit there.

But these are kind of issues in the design and the development of systems.

Another thing we want to have is, and I think this is a key 1, is accurate information.

And it's accurate information on the intent of the commander, or The outcome, what's the outcome we're trying to achieve?

Third 1, there's only 4, so it won't take long.

Third 1 is timely intervention, human intervention, it should be.

It'd be good if we could turn it off at some point, if it's gonna be a very long acting system, it'd be good if we could turn it off, Maybe.

And the fourth 1 is just a sort of framework of accountability.

So we're thinking that basic elements of human control can be broken down into these areas.

Some of them are about the technology itself, how it's designed and made, how do you verify and validate that it's gonna do what the manufacturers have said it's gonna do, can you understand it?

This 1 I think is the key 1 in terms of thinking about the issue and this is what I'm gonna talk about a bit more now.

But accurate information on what's the commander's intent?

What do you want to achieve in the use of this system?

I mean, this makes a big difference how it works.

These factors here involve, what are the target profiles that it's gonna use?

On the landmine, of course, it was just pressure, right?

Pressure is being taken as a, pressure on the ground is being taken as a proxy for a military target, for a human, who we're gonna assume is a military target.

But in these systems, we're gonna have different target profiles, different heat shapes, different patterns of data that the system is going to operate on the basis of.

What sort of actual weapon is it going to use to apply force?

It makes a difference if it's going to just fire a bullet from a gun or if it's going to drop a 2,000 pound bomb.

I mean that has a different effect and the way in which you envisage and sort of control for those effects is going to be different in those different cases.

And finally, very importantly, these issues of context, information on the context in which the system will operate.

Context of course includes, are there going to be civilians present in the area?

Are there going to be other objects in the area that may present a similar pattern to the proxy data?

You know, if you're using a heat shape of a vehicle engine, it might be aimed at a tank, but if there's an ambulance in the same location, is an ambulance's vehicle engine heat shape sufficiently similar to the tank to cause some confusion between the 2?

And context of course varies in different, obviously varies in different environments, but I think we can see different domains in this area as well, which is significant.

That operating in the water or in the ocean, you've probably got a less cluttered environment, a less complex environment than if you're operating in an urban area.

So that's another factor that needs to be taken into, into account in, in this.

So I just wanted to talk a little bit about some existing systems perhaps and think about them in the context of this, these sort of set of issues here.

1 system that you may, may be aware of is, It's on a boat.

Okay, so something like the Phalanx anti-missile system, it's on a boat, but there's various anti-missile systems, I mean, the details don't matter in this context.

These are systems that a human turns it on.

So a human is choosing when to turn it on and a human turns it off again.

But when it's operating, the radar is basically scanning an area of, an area of sky up here.

And it's looking for fast moving incoming objects because basically it's designed to automatically shoot down incoming missiles or rockets, right?

So thinking about these characteristics, you know what the outcome you want is, you want your boat not to get blown up by an incoming missile, and you wanna shoot down any incoming missiles.

You know how the technology works because you know that it's basically using radar to see incoming fast moving signatures.

And you have a pretty good idea of the context because the skies are fairly uncluttered comparatively and you'd like to think that any fast moving incoming objects towards you here are probably gonna be incoming missiles.

1 of these systems shot down an Iranian passenger airliner by accident, which is obviously a significant accident.

See all Lex Fridman transcripts on Youtube

MIT AGI: Autonomous Weapons Systems Policy (Richard Moyes)