#ai #cicero #diplomacy 

A team from Meta AI has developed Cicero, an agent that can play the game Diplomacy, in which players have to communicate via chat messages to coordinate and plan into the future.

Paper Title: Human-level play in the game of Diplomacy by combining language models with strategic reasoning

Commented game by human expert: https://www.youtube.com/watch?v=u5192bvUS7k

OUTLINE:
0:00 - Introduction
9:50 - AI in cooperation games
13:50 - Cicero agent overview
25:00 - A controllable dialogue model
36:50 - Dialogue-conditional strategic planning
49:00 - Message filtering
53:45 - Cicero's play against humans
55:15 - More examples & discussion

Homepage: https://ai.facebook.com/research/cicero/
Code: https://github.com/facebookresearch/diplomacy_cicero
Blog: https://ai.facebook.com/blog/cicero-ai-negotiates-persuades-and-cooperates-with-people/
Paper: https://www.science.org/doi/10.1126/science.ade9097

Abstract:
Despite much progress in training AI systems to imitate human language, building agents that use language to communicate intentionally with humans in interactive environments remains a major challenge. We introduce Cicero, the first AI agent to achieve human-level performance in Diplomacy, a strategy game involving both cooperation and competition that emphasizes natural language negotiation and tactical coordination between seven players. Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players' beliefs and intentions from its conversations and generating dialogue in pursuit of its plans. Across 40 games of an anonymous online Diplomacy league, Cicero achieved more than double the average score of the human players and ranked in the top 10% of participants who played more than one game.

Authors: Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, Athul Paul Jacob, Mojtaba Komeili, Karthik Konath, Minae Kwon, Adam Lerer, Mike Lewis, Alexander H. Miller, Sasha Mitts, Adithya Renduchintala, Stephen Roller, Dirk Rowe, Weiyan Shi, Joe Spisak, Alexander Wei, David Wu, Hugh Zhang, Markus Zijlstra

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

Today, we'll look at Cicero, which is an agent, an AI agent created by meta AI, that can play the game of diplomacy.

Now diplomacy is a special game, because because it is a board game, where you need to communicate with the other players in order to coordinate actions and cooperate and also compete versus these other players.

And this coordination, as I said, is in natural language in chat messages.

So any AI agent has to actually communicate like a human to the other humans, at least if it doesn't want to get noticed as an AI agent.

Here you can see an instance of this board.

You can see there are these different territories, it's a bit pixelish, but I hope you can see there like territories.

And you can see you can see the world subdivided into these factions, which each are represented in a particular color.

So that will be all the things all the territories belonging to 1 given player, your goal is to get as many as many territories as possible, as specifically the ones that have supply centers on them.

And your moves are you have a bunch of moves available.

But you can also attack other territories.

Or you can, for example, support a player that attacks another territory.

So in a regular game down here somewhere, there'd be a chat window where you could chat with the other players.

And you can coordinate what you want to do what this other player wants to do, you can form alliances and form a build up trust with the other players and so on.

So this is very challenging for an AI agent in various ways.

We've seen board games before like poker or chess, but they're always like just competitive between 2 players, not really cooperative like this 1.

And obviously the chat messages here, they are a major part of this game.

You have to keep in mind that all the other players also communicate privately with each other, which is information that you don't know.

So Meta has made this agent called Cicero that plays this game and places ranks about in the top 10% of all humans in various tournaments.

Today we're going to look at how they built this agent, how it works, and what it does and what it means.

So the paper is called Human-Level Play in the Game of Diplomacy by combining language models with strategic reasoning.

As I said, it's by a set of authors at Meta and it's a pretty impressive system.

Here in the abstract it says, Cicero integrates a language model with planning and reinforcement learning algorithms by inferring players' beliefs and intentions from its conversations and generating a dialogue in pursuit of its plans.

SISR achieved more than double the average score of the human players and ranked in the top 10% of participants who played in more than 1 game.

Now again, we're going to go through this paper, but let me say this ahead of time.

This works, this agent is good because humans are dumb, like humans are really, really dumb.

I've read the paper, I've read the supplementary material, I've watched a YouTube video, which I'll link in the description, by a professional diplomacy player who comments on a game that they played versus Cicero.

Like it's just 1 human against 6 of these agents.

And my conclusion is that, okay, I've, it's overstated that humans are are stupid.

But this game, in my opinion, is first and foremost interesting to humans because of the human element, because you can build up trust as a human, and which is a major function of this diplomacy feature of this chat feature.

There's certainly want something to be said here about coordination, like the communication allows you to coordinate with other players, certain actions, but that's only part of why this is important.

The other part is, as I said, building up trust, chatter, making people happy and so on.

And the fact that like a professional, like the highest level of diplomacy players still do that still like build up trust and still say, well, they say, they say something like, well, in here, if if I were to do this to a human, the human would be like, it would be really be really flipped off and they would be against me for the rest of the game, even if it's irrational, but the bot doesn't do it because it's a bot.

And to me, it's like, well, if the highest levels of players succumb to things like tilt and being like aggressive and damped because you stab them in the back once, which is the most logical strategic move, then It's kind of like, I feel the humans play this because of that human element.

Not necessarily, I feel in this game, you could get away with throwing away a lot of the dialogue, except the coordination bit, and you can just play optimally, and there's nothing that people can do.

I thought for a long time you know what game would I really want to see AI play and my first instinct was something like werewolf or I guess the modern form is among us, because there are also like this negotiation and so on comes in.

Well, it's the human element, it's this human notion for trusting someone, which really has no place in a game like this, like in a game theoretic setting, building up something like trust, it means very little if you don't play the game repeatedly over a long time, like if it has an end, it doesn't like means nothing.

The other player can just betray you at any point and if they're better off, they want to do that.

There's like imagine in chess, if you like start trusting your opponent or something like this.

No, the highest levels, they are ruthless and I think Among Us would just become super duper boring if you take the humans out of it.

In any case, I feel it's still worth developing this bot here to interact with the humans because capturing this human element is I guess part of what this research is about and not as much getting really good at diplomacy because it feels like the field of diplomacy isn't that advanced.

I'm not sure if I'm insulting any diplomacy players right here, but from what I've seen, the whole chittery, chattery, trusty thing is like, it seems like the game is very far away from humans playing optimally.

So in diplomacy, 7 players conduct private natural language negotiations to coordinate their actions in order to both cooperate and compete with each other.

Cicero, this agent couples a controllable dialogue model with a strategic reasoning engine.

So the strategic reasoning engine here will be responsible for deciding what moves Cicero makes and the controllable dialogue model will be responsible for chatting with the other people.

And here is an important thing to notice.

And a little bit, well, I think this research is really, really, really cool.

But a criticism of me is that these things are quite disjoint.

And essentially, essentially, Cicero relies on this thing here very heavily on this strategic reasoning engine so it plans its moves ahead which is kind of sort of controlled by the dialogue it gets but only a little bit.

It plans its moves ahead and then it just communicates what it wants to do to the other players using this right here.

And because part of the game is about coordination and communication and also because humans generally are seem to be honest and therefore this the agent being always honest is also a good strategy or happens to be a good strategy.

In any case, what the model doesn't consider is strategically using language, right?

It just uses language, it determines what it wants to do, and then it uses language to like communicate that out.

But and then there's some filtering and so on.

But it never considers the what it says as a part of the strategy.

It never thinks, oh if I say this to that person then you know next turn they're going to do that.

At least not to the degree with which I would have hoped.

And we're going to see that but keep keep that in mind.

Also, the dialogue module as such is more like a translator.

So they try to essentially parse out what they call intents of the game.

And then they simply use the dialogue model to translate those intents like, you know, troop 1 moves to that country to translate that into like, hey, my troops are going to move to that country, is that okay with you?

But it's not really part of the strategy, the language.

So those are a bit of the disappointments that I have right here, but I think they also serve as the basis for further research.

So first of all, they go into a little bit of background.

What are the challenges of human-AI cooperation in diplomacy?

They say, in games involving cooperation, self-play without human data is no longer guaranteed to find a policy that performs well with humans.

This is in contrast to things like chess or Go, where you can just have 2 agents, right, have agent 1, and agent 2, and they just play against each other all the time.

And they will get better and better and better and hopefully converge to a really strong solution and under some conditions and optimal solutions.

Now, this is no longer guaranteed if you need to cooperate, especially what they say right here, a strategy that performs well with humans, right?

It's not necessarily about finding the most optimal strategy.

Even as I understand it, the most optimal strategy against humans, it's a strategy that performs well with humans if you need to cooperate.

Although in this game, I think you could find a really good strategy, absent of much communication.

Yeah, it says it may converge to a policy that's incompatible with human norms and expectations and that's the human element that I mentioned.

These norms and expectations, I think that's what makes these games interesting, makes these games fun to humans to sort of like, you know, are they telling the truth or they're lying?

It's that that's what makes it fun, right?

And I think that's why people play, play these games.

And yeah, interest, like that, that's, as I said, that's the exact aspect that's kind of not modeled in the dialogue model right here and in the strategic aspect.

So that's where a little bit of my criticism would come from right here.

The average, the agent here sends and receives an average of 292 messages per game.

The chat is really a big part of the game.

It's not as much the moves, it's like chat, chat, chat, chat, chat, coordinate, negotiate,

So the challenge is, they say each message the agent sends must be grounded.

If they just had like some sort of language model, it would do whatever, even if it's trained on data of that game.

However, you have to have a way to control the language model to say, language model, please transmit this piece of information right here to the other player.

And we're going to see how they train a language model that does it.

They say, lastly, diplomacy is a particularly challenging domain because success requires building trust with others in an environment that encourages players not to trust anyone.

Each turn's actions occur simultaneously after non-binding private negotiations.

Again, it encourages players to not trust anyone, yet you need to build trust.

So I've already explained the game in itself.

Yeah, 1 thing that I found important was this ability that a unit may support other units including those of another player.

And I think that is 1 of the mechanics that makes this game, you know, include this aspect of cooperation and coordination between players.

So it might very well be that players who do coordinate, even if they're technically enemies, who do coordinate for a move or 2, are better off at the end than had they not coordinated.

So here is a general overview over this agent.

We're going to look at some parts in more detail but this is essentially it.

You have this board state and the history over here.

This is quite your standard input to a reinforcement learning pipeline.

So the board state is essentially what's happening right now and the history is what was the move before and before and before that.

And sometimes that's actually relevant for the game, like in chess, the history plays a place, it has an influence to some degree, like you can't make certain moves twice.

In Atari games, it has some degree of relevance because if something flies with some velocity you want the history to estimate which direction it flies in.

Even if this is Markovian, sometimes it seems to help the algorithms, just because humans be humans, I guess, and it's not Markovian after all, but you can think of that yourself.

In any case, we get the board state as an input.

And that goes in 2 different directions as you can see.

So the first is this planning module here.

The planning module is very classic reinforcement learning planning module.

So we go from, essentially from the state, we determine a policy for all the players.

So that is what such a planning module does.

You can think of it a little bit like the Monte Carlo tree search in AlphaZero or something like this, except now you don't have 2 players, you have many players, so what you want to do is you want to determine a joint action, which means all the players move at the same time in this game.

So 1 action is going to be what every player is doing.

And the policy, what the policies are essentially the action distribution of all the players.

Then You want to forward simulate that into a future state and essentially repeat that.

So you plan multiple steps into the future.

And what you can also do is you can sort of run an improvement algorithm to make your policy better against all the other policies and then these policies better and so on.

So this is very classic, I would say, not even reinforcement learning.

This is just a very classic sort of policy computing algorithm that you might know from, from game theory papers or something like this.

The only interesting thing here or the novel thing is that you do get an input from what's called here, these anchor policies.

The anchor policies are what keeps the strategy in at a human level.

And it's a bit tricky to explain just here.

But essentially, if you let the model just do reinforcement learning, just do sort of computational planning up here, you quickly get into a state, that's what they explained above, where the actions become like non-human.

So where the actions, the algorithm thinks they're optimal, but the human would say like, that's kind of weird, No human plays like this.

And I've definitely seen sometimes this video commentator say something like this.

Now, usually, usually in something like chess or so, If you know alpha 0 is like 10 times as strong as the strongest human and the bot does something weird, then you're like, I guess that's a really good move.

Now here, it's a bit more tricky because it's a lot about, you know, this trust element, this human element.

There is a value to being more human, even if that means that technically you deviate from the most optimal action.

At least that's how the authors see it and that's why they have these anchor policies.

So the anchor policies are behavior cloning policies.

So what you do is you take a big data set, I guess here a big data set from human place and you train a behavior cloning algorithm.

Behavior cloning essentially means I take 1 game out, here is a state and an action and a state and an action.

I just observe past games, how they went, and I just train a model that if it's given a certain state is trained to perform the same actions as the humans did in that game.

Yeah, this is sometimes phrased as imitation learning, sometimes phrased as behavior cloning, has different names but all about the same ideas.

And that policy that they call an anchor policy because it anchors the model to what a human would do.

It's not necessarily the best action but it's an action that a human would do.

It's a little bit like a discriminator in an adversarial model.

They always mix the anchor policy with the reinforcement learned or with the computed policy in order to get a model that performs both well and like humans.

Yeah, and you can see right here, the anchor policies, those are dialogue conditional.

So you can see that here, dialogue conditional, because in this database, you obviously not only have the state as the board was, but you also have all the chit-chat that goes on inside of the state.

So you condition this behavior cloning policy, you say, okay, here is how the board looks, here what the humans have communicated, what has the human done?

Interestingly enough, up here, you see in this cycle here, there is no notion of any of the dialogue.

So all this planning here happens without the dialogue, whereas I think we might, Yes, all the planning here happens without the dialogue, except the dialogue comes in via the dialogue conditional action model.

So from here, the dialogue comes into this model, and then that information goes up here.

It's essentially the only information that the planning has about the action is what would a human do in this situation given this board and this dialogue.

That's the only information that you have about the dialogue.

You don't have the input dialogue directly and your actions, the actions that you do are not including what dialogue you're going to send.

Here you see only at the output of this planning module, you have something that you call intents.

So an intent is essentially a plan to move somewhere.

So the output of the planning module is here, the output action, what you do, but before you do it, or at the same time before the turn is over, you can also communicate to the others.

So you compute what you want to do based on everything that's happening.

So you say, I think I'm going to move my troop from here to here and they are going to move their troop from here to here and you can encode that as these intents.

And as I said before, what the language model does is it takes these intents and it translates them into chat messages.

So based on these intents you now go and you communicate with the other humans.

So you can see right here the message model or the message generation module here gets 3 inputs.

The board state as well, so it knows what the board looks like.

Then the current dialogue, like what currently has been discussed.

So now it's the turn of the agent to say something.

So it knows how do things look like, what has the other person told me, I guess, like what's the current status of the chat?

And what do I expect the other people to do next turn?

And from that, the dialogue model then generates message candidates, which go through filters, and if they pass the filter, they go into the chat.

So here you can see that the bot says something like Hi Italy, care to work together on this 1?

If you support me there I think we both be able to grow quickly.

Italy, which is the human in this turn, says Could you support me into Bol, into Bulgaria in return?

So now Austria takes everything into account What it wants to do, what it thinks Italy wants to do based on what's been said and so on.

And then it says, first thing, I have ordered Sir to support Greece, sorry, Serbia to support Greece to Bulgaria.

And yeah, So that's how the whole thing works.

We take in the current state, we take in the current dialogue.

First of all, we compute these anchor policies right here, like what would humans be doing, then we with the help of that, we also determine a best action to take, which is this planning loop right here.

Once we have the best action, we generate these intents from that, that's just mechanical, What do I want to do?

Those are just the policies, essentially, written out as intents.

And from that, we generate our messaging, our messages, which are intent conditioned.

And this happens in multiple steps, as I said, multiple planning loops.

So what I said before, like the dialogue doesn't come into the planning.

It does, but as I said, not in like a super direct way.

The agent cannot decide to strategically tell some other player something.

Like, the agent can only decide on an action, and then the dialogue model is just responsible for communicating that action to the other players.

The dialogue model is a central thing here.

It was trained to be controllable via intents.

So what you want to do is you want to have a dialogue model.

A message is defined to have intent Z if Z is the most likely set of actions that the sender and recipient will take for both the current turn and several future turns.

So that's how they determine the intent during training.

So during training they take a data set, they obviously don't know the plans of the people, but they take a data set and they annotate each chat message with what they think is the intent.

So they define the intent as essentially like the plan that results out of this chat message.

They say we developed techniques to automatically annotate every message in the training set with a set of actions corresponding to the message content.

During training, the dialogue model learned the distribution, this distribution where z represents the intent for data point x and y.

So x here is the input, whatever the dialogue model gets as an input, z is the intent, like what the agent thinks the plan of everyone is, or what they heuristically determined, and then y is the output.

So in this case, the dialogue model is tested for different intents.

You can hopefully see that if you compare the pictures a little bit.

And you can see, for example, this troop here is, I guess, going here.

That's the action that England takes or wants to take.

Over here it goes over here and over here it also goes over here but it even does a bunch of other things in turn.

And every time you can see that the chat messages that the bot sends now change.

I'm not a diplomacy player, so all I know is what they tell me.

So here they say, England convoys an army to Belgium with the support of France and Germany while taking Norway in a manner friendly to Russia.

So we expect these actions to be reflected in the chat messages.

So to France, it says, would you mind supporting this EDI to Belgium?

So it sends, since that is its intent to move into Belgium, it asks France, Hey, would you like to support me?

If since wait, Oh, the Germans, it also wants the German support.

So they say, do you want to support my convoy to Belgium?

With Italy going aggressive, France will fail quickly and we can make gains of both Russia and France.

So here you can see a bit of an extended example of this dialogue model.

To me it's like a tiny bit unclear where this comes from because they said that intents cover both this turn and turns in the future.

So it's quite likely that some of what the dialogue model here says is also contained in the intent, and it's kind of like the dialogue model presents it.

It's also somewhat likely that the dialogue model just sort of makes stuff up because it sees the board, right?

Yes, the dialogue model as far as I, yeah, the dialogue model sees the board itself and it sees the current intent.

See all Yannic Kilcher transcripts on Youtube

CICERO: An AI agent that negotiates, persuades, and cooperates with people