- The speaker, Rosalind Park, introduces Natasha, who will be presenting her thesis defense.
- Natasha has been studying machine learning for 5 years and is interested in incorporating social learning into AI agents.
- She discusses the problem of generalization in machine learning models and the need for intrinsic motivation in reinforcement learning.
- Natasha presents a project on multi-agent social learning, where agents learn from each other while training independently.
- The project focuses on giving agents intrinsic social motivation for influencing the actions of other agents.
- She explains how the agents use a predictive model to simulate counterfactual actions and measure the causal influence of their actions on other agents.
- The reward for influencing another agent is related to the mutual information between agents' actions.
- The speaker acknowledges the potential problem of agents influencing each other in a non-cooperative way and discusses approaches to address this issue.
- The project is tested in two environments, Harvest and Cleanup, which represent cooperation challenges.
- The speaker concludes by showing a shelling diagram that illustrates the payoff for cooperative and non-cooperative strategies in the environments.

- The video discusses the use of an influence reward in multi-agent reinforcement learning (RL) to promote cooperation and coordination among agents.
- The influence reward leads agents to communicate about the presence of food in the environment, similar to a bee waggle dance.
- Training explicit communication protocols with the influence reward improves collective return and meaningful communication.
- Agents that are more easily influenced tend to achieve higher individual environmental rewards.
- The influence reward is a unified method for promoting coordination and communication in multi-agent settings, allowing agents to learn socially from each other while training independently.
- Learning from humans in AI systems can help improve their performance and align their actions with human preferences.

- The project focuses on creating an AI agent that can sense human preferences in text conversations.
- The team built a platform for people to chat with the AI bots and collect human interaction data.
- They hypothesize that learning from implicit cues like sentiment is more effective than explicit manual ratings.
- The rewards for the AI agent include detecting sentiment, making the user laugh, asking questions, and staying on topic.
- Training the AI model with reinforcement learning is challenging due to the large action and state spaces.
- Learning from human interaction data requires good off-policy reinforcement learning.
- Testing the model carefully before deploying to humans is necessary to avoid inappropriate behavior.
- Batch reinforcement learning, without the ability to explore, presents additional difficulties.
- Several baselines were tested, but they did not perform well in terms of generating realistic language.
- The team proposes adding terms to the Q action value function to minimize the divergence of the RL policy from the pre-trained language model.

- The study focuses on using KL control from a pre-trained prior to handle off-policy, no exploration problem in batch RL.
- Several enhancements are proposed, including entropy regularization and replacing hard max with log sum x.
- KL control models significantly outperform baseline models in terms of ratings, votes, and reward elicited from humans.
- Batch Q models trivially exploit easy rewards, while KL control models shift their distribution to polite, supportive, and cheerful language.
- Sentiment-based reward leads to the highest quality conversation, and implicit signals are more scalable than explicit button presses.
- Another project explores learning from implicit signals from facial expressions using Sketch RNN.
- The latent constraints model is trained to understand which part of the representation space makes people smile and frown less.
- The model is applied to improve the quality of sketching cats and rhinoceroses, with significant improvements observed.
- The study is the first to show that generative models can be improved using facial expression feedback.

- The author is researching ways to detect affective signals, such as facial expressions and body language, in order to predict people's moods.
- They want to use this information to provide personalized predictions of people's moods and track long-term history to potentially detect depression.
- The accuracy of current models in predicting mood outcomes is low because they do not account for individual differences.
- The author proposes using multitask learning to personalize the models for each individual, resulting in significant performance gains.
- The models learned that individual differences play a significant role in mood, such as the importance of weather or routine for some people.
- The hierarchical Bayesian model identified different clusters of individuals based on their stress levels and personality traits.

- The study found that studious people who text a lot on campus are more stressed the next day, showing the importance of personalization in models.
- The researchers used a Gaussian process approach with domain adaptation to predict stress levels and adjusted the mean and variance of the Gaussian process based on individual data.
- Neural networks were also used, with the loss function changed, showing improved performance in regression and model fit.
- The researcher emphasized the power of social learning and how it can benefit AI systems by learning from humans.
- The integration of social learning and affective signal detection can lead to the development of multi-agent policies that can quickly coordinate with other agents and humans in cooperative environments.
- The researcher expressed gratitude to advisors, collaborators, and friends for their support throughout the thesis.

- The speaker praises the dissertation and asks about the concept of "social" in relation to agents and humans
- The speaker suggests that understanding and inferring intentions is important for coordination among agents and humans
- The speaker asks about the importance of compositional communication and the use of discrete symbols versus continuous vectors
- The speaker discusses the potential of using natural language in agent communication and coordination with humans
- The speaker asks about the possibility of testing if RL agents can explore higher abstract spaces using language
- The speaker suggests using generative adversarial imitation learning to improve reward functions in agent training.

- Natasha presents her thesis on incorporating social learning into AI agents and the problem of generalization in machine learning models.
- She discusses a project on multi-agent social learning, focusing on intrinsic social motivation and using a predictive model to measure causal influence.
- The project is tested in cooperation challenges and aims to promote cooperation and coordination among agents.
- Another project focuses on creating an AI agent that can sense human preferences in text conversations, using reinforcement learning and implicit cues.
- Learning from implicit signals from facial expressions is explored to improve generative models.
- The author's research aims to detect affective signals and personalize models for predicting mood outcomes.
- Social learning and affective signal detection can benefit AI systems in cooperative environments.
- The speaker discusses the importance of understanding intentions, compositional communication, and the use of natural language in agent coordination.
- The possibility of exploring higher abstract spaces using language and using generative adversarial imitation learning to improve reward functions is also mentioned.

- The speaker discusses the idea of using inverse reinforcement learning to observe good conversations and determine what makes them enjoyable.
- They mention the importance of learning from data and evaluating the policy before deployment to avoid adverse effects.
- The speaker discusses the challenges of predicting inappropriate behavior and the need for human assessment in determining if a policy is safe and appropriate.
- They mention the iterative process of updating the policy based on feedback and the potential use of techniques like importance sampling.
- The limitations of using model-free reinforcement learning are discussed, and the speaker suggests that modeling the world and using unsupervised learning could be promising alternatives.
- The speaker expresses their excitement about reinforcement learning as a broad paradigm that can describe various AI scenarios.
- They mention the potential of using latent characteristics of people to adapt RL models and the possibility of updating these characteristics based on feedback.
- The concept of intrinsic motivation is discussed, acknowledging the blurry line between intrinsic motivation and reward engineering.
- The speaker believes that intrinsic motivation is meaningful but acknowledges that not all implementations are generalizable to different environments.

- The speaker discusses their work on off-policy learning and the challenges of dealing with a high-dimensional action space.
- They emphasize the importance of having a reasonable pre-trained model to avoid issues with overestimation and extrapolation errors.
- They mention that they are exploring the use of hierarchical models to reduce complexity and allow the model to choose conversational goals.
- The speaker highlights the difficulty of credit assignment at the word level and suggests that credit assignment at the sentence level makes more sense for understanding conversational intention.
- They express the desire for more time to delve into projects in greater detail and prioritize quality over quantity in research papers.
- The speaker discusses the potential of integrating reinforcement learning and gradient descent learning to improve model-based RL.
- They mention the benefits of pre-training models for representation learning and suggest aligning language understanding with acting in the world for more general intelligence.
- The speaker contemplates the possibility of starting with a restricted set of words and gradually expanding it in a curriculum-style learning process.
- They suggest that affect sensitivity and empathy are important for improving human learning and understanding intentions in human interaction.

- The speaker discusses the importance of perspective taking and modeling intentions in understanding and coordinating with others.
- Affective signals can be used to update and understand what others are trying to communicate or do.
- The speaker thanks Natasha for her presentation and mentions that there will be ice cream in the affective computing area.

The speaker discusses the use of inverse reinforcement learning to analyze good conversations and determine their enjoyable aspects. They emphasize the importance of learning from data and evaluating policies before deployment. The challenges of predicting inappropriate behavior and the need for human assessment are mentioned. The iterative process of updating policies based on feedback and the limitations of model-free reinforcement learning are discussed. The speaker expresses excitement about reinforcement learning as a broad paradigm and mentions the potential of using latent characteristics of people. They discuss intrinsic motivation, off-policy learning, and the challenges of dealing with a high-dimensional action space. The importance of reasonable pre-trained models, hierarchical models, and credit assignment at the sentence level is highlighted. The speaker desires more time for detailed projects and prioritizes quality over quantity in research papers. They contemplate integrating reinforcement learning and gradient descent learning, pre-training models, and gradually expanding word sets. The importance of affect sensitivity, empathy, perspective taking, and modeling intentions in human interaction is discussed. The speaker thanks Natasha for her presentation and mentions ice cream in the affective computing area.

Presentation of my thesis "Towards Social and Affective Machine Learning" https://natashajaques.ai/publication/social-and-affective-machine-learning/

I'm Rosalind Park, I'm the thesis advisor for the Jakes, and this is a very happy, hopefully, occasion where Natasha's going to be presenting the defense for her thesis, quite a large body of work.

She's gonna present for probably a little bit more than 45 minutes.

Then we will open it up to Q&A to the audience.

If you have to ask a clarifying question as we go, that's fine too, but we wanna save real conversational discussion questions for the end.

Then at some, probably after about 20 to 30 minutes of public back and forth, we will chase out the public and the committee will meet with Natasha.

After that, if she's determined to have passed her defense, there will be a small celebration in our group around the corner back over there to which all will be invited.

And I'm sorry, Doug and Nando won't be able to come for that part.

So without further ado, you've got a mic on.

Nando is has joined but Doug is still joining.

I'm excited to be here Wow, Thank you guys so much for coming.

And if you're wondering who Doug and Nando are, I just want to mention, Thanks to my committee for being here.

Doug Eck and Nando DeFritis are at Google Brain and DeepMind.

So very excited that they're able to join us as well.

So I've been studying the problem of machine learning for the last 5 years here at MIT.

And if you know me at all, you'll know that I'm very willing to jump into an excited rant about how amazing machine learning is and how we're making so much progress, and it's so exciting.

Interest in machine learning, and in particularly deep learning, has been exploding over the last few years because we've able- we've been able to tackle new problems that we weren't able to tackle before.

And so, you know, I'm very hyped about this, but, we know that there's still huge problems with these techniques.

So for 1 of the biggest problems is generalization.

If you train a model to play an Atari video game that looks like this, and you just slightly switch the colors, it just collapses.

And that's just a far cry from a model that could play this video game and this 1 at the same time.

So we know that these models are not good at generalizing to new problems.

So this is particularly a problem in reinforcement learning, which is an area that I'm really excited about.

And even though we've made impressive progress in reinforcement learning in the last few years, like we've beaten the world champion at the game of Go, a human champion, again generalization is still a problem.

So 1 way to overcome this is to think about intrinsic motivation for reinforcement learning.

So this is where you design a reward function that the agent is trying to optimize that's environment agnostic.

And it could cause this agent to learn across many different tasks.

So a couple of areas where we can develop intrinsic motivation are, for example, curiosity.

So if an agent is driven to discover novelty in its environment, then perhaps it will learn across many environments.

Similarly, empowerment rewards an agent for having high mutual information between its own actions and its future state.

So basically it wants to be able to manipulate the environment, it wants to have power over the environment.

But if I look at this, I think the things that motivate people aren't limited to a drive for novelty and a drive for power.

I would argue that people are very motivated by other people.

So if you guys were looking at me like this audience here, I would be extremely motivated to go to the next slide.

So humans are really good social learners.

I would argue that social learning is a key component of why humans are so smart.

And if you don't believe me, I've thrown up a lot of citations here that argue that social learning not only drives our cognitive development, but it's actually tied to our ability to transmit knowledge and evolve culturally, which is probably why I'm standing here talking to you about deep reinforcement learning, for example.

So the focus of this thesis is broadly about how to incorporate forms of social learning into AI agents.

And the first project I'm going to talk about is a multi-agent project.

How could we develop a form of social learning among the agents?

And In this project, we wanted the agents to be able to learn socially from each other, but still train independently.

And so we chose to do this by giving them intrinsic social motivation for having a causal influence over the actions of another agent.

Well, if we focus on learning from other agents actions, then we don't need to observe the reward for getting they're getting from the environment.

And we think this is a more realistic assumption when we think about applications for these types of systems like autonomous driving.

So with autonomous vehicles there's likely to be cars that are manufactured by Tesla and Waymo.

And the Waymo car isn't going to be willing to share its proprietary reward function that it's optimizing with the Tesla car.

So how could those cars still learn from each other just by observing what they're doing in the environment, such as whether they turn left.

And we'll show that this causal influence reward is actually related to rewarding the mutual information between agents' actions.

And we have the intuition that this will help agents learn more coordinated behavior with each other.

That's all fine, but how can we actually make this work?

Well, what we do is we start with a standard reinforcement learning agent that's inputting something from the environment, and it's having a recurrent module that it can understand what's been happening in the past.

It's outputting a policy about how to act in the environment.

And then we equip it with an extra model.

And we're calling this a model of other agents.

And what this model is doing is it's predicting the actions of the other agents.

So it predicts at the next time step what every other agent is going to do conditioned on the agent's own action.

Well it's actually able to then simulate counterfactual actions that it could have taken.

So it can retrospectively ask, given everything that was going on at this moment in time, what if I had taken this other action instead?

So it can ask itself, what do I predict this other agent would do if I had done this action instead?

And then what that allows it to do is it can sample a bunch of counterfactual actions that it could have taken, think about all the things it could have done, plug that into its predictive model and sum over that to get this policy of the other agent, what the other agent would do if it wasn't considering this agent at all.

And the divergence, the difference between the other agent's default policy and its policy conditioned on this agent's action is a measure of the causal influence of this agent's action on the other agent.

And the idea is to give this as a reward to agent A.

So the agents get a reward if they influence the behavior of another agent.

And if you don't believe me that it's the causal influence, that's what this cool graph is for.

So we know from Perl's notion of causality in directed graphs That causality can't pass between an unconditioned collider node.

So the only causal path is this path that goes forward in time between this agent's action at time step t and agent B's action at time step t plus 1.

But we can also show that this influence reward is related to the mutual information between agents' actions.

So this is the formula for the mutual information here.

We see that it requires summing over the probability of the state, for example.

But as we sample many random trajectories from the environment, we actually in expectation over those trajectories recover this mutual influence reward.

So influences the mutual information in expectation.

So we think of this as a novel form of social empowerment.

If you remember, empowerment was the mutual information between an agent's action and its own future state, and this is the mutual information between multiple agents' actions.

Does anybody see a problem with rewarding an agent for causally influencing the actions of another agent?

Does an agent, in order to influence another agent, does the agent always have to help the other agent?

So it might be possible for me to just get in your way, you have to walk around me, I haven't helped you but I've influenced you.

So how do we make sure that the agents are influencing each other in a cooperative way?

The first is that we tested this reward in environments where cooperation is precisely the hard problem.

It's a tragedy of the commons environment.

So what agents are trying to do is pick up these apples and they get a reward for doing so.

But if they greedily harvest the apples too quickly then the apples will be depleted, they won't grow back, everyone's gonna go hungry.

So even though each agent wants to harvest more for themselves, if they all follow that strategy everyone loses.

It's meant to mimic like a spatially and temporally extended prisoner's dilemma.

Agents can make apples appear by cleaning this river, but it's partially observable.

So when they're doing this, they can't actually see if these apples are appearing.

Other agents can easily just exploit them by kind of stealing their apples.

So to make that more concrete, what we're looking at here is called a shelling diagram.

And it's describing the payoff that each agent can get for following, for example, a defecting or exploitative strategy in red, or a cooperative strategy in blue, depending on the number of other agents out of 5 that are cooperating with it.

So if everyone's cooperating, or if no 1 is cooperating.

And you can see that the agents are always going to get a higher payoff if they defect on their fellow comrades here, but of course if everyone follows that strategy again everyone loses.

So these types of long-term trade-offs over repeated iterations are very hard for vanilla RL algorithms to solve.

But what we see is that when we add this reward for agents having influence over each other or having high mutual information, we see that the collective reward obtained by all of the agents is higher.

And that means basically it's not any given agent that's doing better, but all of them are doing better together in this environment, meaning they're harvesting sustainably, for example, and it's evidence that they're cooperating.

How is this influence reward leading them to cooperate?

Oh, and I should mention this actually does, beat previous work.

So, this previous work, agents were able to view each other's reward function, and we actually see superior performance just from this influence reward.

So in this video, We have this purple agent here is the only 1 that's been trained with the influence reward.

And you can see that it behaves a little bit differently than the other agents.

So when there are no apples, the other agents keep exploring to try and find apples.

And in fact, it only ever used 2 moves to navigate the map, and it only ever traverses the map when there are apples present.

How does that allow it to gain influence?

Well, if we take a look at a moment of high influence between the purple agent and this yellow agent here that can only see what's in this partially observable box, we see that there's this delicious apple that's outside of the view of this agent.

But when the purple agent chooses to move towards the apple, which it only moves when there is an apple, that communicates something to this agent that there must be food present, it changes its intended behavior, and this agent is able to gain influence.

And then we see something like that happening again.

So here, this pink agent here has been cleaning the river.

It can't see if any apples have appeared.

But when this agent chooses to stay still, that signals that no apples have appeared, again, changing its intended plan, and the agent gains influence.

So what we see is that this influence reward has led the agent to learn to use its actions to communicate about the presence of food in the environment.

So you can think of it like a little bee waggle dance.

And so the next thing we did was to train explicit communication protocols with this influence reward.

So what we're doing here, here's our normal policy for acting in the environment and this is a policy that says which of several communication symbols the agent should choose to emit and show to the other agents at the next time step.

So it's kind of this cheap talk communication channel.

I can say something, nobody else has to listen to it.

The agents are free to ignore that channel.

And the idea is that we're going to train this communication policy with the influence reward.

So agents get a reward if something they say influences the behavior of another agent.

And we think that this actually overcomes the problem of the influence not being beneficial.

Because the only way an agent that is acting in the environment to maximize its own reward is going to change its behavior based on a communication symbol that it's free to ignore is if that symbol is going to reliably help it achieve its own environmental reward.

So we think that influential communication should benefit the listener.

Well I should have first point out that cheap talk between self-interested agents actually doesn't work.

So we can show theoretically that as agents' interests diverge, the amount of information placed on the channel goes to 0.

And we can show empirically in past work in multi-agent reinforcement learning that agents that are self-interested don't learn to use these types of channels effectively.

But we see that when we add this influence reward, the agents achieve, again, higher collective return, suggesting that it is helping them to coordinate.

And we also see that the communication protocols they learn are more meaningful.

So agents are speaking more consistently about what they're doing in the environment.

And they're also having high mutual information, as we suspected, when there are moments of influence, although that is a very sparse occurrence.

So what you're looking at here is a degree of influence between an influencer agent and an agent being influenced and their individual returns.

And what we see is that agents that are more easily influenced or a better listener, if you will, tend to get higher individual environmental reward.

And this is a reliable finding across many different hyperparameter settings, hundreds of examples of these environments.

So this seems to support our hypothesis that the information being sent over the channel is helping agents to achieve their own environmental reward.

So this slide is to show that these results are reliable.

We ran it not only for 5 random seeds for each hyperparameter setting, but for the top 5 hyperparameter settings.

And we see consistent performance improvements from this influence reward.

I do want to mention that there are some caveats.

So we talked about whether the action-action influence is going to help the other agent, is going to depend on the environment, on the hyperparameter settings.

We think that 1 of the reasons that communication emerged in this setup is that agents were trying to maximize both their environmental reward and their influence and actually just communicating information may be 1 of the cheapest ways to gain some influence without deviating too much from your optimal policy in the environment.

We think that influence on self-interested agents via this cheap talk channel should benefit the listener, but that depends on agents that are interacting repeatedly.

So if in some future work there was some agreed upon communication protocol that agents could use to communicate with a new agent that they would never see again, they could potentially lie and still gain influence.

But of course, if you're playing with the same agent over and over and they lie to you, you're not going to trust them and they will lose influence over time.

So in conclusion, we think of this social influence as a unified method for promoting coordination and communication in multi-agent settings.

And this allows the agents to learn socially from each other but train independently.

And that's actually a big improvement over prior work in MARL, which often relies on agents viewing each other's reward or having a centralized controller that teaches all the agents something.

I want to stop right now if anyone has a question.

For the defense of agents, do they have to infer each other's actions?

Yeah, so we make the assumption that agents can observe each other's actions.

But actually, in the final product, we ended up saying that you can't influence an agent that you can't see.

That's not within your partially observable view of the world.

Also, also via the communication channel.

The communication messages are always sort of heard.

Even if you're outside of the view, they're transmitted to all the agents.

Which we again think is pretty reasonable.

What is the dimension of this communication channel?

We vary the dimensions I think between like 10 symbols and 20 symbols, so pretty small.

But they can actually- the interesting thing about this that we didn't get a chance to analyze, is they could potentially communicate compositional messages.

So if I say AAB, that might mean something different than AAC.

And actually, the measure, the metric that we have of the communication doesn't account for that.

So it would be nice future work to more deeply analyze how the agents are communicating.

We can always take more questions at the end.

So I'm really interested in this idea of social learning because I think it's really powerful.

But it seems like the best and smartest things you could learn from in your environment are going to be humans.

For most tasks that we want an AI to do, like for example just recommend us music, humans are gonna know better how to perform the task.

And I'll give you a motivating example why learning from humans could help.

So this is a real conversation between a user and their Alexa device.

The right way to spell Walmart is W-A-R-M-A-R.

So I would bet you that when this mistake happens and Alexa doesn't meet the need of this person, the way they're going to respond is going to contain some element of frustration.

Whether it's their tone of voice, the words that they use, there's a signal there that Alexa could be using to understand that she did the wrong thing and she better not do it again.

So wouldn't it be cool if we could pick up on these signals and learn from them?

And this is important not just for making deployed systems like Alexa more useful to people, but I would argue it's important for long-term considerations of AI safety.

Because if these models are able to, in a sense, accurately our preferences and adapt to them, then they're going to be more likely to take actions that are aligned with their preferences.

So that was, but the problem is that people don't actually enjoy manually labeling their preferences.

So nobody actually wants to fill out these cards.

How many people have ever trained the Google algorithm by filling this out?

Okay, got a few hardcore people, thanks dad, good job.

I think most people don't actually enjoy doing this.

But in contrast, social feedback is rich and ubiquitous and natural.

You're already making facial expressions.

So how many people have ever used a frustrated tone with their Alexa or Siri or yelled at their Roomba?

So we wanted to make an AI agent that could sense human preferences.

And here we wanted to sense this in text.

So here's an example of me talking to a conversational AI, a chat bot, and you can see that we have this state-of-the-art sentiment detector that was actually developed here in the media lab that can tell that when I say something like, that didn't make sense, that's a negative sentiment.

So couldn't we use this as a reward signal to disincentivize the agent from whatever it just did?

Similarly, When the conversation is going well, it can detect that I'm happy and the agent is doing something right.

I worked on this project with a number of wonderful collaborators in this group, including Asma and Judy, and I'll talk more about that later, and Craig, Agata.

But we ended up building this whole platform to allow people to chat online with our bots and collect this human interaction data.

And so our hypothesis, even though in that interface we allow people to manually tag the bots' responses as good or bad using this upvote, downvote, we think that learning from these explicit manual ratings is not gonna scale as well as learning from these implicit cues like the sentiment.

And so the implicit cues we focused on in this work were not only the sentiment of each utterance, as well as over the course of the conversation, but even whether we could get the user to laugh, how much they typed ha, how many words they typed, how long they bothered to talk to the bot, because we reasoned that if it's totally terrible, you can't really put up with talking to it for too long.

And we also rewarded the distance in embedding space between the bot's answer and the human input to help it stay on topic.

So these are rewards that are elicited from interacting well with a human.

They were based on some research into the psychology of what makes a good conversation.

See all Natasha Jaques transcripts on Youtube

Natasha Jaques PhD Thesis Defense