The following is a conversation with Michael Kearns.

He's a professor at the University of Pennsylvania and a co-author of the new book, "'Ethical Algorithm'", that is the focus of much of this conversation.

It includes algorithmic fairness, bias, privacy, and ethics in general.

But that is just 1 of many fields that Michael is a world-class researcher in, some of which we touch on quickly, including learning theory, or the theoretical foundation of machine learning, game theory, quantitative finance, computational social science, and much more.

But on a personal note, when I was an undergrad, early on, I worked with Michael on an algorithmic trading project and competition that he led.

That's when I first fell in love with algorithmic game theory.

While most of my research life has been in machine learning and human robot interaction, the systematic way that game theory reveals the beautiful structure in our competitive and cooperating world of humans has been a continued inspiration to me.

So for that and other things, I'm deeply thankful to Michael and really enjoyed having this conversation again in person after so many years.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, give it 5 stars on Apple Podcasts, support on Patreon, or simply connect with me on Twitter, Alex Friedman, spelled F-R-I-D-M-A-N.

This episode is supported by an amazing podcast called Pessimists Archive.

Jason, the host of the show, reached out to me looking to support this podcast and so I listened to it to check it out.

And by listened, I mean I went through it Netflix binge style at least 5 episodes in a row.

It's not 1 of my favorite podcasts, and I think it should be 1 of the top podcasts in the world, frankly.

It's a history show about why people resist new things.

Each episode looks at a moment in history when something new was introduced, something that today we think of as commonplace, like recorded music, umbrellas, bicycles, cars, chess, coffee, the elevator, and the show explores why it freaked everyone out.

The latest episode on mirrors and vanity still stays with me as I think about vanity in the modern day of the Twitter world.

That's the fascinating thing about this show, is that stuff that happened long ago, especially in terms of our fear of new things, repeats itself in the modern day, and so has many lessons for us to think about in terms of human psychology and the role of technology in our society.

Anyway, you should subscribe and listen to Pessimist Archive.

And now, here's my conversation with Michael Kearns.

You mentioned reading Fear and Loathing in Las Vegas in high school and having a more, or a bit more of a literary mind.

So what books, non-technical, non-computer science, would you say had the biggest impact on your life, either intellectually or emotionally?

Yeah, I think, well, my favorite novel is Infinite Jest by David Foster Wallace, which actually, coincidentally, much of it takes place in the halls of buildings right around us here at MIT.

So that certainly had a big influence on me.

And as you noticed, like when I was in high school, I actually even started college as an English major.

So was very influenced by sort of that genre of journalism at the time and thought I wanted to be a writer and then realized that an English major teaches you to read but it doesn't teach you how to write and then I became interested in math and computer science instead.

Well, in your new book, Ethical Algorithm, you kinda sneak up from an algorithmic perspective on these deep, profound philosophical questions of fairness, of privacy, in thinking about these topics, how often do you return to that literary mind that you had?

Yeah, I'd like to claim there was a deeper connection, but I think both Aaron and I kind of came at these topics first and foremost from a technical angle.

I mean, I kind of consider myself primarily and originally a machine learning researcher.

And I think as we just watched like the rest of the society, the field technically advance and then quickly on the heels of that kind of the buzzkill of all of the anti-social behavior by algorithms, just kind of realized there was an opportunity for us to do something about it from a research perspective.

More to the point in your question, I mean, I do have an uncle who is literally a moral philosopher.

And so in the early days of our technical work on fairness topics, I would occasionally run ideas behind him.

So, I mean, I remember an early email I sent to him in which I said like, oh, here's a specific definition of algorithmic fairness that we think is some sort of variant of Rawlsian fairness.

And I thought I was asking a yes or no question.

And I got back to a kind of classical philosopher's response, saying, well, it depends.

If you look at it this way, then you might conclude this.

And that's when I realized that there was a real kind of rift between the ways philosophers and others had thought about things like fairness, you know, from sort of a humanitarian perspective, and the way that you needed to think about it as a computer scientist if you were going to kind of implement actual algorithmic solutions.

But I would say the algorithmic solutions take care of some of the low hanging fruit.

Sort of the problem is a lot of algorithms, when they don't consider fairness, they are just terribly unfair.

And when they don't consider privacy, they're terribly, they violate privacy.

Sort of the algorithmic approach fixes big problems, but there's still, you get, when you start pushing into the gray area, that's when you start getting into this philosophy of what it means to be fair, starting from Plato, what is justice kind of questions.

And I mean, I would even not go as far as you went to say that sort of the algorithmic work in these areas is solving like the biggest problems.

And we discussed in the book the fact that really we are, there's a sense in which we're kind of looking where the light is in that, for example, if police are racist in who they decide to stop and frisk, and that goes into the data, there's sort of no undoing that downstream by kind of clever algorithmic methods.

And I think especially in fairness, I mean, I think less so in privacy where we feel like the community kind of really has settled on the right definition, which is differential privacy.

If you just look at the algorithmic fairness literature already, you can see it's gonna be much more of a mess.

And you've got these theorems saying, here are 3 entirely reasonable, desirable notions of fairness.

And here's a proof that you cannot simultaneously have all 3 of them.

So I think we know that algorithmic fairness compared to algorithmic privacy is gonna be kind of a harder problem.

And it will have to revisit, I think, things that have been thought about by many generations of scholars before us.

So it's very early days for fairness, I think.

So before we get into the details of differential privacy and on the fairness side, let me linger on the philosophy a bit.

Do you think most people are fundamentally good?

Or do most of us have both the capacity for good and evil within us.

I tend to think that most people are good and want to do right and that deviations from that are usually due to circumstance, not due to people being bad at heart.

the heads of governments, people at the heads of companies, people at the heads of maybe, so financial power markets, Do you think the distribution there is also most people are good and have good intent?

I mean, my statement wasn't qualified to people not in positions of power.

I mean, I think what happens in a lot of the, you know, the cliche about absolute power corrupts absolutely.

I mean, I think even short of that, having spent a lot of time on Wall Street and also in arenas very, very different from Wall Street, like academia, 1 of the things I think I've benefited from by moving between 2 very different worlds is you become aware that, you know, these worlds kind of develop their own social norms and they develop their own rationales for, you know, behavior, for instance, that might look unusual to outsiders, but when you're in that world, it doesn't feel unusual at all.

And I think this is true of a lot of professional cultures, for instance, And so then your maybe slippery slope is too strong of a word, but you're in some world where you're mainly around other people with the same kind of viewpoints and training and worldview as you.

And I think that's more of a source of kind of abuses of power than sort of, there being good people and evil people, and that somehow the evil people are the ones that somehow rise to power.

So it's the, within the social norms constructed by that particular group of people, you're all trying to do good.

But because it's a group, you might drift into something that for the broader population, it does not align with the values of society.

Yeah, I mean, or not that you drift, but even the things that don't make sense to the outside world don't seem unusual to you.

So it's not sort of like a good or a bad thing, but you know, like, so for instance, you know, on, on, in the world of finance, right, there's a lot of complicated types of activity that if you are not immersed in that world, you cannot see why the purpose of that, you know, that activity exists at all.

It just seems like, you know, completely useless and people just like, you know, pushing money around.

And when you're in that world, right, you're, you're, you, and you learn more, you, your view does become more nuanced, right?

You realize, okay, there is actually a function to, to this activity.

And for, in some cases you would conclude that actually if magically we could eradicate this activity tomorrow, it would come back because it actually is like serving some useful purpose.

It's just a useful purpose that's very difficult for outsiders to see.

And so I think, you know, Lots of professional work environments or cultures, as I might put it, have these social norms that don't make sense to the outside world.

Lots of people look at academia and say, what the hell are all of you people doing?

Why are you paid so much, in some cases, at taxpayer expenses to do, you know, to...

But when you're in that world, you come to see the value for it, but even though you might not be able to explain it to the person in the street.

And in the case of the financial sector, tools like credit might not make sense to people.

It's a good example of something that does seem to pop up and be useful, or just the power of markets and just in general capitalism.

In finance, I think the primary example I would give is leverage, right?

So being allowed to borrow to sort of use 10 times as much money as you've actually borrowed, right?

So that's an example of something that before I had any experience in financial markets, I might have looked at and said, well, what is the purpose of that?

And it is dangerous and it has proven dangerous.

But if the fact of the matter is that on some particular time scale, you are holding positions that are, you know, very unlikely to, you know, lose, you know, your value at risk or variance is like 1 or 5 percent, then it kind of makes sense that you'd be allowed to use a little bit more than you have because you have some confidence that you're not going to lose it all in a single day.

Now, of course, when that happens, we've seen what happens not too long ago.

But the idea that it serves no useful economic purpose under any circumstances is definitely not true.

We'll return to the other side of the coast, Silicon Valley, and the problems there, as we talk about privacy, as we talk about fairness.

At the high level, and I'll ask some sort of basic questions with the hope to get at the fundamental nature of reality.

But from a very high level, what is an ethical algorithm?

So I can say that an algorithm has a running time of using big O notation, N log N.

I can say that a machine learning algorithm classified cat versus dog with 97% accuracy.

Do you think there will 1 day be a way to measure sort of in the same compelling way as the big O notation of this algorithm is 97% ethical?

First of all, let me riff for a second on your specific n log n example.

So because early in the book when we're just kind of trying to describe algorithms period, we say like, okay, what's an example of an algorithm or an algorithmic problem?

You have a bunch of index cards with numbers on them and you want to sort them.

And we describe an algorithm that sweeps all the way through, finds the smallest number, puts it at the front, then sweeps through again, finds the second smallest number.

So we make the point that this is an algorithm and it's also a bad algorithm in the sense that it's quadratic rather than N log N, which we know is kind of optimal for sorting.

And we make the point that sort of like, so even within the confines of a very precisely specified problem, There might be many, many different algorithms for the same problem with different properties.

Like some might be faster in terms of running time, some might use less memory, some might have better distributed implementations.

And so the point is, is that already we're used to, you know, in computer science, thinking about trade-offs between different types of quantities and resources, and there being, you know, better and worse algorithms.

And our book is about that part of algorithmic ethics that we know how to kind of put on that same kind of quantitative footing right now.

So, just to say something that our book is not about.

Our book is not about kind of broad, fuzzy notions of fairness.

It's about very specific notions of fairness.

But if you pick 1 of them, you can do something akin to saying that this algorithm is 97% ethical.

You can say, for instance, for this lending model, the false rejection rate on black people and white people is within 3%.

So we might call that a 97% ethical algorithm, and a 100% ethical algorithm would mean that that difference is 0%.

In that case, fairness is specified when 2 groups, however they're defined, are given to you.

So the, and then you can sort of mathematically start describing the algorithm, but Nevertheless, the part where the 2 groups are given to you, unlike running time, we don't in computer science talk about how fast an algorithm feels like when it runs.

We measure it, and ethical starts getting into feelings.

So for example, an algorithm runs, if it runs in the background, it doesn't disturb the performance of my system, it'll feel nice, I'll be okay with it.

But if it overloads the system, it'll feel unpleasant.

So in that same way, ethics, there's a feeling of how socially acceptable it is, how does it represent the moral standards of our society today.

So in that sense, and sorry to linger on that, first of all, high-level philosophical question is, do you have a sense we'll be able to measure how ethical an algorithm is?

give the impression that you can kind of measure memory speed trade-offs, you know, and that there's a complete mapping from that onto fairness, for instance, or ethics and accuracy, for example.

In the type of fairness definitions that are largely the objects of study today and starting to be deployed, you as the user of the definitions, you need to make some hard decisions before you even get to the point of designing fair algorithms.

1 of them, for instance, is deciding who it is that you're worried about protecting, who you're worried about being harmed by, for instance, some notion of discrimination or unfairness.

And then you need to also decide what constitutes harm.

So for instance, in a lending application, Maybe you decide that falsely rejecting a creditworthy individual, sort of a false negative, is the real harm and that false positives, i.e.

People that are not creditworthy or are not going to repay your loan, that get a loan, you might think of them as lucky.

And so that's not a harm, although it's not clear that if you don't have the means to repay a loan, that being given a loan is not also a harm.

So, you know, the Literature is sort of so far quite limited in that you sort of need to say, who do you want to protect and what would constitute harm to that group?

And when you ask questions like, will algorithms feel ethical?

1 way in which they won't under the definitions that I'm describing is if you are an individual who is falsely denied alone, incorrectly denied alone, all of these definitions basically say like, well, your compensation is the knowledge that We are also falsely denying loans to other people, you know, in other groups at the same rate that we're doing it to you.

And, you know, and so there is actually this interesting, even technical tension in the field right now between these sort of group notions of fairness and notions of fairness that might actually feel like real fairness to individuals, right?

They might really feel like their particular interests are being protected or thought about by the algorithm rather than just, you know, the groups that they happen to be members of.

Is there parallels to the big O notation of worst case analysis?

So is it important to, looking at the worst violation of fairness for an individual, is it important to minimize that 1 individual?

So like worst case analysis, is that something you think about or?

I mean, I think we're not even at the point where

So first of all, you know, we're talking here both about fairness applied at the group level, which is a relatively weak thing, but it's better than nothing, and also the more ambitious thing of trying to give some individual promises.

But even that doesn't incorporate, I think, something that you're hinting at here is what a child might call subjective fairness, right?

So a lot of the definitions, I mean, all of the definitions in the algorithmic fairness literature are what I would kind of call received wisdom definitions.

It's sort of, you know, somebody like me sits around and things like, okay, you know, I think here's a technical definition of fairness that I think people should want or that they should, you know, think of as some notion of fairness, maybe not the only 1, maybe not the best 1, maybe not the last 1.

But we really actually don't know from a subjective standpoint like what people really think is fair.

There's, you know, we just started doing a little bit of work in our group at actually doing kind of human subject experiments in which we, you know, ask people about, you know, we ask them questions about fairness, we survey them, we, you know, we show them pairs of individuals in, let's say, a criminal recidivism prediction setting, and we ask them, do you think these 2 individuals should be treated the same as a matter of fairness?

And to my knowledge, there's not a large literature in which ordinary people are asked about, they have notions of their subjective fairness elicited from them.

It's mainly, you know, kind of scholars who think about fairness, you know, kind of making up their own definitions.

And I think this needs to change actually for many social norms, not just for fairness, right?

So there's a lot of discussion these days in the AI community about interpretable AI or understandable AI.

And as far as I can tell, everybody agrees that deep learning, or at least the outputs of deep learning, are not very understandable.

And people might agree that sparse linear models with integer coefficients are more understandable.

There's very little literature on showing people models and asking them, do they understand what the model is doing?

And I think that in all of these topics, as these fields mature, we need to start doing more behavioral work.

Yeah, which is, so 1 of my deep passions is psychology, and I always thought computer scientists would be the best future psychologists.

In a sense, that data is, especially in this modern world, the data is a really powerful way to understand and study human behavior.

And you've explored that with your theory side of work as well.

Yeah, I'd like to think that what you say is true about computer scientists and psychology from my own limited wandering into human subject experiments, we have a great deal to learn.

Not just computer science, but AI and machine learning more specifically, I kind of think of as imperialist research communities in that, you know, kind of like physicists in an earlier generation, computer scientists kind of don't think of any scientific topic as off limits to them.

They will like freely wander into areas that others have been thinking about for decades or longer.

And, you know, we usually tend to embarrass ourselves in those efforts for some amount of time.

I think reinforcement learning is a good example.

So a lot of the early work in reinforcement learning, I have complete sympathy for the control theorists that looked at this and said, like, okay, you are reinventing stuff that we've known since like the 40s, right?

But in my view, eventually, this sort of computer scientists have made significant contributions to that field, even though we kind of embarrassed ourselves for the first decade.

So I think if computer scientists are gonna start engaging in kind of psychology, human subjects type of research, we should expect to be embarrassing ourselves for a good 10 years or so, and then hope that it turns out as well as some other areas that we've waded into.

So you kind of mentioned this, just to linger on the idea of an ethical algorithm, of idea of groups, sort of group thinking and individual thinking.

1 of the amazing things about algorithms and your book and just this field of study is it gets us to ask, like forcing machines, converting these ideas into algorithms is forcing us to ask questions of ourselves as a human civilization.

So there's a lot of people now in public discourse doing sort of group thinking, thinking like There's particular sets of groups that we don't want to discriminate against and so on.

And then there is individuals, sort of in the individual life stories, the struggles they went through and so on.

Now, like in philosophy, it's easier to do group thinking because you don't, it's very hard to think about individuals, there's so much variability.

But with data, you can start to actually say, you know what, group thinking is too crude.

You're actually doing more discrimination by thinking in terms of groups and individuals.

Can you linger on that kind of idea of group versus individual in ethics?

And is it good to continue thinking in terms

So let me start by answering a very good high-level question with a slightly narrow technical response, which is these group definitions of fairness, like here's a few groups, like different racial groups, maybe gender groups, maybe age, what have you, And let's make sure that, you know, for none of these groups, do we, you know, have a false negative rate, which is much higher than any other 1 of these groups.

So these are kind of classic group aggregate notions of fairness.

And, you know, but at the end of the day, an individual you can think of as a combination of all of their attributes, right?

They're a member of a racial group, they have a gender, they have an age, you know, and many other, you know, demographic properties that are not biological, but that, you know, are still, you know, very strong determinants of outcome and personality and the like.

So 1, I think, useful spectrum is to sort of think about that array between the group and the specific individual and to realize that in some ways, asking for fairness at the individual level is to sort of ask for group fairness simultaneously for all possible combinations of groups.

So in particular, if I build a predictive model that meets some definition of fairness by race, by gender, by age, by what have you, marginally, to get it slightly technical, sort of independently, I shouldn't expect that model to not discriminate against disabled Hispanic women over age 55 making less than $50,000 a year annually, even though I might have protected each 1 of those attributes marginally.

So the optimization actually, that's a fascinating way to put it.

So you're just optimizing, so 1 way to achieve the optimizing fairness for individuals is just to add more and more definitions of groups that each

So you know, at the end of the day, we could think of all of ourselves as groups of size 1, because eventually there's some attribute that separates you from me and everybody from everybody else in the world.

And so it is possible to put, you know, these incredibly coarse ways of thinking about fairness and these very, very individualistic specific ways on a common scale.

And, you know, 1 of the things we've worked on from a research perspective is, you know, so we sort of know how to, you know, in relative terms, we know how to provide fairness guarantees at the coarsest end of the scale.

We don't know how to provide kind of sensible, tractable, realistic fairness guarantees at the individual level.

But maybe we could start creeping towards that by dealing with more refined subgroups.

We gave a name to this phenomenon where you protect, you enforce some definition of fairness for a bunch of marginal attributes or features, but then you find yourself discriminating against a combination of them.

Because like political gerrymandering, you're giving some guarantee at the aggregate level, but when you kind of look in a more granular way at what's going on, you realize that you're achieving that aggregate guarantee by sort of favoring some groups and discriminating against other ones.

And so there are, you know, it's early days, but there are algorithmic approaches that let you start creeping towards that individual end of the spectrum.

Does there need to be human input in the form of weighing the value of the importance of each kind of group?

So for example, is it like, so gender, say, crudely speaking, male and female, and then different races, Are we as humans supposed to put value on saying gender is .6 and race is .4 in terms of in the big optimization of achieving fairness?

I mean, of course, I don't need to tell you that, of course, technically 1 could incorporate such weights if you wanted to into a definition of fairness.

You know, fairness is an interesting topic in that having worked in, in the book being about both fairness, privacy, and many other social norms.

Fairness, of course, is a much, much more loaded topic.

So privacy, I mean, people want privacy, people don't like violations of privacy.

Violations of privacy cause damage, angst, and bad publicity for the companies that are victims of them.

But sort of everybody agrees more data privacy would be better than less data privacy.

And you don't have these, somehow the discussions of fairness don't become politicized along other dimensions like race and about gender and whether we, And you quickly find yourselves kind of revisiting topics that have been kind of unresolved forever, like affirmative action, right?

Sort of like why are you protecting, some people will say, why are you protecting this particular racial group?

And others will say, well, we need to do that as a matter of retribution.

Other people will say it's a matter of economic opportunity.

And I don't know whether any of these are the right answers, but you sort of fairness is sort of special in that as soon as you start talking about it, you inevitably have to participate in debates about fair to whom, at what expense to who else.

I mean, even in criminal justice, where people talk about fairness in criminal sentencing or predicting failures to appear or making parole decisions or the like, they will, you know, they'll point out that, well, these definitions of fairness are all about fairness for the criminals, and what about fairness for the victims, Right?

So when I basically say something like, well, the false incarceration rate for black people and white people needs to be roughly the same, you know, there's no mention of potential victims of criminals in such a fairness definition.

And that's the realm of public discourse.

I should actually recommend, I just listened to people listening, Intelligence Squares debates US edition just had a debate.

They have this structure where you have old Oxford style or whatever they're called, debates, and it was 2 versus 2, and they talked about affirmative action.

And it was incredibly interesting that it's still, there's really good points on every side of this issue, which is fascinating to listen to.

And so it's interesting to be a researcher trying to do, for the most part, technical algorithmic work, but Aaron and I both quickly learned you cannot do that and then go out and talk about it and expect people to take it seriously if you're unwilling to engage in these broader debates that are entirely extra algorithmic, right?

They're not about algorithms and making algorithms better.

They're sort of, as you said, sort of like, what should society be protecting in the first place?

When you discuss the fairness, an algorithm that achieves fairness, whether in the constraints and the objective function, there's an immediate kind of analysis you can perform, which is saying, if you care about fairness in gender, this is the amount that you have to pay for in terms of the performance of the system.

Like, Is there a role for statements like that in a table in a paper?

No, we want to touch that and we do touch it.

See all Lex Fridman transcripts on Youtube

Michael Kearns: Algorithmic Fairness, Privacy & Ethics | Lex Fridman Podcast #50