Full episode with Michael I. Jordan (Feb 2020): https://www.youtube.com/watch?v=EYIKy_FM9x0
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Michael I Jordan is a professor at Berkeley, and one of the most influential people in the history of machine learning, statistics, and artificial intelligence. He has been cited over 170,000 times and has mentored many of the world-class researchers defining the field of AI today, including Andrew Ng, Zoubin Ghahramani, Ben Taskar, and Yoshua Bengio.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

An absurd question, but what is statistics?

So here it's a little bit, it's somewhere between math and science and technology.

So it's a set of principles that allow you to make inferences that have got some reason to be believed.

And also principles allow you to make decisions where you can have some reason to believe you're not gonna make errors.

So all that requires some assumptions about what do you mean by an error?

But after you start making some of those assumptions, you're led to conclusions that yes, I can guarantee that if you do this in this way, your probability of making an error will be small.

Your probability of continuing to not make errors over time will be small.

And probability you found something that's real will be small, will be high.

So decision making is a big part of that.

So the original, so statistics, you know, short history was that, you know, it's kind of goes back, as a formal discipline, you know, 250 years or so.

It was called inverse probability because around that era, probability was developed sort of especially to explain gambling situations.

So you would say, well, given the state of nature is this, there's a certain roulette board that has a certain mechanism in it, what kind of outcomes do I expect to see?

And especially if I do things long amounts of time, what outcomes will I see?

And the physicists started to pay attention to this.

And then people say, well, given, let's turn the problem around.

What if I saw certain outcomes, could I infer what the underlying mechanism was?

And in fact, for quite a while, statistics was called inverse probability.

And I believe that it was Laplace who was working in Napoleon's government, who needed to do a census of France, learn about the people there.

And he analyzed that data to determine policy and said, well, let's call this field that does this kind of thing, statistics, because the word state is in there.

But it's the study of data for the state.

So anyway, that caught on, and it's been called statistics ever since.

But by the time it got formalized, it was sort of in the 30s.

And around that time, there was game theory and decision theory developed nearby.

People that year didn't think of themselves as either computer science or statistics or control or econ, they were all the above.

And so, Von Neumann is developing game theory, but also thinking of that as decision theory.

Wald is an econometrician developing decision theory and then turn that into statistics.

And so it's all about, Here's not just data and you analyze it, here's a loss function, here's what you care about, here's the question you're trying to ask.

Here is a probability model and here is the risk you will face if you make certain decisions.

And to this day in most advanced statistical curricula you teach decision theory as the starting point.

And then it branches out into the 2 branches of Bayesian and Frequentist.

In statistics, what is the most beautiful, mysterious, maybe surprising idea that you've come across?

I mean, there's a bunch of surprising ones.

There's something that's way too technical for this thing, but something called James Stein estimation, which is kind of surprising and really takes time to wrap your head around.

Let me just say a colleague at Steven Stigler at University of Chicago wrote a really beautiful paper on James Stein estimation, which helps to, it's views a paradox, it kind of defeats the mind's attempts to understand it, but you can, and Steve has a nice perspective on that.

So 1 of the troubles with statistics is that it's like in physics, or in quantum physics, you have multiple interpretations.

There's a wave and particle duality in physics.

And you get used to that over time, but it still kind of haunts you that you don't really quite understand the relationship.

The electron's a wave and electron's a particle.

There's Bayesian ways of thinking in frequentist and they are different.

They sometimes become sort of the same in practice, but they are physically different.

And then in some practice, they are not the same at all.

And so it is very much like wave and particle duality.

And that is something you have to kind of get used to in the field.

Yeah, in decision theory, you can make, I have a video that people could see, it's called Are You a Bayesian or a Frequentist?

And kind of help try to make it really clear.

So, you know, decision theory, you're talking about loss functions, which are a function of data, x, and parameter theta.

So you have this function of 2 things you don't know, and you're trying to say, I want that function to be small, I want small loss, right?

So you sort of say, well, I'm gonna average over these quantities or maximize over them or something, so that I turn that uncertainty into something certain.

So you could look at the first argument and average over it, or you could look at the second argument, and average over it, that's Bayesian frequentist.

So the frequentist says, I'm gonna look at the X, the data, and I'm gonna take that as random and I'm gonna average over the distribution.

So I take the expectation loss under X, theta is held fixed, right?

And so it's looking at other, all the data sets you could get, right?

And say how well will a certain procedure do under all those data sets?

So I think it is very appropriate when like you're building a piece of software and you're shipping it out there and people are using all kinds of data sets.

You wanna have a stamp, a guarantee on it that as People run it on many, many data sets that you never even thought about, that 95% of the time it will do the right thing.

The Bayesian perspective says, well, no, I'm gonna look at the other argument of the loss function, the theta part.

Okay, that's unknown and I'm uncertain about it.

So I could have my own personal probability for what it is.

How many tall people are there out there?

I'm trying to infer the average height of the population.

Well, I have an idea roughly what the height is.

So now that loss function has only now, again, 1 argument's gone, now it's a function of X.

And that's what a Bayesian does is they say, well, let's just focus on the particular X we got, the data set we got, we condition on that.

Condition on the X, I say something about my loss.

And the Bayesian will argue that it's not relevant to look at all the other data sets you could have gotten and average over them, the frequentist approach.

And I do agree with that, especially in situations where you're working with a scientist, you can learn a lot about the domain, and you really only focus on certain kinds of data, and you gather your data, and you make inferences.

I don't agree with it, though, in the sense that there are needs for frequentist guarantees.

You're writing software, people are using it out there, you wanna say something.

So these 2 things have to got to fight each other a little bit, but they have to blend.

So long story short, there's a set of ideas that are right in the middle, they're called empirical Bayes.

And empirical Bayes sort of starts with the Bayesian framework.

It's kind of arguably philosophically more reasonable and kosher.

Write down a bunch of the math that kind of flows from that, and then realize there's a bunch of things you don't know, because it's the real world and you don't know everything, so you're uncertain about certain quantities.

At that point ask, is there a reasonable way to plug in an estimate for those things?

And in some cases there's quite a reasonable thing to do, to plug in, there's a natural thing you can observe in the world that you can plug in and then do a little bit more mathematics and assure yourself it's really good.

faith- So based on math or based on human expertise, what are good-

The Bayesian framework allows you to put a lot of human expertise in.

But the math kind of guides you along that path and then kind of reassures you at the end you could put that stamp of approval.

Under certain assumptions, this thing will work.

So you asked the question, what's my favorite or what's the most surprising nice idea?

So 1 that is more accessible is something called false discovery rate, which is You're making not just 1 hypothesis test or making 1 decision, you're making a whole bag of them.

And in that bag of decisions, you look at the ones where you made a discovery.

You announced that something interesting had happened.

All right, that's gonna be some subset of your big bag.

In the ones you made a discovery, which subset of those are bad?

You'd like the fraction of your false discoveries among your discoveries to be small.

That's a different criterion than accuracy or precision or recall or sensitivity and specificity.

Those latter ones are almost all of them have more of a frequentist flavor.

They say, given the truth is that the null hypothesis is true, here's what accuracy I would get.

Or given that the alternative is true, here's what I would get.

So it's kind of going forward from the state of nature to the data.

The Bayesian goes the other direction from the data back to the state of nature.

And that's actually what false discovery rate is.

It says, given you made a discovery, that's conditioned on your data, what's the probability of the hypothesis?

And so the classical frequency look at that.

So I can't know that there's some priors needed in that.

And the empirical Bayesian goes ahead and plows forward and starts writing down these formulas and realizes at some point some of those things can actually be estimated in a reasonable way.

So this kind of line of argument has come out, it's not certainly mine, but it sort of came out from Robbins around 1960.

Brad Efron has written beautifully about this in various papers and books.

And the FDRs, you know, Ben Yamini in Israel, John Story did this Bayesian interpretation and so on.

So I've just absorbed these things over the years and find it a very healthy way to think about statistics.


Speaker 0: An absurd question, but what is statistics?

Speaker 1: So here it's a little bit, it's somewhere between math and science and technology. It's somewhere in that convex hole. So it's a set of principles that allow you to make inferences that have got some reason to be believed. And also principles allow you to make decisions where you can have some reason to believe you're not gonna make errors. So all that requires some assumptions about what do you mean by an error?

What do you mean by the probabilities? But after you start making some of those assumptions, you're led to conclusions that yes, I can guarantee that if you do this in this way, your probability of making an error will be small. Your probability of continuing to not make errors over time will be small. And probability you found something that's real will be small, will be high.

Speaker 0: So decision making is a big part of that.

Speaker 1: Decision making is a big part, yeah. So the original, so statistics, you know, short history was that, you know, it's kind of goes back, as a formal discipline, you know, 250 years or so. It was called inverse probability because around that era, probability was developed sort of especially to explain gambling situations.

Speaker 0: Of course.

Speaker 1: And- Interesting. So you would say, well, given the state of nature is this, there's a certain roulette board that has a certain mechanism in it, what kind of outcomes do I expect to see? And especially if I do things long amounts of time, what outcomes will I see? And the physicists started to pay attention to this. And then people say, well, given, let's turn the problem around.

What if I saw certain outcomes, could I infer what the underlying mechanism was? That's an inverse problem. And in fact, for quite a while, statistics was called inverse probability. That was the name of the field. And I believe that it was Laplace who was working in Napoleon's government, who needed to do a census of France, learn about the people there.

So he went and got and gathered data. And he analyzed that data to determine policy and said, well, let's call this field that does this kind of thing, statistics, because the word state is in there. In French, that's état. But it's the study of data for the state. So anyway, that caught on, and it's been called statistics ever since.

But by the time it got formalized, it was sort of in the 30s. And around that time, there was game theory and decision theory developed nearby. People that year didn't think of themselves as either computer science or statistics or control or econ, they were all the above. And so, Von Neumann is developing game theory, but also thinking of that as decision theory. Wald is an econometrician developing decision theory and then turn that into statistics.

And so it's all about, Here's not just data and you analyze it, here's a loss function, here's what you care about, here's the question you're trying to ask. Here is a probability model and here is the risk you will face if you make certain decisions. And to this day in most advanced statistical curricula you teach decision theory as the starting point. And then it branches out into the 2 branches of Bayesian and Frequentist. But that's, it's all about decisions.

Speaker 0: In statistics, what is the most beautiful, mysterious, maybe surprising idea that you've come across?

Speaker 1: Yeah, good question. I mean, there's a bunch of surprising ones. There's something that's way too technical for this thing, but something called James Stein estimation, which is kind of surprising and really takes time to wrap your head around.

Speaker 0: Can you try to maybe?

Speaker 1: Nah, I think I don't even wanna try. Let me just say a colleague at Steven Stigler at University of Chicago wrote a really beautiful paper on James Stein estimation, which helps to, it's views a paradox, it kind of defeats the mind's attempts to understand it, but you can, and Steve has a nice perspective on that. So 1 of the troubles with statistics is that it's like in physics, or in quantum physics, you have multiple interpretations. There's a wave and particle duality in physics. And you get used to that over time, but it still kind of haunts you that you don't really quite understand the relationship.

The electron's a wave and electron's a particle. Well, the same thing happens here. There's Bayesian ways of thinking in frequentist and they are different. They sometimes become sort of the same in practice, but they are physically different. And then in some practice, they are not the same at all.

They give you rather different answers. And so it is very much like wave and particle duality. And that is something you have to kind of get used to in the field.

Speaker 0: Can you define Bayesian and frequentist?

Speaker 1: Yeah, in decision theory, you can make, I have a video that people could see, it's called Are You a Bayesian or a Frequentist? And kind of help try to make it really clear. It comes from decision theory. So, you know, decision theory, you're talking about loss functions, which are a function of data, x, and parameter theta. So they're a function of 2 arguments.

Neither 1 of those arguments is known. You don't know the data a priori. It's random. And the parameter is unknown. So you have this function of 2 things you don't know, and you're trying to say, I want that function to be small, I want small loss, right?

Well, what are you gonna do? So you sort of say, well, I'm gonna average over these quantities or maximize over them or something, so that I turn that uncertainty into something certain. So you could look at the first argument and average over it, or you could look at the second argument, and average over it, that's Bayesian frequentist. So the frequentist says, I'm gonna look at the X, the data, and I'm gonna take that as random and I'm gonna average over the distribution. So I take the expectation loss under X, theta is held fixed, right?

That's called the risk. And so it's looking at other, all the data sets you could get, right? And say how well will a certain procedure do under all those data sets? That's called a frequent as guarantee. So I think it is very appropriate when like you're building a piece of software and you're shipping it out there and people are using all kinds of data sets.

You wanna have a stamp, a guarantee on it that as People run it on many, many data sets that you never even thought about, that 95% of the time it will do the right thing. Perfectly reasonable. The Bayesian perspective says, well, no, I'm gonna look at the other argument of the loss function, the theta part. Okay, that's unknown and I'm uncertain about it. So I could have my own personal probability for what it is.

How many tall people are there out there? I'm trying to infer the average height of the population. Well, I have an idea roughly what the height is. So I'm gonna average over the theta. So now that loss function has only now, again, 1 argument's gone, now it's a function of X.

And that's what a Bayesian does is they say, well, let's just focus on the particular X we got, the data set we got, we condition on that. Condition on the X, I say something about my loss. That's a Bayesian approach to things. And the Bayesian will argue that it's not relevant to look at all the other data sets you could have gotten and average over them, the frequentist approach. It's really only the data sets you got.

And I do agree with that, especially in situations where you're working with a scientist, you can learn a lot about the domain, and you really only focus on certain kinds of data, and you gather your data, and you make inferences. I don't agree with it, though, in the sense that there are needs for frequentist guarantees. You're writing software, people are using it out there, you wanna say something. So these 2 things have to got to fight each other a little bit, but they have to blend. So long story short, there's a set of ideas that are right in the middle, they're called empirical Bayes.

And empirical Bayes sort of starts with the Bayesian framework. It's kind of arguably philosophically more reasonable and kosher. Write down a bunch of the math that kind of flows from that, and then realize there's a bunch of things you don't know, because it's the real world and you don't know everything, so you're uncertain about certain quantities. At that point ask, is there a reasonable way to plug in an estimate for those things? And in some cases there's quite a reasonable thing to do, to plug in, there's a natural thing you can observe in the world that you can plug in and then do a little bit more mathematics and assure yourself it's really good.

So my

Speaker 0: faith- So based on math or based on human expertise, what are good-

Speaker 1: They're both going in. The Bayesian framework allows you to put a lot of human expertise in. But the math kind of guides you along that path and then kind of reassures you at the end you could put that stamp of approval. Under certain assumptions, this thing will work. So you asked the question, what's my favorite or what's the most surprising nice idea?

So 1 that is more accessible is something called false discovery rate, which is You're making not just 1 hypothesis test or making 1 decision, you're making a whole bag of them. And in that bag of decisions, you look at the ones where you made a discovery. You announced that something interesting had happened. All right, that's gonna be some subset of your big bag. In the ones you made a discovery, which subset of those are bad?

There are false, false discoveries. You'd like the fraction of your false discoveries among your discoveries to be small. That's a different criterion than accuracy or precision or recall or sensitivity and specificity. It's a different quantity. Those latter ones are almost all of them have more of a frequentist flavor.

They say, given the truth is that the null hypothesis is true, here's what accuracy I would get. Or given that the alternative is true, here's what I would get. So it's kind of going forward from the state of nature to the data. The Bayesian goes the other direction from the data back to the state of nature. And that's actually what false discovery rate is.

It says, given you made a discovery, that's conditioned on your data, what's the probability of the hypothesis?

Speaker 0: It's

Speaker 1: going the other direction. And so the classical frequency look at that. So I can't know that there's some priors needed in that. And the empirical Bayesian goes ahead and plows forward and starts writing down these formulas and realizes at some point some of those things can actually be estimated in a reasonable way. And so it's a beautiful set of ideas.

So this kind of line of argument has come out, it's not certainly mine, but it sort of came out from Robbins around 1960. Brad Efron has written beautifully about this in various papers and books. And the FDRs, you know, Ben Yamini in Israel, John Story did this Bayesian interpretation and so on. So I've just absorbed these things over the years and find it a very healthy way to think about statistics.

See all Lex Fridman transcripts on Youtube

What is Statistics? (Michael I. Jordan) | AI Podcast Clips