Full episode with Michael Kearns (Nov 2019): https://www.youtube.com/watch?v=AzdxbzHtjgs
New clips channel (Lex Clips): https://www.youtube.com/lexclips
Once it reaches 20,000 subscribers, I'll start posting the clips there instead. 
(more links below)

For now, new full episodes are released once or twice a week and 1-2 new clips or a new non-podcast video is released on all other days.

Clip from full episode: https://www.youtube.com/watch?v=AzdxbzHtjgs If you enjoy these clips, subscribe to the new clips channel (Lex Clips): https://www.youtube.com/lexclips Once it reaches 20,000 subscribers, I'll start posting the clips there instead. For now, new full episodes are released once or twice a week and 1-2 new clips or a new non-podcast video is released on all other days.
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Michael Kearns is a professor at University of Pennsylvania and a co-author of the new book Ethical Algorithm that is the focus of much of our conversation, including algorithmic fairness, bias, privacy, and ethics in general. But, that is just one of many fields that Michael is a world-class researcher in, some of which we touch on quickly including learning theory or theoretical foundations of machine learning, game theory, algorithmic trading, quantitative finance, computational social science, and more.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

You've worn many hats, 1 of which, the 1 that first caused me to become a big fan of your work many years ago is algorithmic trading.

So I have to just ask a question about this because you have so much fascinating work there.

In the 21st century, what role do you think algorithms have in the space of trading, investment, in the financial sector?

I mean, in the time I've spent on Wall Street and in finance you know, I've seen a clear progression and I think it's a progression that kind of models the use of algorithms and automation more generally in society, which is, you know, the things that kind of get taken over by the algos first are sort of the things that computers are obviously better at than people, right?

So, you know, so first of all, there needed to be this era of automation, right, where just, you know, financial exchanges became largely electronic, which then enabled the possibility of trading becoming more algorithmic because once the exchanges are electronic, an algorithm can submit an order through an API just as well as a human can do at a monitor.

And so, you know, I think the places where algorithmic trading have had the greatest inroads and had the first inroads were in kind of execution problems, kind of optimized execution problems.

So what I mean by that is at a large brokerage firm, for example, 1 of the lines of business might be on behalf of large institutional clients taking what we might consider difficult trades.

So it's not like a mom and pop investor saying, I want to buy 100 shares of Microsoft.

It's a large hedge fund saying, I want to buy a very, very large stake in Apple, and I want to do it over the span of a day.

And it's such a large volume that if you're not clever about how you break that trade up, not just over time, but over perhaps multiple different electronic exchanges that all let you trade Apple on their platform, you know, you will move, you'll push prices around in a way that hurts your execution.

So, you know, this is the kind of, you know, this is an optimization problem, this is a control problem, right?

And so, machines are better, we know how to design algorithms, you know, that are better at that kind of thing than a person is going to be able to do, because we can take volumes of historical and real-time data to kind of optimize the schedule with which we trade.

And similarly, high-frequency trading, which is closely related but not the same as optimized execution where you're just trying to spot very, very temporary mispricings between exchanges or within an asset itself, or just predict directional movement of a stock because of the very, very low level granular buying and selling data in the exchange.

It's kind of like the mechanics of trading.

What about the, can machines do long-term sort of prediction?

So I think we are in an era where clearly there have been some very successful quant hedge funds that are in what we would traditionally call still in the stat arb regime.

Stat arb referring to statistical arbitrage, But for the purposes of this conversation, what it really means is making directional predictions in asset price movement or returns.

Your prediction about that directional movement is good for, you know, you have a view that it's valid for some period of time between a few seconds and a few days.

And that's the amount of time that you're going to kind of get into the position, hold it and then hopefully be right about the directional movement and buy low and sell high as the cliche goes.

So that is a, you know, kind of a sweet spot, I think, for quant trading and investing right now and has been for some time.

When you really get to kind of more Warren Buffett style time scales, right?

Like, you know, my cartoon of Warren Buffett is that, you know, Warren Buffett sits and thinks what the long-term value of Apple really should be and he doesn't even look at what Apple is doing today.

He just decides, you know, You know, I think that this is what its long-term value is, and it's far from that right now, and so I'm going to buy some apple or, you know, short some apple, and I'm going to sit on that for 10 or 20 years.

So when you're at that kind of timescale, or even more than just a few days, all kinds of other sources of risk and information, you know.

So now you're talking about holding things through recessions and economic cycles.

So there you have to understand human nature at a level.

Yeah, and you need to just be able to ingest many, many more sources of data that are on wildly different time scales, right?

So if I'm an HFT, I'm a high-frequency trader, like I don't, I don't, I really, my main source of data is just the data from the exchanges themselves about the activity in the exchanges, right?

And maybe I need to pay, you know, I need to keep an eye on the news, right?

Because you know, that can cause sudden, you know, the, you know, CEO gets caught in a scandal or gets run over by a bus or something that can cause very sudden changes.

But I don't need to understand economic cycles.

I don't need to worry about the political situation or war breaking out in this part of the world.

Because all I need to know is, as long as that's not going to happen in the next 500 milliseconds, then my model is good.

When you get to these longer time scales, you really have to worry about that kind of stuff.

And people in the machine learning community are starting to think about this.

We jointly sponsored a workshop at Penn with the Federal Reserve Bank of Philadelphia a little more than a year ago on, you know, I think the title was something like machine learning for macroeconomic prediction, macroeconomic referring specifically to these longer timescales.

And it was an interesting conference, but it left me with greater confidence that we have a long way to go.

And so I think that people that, you know, in the grand scheme of things, you know, if somebody asked me like, well, whose job on Wall Street is safe from the bots?

I think people that are at that longer, you know, the timescale and have that appetite for all the risks involved in long-term investing and that really need kind of not just algorithms that can optimize from data, but they need views on stuff.

They need views on the political landscape, economic cycles, and the like.

And I think they're pretty safe for a while, as far as I can tell.

seeing a robo Warren Buffett anytime soon.


Speaker 0: You've worn many hats, 1 of which, the 1 that first caused me to become a big fan of your work many years ago is algorithmic trading. So I have to just ask a question about this because you have so much fascinating work there. In the 21st century, what role do you think algorithms have in the space of trading, investment, in the financial sector?

Speaker 1: Yeah, it's a good question. I mean, in the time I've spent on Wall Street and in finance you know, I've seen a clear progression and I think it's a progression that kind of models the use of algorithms and automation more generally in society, which is, you know, the things that kind of get taken over by the algos first are sort of the things that computers are obviously better at than people, right? So, you know, so first of all, there needed to be this era of automation, right, where just, you know, financial exchanges became largely electronic, which then enabled the possibility of trading becoming more algorithmic because once the exchanges are electronic, an algorithm can submit an order through an API just as well as a human can do at a monitor. It can do

Speaker 0: it really quickly. It can read all the data.

Speaker 1: So yeah. And so, you know, I think the places where algorithmic trading have had the greatest inroads and had the first inroads were in kind of execution problems, kind of optimized execution problems. So what I mean by that is at a large brokerage firm, for example, 1 of the lines of business might be on behalf of large institutional clients taking what we might consider difficult trades. So it's not like a mom and pop investor saying, I want to buy 100 shares of Microsoft. It's a large hedge fund saying, I want to buy a very, very large stake in Apple, and I want to do it over the span of a day.

And it's such a large volume that if you're not clever about how you break that trade up, not just over time, but over perhaps multiple different electronic exchanges that all let you trade Apple on their platform, you know, you will move, you'll push prices around in a way that hurts your execution. So, you know, this is the kind of, you know, this is an optimization problem, this is a control problem, right? And so, machines are better, we know how to design algorithms, you know, that are better at that kind of thing than a person is going to be able to do, because we can take volumes of historical and real-time data to kind of optimize the schedule with which we trade. And similarly, high-frequency trading, which is closely related but not the same as optimized execution where you're just trying to spot very, very temporary mispricings between exchanges or within an asset itself, or just predict directional movement of a stock because of the very, very low level granular buying and selling data in the exchange. Machines are good at this kind of stuff.

Speaker 0: It's kind of like the mechanics of trading. What about the, can machines do long-term sort of prediction?

Speaker 1: Yeah. So I think we are in an era where clearly there have been some very successful quant hedge funds that are in what we would traditionally call still in the stat arb regime. What's that? Stat arb referring to statistical arbitrage, But for the purposes of this conversation, what it really means is making directional predictions in asset price movement or returns. Your prediction about that directional movement is good for, you know, you have a view that it's valid for some period of time between a few seconds and a few days.

And that's the amount of time that you're going to kind of get into the position, hold it and then hopefully be right about the directional movement and buy low and sell high as the cliche goes. So that is a, you know, kind of a sweet spot, I think, for quant trading and investing right now and has been for some time. When you really get to kind of more Warren Buffett style time scales, right? Like, you know, my cartoon of Warren Buffett is that, you know, Warren Buffett sits and thinks what the long-term value of Apple really should be and he doesn't even look at what Apple is doing today. He just decides, you know, You know, I think that this is what its long-term value is, and it's far from that right now, and so I'm going to buy some apple or, you know, short some apple, and I'm going to sit on that for 10 or 20 years.

Okay. So when you're at that kind of timescale, or even more than just a few days, all kinds of other sources of risk and information, you know. So now you're talking about holding things through recessions and economic cycles. Wars can break out.

Speaker 0: So there you have to understand human nature at a level.

Speaker 1: Yeah, and you need to just be able to ingest many, many more sources of data that are on wildly different time scales, right? So if I'm an HFT, I'm a high-frequency trader, like I don't, I don't, I really, my main source of data is just the data from the exchanges themselves about the activity in the exchanges, right? And maybe I need to pay, you know, I need to keep an eye on the news, right? Because you know, that can cause sudden, you know, the, you know, CEO gets caught in a scandal or gets run over by a bus or something that can cause very sudden changes. But I don't need to understand economic cycles.

I don't need to understand recessions. I don't need to worry about the political situation or war breaking out in this part of the world. Because all I need to know is, as long as that's not going to happen in the next 500 milliseconds, then my model is good. When you get to these longer time scales, you really have to worry about that kind of stuff. And people in the machine learning community are starting to think about this.

We held a... We jointly sponsored a workshop at Penn with the Federal Reserve Bank of Philadelphia a little more than a year ago on, you know, I think the title was something like machine learning for macroeconomic prediction, macroeconomic referring specifically to these longer timescales. And it was an interesting conference, but it left me with greater confidence that we have a long way to go. And so I think that people that, you know, in the grand scheme of things, you know, if somebody asked me like, well, whose job on Wall Street is safe from the bots? I think people that are at that longer, you know, the timescale and have that appetite for all the risks involved in long-term investing and that really need kind of not just algorithms that can optimize from data, but they need views on stuff.

They need views on the political landscape, economic cycles, and the like. And I think they're pretty safe for a while, as far as I can tell.

Speaker 0: So Warren Buffett's job is

Speaker 1: safe for a little while.

Speaker 0: Yeah, I'm not

Speaker 1: seeing a robo Warren Buffett anytime soon.

Speaker 0: We should give him comfort. You you

See all Lex Fridman transcripts on Youtube

Michael Kearns: Algorithmic Trading and the Role of AI in Investment at Different Time Scales