How Money Work's video on ChatGPT: https://youtu.be/TbBEScH-is4
University of Florida study: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4412788

Let's talk about recent experiments and studies that are using ChatGPT to trade stocks...and why it's probably not a good idea.

If you'd like to support the channel, you can do so at Patreon.com/ThePlainBagel :)

DISCLAIMER:
This channel is for education purposes only and is not affiliated with any financial institution, although Richard does work as an employee for an investment manager. Richard Coffin does not provide recommendations on The Plain Bagel - those looking for investment advice should seek out a registered professional. Richard is not responsible for investment actions taken by viewers

Ladies and gentlemen, welcome to the plain bagel.

If you've been anywhere online in 2023, you know that AI has been a dominating topic, not just within the world of investing, but also the world.

We've all seen just how far reaching the capabilities of the space can be and how people online are really pushing the boundary by putting their favorite characters in Balenciaga gear.

But 1 of the platforms really leading the charge here is ChatGBT and with its impressive responses to questions and queries, it didn't take long for people to ask the question, how can I use this to make money?

And we've since seen the rise of people using ChatGBT to help them trade stocks, either by asking explicitly for strategies, for code, for a trading bot, or just explicitly asking for stock picks.

And while that might sound stupid, and I think you're justified in that evaluation.

There are different articles and even research papers that seem to support the idea that you can use ChatGPT to earn excess return.

1 of the early articles from finder.com pitted ChatGPT against 10 of the UK's most popular funds and found that it was able to outperform over the period analyzed.

And back in April, the University of Florida released a research report highlighting that ChadGBT could earn a 500% return in just over a year's time by using sentiment analysis.

And with CNN highlighting both these things in an article titled, "'Chat GPT Can Pick Stocks Better Than Your Fund Manager'", people have since been really running with this idea.

There are thousands of TikToks of people claiming to make large returns with Chat GPT trading bots.

And a more well-known example is the GPT portfolio, which is a Twitter account that itself started trading with $50, 000 using chat GPT, started outperforming the ESP500 in its first week of performance back in May, and now has $27 million of other people's money trading alongside it.

It's time for a party pooper video, because yes, generative AI is incredible stuff.

And at first glance, the research, the performance around trading with it, look super impressive as well.

But if you take a closer look, It really isn't.

There's a lot of nuance that's really being skipped over when people talk about ChatGPT, picking stocks or investing.

And if you actually read the research report and go through it, there's a pretty big asterisk beside that 500% figure that you see floated online.

So I thought I'd take some time to explain why chat GPT isn't the cheat code to the markets that we've all been hoping for.

Now I probably don't have to explain what chat GPT is, but I do want to explain at a high level how it works.

I'm not going to get into the technicals that I'm not a subject matter expert on, but I do think this is important to the conversation because it really does explain why Chat GPT may not necessarily be good at picking stocks.

Chat GPT is what's called a large language model, Something that takes a vast amount of text data, billions of words from online sources, web pages and Wikipedia articles, and uses that to develop a neural network for understanding the structure and patterns of natural language.

There are then processes to fine tune the model using human feedback, and it leaves us with a program that's really good at language focused tasks, things like summarizing, translation, and yes, answering questions.

And there's no doubt that ChatGPT is impressive on this front, but it's not a stock analyzing system.

Not only is ChatGPT pre-trained on data only going up to 2021, meaning that the data on companies that it might recommend is stale, but the stock picks it comes up with is based entirely on the web pages that are part of that pre-training period.

Even the new feature that allows ChiGPT to search the internet to get more up-to-date information usually just leads it to summarize what it finds on the web pages.

In other words, it's not going behind the scenes in determining the economic forecast, calculating what the expected revenue growth and all these variables are.

In really oversimplified terms, it's just regurgitating what it reads online, which needless to say, isn't necessarily the best source of stock picks.

In fact, a different research report carried out this year actually found that ChatGPT underperformed simple models like linear regression, when using numerical data to predict stock prices.

But we're sure you might ask, what about that report from the University of Florida that showed Chats GPT earning a 500% return?

Well, let's take some time to go over what that research report actually says.

To determine what stocks to buy and sell, the researchers would give a prompt to Chats GPT asking it to analyze a headline to determine whether the headline was positive, negative, or neutral for the company in question.

The headline would then be given a score of negative 1 if it was bad for the company, positive 1 if it was good, and 0 if it didn't know.

Multiple headlines would have their score average for the stock.

And then based on whether that stock was positive or negative, the researchers would buy or short the stock accordingly and exit the position at the next stock market close.

In other words, ChadGBT wasn't actually picking stocks.

It was just indicating whether a headline about that stock was good or bad news for that company.

And then the researchers built a trading strategy around this that more or less just involved holding the position for a day.

Now to its credit, Chad GPT was better at picking up on nuances in headlines to determine whether say bad news for competitor was good news for the stock in question.

And this ability to understand language has been highlighted in the past as well.

There is another research report that showed that Chad GPT was able to interpret Fed speak.

Now, when the Federal Reserve would make an announcement about their intentions moving forward, Chad GPT could pick up on the nuances of the message to determine whether the Federal Reserve was likely thinking about hiking, lowering or keeping rates stable.

But even with that, there are a number of drawbacks that will pretty much prevent anyone from replicating the performance that Chachapiti saw in this research report.

Firstly, the cited figure of 500% ignores any transaction costs, which isn't reasonable.

And as highlighted in the research report, if you incorporate a 25 basis point transaction cost, your return actually drops to about 50% over more than

a year's time, which is a little different.

The report also doesn't mention taxes, which would likely take a further massive chunk out of your returns, given the frequency of trading.

And given that the researchers themselves highlighted that the strategy worked best for smaller companies, specifically micro-cap companies in the bottom 10th percentile of market caps, liquidity would be an important consideration as well.

Since there's a chance you wouldn't actually even be able to find a buyer or seller for these hypothetical trades.

On top of all this, I would also take the entirety of the report with a grain of salt, since it's missing a lot of calculations, basic details like trade sizing, doesn't outline the assumptions and isn't actually peer reviewed.

And finally, it's worth pointing out that regarding that 500% return figure, that return didn't come from chat GPT-4, the current system.

It came from chat GPT 3.5, which is an older, objectively less extensive model.

You see, the researchers tried this strategy under different AI LLMs, including older versions of chatGBT.

And chatGBT4, while getting a better sharp ratio, had nearly half the return of chat GPT 3.5.

In other words, the better chat GPT became at understanding language and doing this sentiment analysis, the worse its returns became.

So then why do we get the strong performance from the system?

If a better sentiment analysis system actually did worse.

There isn't much work in the paper to demonstrate that the results seen were due to superior sentiment analysis versus randomness.

And there is a chance that The reason ChatGPT 3.5 performed the best is because it just happened to select the right stocks, whereas the other AIs happened to not.

Especially when you consider that with the sentiment analysis, ChatGPT wasn't actually that consistent with its accuracy, with the researchers themselves only giving it a 51% likelihood of being accurate with a given prompt.

Quote, it works well because when you're aggregating across multiple companies on multiple days, you get a result.

But for 1 given headline, it's basically a little better than tossing a coin.

And this seems to line up with other stories we've heard about Chachapiti's accuracy, such as those 2 lawyers in

the US who are now facing fines after using Chachapiti to help build their court case and using case citations that Chachapiti seemingly made up.

So it's very feasible that Chachapiti's success here really came down to dumb luck.

Michael Reeves, a popular tech YouTuber, outperformed the Nasdaq by letting a fish make his investing decisions.

And there are countless of other examples of randomly selected portfolios doing better than the market over a given timeframe.

None of which is evidence that you should let an animal pick your stocks, but it just goes to show that chance plays a role.

So just because the trades generated by ChatGPT outperformed in a given year timeframe doesn't mean that ChatGPT has a strategy for outperforming the market.

In fact, now that we've had some time passed, the other stories involved here with ChatGPT trading have turned out to be a bit more lackluster.

That Twitter ChatGPT fund with $50, 000 invested, well, it started lagging the S&P 500 in May and has been lagging the benchmark since.

The portfolio facing off against the 10 UK funds.

It's still outperforming most of them, but again, it's still relatively a short period of time.

Is more so a diss towards the fund managers than is a compliment of ChatGPT.

Isn't even really a fair comparison against some of the funds.

And the ChatGPT portfolio is likewise still underperforming the S&P 500.

And even when you look at ETFs that use AI that are intended for the stock selection process, a lot of their performance has still lagged the S&P 500.

And part of the reason why it's so difficult to beat the market with an automated system like this is because a lot of the opportunities for these models are marginal and fleeting, meaning that they earn you a very small return requiring you to make tens of thousands of trades to take advantage of them.

The majority of market trading volume is already algorithmic.

It's already utilizing computers and AI that's intended for quantitative analysis.

They are all already competing with each other over what are likely smaller and smaller arbitrage and other opportunities in the market.

So while AI that's meant for stock analysis does play a role with quantitative funds, does play a role with many active hedge funds that have it as more of a proprietary thing, it's unlikely that a chat GPT platform, an AI that isn't designed for stock picking that is widely available is going to give you any sort of edge when it comes to trading stocks.

But Richard, what about the TikTok videos, the YouTube ads?

I show people making crazy returns with chat GPT trading.

It's not to say that every example is going to be fraudulent.

There might just be representation bias where you see the 1 good example of it working out amongst the thousands of other stories we don't hear about of people losing money with chat GPT.

But on top of that, a lot of people are just taking advantage of people's ignorance.

How many works recently put out a great video talking about why chatgpt is a breeding ground for scams at the moment and how it really is taking advantage of the layman's misunderstanding of how the system works.

So I'll leave a link for that down below if you want to check it out.

But hopefully with everything I've covered, I've helped to explain why you probably shouldn't put your financial future in the hands of a chatbot anytime soon.

So that's the video and none of it is to discredit the technological breakthrough that is chat to BT.

I think most people can acknowledge that it's a remarkable advancement.

This is also not to say that there is no application for ChatGPT in investing in trading.

However, those applications are really supplemental.

And the moment you start relying on ChatGPT for trades, for stock picks, you enter a really dangerous area where you might as well be throwing a dart at a board.

Even ChattyBT's well-known ability to code trading bots based on very simple prompts does often require the user to highlight what strategy they want this trading bot to use and often ends up being rife with bugs that doesn't actually lead it to work.

So while it really is an interesting time for AI, just because something is technologically advanced, complex, and really remarkable, doesn't mean that's going to be your next gold rush.

And you should really be careful not to let the spectacle and sparkle of something lure you into what's effectively get-rich-quick schemes.

If you did, please make sure to like, subscribe, all that good stuff.

And let me know your thoughts on Chad2BT trading.

No judgments, despite the intent of this video.

Thanks for joining me today, and as always, be safe out there.


Speaker 0: Ladies and gentlemen, welcome to the plain bagel. I'm your host, Richard Coffin. If you've been anywhere online in 2023, you know that AI has been a dominating topic, not just within the world of investing, but also the world. We've all seen just how far reaching the capabilities of the space can be and how people online are really pushing the boundary by putting their favorite characters in Balenciaga gear. But 1 of the platforms really leading the charge here is ChatGBT and with its impressive responses to questions and queries, it didn't take long for people to ask the question, how can I use this to make money?

And we've since seen the rise of people using ChatGBT to help them trade stocks, either by asking explicitly for strategies, for code, for a trading bot, or just explicitly asking for stock picks. And while that might sound stupid, and I think you're justified in that evaluation. There are different articles and even research papers that seem to support the idea that you can use ChatGPT to earn excess return. 1 of the early articles from finder.com pitted ChatGPT against 10 of the UK's most popular funds and found that it was able to outperform over the period analyzed. And back in April, the University of Florida released a research report highlighting that ChadGBT could earn a 500% return in just over a year's time by using sentiment analysis.

And with CNN highlighting both these things in an article titled, "'Chat GPT Can Pick Stocks Better Than Your Fund Manager'", people have since been really running with this idea. There are thousands of TikToks of people claiming to make large returns with Chat GPT trading bots. And a more well-known example is the GPT portfolio, which is a Twitter account that itself started trading with $50, 000 using chat GPT, started outperforming the ESP500 in its first week of performance back in May, and now has $27 million of other people's money trading alongside it. And I figured, you know what? It's time for a party pooper video, because yes, generative AI is incredible stuff.

And at first glance, the research, the performance around trading with it, look super impressive as well. But if you take a closer look, It really isn't. There's a lot of nuance that's really being skipped over when people talk about ChatGPT, picking stocks or investing. And if you actually read the research report and go through it, there's a pretty big asterisk beside that 500% figure that you see floated online. So I thought I'd take some time to explain why chat GPT isn't the cheat code to the markets that we've all been hoping for.

Sorry. Now I probably don't have to explain what chat GPT is, but I do want to explain at a high level how it works. I'm not a data scientist. I'm not going to get into the technicals that I'm not a subject matter expert on, but I do think this is important to the conversation because it really does explain why Chat GPT may not necessarily be good at picking stocks. Chat GPT is what's called a large language model, Something that takes a vast amount of text data, billions of words from online sources, web pages and Wikipedia articles, and uses that to develop a neural network for understanding the structure and patterns of natural language.

There are then processes to fine tune the model using human feedback, and it leaves us with a program that's really good at language focused tasks, things like summarizing, translation, and yes, answering questions. And there's no doubt that ChatGPT is impressive on this front, but it's not a stock analyzing system. Not only is ChatGPT pre-trained on data only going up to 2021, meaning that the data on companies that it might recommend is stale, but the stock picks it comes up with is based entirely on the web pages that are part of that pre-training period. Even the new feature that allows ChiGPT to search the internet to get more up-to-date information usually just leads it to summarize what it finds on the web pages. In other words, it's not going behind the scenes in determining the economic forecast, calculating what the expected revenue growth and all these variables are.

In really oversimplified terms, it's just regurgitating what it reads online, which needless to say, isn't necessarily the best source of stock picks. In fact, a different research report carried out this year actually found that ChatGPT underperformed simple models like linear regression, when using numerical data to predict stock prices. But we're sure you might ask, what about that report from the University of Florida that showed Chats GPT earning a 500% return? Well, let's take some time to go over what that research report actually says. To determine what stocks to buy and sell, the researchers would give a prompt to Chats GPT asking it to analyze a headline to determine whether the headline was positive, negative, or neutral for the company in question.

The headline would then be given a score of negative 1 if it was bad for the company, positive 1 if it was good, and 0 if it didn't know. Multiple headlines would have their score average for the stock. And then based on whether that stock was positive or negative, the researchers would buy or short the stock accordingly and exit the position at the next stock market close. That's it. In other words, ChadGBT wasn't actually picking stocks.

It was just indicating whether a headline about that stock was good or bad news for that company. And then the researchers built a trading strategy around this that more or less just involved holding the position for a day. Now to its credit, Chad GPT was better at picking up on nuances in headlines to determine whether say bad news for competitor was good news for the stock in question. And this ability to understand language has been highlighted in the past as well. There is another research report that showed that Chad GPT was able to interpret Fed speak.

Now, when the Federal Reserve would make an announcement about their intentions moving forward, Chad GPT could pick up on the nuances of the message to determine whether the Federal Reserve was likely thinking about hiking, lowering or keeping rates stable. But even with that, there are a number of drawbacks that will pretty much prevent anyone from replicating the performance that Chachapiti saw in this research report. Firstly, the cited figure of 500% ignores any transaction costs, which isn't reasonable. And as highlighted in the research report, if you incorporate a 25 basis point transaction cost, your return actually drops to about 50% over more than

Speaker 1: a year's time, which is a little different.

Speaker 0: The report also doesn't mention taxes, which would likely take a further massive chunk out of your returns, given the frequency of trading. And given that the researchers themselves highlighted that the strategy worked best for smaller companies, specifically micro-cap companies in the bottom 10th percentile of market caps, liquidity would be an important consideration as well. Since there's a chance you wouldn't actually even be able to find a buyer or seller for these hypothetical trades. On top of all this, I would also take the entirety of the report with a grain of salt, since it's missing a lot of calculations, basic details like trade sizing, doesn't outline the assumptions and isn't actually peer reviewed. And finally, it's worth pointing out that regarding that 500% return figure, that return didn't come from chat GPT-4, the current system.

It came from chat GPT 3.5, which is an older, objectively less extensive model. You see, the researchers tried this strategy under different AI LLMs, including older versions of chatGBT. And chatGBT4, while getting a better sharp ratio, had nearly half the return of chat GPT 3.5. In other words, the better chat GPT became at understanding language and doing this sentiment analysis, the worse its returns became.

Speaker 1: So then why do we get the strong performance from the system? If a better sentiment analysis system actually did worse.

Speaker 0: Probably chance. There isn't much work in the paper to demonstrate that the results seen were due to superior sentiment analysis versus randomness. And there is a chance that The reason ChatGPT 3.5 performed the best is because it just happened to select the right stocks, whereas the other AIs happened to not. Especially when you consider that with the sentiment analysis, ChatGPT wasn't actually that consistent with its accuracy, with the researchers themselves only giving it a 51% likelihood of being accurate with a given prompt. Quote, it works well because when you're aggregating across multiple companies on multiple days, you get a result.

But for 1 given headline, it's basically a little better than tossing a coin.

Speaker 1: And this seems to line up with other stories we've heard about Chachapiti's accuracy, such as those 2 lawyers in

Speaker 0: the US who are now facing fines after using Chachapiti to help build their court case and using case citations that Chachapiti seemingly made up. So it's very feasible that Chachapiti's success here really came down to dumb luck. Michael Reeves, a popular tech YouTuber, outperformed the Nasdaq by letting a fish make his investing decisions.

Speaker 1: And there are countless of other examples of randomly selected portfolios doing better than the market over a given timeframe. None of which is evidence that you should let an animal pick your stocks, but it just goes to show that chance plays a role. So just because the trades generated by ChatGPT outperformed in a given year timeframe doesn't mean that ChatGPT has a strategy for outperforming the market. In fact, now that we've had some time passed, the other stories involved here with ChatGPT trading have turned out to be a bit more lackluster. That Twitter ChatGPT fund with $50, 000 invested, well, it started lagging the S&P 500 in May and has been lagging the benchmark since.

Speaker 0: The portfolio facing off against the 10 UK funds. It's still outperforming most of them, but again, it's still relatively a short period of time. Is more so a diss towards the fund managers than is a compliment of ChatGPT. Isn't even really a fair comparison against some of the funds. And the ChatGPT portfolio is likewise still underperforming the S&P 500.

And even when you look at ETFs that use AI that are intended for the stock selection process, a lot of their performance has still lagged the S&P 500. And part of the reason why it's so difficult to beat the market with an automated system like this is because a lot of the opportunities for these models are marginal and fleeting, meaning that they earn you a very small return requiring you to make tens of thousands of trades to take advantage of them. And they disappear over time. The majority of market trading volume is already algorithmic. It's already utilizing computers and AI that's intended for quantitative analysis.

They are all already competing with each other over what are likely smaller and smaller arbitrage and other opportunities in the market. So while AI that's meant for stock analysis does play a role with quantitative funds, does play a role with many active hedge funds that have it as more of a proprietary thing, it's unlikely that a chat GPT platform, an AI that isn't designed for stock picking that is widely available is going to give you any sort of edge when it comes to trading stocks. But Richard, what about the TikTok videos, the YouTube ads? I show people making crazy returns with chat GPT trading.

Speaker 1: Mostly scams. It's not to say that every example is going to be fraudulent. There might just be representation bias where you see the 1 good example of it working out amongst the thousands of other stories we don't hear about of people losing money with chat GPT. But on top of that, a lot of people are just taking advantage of people's ignorance. How many works recently put out a great video talking about why chatgpt is a breeding ground for scams at the moment and how it really is taking advantage of the layman's misunderstanding of how the system works.

So I'll leave a link for that down below if you want to check it out. But hopefully with everything I've covered, I've helped to explain why you probably shouldn't put your financial future in the hands of a chatbot anytime soon. So that's the video and none of it is to discredit the technological breakthrough that is chat to BT. I think most people can acknowledge that it's a remarkable advancement. This is also not to say that there is no application for ChatGPT in investing in trading.

However, those applications are really supplemental. And the moment you start relying on ChatGPT for trades, for stock picks, you enter a really dangerous area where you might as well be throwing a dart at a board. Even ChattyBT's well-known ability to code trading bots based on very simple prompts does often require the user to highlight what strategy they want this trading bot to use and often ends up being rife with bugs that doesn't actually lead it to work. So while it really is an interesting time for AI, just because something is technologically advanced, complex, and really remarkable, doesn't mean that's going to be your next gold rush. And you should really be careful not to let the spectacle and sparkle of something lure you into what's effectively get-rich-quick schemes.

Thank you for watching. I hope you found this video helpful. If you did, please make sure to like, subscribe, all that good stuff. It does help the channel tremendously. And let me know your thoughts on Chad2BT trading.

Is it something you've tried yourself? No judgments, despite the intent of this video. Thanks for joining me today, and as always, be safe out there.

See all The Plain Bagel transcripts on Youtube

Using ChatGPT to Trade Stocks - Let's Talk