My guest this episode is Jeff Yan, founder of Chameleon Trading.  
 Jeff began his career in high frequency trading at Hudson River Trading but soon moved over to the world of crypto where he built one of the largest market making firms in the space.
 After Jeff gets me up to speed with the basics of high frequency market making, we dive into some of the more esoteric components, particularly with respect to centralized crypto exchanges.  These include infrastructure quirks, adversarial algorithms, and why HFT P&L might actually be predictive of medium-term price movement.
 In the back half of the conversation, Jeff explains the problems he sees with current decentralized exchanges and introduces Hyperliquid, a new decentralized trading platform built on its own blockchain to provide performant order book execution for perpetual futures.
 Please enjoy my conversation with Jeff Yan.

Real quick, think about how you found out about this podcast.

You saw it shared on LinkedIn or someone sent it to you directly.

Well, that's how this podcast grows through word of mouth.

So no matter how you found it, if you could share it with someone new, it would mean the world to me.

I'm Corey Hofstein, and this is Flirting With Models, the podcast that pulls back the curtain to discover the human factor behind the quantitative strategy.

Korey Hofstein is the co-founder and chief investment officer of Newfound Research.

Due to industry regulations, He will not discuss any of Newfound Research's funds on this podcast.

All opinions expressed by podcast participants are solely their own opinion and do not reflect the opinion of Newfound Research.

This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions.

Clients of Newfound Research may maintain positions and securities discussed in this podcast.

For more information, visit thinknewfound.com.

My guest this episode is Jeff Yan, founder of Chameleon Trading.

Jeff began his career in high frequency trading at Hudson River, but soon moved over to the world of crypto, where he built 1 of the largest market making firms in the space.

After Jeff gets me up to speed with the basics of high frequency market making, we dive into some of the more esoteric components, particularly with respect to centralized crypto exchanges.

These include infrastructure quirks, adversarial algorithms, and why HFTPNL might actually be predictive of medium-term price movements.

In the back half of the conversation, Jeff explains the problems he sees with current decentralized exchanges and introduces Hyperliquid, a new decentralized trading platform built on its own blockchain to provide performant order book execution for perpetual futures.

Please enjoy my conversation with Jeff Yan.

Hitting it off early in the season with some crypto here.

We're gonna see how the audience responds, but I think this is gonna be a fun 1 because we're talking high frequency trading, maybe even digging a little into the secrets of high frequency trading.

And then in the back half of the conversation, gonna be talking about protocol design, which is a whole different new area of quant thinking.

Let's start with the typical stuff for guests who maybe don't know who you are or haven't caught on to your quickly growing Twitter stream.

My story probably sounds pretty similar for a lot of HFT folks out there.

I graduated from Harvard, did computer science and math, went straight to Hudson River Trading, which is 1 of the bigger market makers in TradFi.

Got to work on the most interesting problems.

But 2018 came along and with it, the crypto mania of building smart contracts on Ethereum, read the yellow paper and it just clicked.

And I knew that this was going to be the future.

And so I left to build sort of an L2 exchange protocol.

It was in the format of a prediction market because back then Augur had found a good market fit, but we were interested in the exchange technology.

So raise the money, move out to San Francisco to build this thing, built a team, but kind of shut it down after a few months because we realized it was not the right time.

A lot of regulatory uncertainty and really couldn't find users.

We barely knew how smart contracts worked.

Really were interested in speculating on tokens and not really DeFi at the time.

So shut that down, did a little soul searching, traveled, and ultimately decided I wanna go back into trading because the day-to-day was a lot more interesting than struggling to find product market fit.

So contemplating going back into the industry, joining some company, but thought maybe I would, since I knew all this about crypto from building, I would try to trade crypto first.

And so started as a bit of a side project, but quickly saw the opportunity there and scaled it up really way faster than I thought was possible.

I was surprised by how inefficient the markets were.

So, been heads down building that for, maybe at this point, almost 3 years.

Really seriously started early 2020, which was great timing.

So as the market 10X, even like 100X in volume, we kind of grew with it.

Ultimately, our market share ended up being 1 of the biggest centralized exchange market makers.

And about a year ago, we started looking at DeFi trading and sort of really reminiscent of when we started centralized exchange trading and that there were a ton of inefficiencies.

But in this case, the protocols themselves were quite poorly designed.

And we also saw this demand for truly decentralized product.

After the whole FTX thing, people were finally catching on to the not your keys, not your coins, counterparty risk, that kind of stuff.

Basically, it clicked for us that this was now the right time to build a decentralized exchange.

And so we've been sort of at that for maybe a quarter, a little more than that now.

And the HFT stuff is still running in the back autopilot maintenance mode but we're really focused and excited about building this DEX at this point.

Well a lot to unpack there and stuff we'll get into the conversation I'm excited to talk about learnings you had in the high frequency space and how that ultimately has influenced your design of this DEX.

But I want to start with the basics of high-frequency.

When I talk to people who are in high-frequency trading It seems like 1 of the biggest decisions you have to make is this concept of making versus taking.

It seems to be a very clear line in the sand of very differentiated strategies and what it takes to succeed with each of those types of strategies.

So I was hoping maybe you could explain the differences and how those differences have implications on the choice of strategy, design, infrastructure, and need, and even the research process.

I think this is the first big decision that you need to make when you're starting HFT.

And I think high level, I will say, there are more similarities than there are differences.

At the end of the day, you are doing this very infrastructure intensive, latency sensitive trading, but in many regards, they are opposites as well.

So the first big difference is that I would say making is more infrastructure heavy and taking is more stats, math, model heavy.

I think the best way to decide between the 2 is just what sort of work, What sort of research are you inclined towards?

Maybe I like as a concrete example, like when you're market making, you're kind of at the whim of people coming in and picking you off.

You often have large implicit exposure by being levered up and having open orders over many instruments, many price levels.

And if you screw up, really, that heavy tail is really gonna be painful.

Whereas you can have a strategy that takes once a day, and it can be a really good strategy, and it can be high frequency.

It could be news-based, it could be some sort of niche signal, but you have the luxury.

And because of that, you have the luxury to be much smarter.

If your thing is slow most of the time, doesn't trigger most of the time, that's okay, as long as When you do trade, it's good.

But with making, if you're doing well 99% of the time, 1% of the time you're a little slow and can't keep up with the data, or you're going to lose enough money during that

of the time to negate your P&L from the other 99%.

That's the fundamental infrastructure versus model difference between the 2.

Is it too simplistic to say with taking, you expect the market to move in the direction in which you're trading because you're willing to cross the bid-ask spread and so you expect the market to keep moving versus making, you're hoping the market doesn't move.

Someone's crossing the spread to meet you.

And then you're hoping the market doesn't move so that you can then sell across the spread again.

Like 1 is hoping the market almost stays flat in the timeframe of the trade and 1 is hoping there's a directional move.

So with HFT, we like to mark out to pretty short time horizons, but this is kind of true in general, like no matter what frequency you're trading at, the instant you take, you're actually suffering a loss.

So you're marking to mid your instance, suffering loss and you're only going to be profitable on average if, like you said, over whatever her predictive horizon you have, the price on average compensates for that immediate loss plus fees.

Whereas making the initial P&L is the highest it will ever be.

You just made a spread, but you're banking on that not being on average average selection.

And so necessarily when you make, if you sort of average out markouts of your trades, that P and L will decay over time, but your hope is that it doesn't decay past 0.

In our pre-call, you mentioned that 1 of the most difficult aspects of scaling up your business was actually not on the research side, but on the infrastructure side.

I saw on Twitter you said something to the effect of knowing how to normalize data isn't gonna print you money, but without it, you definitely won't.

Hoping you could talk about maybe some of the biggest lessons you learned in the infrastructure side of the equation and why you think it's so important.

So your question sort of has 2 parts and they're pretty tied.

There's the trading infrastructure and then there's the research infrastructure.

And so like data cleaning falls in the research, it's more like statistical practices.

Whereas trading infrastructure is like pretty unique high-frequency trading.

The stats stuff is I guess more well known, but probably worth emphasizing that it's the regime of noise, sort of signal to noise and high frequency trading is orders of magnitude higher than most things people study in academia.

And so filtering outliers is exponentially more important.

If you don't think about this stuff correctly, if you really just ignore all the outliers, then your model is going to be screwed over when the sort of black swan tail events do happen.

But if you don't filter them and normalize them correctly, then the outliers are gonna basically determine your entire model.

Concretely, I think, depending on what you're doing, using things like percentiles can be a lot more robust than using the actual values.

When we are using actual values, then are you throwing out outliers?

These kinds of things have very big effects.

On the infrastructure side, I think the biggest lesson we learned, sounds kind of silly, but you really need to learn this firsthand.

You might think you're super smart, you have this great pipeline that will clean the data and give you the inputs you want to your models.

But I'd say it's impossible to spend too much time looking at data, like you're always gonna learn something new.

And so when starting out, it's just like, write down all the raw stuff you're getting from the exchanges and just comb through it.

I think a super pretty crazy example of this is at some point, some exchange had some bug on their feed machines and flipped the price and size fields on, I forget if it was the book stream or the trade stream, but regardless, it completely messed up our internal counting code.

Imagine like Bitcoin's price and size being flipped.

And I think a lot of firms probably shut down immediately or quickly recover and like switch from alternative data source.

But things like that, you really want to be close to the raw data because no matter what logic you write, it's not going to be perfectly robust.

I guess another tip is really focus on timestamps.

So exchanges will often give you a bunch of timestamps with their data and it's kind of up to you to figure out exactly what they mean with each timestamp.

And this is important in terms of understanding the black box in terms of your latency, like what are you measuring exactly and seeing if you're keeping up, for example, or if they're sending you garbage.

Timestamps are a great way to distinguish between these different cases.

1 of the things that I see discussed a lot among high-frequency traders is this concept of fair.

I know it's something you've written about a few times talking about making sure someone's trading around fair.

I think fair means something slightly different for every trading firm.

It kind of speaks to the style of trading they're doing.

But at a high level, what's in common is that FAIR sort of incorporates your modeling into a predicted price.

It's a really useful abstraction because it splits this problem of writing a profitable strategy into 2, I would say comparably difficult pieces, depending on your strategy.

And that is the predicting the price piece and the executing your orders piece.

I guess this kind of goes back to the making versus taking question you asked earlier, but like making is heavier on the execution side, whereas taking is heavier on the modeling side.

And so basically taking your spent almost all your time thinking about this fair price.

And I think what goes into it is really up to you as a trader.

Like what kinds of data do you think you have an edge processing over, over the market?

I guess another thing is like, there doesn't have to be 1 fair price.

You might have multiple fares as inputs to this more machine learning style trading.

You might have like a 1 second prediction and 1 day prediction and your execution strategy may use these in different ways.

The optimization problem can be different in P&L space.

But I think when it's starting out, you can get very far by just doing a clean cut and saying, all right, I'm gonna put my work into first just coming up with a number, which is what I think I will trade around.

Like I'll quote around this, I'll use this number to cross the spread.

This will just be like my Oracle and then working around, okay, I have like this Oracle price it's a given to me.

What's the best way I can execute around it.

And so could that be something as simple as you're looking at 1 exchange and you might say, just throwing this example out there, Almost all the liquidities at Binance, I'm just going to assume the price at Binance is fair and then if other exchanges are lagging that by milliseconds or seconds you might be using Binance as fair and okay I can cross the spread at OKX or something like that because you're expecting this catch up across a different exchange.

And then there's other maybe statistical ways of estimating fair where you're not taking truth from 1 exchange but you're trying to use other market book related signals to come up with a fair?

So I think using the most liquid venue as the fair is a really good first approximation.

And I think before I started crypto, I think way back in the day, this was probably the best way to go about it because there were 10% arbitrages between the exchanges.

The problem was like, how do you move money between them?

And so this would have worked super well.

These days, it's been an interesting trajectory where there's been splitting, splintering of liquidity and then some sort of consolidation towards Binance, especially recently.

And so the thing you mentioned is probably a very good place to start is like just use Binance as fair.

That being said, I think you need to be careful when just using an external source as a fare, then maybe OKX is lagging a couple of milliseconds behind.

And maybe, it's not gonna be this simple these days, but let's say there was just like an opportunity to close the ARB each time Binance moved because nobody was lifting orders on OKX.

So you do that, and it'll work most of the time, but then it's crypto.

So OKX maybe, okay, they go into wallet maintenance and it's no longer possible to withdraw or deposit this coin, at least between Binance and OKX.

And now suddenly you'll see the ARB can't be closed and the markets diverge.

And if your fare is just Binance price, then you might get screwed.

So there's always subtlety, even in the super simple example.

It's never gonna be as simple as, okay, here's a number that I pull from some feed and that's my fare.

But it's certainly a good first approximation.

That leads nicely into where I want to go next which was around the idiosyncrasies of crypto exchanges.

And just that historically, reputationally, they are notoriously unreliable from a technology standpoint.

You gave the example earlier of the dirty data where price and volume got swapped, broken APIs, poor documentation, not all the API endpoints are always documented, some of them are hidden, sometimes you can have different parameters that no 1 actually knows about.

I think you had a great Twitter example about that recently, about being able to skip the risk engine or have a risk engine run in parallel.

Stuff that is completely undocumented that is interesting examples of orthogonal alpha that doesn't necessarily have to do with price prediction around fare.

How much alpha is there in things like simply understanding the API better than your competitors or measuring the latency of endpoints correctly versus say more traditional statistical alphas where you're trying to use the order book to guess pressure and direction.

The tweet you're referring to, I think was 1 of my more popular ones.

Which I still don't know whether it was an April Fool's joke, by the way.

I guess April Fool's has passed, so I'm allowed to say it was a joke, but it's closer to reality than people think.

So I think the real joke is that it's actually kind of true.

I've been meaning to do a follow up on that.

I should go tweet that after this podcast.

I think when you work at a quant company, you start to develop preferences.

Or maybe you come in with preferences of what you want to work on.

Like, oh yeah, I studied math, so I'm just gonna make cool machine learning models and find signals and generate alpha.

That's what matters because that's the hardest thing to do.

And I think that kind of attitude maybe works at a big company because people are so specialized, but if you're trying to run HFT on your own, then you're not gonna get anywhere, that attitude.

So the sort of dirty work that you're mentioning, understanding the API as well, seeing what's missing in the documentation, measuring latencies, this kind of stuff is super important.

My mental model for high frequency trading is, or really just like things in life, is that it's like a product of many numbers.

So as a quant, you know, I still wanna be quantitative about it.

Your efforts into different bins are additive in those bins.

You might make your model a little bit better.

Maybe you like spend 10 X time and like make 10 X the Delta there.

But at the end of the day, it's the product.

So it's like infrastructure times model, for example, as a concrete example, if your infrastructure is at 1 and your modeling is at 10, then where are you going to spend your unit of work?

Like, obviously you can mathematically see, you should always work on the thing that is smallest and the tough thing with HFT is like, it's kind of hard to know what these things are in the formula for like, multiplying together.

When we started, we thought it would be modeling work, but it's important to sort of have this meta analysis of like, wait, am I actually doing the most important things?

And you quickly realize that it's not obvious.

And there's a lot of edge in just like knowing what to work on.

It's always about getting the lowest hanging fruit, the 80, 20 principle.

I think, especially during when things are going well, it's easy to fall into the trap of like, all right, I got the basics down, like let me go do some like cool machine learning research, do the innovative stuff.

Not that there isn't any alpha there, but it's a lot of work for like diminishing returns.

And so when you're on a small team and there are still a lot of opportunities and your strategy is doing well, it's always good to like actually ask yourself and be honest.

Don't Be convinced by what the data tells you, you need to work.

For those who are keen on starting out in high frequency trading in crypto, you've recommended that they either just go make markets on Binance and focus on alpha generation, which I sort of interpreted as taking, not making, or picking some long tail exchange and trying to figure out the infrastructure quirks around that long tail exchange, and that's a good source of edge.

Can you elaborate on why you think these are the 2 best avenues and how the approaches differ?

It's a bit like the bell curve meme, and you just don't want to be that guy in the middle.

In this case, if you view the bell curve as like the exchanges, then the big problem is the middle exchanges, maybe say like, you know, rank 2 through 7 or something, you have a lot less volume than Binance, but about the same level of competitiveness and toxic flow.

And the flow can be worse than Binance because at least Binance, as we know, the reason their volume is so high is that they have complete stranglehold over retail volume.

I don't know how they do it, but they do.

That's just the numbers speak for themselves.

And so you don't get that padding, that nice mix of toxic and retail flow.

The big HFT firms have all onboarded the top, I don't know how many, like let's say top 15, they've definitely onboarded.

And so they're gonna be trading full capacity bigger strategies and you're not gonna get much juice there.

So if you're willing to like challenge yourself to do that super scalable, large centralized exchange trading strategies, then just start with finance and it will generalize as well as it does.

And there's no point in starting in the middle.

But the other thing you mentioned is like, yeah, you can also be like super left the bell curve.

There's no shame in just like finding a super small opportunity, something that is overlooked by the big players or just doesn't have enough capacity for it to be worth their time.

And I think niche infrastructure is a super good example of this.

So just like with DEXs, the protocol designs can be just outright dumb.

You can see this to a lesser extent on a lot of how smaller centralized exchanges go to write their tech.

And if you're the only 1 who has this insight into how that works, then that can be a strategy.

Infrastructure is actually like often a big source for alpha, and there's not such a clean line between the 2.

It's not really a problem, but like you might be concerned, oh, this doesn't generalize.

Like, okay, I understand how the tech on this random small exchange works.

And yes, that's true, but I think people undervalue just having something that works live.

That should just be everyone's number 1 priority, and it really shouldn't matter how small.

I guess there's sort of a floor on how small it can be, unless you're looking at super weird things.

If you're trading something on a volume, you're gonna make some money, and if that is high sharp, consistent, and robust to tail events, then you've got something that 99% of people don't.

And yeah, maybe the exact strategy doesn't generalize, but in my experience, you get the reps in for the full research loop, like putting things into production, you learn so much during that, that then even just scrapping it and going for Binance at that point will be just orders of magnitude easier.

And also like often little things, like yeah maybe the tech isn't like exactly the same on other exchanges, but you start to notice these principles and you start to get this fountain of, or just like endless stream of ideas from other things that already work, And those types of ideas tend to be way better than things you just thought kind of thin air.

I think there's a lot of value to both approaches.

See all Flirting with Models transcripts on Applepodcasts

Jeff Yan - High Frequency Crypto Market Making & the Hyperliquid Exchange (S6E2)