Rust x Ethereum Day 2023 at The Foundry SF

Storm Silvkoff presents on "Cryo and Data Endgame"

And for the past few years, I've been experimenting with many different approaches for analyzing crypto data and for building the surrounding crypto data infrastructure.

And things are very exciting right now in the data space.

And this is thanks to a bunch of new Rust tools that I think will fundamentally change the way that we sort of think about crypto data.

the approaches that I've been experimenting with and sort of give my take on where I think the space is going.

So a fun buzzword in the Zeitgeist right now is the end game.

It's the idea of what is the ultimate system that we're working toward?

And so in the context of data infrastructure,

the data space that we need to solve or that an endgame solution needs to solve.

So first of all, right now, even though blockchains are public ledgers, actually collecting crypto data from these blockchains can take a huge amount of time and energy and expertise.

Instead, we want this process to be fast and cheap and easy.

And then there's also just basic data concerns, like we want our systems to be modular, scalable, robust.

So we have a pretty good idea of the properties that we want our systems to have.

So 1 of the main arguments that I want to make today is that we're not actually that far from solving these problems.

So first, I'm going to talk a little bit more about the problem space.

And then I'll sketch out what I think a proto-endgame architecture might look like.

And then I'll finish up by detailing a few tools in this architecture.

So, when I say data here, I'm specifically talking about the infrastructure

that makes on-chain data available to all the downstream research and applications.

So I'm not talking about consensus or validating blocks.

blocks are validated, how do we get those blocks to all the people that want it?

And making this process fast and cheap and easy is just a means to an end.

So monitoring, alerting, accounting, quantitative modeling.

And so there's going to be a lot of trade-offs that we have to navigate for this.

But at the same time, there can still be a shared set of foundations that all of these applications are built on top of.

And I think this is where Rust fits in and has a really interesting role to play.

So everyone knows Rust is really good for building infrastructure.

But the thing is, most people don't want to learn Rust.

And this is especially true among data analyst types, people that are doing different types of research.

People often prefer to work in higher level languages, like Python, like SQL.

But this is actually a great arrangement for Rust, because Rust is good at building composable systems that these higher-level languages can take advantage of.

And this has already happened in the Python ecosystem.

So many of the classic Python libraries now have new versions that are powered by Rust under the hood.

These new versions are faster, they're more robust, they often have better APIs, and so even though most Python users don't know Rust, Rust is still able to elevate the Python ecosystem to the point where people actually prefer it when libraries are written in Rust.

So, I think the dynamic looks something like this.

We build the foundational data infrastructure using tools like Rust.

And then once the foundation is really solid, this enables an entire ecosystem of new tools to grow on top of it.

So people that are using the higher level languages can just get the data quickly, easily, without worrying about infrastructure.

And this isn't just data scientists, this is anybody that needs data.

And this is ultimately the purpose of the data infrastructure.

I want to go over some of the tools and concepts now that can improve the typical crypto data workflow.

And I'll show how these assemble into an architecture that we use for internal data work at Paradigm.

And it's not quite an endgame system, but it's very flexible and it ticks a lot of the boxes that we've discussed so far.

So this is a high-level view of how data flows in the crypto ecosystem.

So it all starts at the top, where EVM nodes essentially define the ground truth of what happens on chain.

And then data from the EVM nodes can either go to indexers, or

it can go directly to the end users and the applications.

And right now, there are many, many, many different tools for implementing each part of this flowchart.

But the really cool development in the past 6 months is that this process can now be performed almost entirely using tools built in Rust.

And even though Reth is still in alpha, there's already a lot of tools for extracting the data out of Reth, either indexing it or delivering it directly to its final destination.

So just to name a few of these, Reth indexer is a way to query Reth data from Postgres, We heard about that a little bit.

And ethers.reth is a way to use RET's internal database as ether's middleware.

So a lot of these tools are still very new, but they already show a huge amount of promise, especially when it comes to their performance characteristics.

And then in the bottom half of the diagram here, we have a bunch of tools that aren't specific to crypto.

These are just Rust tools that have become really popular in the data engineering ecosystem.

And to really appreciate the advantages that you get with these, I think it would be helpful to explain 3 modern trends in data engineering.

So data engineering is something where Web 2.0 companies have made really amazing progress over the years that hasn't really percolated into crypto yet.

So there's a lot of low-hanging fruit by just looking at what's going on in Web 2.0 data engineering and adapting those best practices over here.

And each of these things I've listed can be a huge amplifier on the metrics that we care about.

It's 1 of the main ways that different programs on your computer can talk to each other.

And the big innovation here is a standardized IPC format called Aero.

Aero is kind of what the data ecosystem is uniting around as its common language for data programs to talk to each other, and what you get with Arrow is the ability to share data between different libraries and processes without needing to make copies.

And this is a huge unlock for efficiency and interoperability.

So it means that different programs can operate on the same underlying memory.

You can take a panda's data frame and convert it to a polar's data frame instantaneously, without needing any serialization.

Arrow is also becoming a common way for databases to ingest their data and to expose their data to others.

So, when you're making new data tools, it's good to ask yourself, how can I make my tool speak Arrow, because then you immediately get connected to this entire other ecosystem of other tools?

So, Arrow is only for when you're operating in memory.

When you're reading or writing to disk, the most common modern format is Parquet.

Parquet can be seen as an extreme evolution of CSV.

Most of you have probably seen CSV files before.

Every row in the file, or every line in the file, corresponds to a row in a table.

Well, Parquet stores tables too, but it does so in a much more sophisticated way.

So Parquet will break down the rows and the columns into separate chunks, and it will compress those chunks and it will index those chunks so that you get a very compact representation, and it can still be used for efficient queries that only read the parts of the file that are relevant to you.

So this leads to huge benefits for storage size and for speed, but really the coolest thing about Parquet is the modern ecosystem of tools that is forming around it, which brings us to our third data engineering trend which is the separation of storage versus compute.

So the traditional architecture for databases is that every database has its own custom storage format.

And each of these databases also has its own custom query engine that's run on the same machine that stores the files.

But more recently, these things have changed, and the storage and the computation have been decoupled.

So there are now standard storage formats, like Parquet, that are compatible with multiple query engines.

And the data files and the query engine no longer have to live on the same machine.

And this decoupling means that you can now sort of mix and match your storage versus computing solutions.

And you can scale each 1 separately, depending on your needs.

And you can also seamlessly go from querying files on your laptop to querying files on S3 without needing to change any code.

So with this, you get an easy path toward scalability.

And it's also flexible for your architecture to evolve over time.

So combining all these concepts, this is the proto-endgame architecture that I've been experimenting with.

And this is what I use for most of my internal data work at Paradigm, and it's been working pretty well.

I have a Rust tool called Cryo that extracts data from a ref node into parquet files, and then to analyse this data, I use Polars which is a Rust data frame library with a really nice query engine.

And if I want to avoid this middle step and avoid touching disk, Cryo can also deliver the data directly to pollers using Aero IPC.

So this architecture is a really good fit for crypto data in particular.

Most of the time, I just run things on my laptop, and it executes really quickly.

And then when I need more power, I can run this exact same architecture on a big cloud node.

And since your database here is just a bunch of immutable parquet files, it's very easy to keep data synced across multiple machines.

So just a little bit more about Cryo and Polars.

So it's basically a Rust rewrite of a bunch of Python code I had lying around, and this was my main motivation for learning Rust in the first place.

I was trying really hard to optimize this Python code, but Python gets really slow, it tends to be really fragile when you're doing a lot of parallelism.

The basic operation here is something like this on

the data set that you want to collect, the block range you want to collect it over, and you can do this either on command line or there's a Python interface or a Rust interface as well.

So as an example of usage, you can tell Cryo collect all the contracts that have been deployed on chain.

Cryo will spin up a bunch of worker tasks that collect the data in parallel and save the results as parquet files.

And then You can query these parquet files by pointing your query engine to this data directory.

Cryo gives you a huge amount of control over how the data is acquired, formatted, filtered, et cetera.

So Cryo is designed to be very composable and modular so that you can combine it with other tools.

So, this is an example of combining it with a cron job.

You just tell cron, hey, run this cryo command every 10 minutes, or however much you want, And this will maintain a live copy of your desired data set in whatever data directory.

And cryo is also item potent, so it will only collect

the data that is currently missing from the data set.

Cryo can collect most of the standard EVM data sets.

And thanks to parquet compression, you can fit most of these data sets, or even all of these data sets, on just a laptop.

So for example, the data set of all traces in Ethereum history is more than 5 terabytes on Google BigQuery.

But it's less than 500 gigabytes when you store it using Parquet.

So Cryo can also collect these datasets very quickly.

For example, Bantag was able to collect this traces dataset in under 10 hours from an Eragon node.

A few years ago, this process would have taken weeks.

So Cryo is still under active development.

So first of all, Cryo is currently using RPC to collect data, and this is nice because it works with any EVM node, but the performance will get a lot better once it has a direct connection to REST's internal database.

Number 2, Cryo will soon be able to collect a lot more data sets.

Basically, any information that you can get over RPC, Cryo will be able to build a dataset out of it.

And number 3, we're going to upload a copy of each of these Cryo datasets to the paradigm data portal so that you can access this data, even if you don't have access to an archive node.

And then very quickly, Polars is the other tool that I will talk a little bit about.

So Polars is a Rust data frame library for Python, which means that it's for reading and transforming large tables of data.

When you have a directory of Parquet files, pollers can run complex queries on those files in a really efficient way.

And something that's really cool is you can still query these files, even if they're too big to fit in memory.

So you can run queries on the 500 gigabyte traces dataset from your laptop.

I definitely recommend checking out Polars if you're doing any data work in Python.

So as an example, let's say you collected the parquet dataset of Ethereum contracts.

Polars can run just about any query you want on this data and it can do it quickly, so you could, if you wanted, you could load all the addresses of all contracts into Python from the files in under a second.

You can also do things like get all the contracts deployed by an EOA, get all the contracts deployed with a certain byte code.

All these things are very quick, and it's a similar story for all the other datasets that Cryo can collect.

And something that I've noticed in myself and others using this style of architecture is that it really changes the way that you think about your data.

When it's totally frictionless to explore your data, you tend to explore it more.

And you tend to gain a deeper level of understanding because you're exploring it more efficiently.

How do these things fit together into the data end game?

I still think there's a lot of work to do, but there's a huge amount of progress being made on all the important fronts.

So with modern tooling, we get things that are extremely fast, extremely cheap, not quite effortless, but it's getting easier all the time.

More and more data is becoming available, you get formats that are open and interconvertible, and you can run the same architecture on your laptop or in the cloud.

So I think that if this rate of progress continues, we're going to like each of these issues is going to reach a near optimal state in a short to medium-term time frame.

And with that, we're going to have sort of a new foundation that a rich data ecosystem can grow on top of.

I really thank you for your kind of these amazing repositories.

what I found is that the current of the parquet is now representing in the binary format if it's not a big intype.

is to transform those binary into the hex binary format and then to decode according to the ABI files that I want to decode it into.

kind of a painful time to process in my Spark clusters.

So is there any kind of more plans to implement those ABI or hectic decoding features in the cryo?

Yeah, so, this is actually something that's being worked on right now by Eric from Zora.

We have, this is definitely like a known problem that we want to tackle head on.

So we're going to get direct decoding of logs.

We're going to get direct decoding of different function calls and traces.

We had talked in the past about this not being well suited for maintaining live to the tip chain sync.

I was wondering if you'd figured out any solutions for maintaining some kind of batching process that combines the data from the chunks that have had enough blocks to complete with the delta at the tip of the chain that hasn't been completed yet?

So right now, Cryo has an option, a command line option where you can give it a number of blocks where it won't save any blocks that are newer than that age, so if you give it a high number, like 1, 000 blocks, you basically never have to deal with reorgs or anything like that.

to implement some direct reorg logic so that you can actually stay close to the tip without any sort of delay.

That's not currently something we're working on, but it's something we're thinking about.

Right now, Cryo is very sort of focused on bulk historical analysis, but reaching into these more like streaming type use cases, I think it's like a natural evolution and it will probably happen eventually.


Speaker 0: So, hi, I'm Storm. I work on data at Paradigm. And for the past few years, I've been experimenting with many different approaches for analyzing crypto data and for building the surrounding crypto data infrastructure. And things are very exciting right now in the data space. And this is thanks to a bunch of new Rust tools that I think will fundamentally change the way that we sort of think about crypto data.

Speaker 1: So Today, I want to share some of

Speaker 0: the approaches that I've been experimenting with and sort of give my take on where I think the space is going. So a fun buzzword in the Zeitgeist right now is the end game. It's the idea of what is the ultimate system that we're working toward?

Speaker 1: And so in the context of data infrastructure,

Speaker 0: what is the data endgame?

Speaker 1: There's a bunch of problems in

Speaker 0: the data space that we need to solve or that an endgame solution needs to solve. So first of all, right now, even though blockchains are public ledgers, actually collecting crypto data from these blockchains can take a huge amount of time and energy and expertise. Instead, we want this process to be fast and cheap and easy. And then there's also just basic data concerns, like we want our systems to be modular, scalable, robust. So we have a pretty good idea of the properties that we want our systems to have.

The big question is how.

Speaker 1: So 1 of the main arguments that I want to make today is that we're not actually that far from solving these problems. So first, I'm going to talk a little bit more about the problem space. And then I'll sketch out what I think a proto-endgame architecture might look like. And then I'll finish up by detailing a few tools in this architecture. So, when I say data here, I'm specifically talking about the infrastructure

Speaker 0: that makes on-chain data available to all the downstream research and applications. So I'm not talking about consensus or validating blocks. It's once the

Speaker 1: blocks are validated, how do we get those blocks to all the people that want it?

Speaker 0: And making this process fast and cheap and easy is just a means to an end. The real goal here is to enable all

Speaker 1: of the feature or all of

Speaker 0: the applications in this list. So monitoring, alerting, accounting, quantitative modeling.

Speaker 1: People need data for a

Speaker 0: lot of different things.

Speaker 1: And so there's going to be a lot of trade-offs that we have to navigate for this. But at the same time, there can still be a shared set of foundations that all of these applications are built on top of. And I think this is where Rust fits in and has a really interesting role to play. So everyone knows Rust is really good for building infrastructure. It's fast.

It's robust. It has a lot of nice common practices. But the thing is, most people don't want to learn Rust.

Speaker 0: And this is especially true among data analyst types, people that are doing different types of research. People often prefer to work in higher level languages, like Python, like SQL.

Speaker 1: But this is actually a great arrangement for Rust, because Rust is good at building composable systems that these higher-level languages can take advantage of. And this has already happened in the Python ecosystem. So many of the classic Python libraries now have new versions that are powered by Rust under the hood. These new versions are faster, they're more robust, they often have better APIs, and so even though most Python users don't know Rust, Rust is still able to elevate the Python ecosystem to the point where people actually prefer it when libraries are written in Rust. So, I think the dynamic looks something like this.

We build the foundational data infrastructure using tools like Rust. And then once the foundation is really solid, this enables an entire ecosystem of new tools to grow on top of it.

Speaker 0: So people that are using the higher level languages can just get the data quickly, easily, without worrying about infrastructure. And this isn't just data scientists, this is anybody that needs data. And this is ultimately the purpose of the data infrastructure.

Speaker 1: So that's kind of the problem landscape. I want to go over some of the tools and concepts now that can improve the typical crypto data workflow. And I'll show how these assemble into an architecture that we use for internal data work at Paradigm. And it's not quite an endgame system, but it's very flexible and it ticks a lot of the boxes that we've discussed so far. So this is a high-level view of how data flows in the crypto ecosystem.

So it all starts at the top, where EVM nodes essentially define the ground truth of what happens on chain.

Speaker 0: And then data from the EVM nodes can either go to indexers, or

Speaker 1: it can go directly to the end users and the applications. And right now, there are many, many, many different tools for implementing each part of this flowchart. But the really cool development in the past 6 months is that this process can now be performed almost entirely using tools built in Rust.

Speaker 0: And it looks something like this. So

Speaker 1: it all starts with a Reth node. We heard about this earlier today. And even though Reth is still in alpha, there's already a lot of tools for extracting the data out of Reth, either indexing it or delivering it directly to its final destination. So just to name a few of these, Reth indexer is a way to query Reth data from Postgres, We heard about that a little bit. RETDBpy is a way to do it from Python.

And ethers.reth is a way to use RET's internal database as ether's middleware. So a lot of these tools are still very new, but they already show a huge amount of promise, especially when it comes to their performance characteristics. And then in the bottom half of the diagram here, we have a bunch of tools that aren't specific to crypto. These are just Rust tools that have become really popular in the data engineering ecosystem. And to really appreciate the advantages that you get with these, I think it would be helpful to explain 3 modern trends in data engineering.

So data engineering is something where Web 2.0 companies have made really amazing progress over the years that hasn't really percolated into crypto yet. So there's a lot of low-hanging fruit by just looking at what's going on in Web 2.0 data engineering and adapting those best practices over here. And each of these things I've listed can be a huge amplifier on the metrics that we care about. So starting with standardized IPC. IPC is inter-process communication.

It's 1 of the main ways that different programs on your computer can talk to each other. And the big innovation here is a standardized IPC format called Aero. Aero is kind of what the data ecosystem is uniting around as its common language for data programs to talk to each other, and what you get with Arrow is the ability to share data between different libraries and processes without needing to make copies. And this is a huge unlock for efficiency and interoperability. So it means that different programs can operate on the same underlying memory.

You can take a panda's data frame and convert it to a polar's data frame instantaneously, without needing any serialization. Arrow is also becoming a common way for databases to ingest their data and to expose their data to others. So, when you're making new data tools, it's good to ask yourself, how can I make my tool speak Arrow, because then you immediately get connected to this entire other ecosystem of other tools? So, Arrow is only for when you're operating in memory. When you're reading or writing to disk, the most common modern format is Parquet.

Parquet can be seen as an extreme evolution of CSV. Most of you have probably seen CSV files before. You have a file. Every row in the file, or every line in the file, corresponds to a row in a table. And CSVs are all about storing tables.

Well, Parquet stores tables too, but it does so in a much more sophisticated way. So Parquet will break down the rows and the columns into separate chunks, and it will compress those chunks and it will index those chunks so that you get a very compact representation, and it can still be used for efficient queries that only read the parts of the file that are relevant to you. So this leads to huge benefits for storage size and for speed, but really the coolest thing about Parquet is the modern ecosystem of tools that is forming around it, which brings us to our third data engineering trend which is the separation of storage versus compute. So the traditional architecture for databases is that every database has its own custom storage format. So Postgres has its own format.

MySQL has its own format. SQLite has its own format. And each of these databases also has its own custom query engine that's run on the same machine that stores the files. But more recently, these things have changed, and the storage and the computation have been decoupled. So there are now standard storage formats, like Parquet, that are compatible with multiple query engines.

And the data files and the query engine no longer have to live on the same machine. And this decoupling means that you can now sort of mix and match your storage versus computing solutions.

Speaker 0: And you can scale each 1 separately, depending on your needs.

Speaker 1: And you can also seamlessly go from querying files on your laptop to querying files on S3 without needing to change any code. So with this, you get an easy path toward scalability. And it's also flexible for your architecture to evolve over time. So combining all these concepts, this is the proto-endgame architecture that I've been experimenting with. And this is what I use for most of my internal data work at Paradigm, and it's been working pretty well.

I have a Rust tool called Cryo that extracts data from a ref node into parquet files, and then to analyse this data, I use Polars which is a Rust data frame library with a really nice query engine. And if I want to avoid this middle step and avoid touching disk, Cryo can also deliver the data directly to pollers using Aero IPC. So this architecture is a really good fit for crypto data in particular. Most of the time, I just run things on my laptop, and it executes really quickly. And then when I need more power, I can run this exact same architecture on a big cloud node.

I don't need

Speaker 0: to modify any code, and it just works.

Speaker 1: And since your database here is just a bunch of immutable parquet files, it's very easy to keep data synced across multiple machines. So just a little bit more about Cryo and Polars. Cryo is a bulk data extraction tool. So it's basically a Rust rewrite of a bunch of Python code I had lying around, and this was my main motivation for learning Rust in the first place. I was trying really hard to optimize this Python code, but Python gets really slow, it tends to be really fragile when you're doing a lot of parallelism.

So yeah, I ended up with cryo. The basic operation here is something like this on

Speaker 0: the command line where you specify

Speaker 1: the data set that you want to collect, the block range you want to collect it over, and you can do this either on command line or there's a Python interface or a Rust interface as well. So as an example of usage, you can tell Cryo collect all the contracts that have been deployed on chain. Cryo will spin up a bunch of worker tasks that collect the data in parallel and save the results as parquet files. And then You can query these parquet files by pointing your query engine to this data directory. Cryo gives you a huge amount of control over how the data is acquired, formatted, filtered, et cetera.

So Cryo is designed to be very composable and modular so that you can combine it with other tools. So, this is an example of combining it with a cron job. You just tell cron, hey, run this cryo command every 10 minutes, or however much you want, And this will maintain a live copy of your desired data set in whatever data directory. And cryo is also item potent, so it will only collect

Speaker 0: the data that is currently missing from the data set.

Speaker 1: Cryo can collect most of the standard EVM data sets. And thanks to parquet compression, you can fit most of these data sets, or even all of these data sets, on just a laptop. So for example, the data set of all traces in Ethereum history is more than 5 terabytes on Google BigQuery. But it's less than 500 gigabytes when you store it using Parquet. So Cryo can also collect these datasets very quickly.

For example, Bantag was able to collect this traces dataset in under 10 hours from an Eragon node. A few years ago, this process would have taken weeks. So Cryo is still under active development. We have a lot of features on the way. So first of all, Cryo is currently using RPC to collect data, and this is nice because it works with any EVM node, but the performance will get a lot better once it has a direct connection to REST's internal database.

Number 2, Cryo will soon be able to collect a lot more data sets. Basically, any information that you can get over RPC, Cryo will be able to build a dataset out of it. And number 3, we're going to upload a copy of each of these Cryo datasets to the paradigm data portal so that you can access this data, even if you don't have access to an archive node. And then very quickly, Polars is the other tool that I will talk a little bit about. So Polars is a Rust data frame library for Python, which means that it's for reading and transforming large tables of data.

When you have a directory of Parquet files, pollers can run complex queries on those files in a really efficient way. And something that's really cool is you can still query these files, even if they're too big to fit in memory. So you can run queries on the 500 gigabyte traces dataset from your laptop. I definitely recommend checking out Polars if you're doing any data work in Python. So as an example, let's say you collected the parquet dataset of Ethereum contracts.

Polars can run just about any query you want on this data and it can do it quickly, so you could, if you wanted, you could load all the addresses of all contracts into Python from the files in under a second. You can also do things like get all the contracts deployed by an EOA, get all the contracts deployed with a certain byte code. All these things are very quick, and it's a similar story for all the other datasets that Cryo can collect. And something that I've noticed in myself and others using this style of architecture is that it really changes the way that you think about your data. When it's totally frictionless to explore your data, you tend to explore it more.

And you tend to gain a deeper level of understanding because you're exploring it more efficiently. So that's all I have for today. How do these things fit together into the data end game? I still think there's a lot of work to do, but there's a huge amount of progress being made on all the important fronts. So with modern tooling, we get things that are extremely fast, extremely cheap, not quite effortless, but it's getting easier all the time.

More and more data is becoming available, you get formats that are open and interconvertible, and you can run the same architecture on your laptop or in the cloud. So I think that if this rate of progress continues, we're going to like each of these issues is going to reach a near optimal state in a short to medium-term time frame. And with that, we're going to have sort of a new foundation that a rich data ecosystem can grow on top of. Thanks for your attention. Thanks for your attention.

Speaker 2: We have time for 2 questions. Hi. I'm using Crayo in production stage. I really thank you for your kind of these amazing repositories.

Speaker 3: And I want to ask questions like

Speaker 2: what I found is that the current of the parquet is now representing in the binary format if it's not a big intype. So what I did on my PySpark code

Speaker 3: is to transform those binary into the hex binary format and then to decode according to the ABI files that I want to decode it into. And it gives some

Speaker 2: kind of a painful time to process in my Spark clusters. So is there any kind of more plans to implement those ABI or hectic decoding features in the cryo? Thank you.

Speaker 1: Yeah, so, this is actually something that's being worked on right now by Eric from Zora. We have, this is definitely like a known problem that we want to tackle head on. So we're going to get direct decoding of logs. We're going to get direct decoding of different function calls and traces. All that's on the roadmap.

Speaker 4: I've got a question. We had talked in the past about this not being well suited for maintaining live to the tip chain sync. I was wondering if you'd figured out any solutions for maintaining some kind of batching process that combines the data from the chunks that have had enough blocks to complete with the delta at the tip of the chain that hasn't been completed yet?

Speaker 1: Yeah. So right now, Cryo has an option, a command line option where you can give it a number of blocks where it won't save any blocks that are newer than that age, so if you give it a high number, like 1, 000 blocks, you basically never have to deal with reorgs or anything like that. That's easy mode.

Speaker 0: Eventually it would be nice

Speaker 1: to implement some direct reorg logic so that you can actually stay close to the tip without any sort of delay. That's not currently something we're working on, but it's something we're thinking about. Right now, Cryo is very sort of focused on bulk historical analysis, but reaching into these more like streaming type use cases, I think it's like a natural evolution and it will probably happen eventually.

See all Paradigm transcripts on Youtube

Rust x Ethereum Day - Cryo and Data Endgame