Rust x Ethereum Day 2023 at The Foundry SF

Emily Hsia presents on "Shadow"

A shadow event is a gasless event logged in an isolated fork that shadows mainnet.

You can think of shadow as unlocking permissionless edit mode on any contract on mainnet.

So this means that you can add custom event logic that can include and access any on-chain state, including internal state and private variables, and it's all gasless.

And once you've written this new event logic, you can go back in time and backfill historical data that you didn't get before.

So what this means is that if you need on-chain data, instead of writing convoluted SQL pipelines or assembly script, you can write code close to the metal, directly in the smart contract in Solidity to get the exact data in the exact format that you want.

When you want to add a shadow event, The first step is to spin up your own Shadow Fork.

A Shadow Fork listens to transactions happening on mainnet

And this is completely isolated from mainnet.

So gas costs and contract size limits are completely removed, and you can go crazy here.

In the hosted product, your shadow fork is near real-time, And

it has all the same exact state aside from the additional event logs that you added.

And if you added any additional storage slots, that's the big difference.

But everything else, exactly the same as mainnet.

Second step is to edit the source code of the contract that you want to shadow, and you can shadow any verified contract, or you can shadow unverified contracts that you have the source code for.

And then the third step is to deploy your shadow contract onto your shadow fork and then once deployed, front fills start immediately and then

And then finally, you access your shadow events like you would your normal mainnet data via JSON RPC.

We support all the standard ETH RPC JSON endpoints.

And we also support other access models like GraphQL, webhooks, and SQL tables and parquet tables.

So using a Shadow Fork is pretty simple, but there's a lot going on behind the scenes and under the hood to make this really fast and really performant.

You might actually do this multiple times a day when you run your Foundry tests.

So we took a lot of inspiration from Foundry.

And we're actually using certain crates in Foundry to help us make this blazingly fast, performant, and parallelizable.

So if you're building crypto native infrastructure, there's actually a lot of alpha in Foundry that you should consider leveraging.

The 1 that I'm going to highlight today is called the Backend Handler, which is a highly multi-threaded fork backend used in Foundry to run multiple fork simulations.

So when you do vm.createfork and run a simulation, this is what's happening under the hood.

So basically what it does is when you spin up multiple forks, there is a single instance of the backend handler that is basically fulfilling all of the JSON RPC requests.

And so this is doing really a lot of the heavy lifting.

The back-end handler uses Tokyo channels to dedupe in-flight requests and then caches the previously requested results.

And you can thank this module for not blowing through your alchemy compute limits every time you run your foundry tests.

In this snippet of code, we are fetching for a storage slot at a particular address in the backend handler.

Just first checks if the value is in the cache.

Otherwise, it knows that we need to request this data.

So once we know we need to request this data, what it does is it basically checks first if we have a pending in-flight request, if we've already requested for this storage slot, and we don't need to send another request.

If we haven't seen it before, then We add it to the queue of pending requests to be processed.

And then finally, all this is doing is just pulling for all the requests that are in flight.

If it sees a request finally come back, it just sends it back to all the listeners via the Tokyo Center channel.

So huge shout out to Matt on the core Foundry team for designing and building all this code that I showed today.

And I highly recommend reading through this code if you're building really complicated high I.O.

And if you need any weekend reading for this weekend, this PR, I don't know if you can see it, but it's PR 1715 in Foundry.

It's a really good 1 to read if you are curious about how forking works, and also it has an example of how a VM code works from front to back.

So if there's 1 takeaway that I wanted to give to you guys for this talk is that The name

of the game for building in Ethereum is composability.

And composing on top of Foundry and Reth is the new meta for building crypto-native infrastructure.

So if you're building infra, you should really be thinking about using Foundry and Reth as tools in your tool belt.

And if you're interested in using these tools with us in prod systems, come find me afterwards and

lot more that we could have covered here.

It was only 5 minutes, so feel free to find me afterwards if


Speaker 0: Hi, everyone. I am Emily. I'm the CTO of Shadow, and I'm going

Speaker 1: to be talking about shadow events today. So first, what is a shadow event? A shadow event is a gasless event logged in an isolated fork that shadows mainnet. And why are they cool? Why are they valuable?

Speaker 0: You can think of shadow as unlocking permissionless edit mode on any contract on mainnet. So this means that you can add custom event logic that can include and access any on-chain state, including internal state and private variables, and it's all gasless. And once you've written this new event logic, you can go back in time and backfill historical data that you didn't get before. So what this means is that if you need on-chain data, instead of writing convoluted SQL pipelines or assembly script, you can write code close to the metal, directly in the smart contract in Solidity to get the exact data in the exact format that you want. Cool, so how does it all work?

Speaker 1: When you want to add a shadow event, The first step is to spin up your own Shadow Fork. A Shadow Fork listens to transactions happening on mainnet

Speaker 0: and then executes all those transactions

Speaker 1: on your fork. And that's why we call it a Shadow Fork.

Speaker 0: And this is completely isolated from mainnet. So gas costs and contract size limits are completely removed, and you can go crazy here.

Speaker 1: In the hosted product, your shadow fork is near real-time, And

Speaker 0: it has all the same exact state aside from the additional event logs that you added. And if you added any additional storage slots, that's the big difference. But everything else, exactly the same as mainnet.

Speaker 1: Second step is to edit the source code of the contract that you want to shadow, and you can shadow any verified contract, or you can shadow unverified contracts that you have the source code for. And then the third step is to deploy your shadow contract onto your shadow fork and then once deployed, front fills start immediately and then

Speaker 0: we will kick off a backfill process. And then finally, you access your shadow events like you would your normal mainnet data via JSON RPC. We support all the standard ETH RPC JSON endpoints. And we also support other access models like GraphQL, webhooks, and SQL tables and parquet tables. Cool.

So using a Shadow Fork is pretty simple, but there's a lot going on behind the scenes and under the hood to make this really fast and really performant. And forking mainnet is not a new idea. You might actually do this multiple times a day when you run your Foundry tests. So we took a lot of inspiration from Foundry. And we're actually using certain crates in Foundry to help us make this blazingly fast, performant, and parallelizable.

So if you're building crypto native infrastructure, there's actually a lot of alpha in Foundry that you should consider leveraging. The 1 that I'm going to highlight today is called the Backend Handler, which is a highly multi-threaded fork backend used in Foundry to run multiple fork simulations. So when you do vm.createfork and run a simulation, this is what's happening under the hood. So basically what it does is when you spin up multiple forks, there is a single instance of the backend handler that is basically fulfilling all of the JSON RPC requests. And so this is doing really a lot of the heavy lifting.

The back-end handler uses Tokyo channels to dedupe in-flight requests and then caches the previously requested results. And you can thank this module for not blowing through your alchemy compute limits every time you run your foundry tests. So here's just a little snippet of code. In this snippet of code, we are fetching for a storage slot at a particular address in the backend handler. So you can see it's pretty simple.

Just first checks if the value is in the cache. If it is, easy. We'll just send it right back. Otherwise, it knows that we need to request this data. So once we know we need to request this data, what it does is it basically checks first if we have a pending in-flight request, if we've already requested for this storage slot, and we don't need to send another request.

If we haven't seen it before, then We add it to the queue of pending requests to be processed. And then finally, all this is doing is just pulling for all the requests that are in flight. If it sees a request finally come back, it just sends it back to all the listeners via the Tokyo Center channel. So huge shout out to Matt on the core Foundry team for designing and building all this code that I showed today. And I highly recommend reading through this code if you're building really complicated high I.O.

Systems. And if you need any weekend reading for this weekend, this PR, I don't know if you can see it, but it's PR 1715 in Foundry. It's a really good 1 to read if you are curious about how forking works, and also it has an example of how a VM code works from front to back. So if there's 1 takeaway that I wanted to give to you guys for this talk is that The name

Speaker 1: of the game for building in Ethereum is composability. And composing on top of Foundry and Reth is the new meta for building crypto-native infrastructure.

Speaker 0: So if you're building infra, you should really be thinking about using Foundry and Reth as tools in your tool belt. And if you're interested in using these tools with us in prod systems, come find me afterwards and

Speaker 1: we can chat. And there's a

Speaker 0: lot more that we could have covered here. It was only 5 minutes, so feel free to find me afterwards if

Speaker 1: you have any other questions.

See all Paradigm transcripts on Youtube

Rust x Ethereum Day - Shadow