Rust x Ethereum Day 2023 at The Foundry SF

James Prestwich presents on "Alloy"

I'm here to talk about Alloy, which is the newest member of the Rust Ethereum stack, still, I think.

Alloy was started early this year when we decided we just wanted to trash all of ethers and rewrite everything from the ground up, eliminate all parody dependencies, and use proper modern Rust to do these things.

The parody libraries had the unfortunate downside that they were written in like 2015, and Rust as a language barely existed back then.

So these days we have Async await, we have Gats, we have const generics, we have all these nice features that we wanted to integrate into Ethers.

And we were just better Rust developers than we were 3 years ago when we started this project.

So Alloy is now a collection of the most powerful, fastest, most intuitive libraries for Ethereum developers.

It is a successor to the Ethers project and will eventually cover most of the scope of Ethers.

Some of the things like Ethers Sol-C we're going to punt right out of the project because I don't want to handle that.

That's going to become Evalier and Matt's problem, I think.

So right now, Alloy is primarily me and Danny.

Danny I don't think was able to make it today.

But over the last 8 months, give or take, that we've been working on this, we've put a huge amount of effort into bringing it up to snuff.

It's being integrated into places like Optimism, Arbitrum, Risk 0, Zeth.

It's making its way across the ecosystem pretty quickly, which makes us feel really good about ourselves.

We've tried to keep this to low-level useful things in the Ethereum ecosystem.

We've focused primarily on the interface between Rust and the EVM.

That means the primitive types, Solidity type representations, JSON ABI files, ABI encoding and decoding.

We feel that we have done the best job of this out of any libraries out there in any language, no matter who wrote them or what they do.

And I'm going to talk a little bit about what the components we built are and why we think they're better than everything else by wide margin.

So the first thing we wanted to do was get away from Parity's Ethereum types.

These were written, again, 7, 8 years ago, barely updated since then.

They have a number of deficiencies and cause dozens of Telegram support questions every day that we just have to answer with, you know, use the debug formatter.

I've said use the debug formatter like 50 times a week for the last 3 years.

So, we decided to start with the primitive types.

This means addresses, uints, signed ints, fixed byte arrays, and dynamic byte arrays.

We based it on Remco's work, RUINT, which I think he's gonna be talking about in a little bit.

So you're kinda going down through the stack here, start with Reth, work down through the Alloy, and then into RUINT that everything's based on.

Alloy primitives is substantially complete.

We have a lot of support for different features and is being used in Revm, Reth, and other places.

After Alloy, we wanted to get into ABI encoding, and we decided to shave the yak first.

So We want to make it really easy for Rust and Solidity to communicate.

That means that we need an idea of what it means to communicate with Solidity.

And I'm going to get into why the Solidity syntax parser is important in a minute.

But basically, you know, Danny just went out and one-man-armied an entire Solidity-asked representation, just parsing Solidity source into tokens that Rust can operate on effectively.

So you can actually just load your contract into Rust right now.

We'll parse everything out, we won't compile it, but we will have an AST representation.

Okay, we wanted to make ABI encoding easy and fast.

ABI encoding is schema-based, so the only way to build an encoder is if you know the types that you're encoding and decoding.

So it kind of follows from there that the first thing you do is make a representation of the types that you're operating on.

In practice, this is all solidity types, you know, the bytes 32, uint 256, solidity structs, errors, events.

So we went ahead and wrote a static representation of all of those types in Rust.

Anything that you can represent as a solidity type, you can now represent as a Rust type.

All of these types are zero-sized, they disappear at runtime, and we use them to make very well-optimized static ABI coding routines.

Rather than like loading everything into some dynamic type representation, we just generate a coding routine at run time for every type that you actually use in your programme.

So, you are actually representing your solidity type system that's in your contract in Rust, and letting the Rust compiler hyper-optimise your encoding routine.

This makes us 2 to 3 times faster than the next best encoding routine and more intuitive to use because you can actually just write solidity.

As you see here on the slide, we've got that bool2.

We represent that solidity in Rust, and that's what we work with.

So, in order to make this even easier, because you don't want to be writing fixed array bool to, you know, like, the complex Rust type name for this simple Solidity pool too.

You just wanna copy-paste Solidity into your code base.

So we wrote a procedural macro that lets you just copy Solidity into your code base.

You can copy entire interfaces, contracts, everything into your Rust code base.

It will automatically generate all of the sole type types, expose them to you, and let you just move them around, use them like native Rust types.

And this is powered by the Syn Solidity AST parser that we talked about earlier.

We are moving towards a world in which you just point Rust at your foundry project.

It automatically loads up every type that you use and has a static hyper-optimized encoder for everything and you don't even have to think or worry about it.

So it allows you to embed your Solidity contracts in a Rust code base and not even think about encoding decoding.

No more loading up JSON files to get interfaces, no more expensive run time parsing of these things.

It all happens at compile time and it makes the fastest possible ABI encoder.

In case you do want to do things at runtime, because you need to do like a MevBot, or you need to do EIP 712 signing, or something like that, we also wrote a dynamic ABI encoder.

This means a dynamic runtime representation of slow-to-deduce type systems and values.

The purpose here is to allow for the sign-typed data RPC requests, which I think is also Remco's fault somehow.

You don't know when you compile your signer what you're going to sign.

In order to handle that, you need to be able to represent those types at runtime.

So while Alloy sole types has no run time overhead from the type system, Alloy dine ABI lets you be a lot more flexible, operate on run time types, but have all the overhead that comes with that.

Nevertheless, We're doing the same thing that ETH ABI did, but we're doing it significantly faster.

Danny optimizes everything just crazy, way better than I can.

So even though we're not having all the benefits of static routines, it's still faster than the alternatives.

So, that's all of Alloy that exists right now that you can go out on GitHub and use and interact with and is ready for testing consumption.

The thing I'm working on right now is Alloy Next.

This is the part of Alloy that replaces Ether's provider networking JSON RPC stack.

So this is the part of Alloy that's actually going to talk to the nodes.

This is like the HTTP, the web sockets, the IPC code.

The main thing we did here is we're swapping out custom network code for tower.

Tower is a commonly used service model in Rust.

It lets us get a bunch of ecosystem code for free and we no longer have to maintain things like quorum or rate limiters in-house.

That's going to let us cut several thousand lines of code out of the ethers project which I love cutting code.

I love it when other people maintain useful code.

It involves these huge tower associated type blocks, so I'm not showing it here.

The next part of the stack is alloy networks.

When building ethers, these people went out and very rudely started building other block chains besides Ethereum.

We have these roll-ups and stuff now, and everybody who made another blockchain decided to get clever and start changing the transaction formats, and that means changing the JSON RPC responses and arguments and everything, and before you know it, you have this data type explosion where every network uses a different transaction, so you have to have a different object, and ethers is not built for that.

Ethers middleware assumes every transaction is a transaction.

It doesn't check that it's an optimism transaction or an arbitrum transaction.

So new in Alloy is the network abstraction.

A network describes the types of the JSON RPC.

So the Optimism network type is free at runtime, but it includes the type information for the Optimism RPC.

It tells you what Optimism says a transaction is, or Optimism says a receipt is.

The provider is abstract over the network and over the transport.

So you can have a provider that connects to optimism via HTTP.

If you look down here in the estimate gas, you can see that it uses the network's transaction request object.

So when you estimate gas, you're providing the type that optimism says is a transaction request.

This lets us use the same middleware methods and code, but swap out the actual types that you're working with, so that you can specialise your code for optimism if you want to.

Or use Ethereum, or provide your own network.

The goal is to make the RPC commands that we must work with because that's how nodes work, more flexible in our code base, and the specific user problem we're solving is as of today, you cannot use Optimism and Ethereum in the same binary with ethers.rs.

In order to enable optimism, you have to make it not work for Ethereum any more.

After alloy providers with the network abstraction, you can have connections to both networks in the same program at the same time.

And As a bonus, the traits are now object safe, so you can abstract over what middleware you're using and you don't have to manually manage that any more, and every user-facing part of the API gets simpler.

That means I explained it really well, right?

I mean, if you shout at me, I can probably hear you.

impressive to get the, using the procedural macro to support solidity.

That's, I cannot imagine how many dirty hacks there.

So it's actually cleaner than you would think.

It's about as clean as you can make a procedural macro.

Due to the constraints of procedural macros, you can never make them really clean, but this one's not bad, relatively.

We don't currently have plans to support any other languages than Solidity, but we have discussed it, and the stack is designed so that you could write a SYN Viper or a SYN Yule if those languages support the Rust token model as well as Solidity does.

With Vypr, it's very complex because white space is significant and the Rust token model doesn't support that, so you would need to do a lot more of the tokenisation and parsing work than you need to do with Solidity.

Our long-term goal is to have essentially an intermediate representation that you build these Solidity things into the intermediate representation and then expand from there so that we can be a lot more flexible and build interesting transpilers and stuff.

We're operating on asks, we're not operating on bytes.

Where does RLP and SSZ fit into Alloy, or does it?

We have the RLP crate that we took over maintenance from Reth, which is a great implementation.

So that is officially part of Alloy as well.

It's just significantly less interesting to talk about on stage.

The focus for Alloy is primitive types and interacting with the EL.

And SSZ doesn't directly further those goals.

But it is the kind of low-level, pure, primitive that we want to eventually support.

Are you keeping the middleware structure for customization or have you played around with any other ways of doing that?

We are likely to keep it but improve the abstraction.

This is another thing where we home rolled that middleware abstraction and we're not happy with it.

So the idea is to have the provider's trait replace the middleware trait and to have a similar but better abstracted middleware stacking system.

If anyone has an alternative to that, I do not love the middleware stacks that we produce.

The goal of this is to make it simpler, so instead of a giant concrete type, we box, or we type erase and make object trade objects.

That makes it a lot simpler to manage the types of your middleware.

We also have been experimenting with a tower-like layering abstraction, so you can have a layer builder that makes it easier to instantiate these things.

But I don't love the core middleware abstraction.

None of us seem to have come up with anything better.

Yeah, The middleware is all tech that blame 2020.

If you have a better suggestion than middleware stacking, let me know.


Speaker 0: Go. All right, so I'm James Prestwich. I'm here to talk about Alloy, which is the newest member of the Rust Ethereum stack, still, I think. Alloy was started early this year when we decided we just wanted to trash all of ethers and rewrite everything from the ground up, eliminate all parody dependencies, and use proper modern Rust to do these things. The parody libraries had the unfortunate downside that they were written in like 2015, and Rust as a language barely existed back then.

So these days we have Async await, we have Gats, we have const generics, we have all these nice features that we wanted to integrate into Ethers. And we were just better Rust developers than we were 3 years ago when we started this project. So Alloy is now a collection of the most powerful, fastest, most intuitive libraries for Ethereum developers. It is a successor to the Ethers project and will eventually cover most of the scope of Ethers. Some of the things like Ethers Sol-C we're going to punt right out of the project because I don't want to handle that.

That's going to become Evalier and Matt's problem, I think. Thank you, buddy. So right now, Alloy is primarily me and Danny. Danny I don't think was able to make it today. But over the last 8 months, give or take, that we've been working on this, we've put a huge amount of effort into bringing it up to snuff.

Alloy today is usable and useful. It's being integrated into places like Optimism, Arbitrum, Risk 0, Zeth. It's making its way across the ecosystem pretty quickly, which makes us feel really good about ourselves. Alloy is a collection of libraries. We've tried to keep this to low-level useful things in the Ethereum ecosystem.

We've focused primarily on the interface between Rust and the EVM. That means the primitive types, Solidity type representations, JSON ABI files, ABI encoding and decoding. We feel that we have done the best job of this out of any libraries out there in any language, no matter who wrote them or what they do. And I'm going to talk a little bit about what the components we built are and why we think they're better than everything else by wide margin. And it's not just because of our egos.

So the first thing we wanted to do was get away from Parity's Ethereum types. These were written, again, 7, 8 years ago, barely updated since then. They have a number of deficiencies and cause dozens of Telegram support questions every day that we just have to answer with, you know, use the debug formatter. I've said use the debug formatter like 50 times a week for the last 3 years. So, we decided to start with the primitive types.

This means addresses, uints, signed ints, fixed byte arrays, and dynamic byte arrays. We based it on Remco's work, RUINT, which I think he's gonna be talking about in a little bit. So you're kinda going down through the stack here, start with Reth, work down through the Alloy, and then into RUINT that everything's based on. Alloy primitives is substantially complete. We have a lot of support for different features and is being used in Revm, Reth, and other places.

After Alloy, we wanted to get into ABI encoding, and we decided to shave the yak first. So We want to make it really easy for Rust and Solidity to communicate. That means that we need an idea of what it means to communicate with Solidity. And I'm going to get into why the Solidity syntax parser is important in a minute. But basically, you know, Danny just went out and one-man-armied an entire Solidity-asked representation, just parsing Solidity source into tokens that Rust can operate on effectively.

So you can actually just load your contract into Rust right now. We'll parse everything out, we won't compile it, but we will have an AST representation. Okay, we wanted to make ABI encoding easy and fast. ABI encoding is schema-based, so the only way to build an encoder is if you know the types that you're encoding and decoding. So it kind of follows from there that the first thing you do is make a representation of the types that you're operating on.

In practice, this is all solidity types, you know, the bytes 32, uint 256, solidity structs, errors, events. So we went ahead and wrote a static representation of all of those types in Rust. Anything that you can represent as a solidity type, you can now represent as a Rust type. All of these types are zero-sized, they disappear at runtime, and we use them to make very well-optimized static ABI coding routines. Rather than like loading everything into some dynamic type representation, we just generate a coding routine at run time for every type that you actually use in your programme.

So, you are actually representing your solidity type system that's in your contract in Rust, and letting the Rust compiler hyper-optimise your encoding routine. This makes us 2 to 3 times faster than the next best encoding routine and more intuitive to use because you can actually just write solidity. As you see here on the slide, we've got that bool2. We represent that solidity in Rust, and that's what we work with. So, in order to make this even easier, because you don't want to be writing fixed array bool to, you know, like, the complex Rust type name for this simple Solidity pool too.

You just wanna copy-paste Solidity into your code base. So we wrote a procedural macro that lets you just copy Solidity into your code base. You can copy entire interfaces, contracts, everything into your Rust code base. It will automatically generate all of the sole type types, expose them to you, and let you just move them around, use them like native Rust types. And this is powered by the Syn Solidity AST parser that we talked about earlier.

We are moving towards a world in which you just point Rust at your foundry project. It automatically loads up every type that you use and has a static hyper-optimized encoder for everything and you don't even have to think or worry about it. So it allows you to embed your Solidity contracts in a Rust code base and not even think about encoding decoding. No more loading up JSON files to get interfaces, no more expensive run time parsing of these things. It all happens at compile time and it makes the fastest possible ABI encoder.

In case you do want to do things at runtime, because you need to do like a MevBot, or you need to do EIP 712 signing, or something like that, we also wrote a dynamic ABI encoder. This means a dynamic runtime representation of slow-to-deduce type systems and values. The purpose here is to allow for the sign-typed data RPC requests, which I think is also Remco's fault somehow. You don't know when you compile your signer what you're going to sign. That's just the way it works.

In order to handle that, you need to be able to represent those types at runtime. So while Alloy sole types has no run time overhead from the type system, Alloy dine ABI lets you be a lot more flexible, operate on run time types, but have all the overhead that comes with that. Nevertheless, We're doing the same thing that ETH ABI did, but we're doing it significantly faster. Danny optimizes everything just crazy, way better than I can. So even though we're not having all the benefits of static routines, it's still faster than the alternatives.

Okay. So, that's all of Alloy that exists right now that you can go out on GitHub and use and interact with and is ready for testing consumption. Still, you know, early beta. Pre 1.0 release. The thing I'm working on right now is Alloy Next.

This is the part of Alloy that replaces Ether's provider networking JSON RPC stack. So this is the part of Alloy that's actually going to talk to the nodes. The first component is Alloy transports. This is like the HTTP, the web sockets, the IPC code. The main thing we did here is we're swapping out custom network code for tower.

Tower is a commonly used service model in Rust. It lets us get a bunch of ecosystem code for free and we no longer have to maintain things like quorum or rate limiters in-house. That's going to let us cut several thousand lines of code out of the ethers project which I love cutting code. I love it when other people maintain useful code. Thank you.

So this replaces ethers providers. The code is very generic. It involves these huge tower associated type blocks, so I'm not showing it here. The next part of the stack is alloy networks. When building ethers, these people went out and very rudely started building other block chains besides Ethereum.

We have these roll-ups and stuff now, and everybody who made another blockchain decided to get clever and start changing the transaction formats, and that means changing the JSON RPC responses and arguments and everything, and before you know it, you have this data type explosion where every network uses a different transaction, so you have to have a different object, and ethers is not built for that. Ethers middleware assumes every transaction is a transaction. It doesn't check that it's an optimism transaction or an arbitrum transaction. So new in Alloy is the network abstraction. A network describes the types of the JSON RPC.

So the Optimism network type is free at runtime, but it includes the type information for the Optimism RPC. It tells you what Optimism says a transaction is, or Optimism says a receipt is. This lets us build Alloy providers. The provider is abstract over the network and over the transport. So you can have a provider that connects to optimism via HTTP.

If you look down here in the estimate gas, you can see that it uses the network's transaction request object. So when you estimate gas, you're providing the type that optimism says is a transaction request. This lets us use the same middleware methods and code, but swap out the actual types that you're working with, so that you can specialise your code for optimism if you want to. Or use Ethereum, or provide your own network. The goal is to make the RPC commands that we must work with because that's how nodes work, more flexible in our code base, and the specific user problem we're solving is as of today, you cannot use Optimism and Ethereum in the same binary with ethers.rs.

Optimism is a feature. In order to enable optimism, you have to make it not work for Ethereum any more. After alloy providers with the network abstraction, you can have connections to both networks in the same program at the same time. And As a bonus, the traits are now object safe, so you can abstract over what middleware you're using and you don't have to manually manage that any more, and every user-facing part of the API gets simpler. So That's the slides for today.

Does anyone have questions? That means I explained it really well, right? Yeah. Do we have microphones around? Who's carrying...

I mean, if you shout at me, I can probably hear you. Oh, sorry. It's very

Speaker 1: impressive to get the, using the procedural macro to support solidity. That's, I cannot imagine how many dirty hacks there. But any plan to support YOL? Or?

Speaker 0: So it's actually cleaner than you would think. It's about as clean as you can make a procedural macro. Due to the constraints of procedural macros, you can never make them really clean, but this one's not bad, relatively. We don't currently have plans to support any other languages than Solidity, but we have discussed it, and the stack is designed so that you could write a SYN Viper or a SYN Yule if those languages support the Rust token model as well as Solidity does. With Vypr, it's very complex because white space is significant and the Rust token model doesn't support that, so you would need to do a lot more of the tokenisation and parsing work than you need to do with Solidity.

Our long-term goal is to have essentially an intermediate representation that you build these Solidity things into the intermediate representation and then expand from there so that we can be a lot more flexible and build interesting transpilers and stuff. We keep it at the code gen level. We're operating on asks, we're not operating on bytes. Not operating on bytes.

Speaker 1: Great, thanks.

Speaker 0: Where does RLP and SSZ fit into Alloy, or does it? RLP is part of Alloy. We have the RLP crate that we took over maintenance from Reth, which is a great implementation. So that is officially part of Alloy as well. It's just significantly less interesting to talk about on stage.

SSZ we have not in-house for Alloy. We may in the future. The focus for Alloy is primitive types and interacting with the EL. And SSZ doesn't directly further those goals. But it is the kind of low-level, pure, primitive that we want to eventually support.

1 last. Are you keeping the middleware structure for customization or have you played around with any other ways of doing that? We are likely to keep it but improve the abstraction. This is another thing where we home rolled that middleware abstraction and we're not happy with it. So the idea is to have the provider's trait replace the middleware trait and to have a similar but better abstracted middleware stacking system.

If anyone has an alternative to that, I do not love the middleware stacks that we produce. The goal of this is to make it simpler, so instead of a giant concrete type, we box, or we type erase and make object trade objects. That makes it a lot simpler to manage the types of your middleware. We also have been experimenting with a tower-like layering abstraction, so you can have a layer builder that makes it easier to instantiate these things. But I don't love the core middleware abstraction.

None of us seem to have come up with anything better. Yeah, The middleware is all tech that blame 2020. If you have a better suggestion than middleware stacking, let me know. Please, someone, please help. Round of applause for James, please.

Thank you.

See all Paradigm transcripts on Youtube

Rust x Ethereum Day - Alloy