- The meeting discusses updates and developments on DevNet 10, including latency analysis and stress tests.
- The team has found stability in DevNet 10, with some edge cases and latency issues identified.
- The map workflow is set up on DevNet 10, but there are debugging issues with Tersec on Nimbus and the relay.
- DevNet 11 launch may be postponed due to a consensus fact change.
- Ethereum on ARM testing shows similar results to the team's ARM testing, which is positive for future testing.
- A discussion on the consensus specs PR for adding Bob sidecar inclusion proof takes place, with agreement on its benefits and improvements.
- The addition of the block header in the sidecar prevents equivocation and makes API interactions and client implementation easier.
- The change also eliminates the construction of inconsistent blob sidecars and improves security.
- The drawbacks include the need for changes in all APIs, which will require time and coordination.

- The discussion is about separating block propagation and data availability sampling in order to make it more compatible with future changes.
- There is overwhelming support for the proposed change, with only a little reservation from Nimbus.
- The relay and builder APIs will require the most attention and time to implement.
- The relay will need to build sidecars and the CL will need to build proofs, which will increase complexity and testing requirements.
- There are concerns about the implementation and testing complexity, but the change is seen as a positive long-term improvement.
- The relay can potentially build proofs for the builder's sidecars, but this would require an extra round of communication.
- The proposed solution involves the builder sending everything to the relay, the relay sending the header to the proposer, and the builder assembling everything and broadcasting.
- The relay and proposer can work in parallel to handle the commitments and make the necessary blocks and proofs.
- Overall, the change is considered beneficial but will require careful coordination and testing.

- The participants discuss making changes to the builder blobs and beacon API.
- They discuss the timeline for implementing these changes, including shutting down DevNet 10 and launching DevNet 11.
- They consider the possibility of a ShadowFox and decide to do a girly ShadowFox using DevNet 10 images.
- The participants agree to have initial spec drafts ready for discussion on Thursday's consensus layer call.
- They plan to target DevNet 12 in approximately 2 to 3 weeks.
- They mention DevConnect in 2 weeks and express enthusiasm for working on the modifications.
- Overall, the participants seem supportive of the proposed modifications and plan to continue discussions and implementation.

- The meeting discussed updates and developments on DevNet 10, including latency analysis and stress tests.
- Stability in DevNet 10 has been found, with some identified edge cases and latency issues.
- There are debugging issues with Tersec on Nimbus and the relay, but the map workflow is set up on DevNet 10.
- The launch of DevNet 11 may be postponed due to a consensus fact change.
- Testing of Ethereum on ARM shows positive results for future testing.
- A discussion took place on the consensus specs PR for adding Bob sidecar inclusion proof, with agreement on its benefits and improvements.
- The addition of the block header in the sidecar improves security and API interactions.
- Separating block propagation and data availability sampling is discussed for compatibility with future changes.
- The proposed change has overwhelming support, with only minor reservations from Nimbus.
- The relay and builder APIs will require the most attention and time to implement.
- Concerns about implementation and testing complexity are raised, but the change is seen as a positive long-term improvement.
- The proposed solution involves the relay and proposer working in parallel to handle commitments and make necessary blocks and proofs.
- Changes to the builder blobs and beacon API are discussed, with plans to have initial spec drafts ready for discussion.
- The timeline for implementing these changes, shutting down DevNet 10, and launching DevNet 11 is considered.
- A girly ShadowFox using DevNet 10 images is planned.
- DevConnect in 2 weeks is mentioned with enthusiasm for working on modifications.
- Overall, participants are supportive of the proposed modifications and plan to continue discussions and implementation.

https://github.com/ethereum/pm/issues/890

Okay, I think most folks here are going to join.

Welcome to the DynKoon interop testing call number 34.

Are there any new updates or developments since we spoke last week on the DevNet?

Yeah, so we've been running like a series of experiments on DevNet 10.

The latest 1 has a latency analysis over here.

So what we did was on the first version we just had DevNet 10.

Nothing crazy on it, just targeting like 3 blobs.

After we did that, we ran the stress test and the stress test included targeting like 6 blobs all the time, collected as much data as we could.

Then we had the blobber that was sending invalid blobs through a percentage of the network.

And on Friday, we turned that off and then we switched to a different mode that the blobber had, which is added delay of roughly 2 seconds to roughly 10% of the blobs on the network before gossip.

And then the latency analysis is the result of that 1.

It seems like everything's mostly stable.

And I think, yeah, Enrico said he also found something in Teku, and that tends to happen if there's a higher latency for the last blob before the EPUB transition.

So it also makes sense that it doesn't happen very often, which could be an explanation for some of these unnatural spikes that we see in the graphs.

But yeah, that's mostly what we have on DevNet 10.

Does anybody have any questions or thoughts on the analysis that was done?

Or any client teams wanna share any insights?

We got the map workflow set up on DevNet 10.

I think every single CL has proposed at least 1 slot with a blob with MEV on DevNet 10.

There is an issue that we're still actively debugging with Tersec on Nimbus and the relay.

We're yet to figure out why, but Yeah, I think other than that, we're mostly good on Mav.

We were gonna launch DevNet 11 today, but because of the new consensus fact change, we might postpone that.

I guess it'll be a part of the discussion and then we can talk about what the DevNet cycle should look after that discussion maybe.

Yeah, we leave that Matan up until probably tomorrow.

Or if you guys can think of other scenarios or data that we want to collect, then let us know and we do our best to get that done by tomorrow.

And yeah, 1 other tiny thing is the guys from Ethereum on ARM also took part in testing this time.

And their ARM testing basically shows the same numbers that we found on our ARM testing, which is also good news in the future.

But yeah, basically Blobs and ARM is fine.

We didn't run any validators, but we plan to run validators on DevNet 11 on ARM also.

Any questions or comments for clients teams on the analysis?

Any further questions or comments on DevNet 10?

Next up on the agenda is to chat about this consensus specs PR, add Bob sidecar inclusion proof.

Does somebody here feel comfortable giving an overview on that?

Yeah, I can give a small update or summary.

So essentially the way that we were doing this before, like currently, is kind of reflecting how we handle the attestation gossip.

And yeah, what that basically means is that you get like all these blob sidecars and it's not immediately clear, for example, if like this message is an equivocation.

And this change, basically, puts the sign beacon block header in the sidecar as well.

And Yeah, I think everyone is pretty much in agreement that this is the right direction to move in.

Does anyone have any questions or feel differently?

1 thing I would add is that I started implementing it over the weekend.

And yeah, I will say besides, I guess as we do more testing, right?

Thanks to all the blobber testing that we realized, there is some concern with equivocation blobs because they come for free and there could be splitting attack with a fortress with that and they're just a bunch of unknowns moving forward, right?

Aside from that, it does make the API interactions much cleaner, and it's both beacon API and the builder API, because you no longer have to sign the block.

It also makes the validator client implementation easier.

We don't have to implement a different endpoint for the web presigner.

So I think overall, it's a really big improvement because it basically enables a cleaner code on the client side.

I also agree and think that this is a good addition because Again, there is a lot of blocks or content handling that is sort of a code spaghetti we have out there and this will totally get rid of that as well as it makes sense to link the blobs with the block.

of calls ago, I brought up like an edge case that would have like bricked import on lighthouse and had to do with like essentially constructing a blob sidecar where your block root doesn't actually contain the KZG commitments that you claim it contains, but then you can like propagate these across the network and no 1 can tell whether they're valid or not.

But you can't do that anymore if we include the block header.

So on top of like eliminating or like adding the equivocation penalty, it's also we're making this construction of inconsistent blob sidecars.

You can immediately identify it's invalid.

So that's another big plus for this change.

And with regard to like all the APIs, like they are all cleaner, but I will say like, they are all going to have to be changed.

So that is going to take like a bit of time and coordination.

That is the drawback of making this change.

It just makes everything a lot simpler so it's easier to get right and also yeah basically just makes all costs of security much better.

In the PI it was mentioned there were some cons too because this was considered before as well and was not adopted.

So I would like to hear what were the cons.

I think the people who would possibly be against this are not on the call, so that's a little tricky.

Was this specific idea of having the header with the proof proposed before, because I mainly just remembered the discussion about including the blobs with the block, combining them together.

I guess we don't have enough people here.

Yeah, I mean, it would be good to hear what the cons are.

I mean, I will say that I think the original thought here about separating this from the original block propagation, it was this idea that it would make it a bit more forward compatible with another data availability sampling solution.

So you wouldn't need to modify the block, the beacon block propagation at all, if you were to then add full dink sharding at a later point.

And I think that having a separate message here and having the blobs proof doesn't really affect the normal beacon block propagation that much anyway.

So it continues to adhere to that principle that we were trying to design for.

But if this was thought previously, it would be nice to hear what people thought negatively about it back then.

So maybe by the end of the day, those people, somebody could ping those people and have them share their thoughts.

I know we heard from, I think 3 consensus clients.

Did Techoo or Nimbus thumbs up this also?

Yeah, I think there was just like a little bit of reservation from Nimbus, but I'm not sure they like thumbs down or anything.

It seems like it's pretty overwhelming support.

And if the main downside is that this is going to take a little bit of time to get implemented and update all the APIs, I think it's the cost we have to pay for doing the robust thing.

So I think Danny will wait until the end of the day to see if there's any additional questions or concerns and try and cut the spec with the update in

All right, POTUS, just pop in here whenever you get your mic working again.

Otherwise, we can talk about DevNet planning going forward.

Well, can we talk about just like, I think like the beacon API and the builder API are probably the most critical path here.

They will take the longest to spec out and perhaps the longest to implement as well.

And besides that, we should also alert the relayer as well.

I'm not sure when the next MEV boost code is, but yeah, they also, I suspect a lot of, like most of the changes will be on their end, because now they have to add this inclusion group to the Blob Stack cart, which before they would just kind of like, they don't have to do much, but now they have to do slightly more.

So I can take care of the builder specs updates.

And yeah, I mean, there's a little bit more that the relay will have to do, but also I think this is like such a big improvement at the CLA that it's just going to happen.

And it's really not that much extra work.

So this thing of mentioning Dan Brad, just came up with a private conversation of myself and Jacek.

And he mentioned that this was thought of.

So since I am just a third person and I wasn't there in the first conversation, I would just ping either Dankrank or Jacek and just clarify this if there's any reservation.

As for the relay, I do expect that this change is going to take us in a position, it's going to put us in a position that is slightly more complicated perhaps because we're going to need to test with them.

So as of today, we only make 1 block, we can make it in parallel and we make only 1 object that we build and we may or may not use it with a builder and we can just quickly swap a payload and put it in that blinded block.

With this change, I think the path with the relay is going to be quite different.

We're going to send back the signed header, and then the relay, I suspect, is going to build those sidecars, the blob sidecars.

So clients will have to keep track of a bunch of objects.

On your local payload, you're going to have to have your locally built sidecars with their own proofs that you might never use because the builder is going to send you those anyways.

But still just to be prepared, you need to start building a bunch of things in parallel that we currently don't do.

So I think this is going to take some time to, so I'm just echoing what everyone is saying.

It's going to take a long time, actually, to produce, but it's a very good change.

Sorry, I don't think I quite understood what the difficulty on the relay side was.

The way I understand it is that you're saying that the CLs would have to be building proofs that they wouldn't normally be doing right now.

**Lucie** Yeah, so typically what you do is you build a block locally and you wait for the builder's block at the same time.

And you typically just ditch your local block that you just use to compare values, to compare whether or not it's better for you to use it or not.

But typically you just ditch your local block.

Now we're gonna have to be building the local block and proofs for the block transactions that our local blocks might use.

And also build this send assigned header to the relay.

And then the relay is going to be responsible for building the sidecars.

So there's many more objects that are going to be like having to be built in parallel that we currently don't do.

And is the concern processing or is the concern just implementation and complexity?

Oh, It's just complexity of implementation and testing.

to have to have a lot of unit tests and a lot of hype tests.

And I think everyone here agrees that in the long term, it might take us 1 more, but in the long term, this is a much better place to be.

But CL can also build the proofs for the builder sidecars and basically send it over to the builder so builder doesn't really has to do anything.

So, I mean, if the, so that's going to require 1 extra round of communication.

The CL doesn't have the blocks for the builder's payload.

So, what's going to have to happen, I suspect that the best way here is to rely on the relay to build those proofs, because the relay doesn't want to have 1 extra round neither with the builder nor with the proposer.

And since we are relying on the builder to do this, then I'm sorry, on the relay to do this, then I suspect that this is going to put us in a precarious situation where we need the relay to be ready to be testing this.

Which is something that has been bugging us on every fork.

But Builder can still give the blinded side cars like how they do in get header right now.

Right, so that would be another way so that the builder sends everything to the relay.

The relay could send the proposer everything for the proposer to propose, but that's not going to work.

So I think the right way of doing this is going to be as follows.

The builder is going to send everything to the relay.

The relay is going to send the header to the proposer.

The proposer is going to return it back signed.

And then the builder is going to assemble everything and broadcast.

I'm sorry, the relay is going to assemble everything and broadcast.

Yes, so put as eat like relay and proposer can do this both at the same time in parallel.

So the case that you commitments are sent from relay to proposer cause they need to make the block and then that's all they need to make the proofs.

So there's no like blocking on getting the blobs from the relay or something like that.

I just wanted to add, we do this today already.

We basically build the local blobs, and then we try to sign the build of blobs.

And if the builder's bid is too low or the override flag is enabled true, then we'll switch to broadcasting the local blob.

So that's not much difference from this perspective.

Yeah and instead of signing we can just build the proofs for the builder blobs and send them to the builder when the time is to submit BlindedBlock.

Yeah, definitely it's much better this thing that we're only signing 1 message now.

Because I figured the majority of the changes will be on the beacon API side.

Was that you volunteering to be the owner, Terrence?

I was hoping that someone was more competent, yes, to do this.

I can volunteer as tribute to make a PR at least.

What, What's the next steps, I guess, for putting all of this together?

I think let's just assume that this change is accepted into the spec.

We'll give it to the end of the day and let Danny merge it.

But assuming that does happen, where do we go from here?

What kind of target are people feeling for getting back into the DevNet cycle?

I mean, I think the first thing is we have to get the API changes up and like have discussions around those.

And then like, once we agree on those, we can implement them.

So I feel like it's like at least a week or 2 for the specs, then a week or 2 for the implementation.

And then at that point, like, try to test it on DevNet.

Because, I mean, the more important change and discussion that we seem to require will be on the builder side.

And that is something I think is more contentious and has more design options.

If that's the case, would we be okay with just getting rid of DevNet 10?

Because it's way too big to just have running for that long.

We can have DevNet 11 that's basically the exact same thing.

So not including any of the changes we spoke about just now.

And then we can leave that running until we're ready.

Dev nets can still be used for latency kind of analysis.

And I don't think we need a new dev net for it.

Because I mean, the main reason of getting rid of DevNet 10 is it's expensive.

Yeah, I mean, if basically expenses can be brought down for DevNet 11.

Else let's continue with DevNet 10 and sort of get the latency thing also figured out.

Is there any value in doing ShadowFox at this time?

Because that's also, I mean, it's again a cost thing, it's not an effort thing.

We can do a girly ShadowFox this week if there's something to learn from it.

Or would it be better to just wait for all of these changes and then do it later on.

I think we should try just 1 shadow fork.

Okay, so we'll do a girly shadow fork just using the exact DevNet 10 images.

Does it make sense to wait for the shadow fork once this beacon API is merged in?

There's not much point doing it right now, in my opinion.

I think we should do it even before DevNet 11 so as to get the familiarity with if there are any other issues out there lurking.

Okay, so the timeline now is tomorrow we're going to shut down DevNet 10 and launch DevNet 11.

That will be running for the next 2 or 3 weeks, then have them at 12 to test out this new beacon API.

Then probably reconsider timelines for Girdley Fork After that.

That seems to be the consensus right now.

We do have a consensus layer call this week on Thursday.

Is there any chance that we can have some initial stabs at the spec ready to discuss in that call?

And I think, yeah, that would give us some time to discuss some questions and concerns synchronously.

And hopefully sometime early next week maybe get them merged in custom releases and then get people to start implementing them.

And we will just see on the next, I guess, yeah, we'll just take it next week and see when we wanna start targeting DevNet 12.

But roughly thinking like on the order of 2 to 3 weeks from now.

Yeah, DevConnect is in 2 weeks, So we'll all be in a building that Tim will find and provide.

It will bring us food and water, and we will be working on this I think.

Okay is there any other things that we were hoping to discuss on this call?

It sounds like everybody's pretty pro this modification to the blob sidecar message.

Again, we'll wait until the end of the day, and I assume Danny's going to merge it, but just to give the people who might be dissenting some time to share, and try to have some specs ready for the builder and the beacon API by Thursday to discuss.

Last questions or comments before we wrap up?

See all Ethereum transcripts on Youtube

Dencun Interop Testing Call #34