- The speaker has completed an audit of the Vypr codebase and wants to share a technical walkthrough of how it works.
- They discuss the setup.py file and how it holds the entry points for compiling using Vypr.
- They explain the different arguments that can be passed to the compiler, such as version, formats, storage layout, EVM version, optimization, etc.
- They dive into the main entry point for the compiler and explain how it works, including generating the abstract syntax tree and intermediate representation.
- They provide links to articles that can provide a deeper understanding of Vypr.

- The video provides an overview of the steps involved in generating an abstract syntax tree (AST) from Vyper source code.
- The process starts with pre-parsing, which ensures that certain elements like pragma lines are set properly and makes modifications to the code.
- The code is then parsed using Python's AST module, with some additional steps to align Vyper code with Python code.
- Annotation is done to add information about the source code and previous types.
- The code is translated into a Vyper AST, with some modifications and checks for unsupported types.
- Once the AST is generated, it can be inspected and further processed.
- The next step is to generate a folded AST, which involves validation and folding of constants.
- Folding simplifies the code by plugging in constants where appropriate, reducing runtime operations.

- The process of folding in programming involves performing operations at compile time instead of runtime to save on gas cost.
- The folding process involves validating and replacing constants, literal operations, subscripts, and built-in functions.
- The process is repeated until no changes are made.
- After folding, the program validates the semantics by preparing a namespace and checking variables, functions, and interfaces.
- The program also generates getter functions for variables tagged as public internally.

- The code includes a class that performs various validations and transformations on an Abstract Syntax Tree (AST).
- The class uses a visit function to traverse the AST and call specific visit functions based on the type of node.
- The validations include checking for correct implementation, variable declarations, events, and enums.
- There is a bug related to the implementation of payable functions in interfaces.
- The code also adds getters to public variables, removes certain elements from the AST, and validates functions.
- The data positions are set based on the storage layout provided or determined by the code.
- The code handles reentrancy locks for functions and allocates storage slots for variables.
- The next step involves generating an intermediate representation (IR) based on the folded AST.
- The IR generation includes optimizations for code size and gas usage.
- The functions in the code are sorted to ensure proper order of execution.
- The code is complex and involves extensive debugging and understanding of Vyper syntax.

- The video discusses the process of sorting and organizing functions in a linear manner.
- Functions are categorized as internal or external, runtime, or constructor.
- Memory allocation is done in a way that each function can only access its allocated memory slice.
- The video explains the creation of code for reentrancy checks and the generation of intermediate representation (IR).
- The parsing of function bodies is a crucial part of the process, where expressions, statements, and assignments are parsed into IR.
- Various types of statements like if statements, assert statements, for loops, etc. are parsed differently.
- The parsed code is then assembled into a representation of the function, including a cleanup routine if necessary.
- The process is repeated for each internal function, and all functions are eventually put together to represent the entire contract.

- The video discusses the creation of function dispatchers in Ethereum.
- There are three types of dispatchers: linear, sparse, and dense.
- The linear dispatcher checks each function selector sequentially.
- The sparse dispatcher uses buckets to distribute function selectors.
- The dense dispatcher finds a magic number to ensure efficient distribution of function selectors.
- The dispatchers include checks for call data size, payable status, etc.
- The process also handles fallback and constructor functions.

- The speaker provides a technical walkthrough of the Vypr codebase audit.
- They explain the setup.py file and its role in compiling with Vypr.
- Different arguments for the compiler are discussed, such as version, formats, storage layout, etc.
- The main entry point for the compiler is explained, including generating the abstract syntax tree and intermediate representation.
- Links to articles for a deeper understanding of Vypr are provided.
- The video outlines the steps of generating an abstract syntax tree from Vyper source code.
- Pre-parsing, parsing, annotation, and translation steps are involved in generating the AST.
- Folding is done to simplify the code by plugging in constants where appropriate.
- Semantics are validated, and getter functions are generated for public variables.
- A class is used to traverse and validate the AST.
- Data positions are set based on storage layout, and reentrancy locks and storage slots are handled.
- An intermediate representation is generated with optimizations for code size and gas usage.
- Functions are sorted and organized, and function dispatchers in Ethereum are discussed.
- Three types of dispatchers (linear, sparse, and dense) are explained.
- The process also handles fallback and constructor functions.

- The piece discusses the process of optimization at the IR level in a programming language called Viper.
- The optimization process involves recursively calling an optimize function on different parts of the code to make improvements.
- Different types of optimizations are performed, such as merging consecutive memory operations and simplifying arithmetic operations.
- The piece also mentions the use of a formal verification tool called Howmost to ensure the accuracy of the optimizations.
- Finally, the IR output is obtained, which can be further used for generating assembly code.

- The video discusses the process of generating assembly and bytecode in the compilation flow.
- There are two functions involved in these steps: "compile to assembly" and "assembly devium".
- The "compile to assembly" function converts the intermediate representation (IR) into assembly code, which includes opcodes and tags for jumps and instructions.
- The process involves recursive calls and keeping track of variable heights for referencing.
- Different actions are taken for each element in the IR to convert it into assembly code.
- Postambles are added to the assembly code, including error handling and data segment relocation.
- An optimized assembly function is called to make further adjustments based on specific conditions.
- Finally, the assembly code is converted into bytecode by reconciling values and replacing references with actual values.
- The bytecode is then ready for deployment or further processing.

- The last step in the process is translating bytecode into opcodes
- The build opcodes function iterates through the bytecode and converts it into opcodes
- Most opcodes require 2 bytes, but some may require more
- There is a small issue with converting the length of the bytes into hex, resulting in potential value discrepancies
- The opcode translation is not used for deployment and may not be significant

The text discusses the process of optimization in the Viper programming language at the IR level. It involves recursive calls to an optimize function and various types of optimizations. The use of a formal verification tool called Howmost is mentioned. The output is the IR code, which can be converted into assembly code and bytecode. The bytecode is then translated into opcodes, although this translation may not be significant for deployment.

A high level walkthrough of the Vyper compiler's source code.

Timestamps
00:00 Intro & CLI
4:45 Compiler Phases
11:03 Abstract Syntax Tree
19:58 AST Validation & Folding
38:09 Generate IR Nodes
51:03 Function Dispatcher
1:01:59 IR Optimization
1:12:08 IR to Assembly
1:19:29 Assembly to Bytecode

What's up guys, I just finished up this Vypr audit and 1 of the hardest things auditing a codebase this big is just kind of getting your head around the whole thing and how the different pieces work and what they're for.

Now that I've spent a few weeks deep in it, kind of meshing it with my brain.

I figured it'd be useful to brain dump a little bit of that kind of like technical walkthrough of how it all works.

So that anyone looking at this in the future can kind of get a headstart and not spend as much time on that piece as I did.

So I think what I'll do is just dive right in and kind of go through kind of how I did like the first pass of making sense of it.

If it makes sense I may veer off and explain a little bit about tooling setup I used and how I structured testing, and maybe a little bit about some of the loose ends that I wasn't able to have time for that might be interesting to explore in the future.

So if we look at the Viper repo we can see there's a file setup.py that's run when it's installed and this is what holds the entry points.

So if we see like when you're compiling using Viper file name it's calling to this place.

And so this is the argument that everything's getting passed to.

And if we follow that along we can see that basically this is like standard Python parsing where it's grabbing a bunch of arguments to that CLI call.

It's not actually setting it here as far as I can see.

There's like a check later basically because you're only running 1 VyprVersion, right?

So it's checking to basically make sure those align and reverting that doesn't.

Showing gas estimates that just kind of sets a flag so that gas throughout a lot of that compilation process it's gathering it up and changing how it prints it out.

Formats, this is all the different formats, we'll come back to this later, but basically we're gonna put together like an array of formats to output and then the compiler will go through its whole process, saving all the things it needs to save, and then kind of give you back all those different things.

Those could be intermediate representation, the byte code, opcodes, assembly, anything along that process.

So because there's no inline assembly in Vypr, we can't do like an assembly block where we would manually set a storage slot.

So to do that, we have these kind of storage files here, I'll show you an example, where we have a variable type and then what slot it should go in.

So we can pass those as arguments to the compiler and it'll make sure those things are in those slots instead of the usually defined ones.

EVM version, so this is like Paris, Shanghai, like what functionality and basically like what hard work it should be supporting.

Optimization, so there's a few ways to optimize.

It's got this no optimize flag, I think they're gonna get rid of this.

But this optimize flag kind of lets you pass none, gas or code size, and it'll go down different branches based on those.

Okay, I don't need to do a toggle and debug.

So at the end of most Vypr compiled bytecode, there's like a chunk at the end that basically specifies like the compiler version and a few other things that aren't used by the program.

This is just a reporting, it's just reporting.

JSON, this is like just an output format to kind of output all of the different formats together in 1 JSON file instead of choosing them.

If we want to print the intermediate representation, do we want the numbers in hexadecimal versus normal decimal?

So those are the things we can input to the compiler.

And then here we're just kind of grabbing all those and putting them into the right variables and kind of getting them organized so that we can call compile files.

I mean this is all just not super specific to Viper but we're grabbing the source code, if there's storage layouts we're grabbing those storage layouts and saving them as a dictionary.

We're setting the formats that we want, a few that kind of had old names that we want to translate to new names.

And then finally, we're calling this Viper.compile code.

So this is like the main entry point for actually doing the work in the compiler.

So everything here was just kind of gathering up the right arguments so that we can call this function.

And okay, let's just jump straight in there, actually.

This will actually be in, let's go in this way.

So if you're kind of starting to explore this, this is the place to start.

And we can see we're passing the like strings of the source code of what's being compiled, a list of all the output formats.

That's just for handling exceptions, Code for the interfaces that we're importing that we want to make sure we're implementing.

Settings is like optimization version, EBM version, storage layouts if we want to show the gas estimates and if we want to put the metadata.

So if we come down here, we can see we're just kind of like iterating over the different contracts to compile.

And then we're creating this compiler data object.

So if we check that out, we can see that this is a class that holds all those basic same things we spoke about, but this has properties that basically define how to get all the different steps of the process.

So we can see there's a generateAST property, or a function that is cached as a property, that does all the work to make the AST.

AST being like the abstract syntax tree of how the program runs and then returns it.

Same thing with creating the let's see intermediate representation right and returning that.

So this is kind of where all the work is done in this class.

And then basically once we have that and it's done the work, we're going to go through the output formats that we defined And then we're going to call Like that formatter is what we want.

So if we look in this output formats Dictionary we can see that like for each string of an output format that we could have defined, there's a function here that defines what to return for it.

So if we jump back, we're basically saying like compile the code and then, or not compile the code, Create the class with all the data for the compiler and then for each format we want like go through and do all the work and return it.

So if we go in here these Functions are the functions that define everything we want to get back.

So the default is bytecode That's like if we don't say anything, we're just getting back to bytecode And we can trace back to say like how do we get the bytecode in order to see what steps are actually done?

First thing is it's calling compiler data dot bytecode.

And then a Python cache property is basically, it's a function that we can call like it's a property.

So we can see, okay, that's the last step of the process that we're gonna wanna understand.

Well, it's calling self.assembly, another cache property.

And if we go back and say, okay, that's calling generate assembly using the IR nodes as intermediate representation.

And if we look at IR nodes, that's calling IR output.

Okay, that's calling this generate IR nodes function.

And if we jump back, that takes in the folded module, folded module takes in this folded module, which is calling this generate folded AST.

And if we go back from there, we can see, okay, well, how is that, how is the normal AST generated?

So, we kind of just zoom past, backwards through the whole thing.

But what we can see is basically starting from the source code, we're going to do this function, which is generating the abstract syntax tree and setting some optimization stuff.

Then once we have that AST, that will basically be passed into generate folded AST that's going to do some work on unfolding which is like compile time up like stuff that can be done to save runtime work we'll go into that more detail soon So then once we have that folded 1, we have a call that's passed to generate IR nodes.

That's gonna give us intermediate representation.

Then that's passed to generate assembly, which is gonna give us like the pure, like assembly, like in words, but a few things not quite in bytecode.

And then we're gonna generate the bytecode with that.

And finally, this output is just gonna pass it to us as a hex string.

So that's kind of the zoomed out overview.

These 3 articles on the first day I read were helpful for me in kind of getting my head around it in a really zoomed out sense.

So I'd recommend like starting off, just reading this obviously, like we wanna get into a ton more detail with a lot of these.

This one's particularly very good except it kind of skips a lot of the meaty stuff at the end so it goes a great level of detail on the abstract syntax tree and folding but then kind of just jumps to like And then we turn that into IR and optimize it and make it my code.

So I hope he makes another 1 that goes deeper here But this is a good overview to get started and then this 1 was was very very good as well as like a deeper introduction and understanding of the language and just like a more readable, intuitive way to understand a lot of what's in the docs.

And then this was was great as well, just on like a pure like how does the language work level.

So reading those 3 you kind of understand the flow of what I just explained in terms of the different steps.

But what I think is useful for this video is just to dive into all of those steps individually.

So let's start with, and I'm going to just peek back and make sure I'm not missing things.

I've got on my wall here my Big overview of everything that kept me grounded through a lot of this.

So yeah, so perfect First thing we want to do is generate AST Find it there.

Okay, great So if we started this generate AST, what this whole chunk is trying to do is it's trying to take in the source code and it's going to turn it into an abstract syntax tree, which is basically a nested set of, I'm gonna butcher this, but I'll explain it the way I understand it, nested set of like operations or values that explain the logic of what all the code is doing, like completely, but in a way that is easier to kind of like reason about or manipulate or play with later or for the compiler to understand.

The way that Vyprr does this is really clever.

Python has a module that does abstract syntax tree parsing.

And Vypr obviously has a lot of overlap with Python.

And so what is done is there's a little bit of work done up front to change some of the Vypr code to make it align more with Python code.

There are some, some like annotations saved in terms of like which of those changes were made.

Then we just use the Python AST, AST parser to parse the code from there and then we change some things back based on what we annotated before so it's kind of like Saving some of the state use the python parser comment after and tweak it which is very clever and Means that this code itself doesn't have any risk in how the tree is parsed, assuming you think Python's is fine.

So if we go into that, we can see in this phase is they called to byast.parse to ast with settings.

So this is, we are now in this AST folder.

So if you're looking at the Viper repo, you'll see that's 1 of the folders.

This whole folder is everything that's involved with going from source code to that abstract syntax tree.

So first we're calling this pre-parse and if we jump in here, I'll show you what it does it's basically going through the contract and making sure that a lot of the The stuff like the pragma line and stuff are all Set properly.

So so if we check this out For okay in in the comments, so that's where it's looking for the pragma line.

Did we try to optimize or like set the optimization level?

If any of these are set twice it's going to error.

But the other thing it's going to do is call up to validate that version pragma.

So you can see here It's going to go and basically use this this like specifier set that is kind of from Python not from from Viper specifically and it's going to set up based on the version that was entered there and then it's going to check the version that is actually being run against that version to make sure it fits.

I played with some ideas for issues here because 1 of the quirks of this specifier set is that if we get it to pass an empty string then everything will pass.

So I was trying to figure out if there was an issue there, but it looks like it's well covered to make sure that it can't be 0 and it's stripped beforehand.

The other thing is it is making sure we, so the way that Vypr does that kind of like translation before is it takes log statements and re kind of like constitutes them as a yield because yield isn't used and then it takes interfaces structs enums and events and puts them all as classes.

Because that fits the logic of what is going to be parsed.

So first it's making sure we don't actually use those, because then they would accidentally get carried through.

And then it's going to translate all those things over and make this modification offset struct that holds where the values that were shifted were before and what type they were so we know when to shift back.

So once it's pre-parsed then we're gonna just do this Python AST dot parse And I'll show you like Python AST here is up here.

And this is like straight from that library.

So that's, we're going to assume that works fine.

So It'll be passed as none when when the main thing is being passed but occasionally like in some of the built-ins we call back and recompile Like little chunks of code in order to know which intermediate representation to stick in and those built-ins And we don't want that to do a lot of the analysis as if it's the top level.

Because at the top level, we're doing a lot of the semantic analysis to make sure there's no variables declared twice and constants are declared the right way and all this stuff that doesn't apply to a little snippet of code that's being plugged into a function.

So if this is the case, it kind of wraps it in something so that it's not misinterpreted as the top level.

So now we're gonna go back and do that annotation.

Let's see how much detail we want to go into here.

Basically, we're going through and adding in where in the source code it is.

A lot of this stuff is useful when there's an error to sit to highlight and show where it was, but we also want to be annotating a lot of the stuff around what the previous type was and that kind of thing.

I didn't spend as much time on that annotation, so I don't have quite the level of detail.

And then the last thing here is to actually translate it into a VyprAST.

So if we run through here, we can see this is kind of a like recursive thing where for each Type it's going to call back in and and kind of parse it a little bit further So we can see like implements is getting replaced with assignments, same with variable declarations.

It's just kind of going through and like putting it into the right language to work as a Viper AST for the rest of the process rather than the Python 1 that it currently is.

This actually may be where things are shifted back into like events or structs as opposed to classes.

I'm honestly, this was like the very beginning of what I worked on.

So it's probably 4 weeks ago now I don't have quite the memory But it's also just making sure here.

There's some types that are Not supported in Viper, but I could have like squeezed through and what they don't want is for you to write like disallowed Viper had it parsed properly by this Python AST and then somehow sneak through and do something destructive so that's just kind of checking for a lot of those.

Cool okay so it's gonna return that So after all of that is done, we now have this AST.

And what we can actually do here, I'll show an example.

So if we do F AST, this was beautiful, that kind of argument-F that lets you return any type, it's really easy to kind of inspect these things partway through.

So I had this kind of minimal 1 set up and we can see it's just returning these like huge, like basically dictionaries of nodes with types and then kind of sub nodes all the way down.

It's going to do this check on the EVM version.

This is actually, I found an issue here where the EVM version, if it's specified in the file, actually doesn't, or no sorry, in the argument, doesn't actually get passed through because of the way that context is set up globally.

It's like too late for it, but they're trying to check here for making sure that the EVM versions match, the optimization levels match.

Basically, if you set something in a contract and something in the CLI argument, making sure those are equal or otherwise it reverts.

And then if no 1 set any optimization levels ever, it sets this default, which is it defaults to gas.

Cool, okay, so we have an AST, but it's unfolded, right?

So the next step of this process is generate folded AST.

So we're going to pass in Viper module, which we can see here is just the AST that's returned, and we're going to generate this folded AST.

So this does a So This does a few different things it's going to first do some Let's see Yeah some validation And then it's gonna call this fold where all the magic happens.

And folding is basically saying like, okay, if we declare a constant, let's just plug it in and all the places there's not more stuff to do at runtime.

Let's do some literal operations, take care of them now.

Basically, what's all the stuff that we can do right now at compile time that we know is rock solid so that we don't have to add more runtime work later because there's gas cost later and there's not gas cost now.

So this is kind of like validation all the way down through the AST.

We can see, so it's grabbing all the descendants.

That's like everything down the tree and it's calling validate on each of them if it has this validate attribute.

So let's see, what's the easiest way to grab that?

So like, if they're numbers, make sure that they fall within the bounds of what they should.

If they are decimals, make sure they fall within the decimal bounds.

If it's a hexadecimal, make sure there's no underscores and that the length is multiple of 2.

You know, these are like very basic things to catch early.

There's a lot more validation that happens later, but this is like the initial stuff, no empty tuples.

This is 1 of the challenges with this stuff is, do you see that?

Cool, so actually you can see I wrote myself notes here of all the stuff it does.

So insert within the bounds, decimals, hex, even number of characters, no underscores, ASCII strings only under 256, no empty tuples.

So now we're going to copy the whole module over.

The reason we copy it is just that we want to, if someone has entered that they want the unfolded AST as 1 of the things to return, we don't want to change that in place.

So we make a copy, and then we call this fold.

And we can see I took notes for myself here, but actually let's just go into it.

So this is going to go through a number of things, and we'll see this same kind of pattern in the optimizer, where There's a number of specific things we can do and we're going to just kind of go through them and until we can't go through them anymore So first is to replace the built-in constants, so We can see built-in constants here are like these different strings that you could put right into the source code and it's going to go through each of them and just basically say replace constant it's like just like sub it out for what it should be subbed out for Then we are going to go through these 4 things and we're going to do them in a way where it's like we do each of the 4, if anything changed, then we're going to go back to the top and do the 4 again.

And basically that's a way that like sometimes some of these things unlock another 1, we don't wanna just do 1 pass.

So we're gonna just keep going until there's, there's no, nothing changed, and then we're ready to move on.

So the first thing we'll do is replace user defined constants.

If we've got something here, that's like x is a constant, u and 256 is 6, and then down here in a function, so like function, if we said like, return x.

Okay, so in order to make this simpler it's going to just go and take 6 and plug it right in there when it compiles So like the constant basically disappears and the number gets added here.

This is actually another issue, or it's a small issue, that I found that when it does that replacement it skips any tuple constants, which I don't know if they're even supposed to exist, but it skips doing anything and then it deletes it.

Weird thing with compilers is like things that cause errors at compiler time, that compile time are not nearly as bad as things that squeeze through.

So I delete this constant and then I can't find it later.

So it's gonna go through the whole tree and look for any things that are like addition, subtraction, multiplication, whatever, or not, inversion, you can see the list here, comparisons, and it's going to look at both sides are literals, if they're like numbers that were plugged into the source code as opposed to variables, and that includes constants, right, because we've replaced them now with literals, it's going to actually do the operation and put the answer back.

So we can see if we if we go in here, it's right here, it's going to go through and just replace those straight in the tree.

So if we call evaluate, we can see, let's not tell the function that's on the nodes.

So for example, here, let's find an easier 1, a binary operation.

So it's going to say, left and right, they have to be the same type.

If it's not an integer or decimal, then that's not gonna work.

And this unfoldable node, it doesn't actually raise an error.

So that's basically saying it can't be folded, just skip it and don't worry about it.

Yeah, so this is like grabbing what the operation is and then doing that operation on the left or right things, validating that the numbers and bounds and then it's just going to replace it.

I want to say like the type of operation like add is an example of that whatever.

So yeah it's going through and like I said, unfoldable note, it'll catch that and just continue.

So we're replacing all those in the tree.

So subscripts being like going into the index of a list or I guess that's it.

It's going through any of these that have subscripts.

So like an array with an index at the third place, it's going to check and say like, is that a literal list that we know the list already?

Let's just grab that and replace it in the tree as well.

And then the last thing is replacing built-in functions.

So there's this file, which is all the functions that are like built into Python.

So you can actually see a list of them here.

Or if we go to Chrome, we can just say like, Viper built in, I'm sure I've been to this page way too many times.

So this is all of them, the like built in stuff.

Some of that is like, like slices or more ship it shifting hashes.

Some of them are like ways to deploy contracts converting between types, that kind of thing.

So this is like, a lot of interesting stuff here.

It's where many of the issues I found were, because it's kind of, I don't know, it's interesting to look at, because it's kind of like writing source code in a way.

So if you find issues in it or edge cases, those can be problems.

So some of these, let's find an example of 1.

See all Zach Obront transcripts on Youtube

How Does the Vyper Compiler Work? (Technical Walkthrough)