With software becoming more important than ever, hardware is following suit.
As the world generates more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware. 
But how much does this all really cost? In this final segment of our AI hardware series, we tackle that question head on. 

Be sure to check part 1 and 2, where we explore the emerging architectures and the momentous competition for AI hardware.

There's very few computational problems that complex that mankind has actually undertaken.

How do you think about the relationship between compute, capital, and then the technology that we have today?

Yeah, that's the million dollar question or maybe trillion dollar question.

The expectation at the moment is that the cost

of training these models may actually sort of top out or even go down

a little bit as the chips get faster, but we don't discover new training material as quickly.

With software becoming more important than ever, hardware is following suit.

And with the world constantly generating more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware.

But how much does all of this really cost?

In this final segment of our AI hardware series, we tackle that question head on.

But if you're just catching up, be sure to check out part 1 and part 2, where we explored the emerging architectures and the momentous competition for AI hardware.

And today we're joined again by A16Z special advisor, Guido Appenzeller, someone who is uniquely suited for this deep dive as a storied infrastructure expert with experience like.

Intel's data center group dealing a lot with hardware and the low level components.

So it's given me sort of, I think, a good insight how large data centers work, what the basic components are that make all of this AI boom possible today.

Here is Guido touching on the reality of these models and how much they cost today.

Training 1 of these large language models today, it's not a $100, 000 thing, it's probably millions of dollars thing.

Practically speaking, what we're seeing in industry is that it's actually more for tens of millions of dollars.

As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16z fund.

Please note that A16z and its affiliates may also maintain investments in the

For more details, including a link to our investments, please see a16z.com slash disclosures.

In Guido's recent article, Navigating the High Cost of AI Compute, Guido even noted that access to compute resources has become a determining factor for the success of AI companies.

And this is not just true for the largest companies building the largest models.

In fact, many companies are spending more than 80% of their total capital raised on compute resources.

So naturally, This begs the question, is this really sustainable?

The core technology that you're building in the early days towards more a complete product offering, right?

There's just a lot more boxes to check and features to implement and all the administrative parts of the application if you're getting to the enterprise.

So probably you'll have more normal software development that's not AI, right?

Or classic software development happening.

You'll probably also have a larger headcount of people that they have to pay.

So at the end of the day, I would expect as a percentage that they'll go down over time, right?

As an absolute amount, I think it'll be going up for some time just because this AI boom is still just in its infancy.

The AI boom has just begun and in part 2 we discussed how it's unlikely for compute demand to subside anytime soon.

There we also discussed how the decision to own or rent infrastructure can make a non-trivial difference to a company's bottom line.

But there are other considerations when it comes to cost.

Batch size, learning rate, and the duration of the training process all contribute to the final price tag.

How much does it cost to train a model depends on a myriad of factors, right?

Now the good news is we can simplify this a little bit because the vast majority of models that are being used today are transformer models, right?

They've proven to be incredibly versatile.

They're easier to train because they parallelize a little bit better than previous models.

And so in a transformer, you can sort of approximate the inference time as twice the number of parameters, floating point operations, right?

And the training time is about 6 times the number of parameters.

So if you take something like GPT-3, which is OpenAI's big model, they have 175 billion parameters.

So you need twice as much, so 350 billion floating point operations to do 1 inference.

And so based on that, you can sort of figure out how much compute capacity you need, how this is going to scale, how you should price it, you know, how much it will cost you at the end of the day.

This also gives you for model training, an idea of how long the training is going to take.

You know how much your AI accelerator can do in terms of floating point operations per second.

You can sort of theoretically calculate how many operations it is to train your model.

In practice, the math is more complicated because There are certain ways to accelerate that, so maybe you can train with a reduced precision.

But it's also very hard to achieve 100% utilization on these cards.

If you naively implement it, you can probably go below 10% utilization.

But you know, you can probably get into the tens of percent with a little bit of work.

This gives you a rough swag, how much capacity you need for training and for inference, but at the end you probably do want to test it before you make any final decisions on these things.

Make sure that your assumptions hold on how much you need.

Now if all those numbers confused you, that's okay.

We'll walk through a very specific example.

And here's Guido on the computation requirements for training the model and ultimately inference.

That's when you're prompting the already trained model to elicit a response.

So if you just do very naively the math, right, let's start with training, right, we know how many tokens it was trained on, we know how many parameters the model has, so we can do soft napkin math, and you end up with something like 3 times 10 to the 23 floating point operations.

It's like a number with 23 digits, right?

There's very few computational problems that complex that mankind has actually undertaken, right?

Then you can be like, okay, So let's take, say, an A100, the most commonly used card.

We know how many floating point operations it can do per second.

That gives us an order of magnitude estimation, like how much time it would take.

And then we know how much 1 of these cards costs.

Like renting an A100 costs you between, I want to say, between $1 and $4 probably, depending on who you rent it from.

And you end up with something in the order of half a million dollars, with this very naive analysis.

Now, there's a couple of things there, right?

We didn't take into account optimization.

We also didn't take into account that you probably cannot run this at full capacity because of memory bandwidth limitations and network limitations.

And last but not least, you probably need more than 1 run to get this right.

They're probably not going to be full runs and so on.

But this gives you an idea that sort of training 1 of these large language models today, it's not a $100, 000 thing, it's probably millions of dollars thing.

Practically speaking, what we're seeing in industry is that it's actually more for tens of millions of dollars thing.

And that's because you need the reserve capacity, right?

So if I could get all my cards for the next 2 months would only cost me a million dollars, but the problem is they want a 2 year reservation.

And so that basically adds a 0 to my training cost.

Right, and how does that compare to inference?

So inference is much, much, much cheaper.

Basically, my training set for modern text model, for example, the training set is about a trillion tokens, right?

And if I run inference, each word that comes out is 1 token, right?

So it's a factor of a trillion or so faster on the inference part.

If you run the numbers like a large language model, you actually add a fraction of a cent, like a 10th of a cent or a hundredth of a cent somewhere in that ballpark for the inference.

Again, if we just naively look at this, right, for inference, your problem is usually that you have to provision for peak capacity, right?

So if everybody wants to use your model at 9am on a Monday, right, you still have to pay for Saturday night at midnight when nobody's using it.

That increases your cost substantially there.

For some of them, specifically image models, what you can do for inference is that you use much, much cheaper cards because the model is small enough that you can run it on essentially the server version of a consumer graphics card and that can save you a lot of cost.

And unfortunately, as we discussed in part 1, you can't just make up for these inefficiencies by piecing together a bunch of less performant chips, at least for model training.

You need some very sophisticated software for that, right?

Because the overhead of distributing the data between these cards would probably outweigh any saving you'd get from cheaper cards.

For inference, you can often do the inference on a single card, so that's not really a problem.

If you take something like stable diffusion, a very popular model for image generation, that runs on a MacBook, for example, that has enough memory and enough compute power so you can generate an image locally.

So that will run on a relatively cheap consumer card and you don't have to use an A100 for it to do inference.

So when we're talking about the training of

the models, clearly the amount of compute is just drastically more than the inference.

And something else that we've already talked about is the more compute, often, not always, but often the better model.

And so does this ultimately, these factors all ladder up to the idea that heavily capitalized incumbents win this race or this competition?

Or how do you think about the relationship between compute, capital, and then the technology that we have today?

Yeah, that's the million dollar question or maybe trillion dollar question, I don't know.

So first of all, training these models is expensive, right?

For example, we haven't seen yet a really good open source large language model.

And I'm sure part of the reason is that training these models is just really, really expensive.

I mean, there's a bunch of enthusiasts out there that would love to do this, but you need to find a couple of million or $10 million of compute capacity to do it.

It means you need to create a substantial effort before something like that can happen.

All that said, the cost for training these models overall seems to be coming down.

And in part, I think it is because it seems to me like we're becoming data limited, right?

So it turns out there is a correspondence between how big your model is and what the optimal amount of training data is for the model.

Having a super large model with very few data doesn't get you anything, or having a ton of data with a small model also doesn't get you anything.

The size of your brain needs to roughly correspond to the length of your university education here, right?

And what this means is that because some of the large models today already leverage a good percentage of all human knowledge in a particular area.

I mean, if you look at GPT, there was probably trained on something like 10% of the internet, right?

And all of Wikipedia and many books, like a good chunk of all books, right?

So going up by a factor of 10, yeah, that's quite possible.

Going up by a factor of 100, that's not clear if that's possible.

I mean, we as mankind just haven't produced enough knowledge yet that you could absorb all of that into 1 of these large models.

And so I think the expectation at the moment is that the cost of training these models may actually

But we don't discover new training material as quickly.

I mean, unless somebody comes up with a new idea of the general training material.

And so if that assumption is true, I think this means that the moat that's created by these large capital investments is actually not particularly deep, right?

It's more of a speed bump than something that prevents new entrants.

I mean, today, training a large language model is something that is definitely within reach for a well-funded startup, right?

So, and for that reason, we expect to see more innovation in that area in the future.

All right, that is a wrap for our AI hardware series.

We genuinely hope you came away with a little more knowledge about this increasingly important space.

Because if software is indeed eating the world, well, hardware is coming along for that ride.

And as a reminder, if you haven't yet listened to part 1, where we explore the emerging architectures and who's creating them, or part 2, where we dive into the future AI stack and how founders can participate, well, those are already live and ready for consumption.

And as always, thank you so much for listening.

We'd actually like to leave you with a fun fact from GPT-4 itself, commenting on the technology that created it.

And yes, we did fact check this, and this is also AI generated audio from 11 Labs.

ChatGPT and its sibling models are trained on diverse internet text.

However, the exact amount of data used can be hard to comprehend.

If we were to print all of the data used to train these models, it could fill a large library.

Consider that 1 single book may contain around 1000000 characters.

If we estimate that the training data is hundreds of gigabytes of text data, let's take a conservative estimate and say it's 100 gigabytes.

is approximately 1 byte, this would mean the model was trained on approximately

If each book has 1 million characters, then the data used to train chat GPT is equivalent to the text in approximately 100 million books.

of a large library, such as the New York Public Library, which has around 53 million items, not just books, the training data is equivalent to the text in almost twice the number of items in that library.

A quick note to close out that many models today are even bigger, with Llama2, for example, being trained on 2 trillion tokens or about 8 trillion characters.

Thank you so much for listening to our full AI hardware series.

We spent a ton of time trying to get these episodes right.

So if you did enjoy them, go ahead and leave a review or tell a friend.

And you can also look forward to a video animated version of these up on our YouTube channel soon.

But for now, you can find some of our recent videos there, like my conversation with Waymo's chief product officer, Inna Waymo, or a conversation I had at the Aspen Ideas Festival, where we discussed Classroom

As always, thank you so much for listening.


Speaker 0: There's very few computational problems that complex that mankind has actually undertaken.

Speaker 1: How do you think about the relationship between compute, capital, and then the technology that we have today?

Speaker 0: Yeah, that's the million dollar question or maybe trillion dollar question. The expectation at the moment is that the cost

Speaker 2: of training these models may actually sort of top out or even go down

Speaker 0: a little bit as the chips get faster, but we don't discover new training material as quickly.

Speaker 3: With software becoming more important than ever, hardware is following suit. And with the world constantly generating more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware. But how much does all of this really cost? In this final segment of our AI hardware series, we tackle that question head on. But if you're just catching up, be sure to check out part 1 and part 2, where we explored the emerging architectures and the momentous competition for AI hardware.

And today we're joined again by A16Z special advisor, Guido Appenzeller, someone who is uniquely suited for this deep dive as a storied infrastructure expert with experience like.

Speaker 0: Intel's data center group dealing a lot with hardware and the low level components. So it's given me sort of, I think, a good insight how large data centers work, what the basic components are that make all of this AI boom possible today.

Speaker 3: Here is Guido touching on the reality of these models and how much they cost today.

Speaker 0: Training 1 of these large language models today, it's not a $100, 000 thing, it's probably millions of dollars thing. Practically speaking, what we're seeing in industry is that it's actually more for tens of millions of dollars.

Speaker 2: As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16z fund. Please note that A16z and its affiliates may also maintain investments in the

Speaker 3: companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com slash disclosures. In Guido's recent article, Navigating the High Cost of AI Compute, Guido even noted that access to compute resources has become a determining factor for the success of AI companies. And this is not just true for the largest companies building the largest models. In fact, many companies are spending more than 80% of their total capital raised on compute resources.

So naturally, This begs the question, is this really sustainable?

Speaker 0: The core technology that you're building in the early days towards more a complete product offering, right? There's just a lot more boxes to check and features to implement and all the administrative parts of the application if you're getting to the enterprise. So probably you'll have more normal software development that's not AI, right? Or classic software development happening. You'll probably also have a larger headcount of people that they have to pay.

So at the end of the day, I would expect as a percentage that they'll go down over time, right? As an absolute amount, I think it'll be going up for some time just because this AI boom is still just in its infancy.

Speaker 3: The AI boom has just begun and in part 2 we discussed how it's unlikely for compute demand to subside anytime soon. There we also discussed how the decision to own or rent infrastructure can make a non-trivial difference to a company's bottom line. But there are other considerations when it comes to cost. Batch size, learning rate, and the duration of the training process all contribute to the final price tag.

Speaker 0: How much does it cost to train a model depends on a myriad of factors, right? Now the good news is we can simplify this a little bit because the vast majority of models that are being used today are transformer models, right? That was the transformer architecture. Huge breakthrough in AI. They've proven to be incredibly versatile.

They're easier to train because they parallelize a little bit better than previous models. And so in a transformer, you can sort of approximate the inference time as twice the number of parameters, floating point operations, right? And the training time is about 6 times the number of parameters. So if you take something like GPT-3, which is OpenAI's big model, they have 175 billion parameters. So you need twice as much, so 350 billion floating point operations to do 1 inference.

And so based on that, you can sort of figure out how much compute capacity you need, how this is going to scale, how you should price it, you know, how much it will cost you at the end of the day. This also gives you for model training, an idea of how long the training is going to take. You know how much your AI accelerator can do in terms of floating point operations per second. You can sort of theoretically calculate how many operations it is to train your model. In practice, the math is more complicated because There are certain ways to accelerate that, so maybe you can train with a reduced precision.

But it's also very hard to achieve 100% utilization on these cards. If you naively implement it, you can probably go below 10% utilization. But you know, you can probably get into the tens of percent with a little bit of work. This gives you a rough swag, how much capacity you need for training and for inference, but at the end you probably do want to test it before you make any final decisions on these things. Make sure that your assumptions hold on how much you need.

Speaker 3: Now if all those numbers confused you, that's okay. We'll walk through a very specific example. GPT-3. GPT-3 has about 175 billion parameters. And here's Guido on the computation requirements for training the model and ultimately inference.

That's when you're prompting the already trained model to elicit a response.

Speaker 0: So if you just do very naively the math, right, let's start with training, right, we know how many tokens it was trained on, we know how many parameters the model has, so we can do soft napkin math, and you end up with something like 3 times 10 to the 23 floating point operations. That's a completely crazy number, right? It's like a number with 23 digits, right? It's like hard to write down. There's very few computational problems that complex that mankind has actually undertaken, right?

And it's a huge effort. Then you can be like, okay, So let's take, say, an A100, the most commonly used card. We know how many floating point operations it can do per second. We can divide that. That gives us an order of magnitude estimation, like how much time it would take.

And then we know how much 1 of these cards costs. Like renting an A100 costs you between, I want to say, between $1 and $4 probably, depending on who you rent it from. And you end up with something in the order of half a million dollars, with this very naive analysis. Now, there's a couple of things there, right? We didn't take into account optimization.

We also didn't take into account that you probably cannot run this at full capacity because of memory bandwidth limitations and network limitations. And last but not least, you probably need more than 1 run to get this right. You probably need a bunch of test runs. They're probably not going to be full runs and so on. But this gives you an idea that sort of training 1 of these large language models today, it's not a $100, 000 thing, it's probably millions of dollars thing.

Practically speaking, what we're seeing in industry is that it's actually more for tens of millions of dollars thing. And that's because you need the reserve capacity, right? So if I could get all my cards for the next 2 months would only cost me a million dollars, but the problem is they want a 2 year reservation. So really the cost is 12 times as high. And so that basically adds a 0 to my training cost.

Speaker 1: Right, and how does that compare to inference?

Speaker 0: So inference is much, much, much cheaper. Basically, my training set for modern text model, for example, the training set is about a trillion tokens, right? And if I run inference, each word that comes out is 1 token, right? So it's a factor of a trillion or so faster on the inference part. If you run the numbers like a large language model, you actually add a fraction of a cent, like a 10th of a cent or a hundredth of a cent somewhere in that ballpark for the inference.

Again, if we just naively look at this, right, for inference, your problem is usually that you have to provision for peak capacity, right? So if everybody wants to use your model at 9am on a Monday, right, you still have to pay for Saturday night at midnight when nobody's using it. That increases your cost substantially there. For some of them, specifically image models, what you can do for inference is that you use much, much cheaper cards because the model is small enough that you can run it on essentially the server version of a consumer graphics card and that can save you a lot of cost.

Speaker 3: And unfortunately, as we discussed in part 1, you can't just make up for these inefficiencies by piecing together a bunch of less performant chips, at least for model training.

Speaker 0: You need some very sophisticated software for that, right? Because the overhead of distributing the data between these cards would probably outweigh any saving you'd get from cheaper cards.

Speaker 3: Inference, on the other hand...

Speaker 0: For inference, you can often do the inference on a single card, so that's not really a problem. If you take something like stable diffusion, a very popular model for image generation, that runs on a MacBook, for example, that has enough memory and enough compute power so you can generate an image locally. So that will run on a relatively cheap consumer card and you don't have to use an A100 for it to do inference.

Speaker 3: So when we're talking about the training of

Speaker 1: the models, clearly the amount of compute is just drastically more than the inference. And something else that we've already talked about is the more compute, often, not always, but often the better model. And so does this ultimately, these factors all ladder up to the idea that heavily capitalized incumbents win this race or this competition? Or how do you think about the relationship between compute, capital, and then the technology that we have today?

Speaker 0: Yeah, that's the million dollar question or maybe trillion dollar question, I don't know. So first of all, training these models is expensive, right? For example, we haven't seen yet a really good open source large language model. And I'm sure part of the reason is that training these models is just really, really expensive. I mean, there's a bunch of enthusiasts out there that would love to do this, but you need to find a couple of million or $10 million of compute capacity to do it.

And that makes it so much harder. It means you need to create a substantial effort before something like that can happen. All that said, the cost for training these models overall seems to be coming down. And in part, I think it is because it seems to me like we're becoming data limited, right? So it turns out there is a correspondence between how big your model is and what the optimal amount of training data is for the model.

Having a super large model with very few data doesn't get you anything, or having a ton of data with a small model also doesn't get you anything. The size of your brain needs to roughly correspond to the length of your university education here, right? Otherwise, it doesn't work. And what this means is that because some of the large models today already leverage a good percentage of all human knowledge in a particular area. I mean, if you look at GPT, there was probably trained on something like 10% of the internet, right?

And all of Wikipedia and many books, like a good chunk of all books, right? So going up by a factor of 10, yeah, that's quite possible. Going up by a factor of 100, that's not clear if that's possible. I mean, we as mankind just haven't produced enough knowledge yet that you could absorb all of that into 1 of these large models. And so I think the expectation at the moment is that the cost of training these models may actually

Speaker 2: sort of top out or even go down

Speaker 0: a little bit as the chips get faster. But we don't discover new training material as quickly. I mean, unless somebody comes up with a new idea of the general training material. And so if that assumption is true, I think this means that the moat that's created by these large capital investments is actually not particularly deep, right? It's more of a speed bump than something that prevents new entrants.

I mean, today, training a large language model is something that is definitely within reach for a well-funded startup, right? So, and for that reason, we expect to see more innovation in that area in the future.

Speaker 3: All right, that is a wrap for our AI hardware series. We genuinely hope you came away with a little more knowledge about this increasingly important space. Because if software is indeed eating the world, well, hardware is coming along for that ride. And as a reminder, if you haven't yet listened to part 1, where we explore the emerging architectures and who's creating them, or part 2, where we dive into the future AI stack and how founders can participate, well, those are already live and ready for consumption. And as always, thank you so much for listening.

We'd actually like to leave you with a fun fact from GPT-4 itself, commenting on the technology that created it. And yes, we did fact check this, and this is also AI generated audio from 11 Labs. We'll see you next time.

Speaker 4: ChatGPT and its sibling models are trained on diverse internet text. However, the exact amount of data used can be hard to comprehend. If we were to print all of the data used to train these models, it could fill a large library.

Speaker 5: Consider that 1 single book may contain around 1000000 characters. If we estimate that the training data is hundreds of gigabytes of text data, let's take a conservative estimate and say it's 100 gigabytes. Considering that 1 character

Speaker 6: is approximately 1 byte, this would mean the model was trained on approximately

Speaker 0: 100

Speaker 6: billion characters. If each book has 1 million characters, then the data used to train chat GPT is equivalent to the text in approximately 100 million books. If we take the size

Speaker 2: of a large library, such as the New York Public Library, which has around 53 million items, not just books, the training data is equivalent to the text in almost twice the number of items in that library.

Speaker 3: Thanks, ChachiBT. A quick note to close out that many models today are even bigger, with Llama2, for example, being trained on 2 trillion tokens or about 8 trillion characters. Now that is a lot of libraries. Thank you so much for listening to our full AI hardware series. We spent a ton of time trying to get these episodes right.

So if you did enjoy them, go ahead and leave a review or tell a friend. We would so appreciate that. And you can also look forward to a video animated version of these up on our YouTube channel soon. But for now, you can find some of our recent videos there, like my conversation with Waymo's chief product officer, Inna Waymo, or a conversation I had at the Aspen Ideas Festival, where we discussed Classroom

Speaker 0: 2050.

Speaker 3: As always, thank you so much for listening.

See all a16z Podcast transcripts on Applepodcasts

The True Cost of Compute