Here is my conversation with Dario Amodei, CEO of Anthropic.

We discuss:
 - why human level AI is 2-3 years away
 - race dynamics with OpenAI and China
 - $10 billion training runs, bioterrorism, alignment, cyberattacks, scaling, ...

Dario is hilarious and has fascinating takes on what these models are doing, why they scale so well, and what it will take to align them.

Transcript: https://www.dwarkeshpatel.com/dario-amodei
Apple Podcasts: https://apple.co/3rZOzPA
Spotify: https://spoti.fi/3QwMXXU

Follow me on Twitter: https://twitter.com/dwarkesh_sp

---
I’m running an experiment on this episode.

I’m not doing an ad. 

Instead, I’m just going to ask you to pay for whatever value you feel you personally got out of this conversation.

Pay here: https://bit.ly/3ONINtp
---

(00:00:00) - Introduction
(00:01:00) - Scaling
(00:15:46) - Language
(00:22:58) - Economic Usefulness
(00:38:05) - Bioterrorism
(00:43:35) - Cybersecurity
(00:47:19) - Alignment & mechanistic interpretability
(00:57:43) - Does alignment research require scale?
(01:05:30) - Misuse vs misalignment
(01:09:06) - What if AI goes well?
(01:11:05) - China
(01:15:11) - How to think about alignment
(01:31:31) - Is modern security good enough?
(01:36:09) - Inefficiencies in training
(01:45:53) - Anthropic’s Long Term Benefit Trust
(01:51:18) - Is Claude conscious?
(01:56:14) - Keeping a low profile

That could happen in, you know, 2 or 3 years.

What does that imply for Anthropic when in 2 to 3 years, these Leviathans are doing like $10 billion training runs?

I listened to this and I became enlightened.

The compute doesn't flow, like the spice doesn't flow.

It's like, you can't like, like the blob has to be unencumbered, right?

The big acceleration that happened late last year and beginning of this year, we didn't cause that.

And honestly, I think if you look at the reaction to Google, that that might be 10 times more important than anything else.

There was a running joke, the way building AGI would look like is there would be a data center next to a nuclear power plant, next to a bunker.

Okay, today I have the pleasure of speaking with Dario Amodei, who is the CEO of Ventropic.

Dario, thank you so much for coming on the podcast.

First question, you have been 1 of the very few people who has seen scaling coming for years, more than 5 years.

But as somebody who's seen it coming, what is fundamentally the explanation for why scaling works?

Why is the universe organized such that if you throw big blobs and compute at a wide enough distribution of data, the thing becomes intelligent?

I think the truth is that we still don't know.

I think it's almost entirely an empirical fact.

You know, I think it's a fact that you could kind of sense from the data and from a bunch of different places.

But I think we don't still have a satisfying explanation for it.

If I were to try to make 1, but I'm just, I don't know, I'm just kind of waving my hands when I say this, you know, there's this, there's these ideas in physics around like long tail or power law of like correlations or effects.

And so like when a bunch of stuff happens, right?

When you have a bunch of like features, you get a lot of the data in like kind of the early, you know, the fat part of the distribution before the tails.

For language, this would be things like, oh, I figured out there are parts of speech and nouns follow verbs.

And then there are these more and more and more and more subtle correlations.

And so it kind of makes sense why there would be this, every log or order of magnitude that you add, you kind of capture more of the distribution.

What's not clear at all is, why does it scale so smoothly with parameters?

Why does it scale so smoothly with the amount of data?

Why are you can think up some explanations of why it's linear, like the parameters are like a bucket.

And so size of the bucket is proportional to size of the water.

But like, why does it lead to all these, this very smooth scaling?

Our chief scientist, Garrett, Jared Kaplan did some stuff on like fractal manifold dimension that like you can use to explain it.

So there's, there's all kinds of ideas, but I feel like we just don't really know.

And by the way, for the audience who's trying to follow along, by scaling, we're referring to the fact that you can very predictably see how if you go from GPT-3 to GPT-4, or in this case, CLAW-1 to CLAW-2, that the loss in terms of whether it can predict the next token scales very smoothly.

So, okay, we don't know why it's happening, but can you at least predict if empirically, here is the loss at which this ability will emerge, here is the place where this circuit will emerge.

Is that at all predictable or are you just looking at the loss number?

What's predictable is this statistical average, this loss, this entropy.

It's like predictable to like sometimes even to several significant figures, which you don't see outside of physics.

You don't expect to see it in this messy empirical field.

But actually, specific abilities are very hard to predict.

So, you know, back when I was working on GPT-2 and GPT-3, like, when does arithmetic come in place?

You know, it's kind of like you can predict statistical averages of the weather, but the weather on 1 particular day is very, you know, very, very hard to predict.

I don't understand manifolds, but mechanistically, it doesn't know addition yet.

This is another question that we don't know the answer to.

I mean, we're trying to answer this with things like mechanistic interpretability.

I mean, you can think about these things about like circuits snapping into place, although there is some evidence that when you look at the models being able to add things, that, you know, like, if you look at its chance of getting the right answer, that shoots up all of a sudden.

But if you look at, okay, what's the probability of the right answer, you'll see it climb from like 1 in a million to 1 in a hundred thousand to 1 in a thousand long before it actually gets the right answer.

And so there's some, in many of these cases, at least, I don't know if in all of them, there's some continuous process going on behind the scenes.

Does that imply that the circuit or the process for doing addition was pre-existing and it just got increased in salient?

I don't know if like there's this circuit that's weak and getting stronger.

I don't know if it's something that works but not very well.

Like I think we don't know and these are some of the questions we're trying to answer with mechanistic interpretability.

Are there abilities that won't emerge with scale?

So I definitely think that, again, like things like alignment and values are not guaranteed to emerge with scale, right?

It's kind of like, you know, 1 way to think about it is you train the model and it is basically it's like predicting the world, it's understanding the world, it's job is facts, not values, right?

But there's free variables here where it's like, what should you do?

Those, you know, like, they're just, there aren't the bits for that.

There's just like, well, if I started with this, I should finish with this.

If I started with this other thing, I should finish with this other thing.

And so I think that's not going to emerge.

to talk about a lemon in a second, but on scaling, if it turns out that scaling plateaus before we reach human level intelligence, looking back on it, what would be your explanation?

What do you think is likely to be the case if that turns out to be the outcome?

Yeah, so I guess I would distinguish some problem with the fundamental theory with some practical issue.

So 1 practical issue we could have is we could run out of data.

For various reasons, I think that's not going to happen.

But if you look at it very, very naively, we're not that far from running out of data.

And so it's like, we just don't have the data to continue the scaling curves.

I think, you know, another way it could happen is like, oh, we just use up all of our compute that was available and that wasn't enough, and then progress is slow after that.

I wouldn't bet on either of those things happening, but they could.

I think from a fundamental perspective, I personally, I think it's very unlikely that the scaling laws will just stop.

If they do, another reason, again, this isn't fully fundamental, could just be we don't have quite the right architecture.

Like if we tried to do it with an LSDM or an RNN, the slope would be different.

I still might be that we get there, but I think there are some things that are just very hard to represent when you don't have this ability to attend far in the past that transformers have.

If somehow, and I don't know how we would know this, it kind of wasn't about the architecture and we just hit a wall, I think I'd be very surprised by that.

I think we're already at the point where the things the models can't do don't seem to me to be different in kind from the things they can do.

And it just, you know, you could have made a case a few years ago that it was like they can't reason, they can't program, like you could have, you could have drawn boundaries and said, well, maybe you'll hit a wall.

Few other people didn't think we would hit a wall, but it was a more plausible case then.

Like, it could happen tomorrow that it's just like we hit a wall.

I think if that happens, I'm trying to think of what's my, what would really be my, it's unlikely, but what would really be my explanation.

I think my explanation would be, there's something wrong with the loss when you train on next word prediction.

Like, some of the remaining reasoning abilities or something like that, like if you really want to learn, it's a program at a really high level, it means you care about some tokens much more than others.

And they're rare enough that it's like the loss function over focuses on kind of the appearance, the things that are responsible for the most bits of entropy.

And instead, you know, they don't focus on this stuff that's really essential.

And so you could kind of have the signal drowned out in the noise.

I don't think it's going to play out that way for a number of reasons.

But if you told me, yep, you trained your 2024 model, it was much bigger and it just wasn't any better, and you tried every architecture and it didn't work.

I think that's the explanation I would reach for.

Is there a candidate for another loss function if you had to abandon next token prediction?

I think then you would have to go for some kind of RL.

There's things like amplification and debate, right?

These are kind of both alignment methods and ways of training models.

You would have to try a bunch of things, but the focus would have to be on what do we actually care about the model doing, right?

In a sense, we're a little bit lucky that it's like predict the next word gets us all these other things we need.

from your worldview, there's a multitude of different loss functions that it's just a matter of what can allow you to just throw a whole bunch of data at it.

Like the next token prediction itself is not significant.

Yeah, well, I mean, I guess the thing with RL is you get slowed down a bit because it's like, you know, you have to, by some method, kind of, you know, design how the loss function works.

Nice thing with the next token prediction is it's there for you, right?

And so I think it would slow you down if you couldn't scale in just that very simplest way.

You mentioned that the data is likely not to be the constraint.

And, you know, for a number of reasons, I shouldn't go into the details.

But, you know, like there's many sources of data in the world And there's many ways that you can also generate data.

My guess is that this will not be a blocker.

Maybe it'd be better if it was, but it won't be.

There's just many different ways to do it.

And then you would be basically saying something similar to this.

This view that I have probably formed gradually from, I would say like 2014 to 2017.

So I think my first experience with it was my first experience with AI.

So I, you know, I saw some of the early stuff around AlexNet in 2012.

Always kind of wanted to study intelligence, but before I was just like, this isn't really working.

It doesn't seem like it's actually working.

All the way back to 2005, I'd read Ray Kurzweil's work, I'd read even some of Eliezer's work on the early internet back then, and I was like, ah, this stuff kinda looks far away.

Like, I look at the AI stuff of today, and it's like, not anywhere close.

But with Alexander, I was like, oh, this stuff is actually starting to work.

So I joined Andrew Ng's group initially at Baidu.

And the first task that I got set to do, I'd been in a different field.

This was my first experience with AI, and it was a bit different from a lot of the kind of academic-style research that was going on kind of elsewhere in the world, right?

I think I kind of got lucky in that the task that was given to me and the other folks there was just make the best speech recognition system that you can.

So it kind of, it posed the problem in a way that was amenable to discovering that kind of scaling was a solution, right?

That's very different from like, you're a postdoc and it's your job to come up with, you know, what's the best, like, you know, what's an idea that seems clever and new and makes your mark as someone who's invented something.

And so I just quickly discovered that like, you know, I was just trying the simplest experiments.

I was like, you know, just fiddling with some dials.

I was like, okay, try, you know, try adding more layers to the, literally add more layers to the RNN.

What if I add new data and repeat it less times?

And like, I just saw these like very consistent patterns.

I didn't really know that this was unusual or that others weren't thinking in this way.

This was just kind of like, almost like beginner's luck.

And I didn't really think about it beyond speech recognition, right?

You know, I was just kind of like, oh, this is, you know, I don't know anything about this field.

There's zillions of things people do with machine learning, but like, I'm like, weird, this seems to be true in the speech recognition field.

And then I think it was recently, you know, like, just before OpenAI started, that I met Ilya, who you interviewed, 1 of the first things he said to me was, look, the models, they just want to learn.

Like, I kind of like, I listened to this and I became enlightened.

And over the years after this, again, I would be kind of the 1 who would formalize a lot of these things and kind of put them together.

But like, just kind of the, what that told me is that that phenomenon that I'd seen wasn't just some random thing that I'd seen.

You get the obstacles out of their way, right?

You give them enough space to operate in.

You don't do something stupid like condition them badly numerically.

You know what I find really interesting about what you said is there are many people who were aware back at that time, probably weren't working on it directly, but were aware that these things are really good at speech recognition or at playing these constrained games.

Very few extrapolated from there like you and Ilya did to something that is generally intelligent.

What was different about the way you were thinking about it versus how others think that you went from like, is getting better at speech in this consistent way, it will get better at everything in this consistent way.

I mean, at first when I saw it for speech, I assumed this was just true for speech or for this narrow class of models.

I think it was just over the period between 2014 and 2017, I tried it for a lot of things and saw the same thing over and over again.

I watched the same being true with robotics, which many people thought of as a counter example, but I just thought, well, it's hard to get data for robotics, but if we operate within, if we look within the data that we have, we see the same patterns.

And so, I don't know, I think people were very focused on solving the problem in front of them.

Why 1 person thinks 1 way, another person thinks, it's very hard to explain.

I think people just see it through a different lens, you know, are looking like vertically instead of horizontally.

They're not thinking about the scaling, they're thinking about how do I solve my problem.

And well, for robotics, there's not enough data.

And so, you know, and so, you know, that can easily abstract to well, scaling doesn't work because we don't have the data.

And so I don't, I don't know, I just, for some reason, and it may just it may just have been random chance was obsessed with that particular direction.

When did it become obvious to you that language is the means to just feed a bunch of data into these things that, or was it just, you ran out of other things like robotics, there's not enough data.

There's other things, there's not enough data.

I mean, I think this whole idea of like the next word prediction that you could do self-supervised learning, you know, that together with the idea that it's like, wow, for predicting the next word, there's so much richness and structure there, right?

You know, it might say 2 plus 2 equals and you have to know the answer is 4.

And, you know, it might be telling the story about a character and then basically it's posing to the model, you know, the equivalent of these developmental tests that get posed to children.

You know, Mary walks into the room and, you know, puts an item in there and then, you know, Chuck walks into the room and removes the item and Mary doesn't see it, what does Mary think, you know, so like, so the models are going to have to get this right in the service of predicting the next word, they're going to have to solve, you know, solve all these theory of mind problems, solve all these math problems.

And so I, you know, my thinking was just, well, you know, scale it up as much as you can.

You know, there's kind of no limit to it.

And I think I kind of had abstractly that view, but the thing, of course, that like really solidified and convinced me was the work that Alec Radford did on GPT-1, which was not only could you get this language model that could predict things very well, but also you could fine-tune it, you needed to fine-tune it in those days, to do all these other tasks.

And so I was like, wow, this isn't just some narrow thing where you get the language model right, it's sort of halfway to everywhere, right?

It's like, you know, you get the language model right, and then with a little move in this direction, it can, you know, it can solve this, you know, logical dereference test or whatever.

And, you know, with this other thing, you know, it can solve translation or something.

And then you're like, wow, I think there's really something to do it.

Well, 1 thing that's confusing or that would have been hard to see, if you told me in 2018, we'll have models in 2023, like Quote Law 2, that can write theorems in the style of Shakespeare, whatever theory you want, you want, they can a standardized test with open ended questions, you know, just all kinds of really impressive things.

You would have said at that time, I would have said, oh, you have AGI.

You clearly have something that is a human level intelligence.

Where these, while these things are impressive, it clearly seems we're not at human level, at least in the current generation and potentially for generations to come.

What explains this discrepancy between super impressive performance in these benchmarks and in just like the things you could describe versus yeah, generally.

So that, that was 1 area where actually I was not pressing and I was surprised as well.

So when I first looked at GPT 3 and you know, more so the kind of things that we built in the early days at Anthropic.

My general sense was, I looked at these and I'm like, it seems like they really grasp the essence of language.

I'm not sure how much we need to scale them up.

Like Maybe what's more needed from here is like RL and kind of all the other stuff.

Like we might be kind of near the, you know, I thought in 2020, like we can scale this a bunch more, but I wonder if it's more efficient to scale it more or to start adding on these other objectives like RL.

I thought maybe if you do as much RL as you've done pre-training for a 2020 style model that that's the way to go and scaling it up will keep working, but is that really the best path?

And I think it, I don't know, it just keeps going.

Like I thought it had understood a lot of the essence of language, but then, you know, there's kind of further to go.

And so, I don't know, stepping back from it, like 1 of the reasons why I'm sort of very empiricist about AI, about safety, about organizations is that you often get surprised, right?

I feel like I've been right about some things, but I've still, with these theoretical pictures ahead, been wrong about most things.

Being right about 10% of the stuff sets you head and shoulders above many people.

If you look back to, I can't remember who it was, kind of made these diagrams that are like, you know, here's the village idiot, here's Einstein, here's the scale of intelligence, right?

And the village idiot and Einstein are like very close to each other.

Like that, maybe that's still true in some abstract sense or something, but it's not really what we're seeing, is it?

We're seeing that it seems like the human range is pretty broad and we don't hit the human range in the same place or at the same time for different tasks.

Write a sonnet in the style of Cormac McCarthy or something, like, I don't know, I'm not very creative, so I couldn't do that.

But like, you know, that's a pretty high level human skill, right?

And even the model is starting to get good at stuff of, you know, like constrained writing, you know, there's this like, write a, you know, write a page without using the letter E or something, write a page about X without using the letter E.

I think their models might be superhuman or close to superhuman at that.

But when it comes to, I don't know, prove relatively simple mathematical theorems, like they're just starting to do the beginning of it.

They make really dumb mistakes sometimes, and they really lack any kind of broad, like, you know, correcting your errors or doing some extended task.

And so, I don't know, it turns out that intelligence isn't a spectrum.

There are a bunch of different areas of domain expertise.

There are a bunch of different, like, kinds of skills, like memory is different.

I mean, it's all it's all formed in the blob.

It's not complicated, but to the extent it even is on the spectrum, the spectrum is also wide.

If you asked me 10 years ago, that's not what I would have expected at all.

But I think that's very much the way it's turned out.

I have so many questions just as follow up on that.

1 is, do you expect that given the distribution of training that these models get from massive amounts of Internet data versus what humans got from evolution, that the repertoire of skills that elicits will be just barely overlapping, it will be like concentric circles.

Clearly, there's certainly a large amount of overlap, because a lot of the, these models have business applications and many of their business applications are doing things that, you know, are helping humans to be more effective at things.

And, you know, if you think of all the activity that humans put on the internet in text, that covers a lot of it.

But it probably doesn't cover some things.

Like the models, I think they do learn a physical model of the world to some extent, but they certainly don't learn how to actually move around in the world.

But I think, so I think there are some things that the models don't learn that humans do.

See all Dwarkesh Patel transcripts on Youtube

Dario Amodei (Anthropic CEO) - $10 Billion Models, OpenAI, Scaling, & AGI in 2 years