Eliezer Yudkowsky is a researcher, writer, and philosopher on the topic of superintelligent AI. Please support this podcast by checking out our sponsors:
- Linode: https://linode.com/lex to get $100 free credit
- House of Macadamias: https://houseofmacadamias.com/lex and use code LEX to get 20% off your first order
- InsideTracker: https://insidetracker.com/lex to get 20% off

EPISODE LINKS:
Eliezer's Twitter: https://twitter.com/ESYudkowsky
LessWrong Blog: https://lesswrong.com
Eliezer's Blog page: https://www.lesswrong.com/users/eliezer_yudkowsky
Books and resources mentioned:
1. AGI Ruin (blog post): https://lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
2. Adaptation and Natural Selection: https://amzn.to/40F5gfa

PODCAST INFO:
Podcast website: https://lexfridman.com/podcast
Apple Podcasts: https://apple.co/2lwqZIr
Spotify: https://spoti.fi/2nEwCF8
RSS: https://lexfridman.com/feed/podcast/
Full episodes playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist: https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
0:43 - GPT-4
23:23 - Open sourcing GPT-4
39:41 - Defining AGI
47:38 - AGI alignment
1:30:30 - How AGI may kill us
2:22:51 - Superintelligence
2:30:03 - Evolution
2:36:33 - Consciousness
2:47:04 - Aliens
2:52:35 - AGI Timeline
3:00:35 - Ego
3:06:27 - Advice for young people
3:11:45 - Mortality
3:13:26 - Love

SOCIAL:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Reddit: https://reddit.com/r/lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The problem is that we do not get 50 years to try and try again and observe that we were wrong and come up with a different theory and realize that the entire thing is going to be way more difficult than realized at the start.

Because the first time you fail at aligning something much smarter than you are, you die.

The following is a conversation with Eliezer Yudkowsky, a legendary researcher, writer, and philosopher on the topic of artificial intelligence, especially super intelligent AGI and its threat to human civilization.

To support it, please check out our sponsors in the description.

And now, dear friends, here's Eliezer Yudkowsky.

It is a bit smarter than I thought this technology was going to scale to.

And I'm a bit worried about what the next 1 will be like.

Like this particular 1, I think, I hope there's nobody inside there because it would be suck to be stuck inside there.

But we don't even know the architecture at this point, because OpenAI is very properly not telling us.

And yeah, like giant inscrutable matrices of floating point numbers, I don't know what's going on in there.

All we have to go by are the external metrics.

And on the external metrics, if you ask it to write a self-aware FORTRAN green text, it will start writing a green text about how it has realized that it's an AI writing a green text and like oh well, so That's probably Not quite what's going on in there in reality, but we're kind of like blowing past all the science fiction guardrails.

We are past the point where in science fiction, people would be like, well, wait, stop, that thing's alive, what are you doing to it?

We don't have any lines to draw on the sand and say, like, well, when we get this far, we will start to worry about what's inside there.

So if it were up to me, I would be like, okay, like this far, no further, time for the summer of AI where we have planted our seeds and now we like wait and reap the rewards of the technology you've already developed and don't do any larger training runs than that.

Which to be clear, I realize requires more than 1 company agreeing to not do that.

And take a rigorous approach for the whole AI community to investigate whether there's somebody inside there.

Like having any idea of what's going on in there, people have been trying for a while.

It's a poetic statement about if there's somebody in there, but I feel like it's also a technical statement, or I hope it is 1 day, which is a technical statement that Alan Turing tried to come up with with the Turing test.

Do you think it's possible to definitively or approximately figure out if there is somebody in there, if there's something like a mind inside this large language model?

I mean, there's a whole bunch of different sub-questions here.

There's the question of like, is there consciousness?

Should we be worried about how we're treating it?

And then there's questions like, how smart is it exactly?

And we can check how it can do X and how it can do Y.

Unfortunately, we've gone and exposed this model to a vast corpus of text of people discussing consciousness on the internet, which means that when it talks about being self-aware, we don't know to what extent it is repeating back what it has previously been trained on for discussing self-awareness, or if there's anything going on in there such that it would start to say similar things spontaneously.

Among the things that 1 could do if 1 were at all serious about trying to figure this out is train GPT-3 to detect conversations about consciousness, exclude them all from the training datasets, and then retrain something around the rough size of GPT-4 and no larger with all of the discussion of consciousness and self-awareness and so on missing.

Humans are self-aware, we're self-aware all the time, we talk about what we do all the time, what we're thinking at the moment all the time.

But nonetheless, get rid of the explicit discussion of consciousness, I think therefore I am and all that.

And then try to interrogate that model and see what it says.

But nonetheless, I don't know, I feel like when you run over the science fiction guard rails, Like maybe this thing, but what about GPT?

Maybe not this thing, but like what about GPT-5?

Yeah, this would be a good place to pause.

On the topic of consciousness, there's so many components to even just removing consciousness from the data set.

Emotion, the display of consciousness, the display of emotion, feels like deeply integrated with the experience of consciousness.

So the hard problem seems to be very well integrated with the actual surface level illusion of consciousness.

I mean, do you think there's a case to be made that we humans, when we're babies, are just like GPT that we're training on human data on how to display emotion versus feel emotion, how to show others, communicate others that I'm suffering, that I'm excited, that I'm worried, that I'm lonely and I missed you and I'm excited to see you.

That's a communication skill versus the actual feeling that I experience.

So we need that training data as humans too, that we may not be born with that, how to communicate the internal state.

And that's, in some sense, if we remove that from GPT-4's data set, it might still be conscious, but not be able to communicate it.

So I think you're gonna have some difficulty removing all mention of emotions from GPT's data set.

I would be relatively surprised to find that it has developed exact analogs of human emotions in there.

I think that humans will have emotions even if you don't tell them about those emotions when they're kids.

It's not quite exactly what various blank slaytists tried to do with the new Soviet man and all that, but if you try to raise people perfectly altruistic, they still come out selfish.

You try to raise people sexless, they still develop sexual attraction.

We have some notion in humans, not in AIs, of where the brain structures are that implement this stuff.

It is a really remarkable thing, I say in passing, that despite having complete read access to every floating point number in the GPT series, we still know vastly more about the architecture of human thinking than we know about what goes on inside GPT despite having vastly better ability to read GPT.

Do you think that's just a matter of time?

Do you think it's possible to investigate and study the way neuroscientists study the brain?

Which is look into the darkness, the mystery of the human brain by just desperately trying to figure out something and to form models and then over a long period of time actually start to figure out what regions of the brain do certain things, what different kinds of neurons, when they fire, what that means, how plastic the brain is, all that kind of stuff.

You slowly start to figure out different properties of the system.

Do you think we can do the same thing with language models?

Sure, I think that if, you know, like half of today's physicists stop wasting their lives on string theory or whatever.

And go off and study what goes on inside transformer networks.

Then in, you know, like 30, 40 years, we'd probably have a pretty good idea.

Do you think these large language models can reason?

How are they doing that without reasoning?

So you're somebody that spearheaded the movement of rationality, So reason is important to you.

Or is it, like how difficult is the threshold of being able to reason to you?

I mean, in my writings on rationality, I have not gone making a big deal out of something called reason.

I have made more of a big deal out of something called probability theory.

And that's like, well, you're reasoning, but you're not doing it quite right, and you should reason this way instead.

And interestingly, people have started to get preliminary results showing that reinforcement learning by human feedback has made the GPT series worse in some ways.

In particular, it used to be well calibrated.

If you trained it to put probabilities on things, it would say 80% probability and be right 8 times out of 10.

And if you apply reinforcement learning from human feedback, the nice graph of 70%, 7 out of 10, sort of flattens out into the graph that humans use, where there's some very improbable stuff, and likely, probable, maybe, which all means around 40%, and then certain.

It used to be able to use probabilities, but if you apply, but if you try to teach it to talk in a way that satisfies humans, it gets worse at probability in the same way that humans are.

PY I would call it a bug, although such a fascinating bug.

But yeah, so like reasoning, it's doing pretty well on various tests that people used to say would require reasoning, but, you know, rationality is about when you say 80% does it happen 8 times out of 10.

So what are the limits to you of these transformer networks, of neural networks?

If reasoning is not impressive to you, or it is impressive, but there's other levels to achieve.

I mean, it's just not how I carve up reality.

If reality is a cake, what are the different layers of the cake or the slices?

Or you can use a different food if you like.

It's, I don't think it's as smart as a human yet.

I do, like back in the day, I went around saying like, I do not think that just stacking more layers of transformers is going to get you all the way to AGI.

And I think that GPT-4 is past, or I thought this paradigm was going to take us.

And I, you know, you want to notice when that happens.

You want to say like, whoops, well I guess I was incorrect about what happens if you keep on stacking more transformer layers.

And that means I don't necessarily know what GPT-5 is going to be able to do.

So you're saying like your intuition initially is now appears to be wrong.

It's good to see that you can admit in some of your predictions to be wrong.

So because you make several very, throughout your life you've made many strong predictions and statements about reality and you evolve with that.

So maybe that'll come up today about our discussion.

It's a bit ambitious to go through your entire life never having been wrong.

1 can aspire to be well-calibrated, like not so much think in terms of like, was I right, was I wrong, but like when I said 90% that it happened 9 times out of 10.

Yeah, like oops is the sound we emit when we improve.

And somewhere in there, we can connect the name of your blog, Less Wrong.

The name Less Wrong was, I believe, suggested by Nick Bostrom, and it's after someone's epigraph, I actually forget who's, who said, like, we never become right, we just become less wrong.

What's the something, something easy to confess, just error and error and error again, but less and less and less.

So what has surprised you about GPT-4 that you found beautiful, as a scholar of intelligence, of human intelligence, of artificial intelligence, of the human mind?

I mean, beauty does interact with the screaming horror.

But like beautiful moments, well, somebody asked Bing Sidney to describe herself and fed the resulting description into 1 of the stable diffusion things, I think.

She's pretty and this is something that should have been an amazing moment.

The AI describes herself, you get to see what the AI thinks the AI looks like.

Although, the thing that's doing the drawing is not the same thing that's outputting the text.

And it does happen the way that it would happen in the old school science fiction when you ask an AI to make a picture of what it looks like.

Not just because there are 2 different AI systems being stacked that don't actually interact, it's not the same person, but also because the AI was trained by imitation in a way that makes it very difficult to guess how much of that it really understood and probably not actually a whole bunch.

Although GPT-4 is like multi-modal and can like draw vector drawings of things that make sense and like does appear to have some kind of spatial visualization going on in there.

But like the pretty picture of the like girl with the steampunk goggles on her head, if I'm remembering correctly what she looked like.

It just made a description of it and stable diffusion output it.

There's the concern about how much the discourse is going to go completely insane once the AIs all look like that and like are actually look like people talking.

And yeah, there's like another moment where somebody is asking Bing about like, well, I like fed my kid green potatoes and they have the following symptoms, and Bing is like, that's solanine poisoning, and call an ambulance, and the person's like, I can't afford an ambulance, I guess if this is time for my kid to go, that's God's will.

And the main Bing thread gives the message of, I cannot talk about this anymore.

And the suggested replies to it say, please don't give up on your child.

Solanine poisoning can be treated if caught early.

And if that happened in fiction, that would be like the AI cares.

The AI is bypassing the block on it to try to help this person.

Probably not, but nobody knows what's going on in there.

It's part of a process where these things are not happening in a way where somebody figured out how to make an AI care, and we know that it cares, and we can acknowledge its caring now.

It's being trained by this imitation process followed by reinforcement learning on human feedback.

We're trying to point it in this direction, and it's pointed partially in this direction, and nobody has any idea what's going on inside it.

If there was a tiny fragment of real caring in there, we would not know.

It's not even clear what it means exactly.

And things are clear cut in science fiction.

We'll talk about the horror and the terror and the trajectories this can take.

But this seems like a very special moment.

Just a moment where we get to interact with the system that might have care and kindness and emotion, and maybe something like consciousness.

And we don't know if it does, and we're trying to figure that out, and we're wondering about what is, what it means to care.

We're trying to figure out almost different aspects of what it means to be human, about the human condition, by looking at this AI that has some of the properties of that.

It's almost like this subtle, fragile moment in the history of the human species.

We're trying to almost put a mirror to ourselves here.

We are seeing increasing signs bit by bit, but not spontaneous signs, because people are trying to train the systems to do that using imitative learning.

And the imitative learning is like spilling over and having side effects.

And the most photogenic examples are being posted to Twitter, rather than being examined in any systematic way.

So when you are boiling a frog like that, or you're going to get like, first is going to come the Blake-Lemoyne's, like first you're going to have like a thousand people looking at this, and the 1 person out of a thousand who is most credulous about the signs is going to be like, that thing is sentient.

While 999 out of a thousand people think, almost surely correctly, though we don't actually know, that he's mistaken.

And so the first people to say, like, sentience look like idiots.

And humanity learns the lesson that when something claims to be sentient and claims to care, it's fake.

Because we have been training them using imitative learning rather than, and this is not spontaneous, and they keep getting smarter.

Do you think we would oscillate between that kind of cynicism, that AI systems can't possibly be sentient, they can't possibly feel emotion, they can't possibly, this kind of, yeah, cynicism about AI systems, and then oscillate to a state where we empathize with the AI systems, we give them a chance, we see that they might need to have rights and respect and similar role in society as humans.

You're going to have a whole group of people who can just like never be persuaded of that because to them, like being wise, being cynical, being skeptical is to be like, oh, well, machines can never do that.

It's just imitating, it's just fooling you.

And they would say that right up until the end of the world, and possibly even be right, because they are being trained on an imitative paradigm.

And you don't necessarily need any of these actual qualities in order

Have you observed yourself working through skepticism, cynicism, and optimism about the power of neural networks.

What has that trajectory been like for you?

It looks like neural networks before 2006, forming part of an indistinguishable to me – other people might have had better distinction on it – indistinguishable blob of different AI methodologies, all of which are promising to achieve intelligence without us having to know how intelligence works.

You had the people who said that if you just manually program lots and lots of knowledge into the system, line by line, at some point all the knowledge will start interacting, it will know enough, and it will wake up.

You've got people saying that if you just use evolutionary computation, if you try to mutate lots and lots of organisms that are competing together, that's the same way that human intelligence was produced in nature.

So we'll do this and it will wake up without having any idea of how AI works.

You've got people saying, well, we will study neuroscience and we will learn the algorithms off the neurons and we will imitate them without understanding those algorithms, which was a part I was pretty skeptical of because it's hard to reproduce, re-engineer these things without understanding what they do.

And so we will get AI without understanding how it works.

And there were people saying, well, we will have giant neural networks that we will train by gradient descent.

And when they are as large as the human brain, they will wake up.

We will have intelligence without understanding how intelligence works.

And from my perspective, this is all like an indistinguishable blob of people who are trying to not get to grips with the difficult problem of understanding how intelligence actually works.

That said, I was never skeptical that evolutionary computation would not work in the limit.

Like you throw enough computing power at it, it obviously works.

And it turned out that you can throw less computing power than that at gradient descent If you are doing some other things correctly And you will get intelligence without having any idea of how it works and what is going on inside It wasn't ruled out by my model that this could happen.

I wouldn't have been able to call neural networks rather than any of the other paradigms for getting like massive amount, like intelligence without understanding it.

And I wouldn't have said that this was a particularly smart thing for a species to do, which is an opinion that has changed less than my opinion about whether or not you can actually do it.

Do you think AGI could be achieved with a neural network as we understand them today?

The question is whether the current architecture of stacking more transformer layers, which for all we know GPT-4 is no longer doing because they're not telling us the architecture, which is a correct decision.

I had a conversation with Sam Altman, we'll return to this topic a few times.

Would you open source the code, he asked me.

Because I provided as criticism, saying that while I do appreciate transparency, open AI could be more open.

And he says, we struggle with this question.

sell GPT-4 to business backend applications that don't expose it to consumers and venture capitalists and create a ton of hype and like pour a bunch of new funding into the area.

LRW But don't you think others would do it?

If you already have giant nuclear stockpiles, don't build more.

If some other country starts building a larger nuclear stockpile, then sure, even then, maybe just have enough nukes.

You know, these things are not quite like nuclear weapons.

They spit out gold until they get large enough and then ignite the atmosphere and kill everybody.

And there is something to be said for not destroying the world with your own hands, even if you can't stop somebody else from doing it.

But open sourcing, that's just sheer catastrophe.

The whole notion of open sourcing, this was always the wrong approach, the wrong ideal.

There are places in the world where open source is a noble ideal, and building stuff you don't understand that is difficult to control, where if you could align it, it would take time.

You'd have to spend a bunch of time doing it.

That is not a place for open source because then you just have like powerful things that just like go straight out the gate without anybody having had the time to have them not kill everyone.

So can we still make on the case for some level of transparency and openness, maybe open sourcing.

So the case could be that because GPT-4 is not close to AGI, if that's the case, that this does allow open sourcing, of being open about the architecture, being transparent about maybe research and investigation of how the thing works, of all the different aspects of it, of its behavior, of its structure, of its training processes, of the data it was trained on, everything like that, that allows us to gain a lot of insights about alignment, about the alignment problem, to do really good AI safety research while the system is not too powerful.

Can you make that case, that it could be resourced?

I do not believe in the practice of steelmanning.

There is something to be said for trying to pass the ideological Turing test where you describe your opponent's position, the disagreeing person's position, well enough that somebody cannot tell the difference between your description and their description.

Like, okay, well, this is where you and I disagree here.

I do not want, okay, So for 1 thing, if somebody's trying to understand me, I do not want them steel manning my position.

I want them to try to describe my position the way I would describe it, not what they think is an improvement.

Well, I think that is what steel manning is, is the most charitable interpretation.

I don't want to be interpreted charitably.

I want them to understand what I'm actually saying.

If they go off into the land of charitable interpretations, they're like off in their land of like the stuff they're imagining and not trying to understand my own viewpoint anymore.

Well, I'll put it differently then, just to push on this point.

I would say it is restating what I think you understand under the empathetic assumption that Eliezer is brilliant and have honestly and rigorously thought about the point he has made.

So if there's 2 possible interpretations of what I'm saying, and 1 interpretation is really stupid and whack and doesn't sound like me and doesn't fit with the rest of what I've been saying.

And 1 interpretation sounds like something a reasonable person who believes the rest of what I believe would also say, go with the second interpretation.

If on the other hand, there's like something that sounds completely whack and something that sounds like a little less completely whack, but you don't see why I would believe in it, doesn't fit with the other stuff I say, but you know, that sounds like less whack and you can like sort of see, you could like maybe argue it, then you probably have not understood it.

See, okay, I'm gonna, this is fun, because I'm gonna linger on this.

You know, you wrote a brilliant blog post, AGI ruined a list of lethalities, right?

And I would say that some of the points are bigger and more powerful than others.

If you were to sort them, you probably could, you personally, and to me, steel manning means like going through the different arguments and finding the ones that are really the most like powerful, if people like TLDR, like what should you be most concerned about and bringing that up in a strong, compelling, eloquent way, these are the points that Eliezer would make to make the case, in this case, that AI's gonna kill all of us.

But that's what Steel Manning is, is presenting it in a really nice way, the summary of my best understanding of your perspective.

That because to me, there's a sea of possible presentations of your perspective.

And Steel Manning is doing your best to do the best 1 in that sea of different perspectives.

Like these things that you would be presenting as like the strongest version of my perspective.

Do you believe what you would be presenting?

When I see the perspective of a person, there is a part of me that believes it if I understand it.

I mean, especially in political discourse, in geopolitics, I've been hearing a lot of different perspectives on the world.

And I hold my own opinions, but I also speak to a lot of people that have a very different life experience and a very different set of beliefs.

And I think there has to be epistemic humility in stating what is true.

See all Lex Fridman transcripts on Youtube

Eliezer Yudkowsky: Dangers of AI and the End of Human Civilization | Lex Fridman Podcast #368