Full episode with Nick Bostrom (Mar 2020): https://www.youtube.com/watch?v=rfKiTGj-zeQ
Clips channel (Lex Clips): https://www.youtube.com/lexclips
Main channel (Lex Fridman): https://www.youtube.com/lexfridman
(more links below)

Podcast full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4

Podcasts clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

Podcast website:
https://lexfridman.com/ai

Podcast on Apple Podcasts (iTunes):
https://apple.co/2lwqZIr

Podcast on Spotify:
https://spoti.fi/2nEwCF8

Podcast RSS:
https://lexfridman.com/category/ai/feed/

Nick Bostrom is a philosopher at University of Oxford and the director of the Future of Humanity Institute. He has worked on fascinating and important ideas in existential risks, simulation hypothesis, human enhancement ethics, and the risks of superintelligent AI systems, including in his book Superintelligence. I can see talking to Nick multiple times on this podcast, many hours each time, but we have to start somewhere.

Subscribe to this YouTube channel or connect on:
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/lexfridman
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

Let's talk about superintelligence, at least for a little bit.

Yeah, I tend not to get too stuck with the definitional question.

I mean, the common sense to understand, like the ability to solve complex problems, to learn from experience, to plan, to reason, some combination of things like that.

Is consciousness mixed up into that or is it?

Well, I don't think, I think it could be fairly intelligent at least without being conscious probably.

Yeah, that would be like something that was much more, had much more general cognitive capacity than we humans have.

So if we talk about general superintelligence, it would be much faster learner be able to reason much better, make plans that are more effective at achieving its goals, say in a wide range of complex challenging environments.

In terms of, as we turn our eye to the idea of sort of existential threats from superintelligence, do you think superintelligence has to exist in the physical world, or can it be digital only?

Sort of, we think of our general intelligence as us humans, as an intelligence that's associated with the body that's able to interact with the world, that's able to affect the world directly with physically.

I mean, digital only is perfectly fine, I think.

I mean, it's physical in the sense that obviously the computers and the memories are physical.

But it's capability to affect the world sort of- Could

be very strong, even if it has a limited set of actuators, if it can types text on the screen or something like that, that would be, I think, ample.

So in terms of the concerns of existential threat of AI, how can an AI system that's in the digital world have existential risk?

Sort of, and what are the attack vectors for a digital system?

Well, I mean, I guess maybe to take 1 step back, so I should emphasize that I also think there's this huge positive potential from machine intelligence, including superintelligence.

And I want to stress that because some of my writing has focused on what can go wrong.

And when I wrote the book, Superintelligence, at that point, I felt that there was a kind of neglect of what would happen if AI succeeds, and in particular, a need to get a more granular understanding of where the pitfalls are so we can avoid them.

I think that since the book came out in 2014, there has been a much wider recognition of that.

And a number of research groups are now actually working on developing, say AI alignment techniques and so on and so forth.

So I'd like, yeah, I think now it's important to make sure we bring back onto the table the upside as well.

And there's a little bit of a neglect now on the upside, which is, I mean, if you look at, I was talking to a friend, if you look at the amount of information that is available, or people talking, or people being excited about the positive possibilities of general intelligence, that's not, it's far outnumbered by the negative possibilities in terms of our public discourse?

What are, can you linger on that for a little bit?

What are some, to you, possible big positive impacts of general intelligence, super intelligence?

Well, I mean, super intelligence, because I tend to also want to distinguish these 2 different contexts of thinking about AI and AI impacts, the kind of near-term and long-term, if you want, both of which I think are legitimate things to think about, and people should discuss both of them, but they are different and they often get mixed up and then you get confusion.

I think you get simultaneously, like maybe an overhyping of the near term and an underhyping of the long term.

And so I think as long as we keep them apart, we can have like 2 good conversations, but, or we can mix them together and have 1 bad conversation.

Can you clarify just the 2 things we're talking about, the near term and the long term.

it's a blurry distinction, but say the things I wrote about in this book, superintelligence, long-term, things people are worrying about today with, I don't know, algorithmic discrimination or even things, self-driving cars and drones and stuff more near term.

And then, of course, you could imagine some medium term where they kind of overlap and 1 evolves into the other.

But at any rate, I think both, yeah, the issues look kind of somewhat different depending on which of these contexts.

So I think it would be nice if we can talk about the long-term and think about a positive impact or a better world because of the existence of the long-term superintelligence.

Yeah, I mean, I guess it's a little hard to articulate because it seems obvious that the world has a lot of problems as it currently stands.

And it's hard to think of any 1 of those which it wouldn't be useful to have like a friendly aligned super intelligence working on.

So from health to the economic system to be able to sort of improve the investment and trade and foreign policy decisions, all that kind of stuff.

I think AI, especially artificial general intelligence, is really the ultimate general purpose technology.

So it's not that there is this 1 problem, this 1 area where it will have a big impact, but if and when it succeeds, it will really apply across the board in all fields where human creativity and intelligence and problem solving is useful, which is pretty much all fields, right?

The thing that it would do is give us a lot more control over nature.

It wouldn't automatically solve the problems that arise from conflict between humans, fundamentally political problems.

Some subset of those might go away if you just had more resources and cooler tech, but some subset would require coordination that is not automatically achieved just by having more technological capability.

But anything that's not of that sort, I think you just get an enormous boost with this kind of cognitive technology once it goes all the way.

Now, again, that doesn't mean I'm thinking, People don't recognize what's possible with current technology and like sometimes things get overhyped, but I mean, those are perfectly consistent views to hold.

And then it's a very different question of how far are we from that or what can we do with near-term technology?

Yeah, so what's your intuition about the idea of intelligence explosion?

So there's this, you know, when you start to think about that leap from the near term to the long term, the natural inclination, like for me, sort of building machine learning systems today, it seems like it's a lot of work to get to general intelligence.

But there's some intuition of exponential growth, of exponential improvement, of intelligence explosion.

Can you maybe try to elucidate, try to talk about what's your intuition about the possibility of an intelligence explosion, that it won't be this gradual, slow process, there might be a phase shift?

Yeah, I think it's, we don't know how explosive it will be.

I think for what it's worth, seems fairly likely to me that at some point there will be some intelligence explosion, like some period of time where progress in AI becomes extremely rapid, roughly in the area where you might say it's kind of human-ish equivalent in core cognitive faculties, that the concept of human equivalent starts to break down when you look too closely at it.

And just how explosive does something have to be for it to be called an intelligence explosion?

Does it have to be overnight literally, or a few years?

But overall, I guess, if you plotted the opinions of different people in the world, I guess I would be somewhat more probability towards the intelligence explosion scenario than probably the average AI researcher, I guess.

So, and then the other part of the intelligence explosion, or just forget explosion, just progress, is once you achieve that gray area of human level intelligence, is it obvious to you that we should be able to proceed beyond it to get to superintelligence?

Yeah, that seems, I mean, as much as any of these things can be obvious, given we've never had 1, people have different views, smart people have different views.

It's like There's some degree of uncertainty that always remains for any big futuristic, philosophical, grand question that just we realize humans are fallible, especially about these things.

But it does seem, as far as I'm judging things based on my own impressions, it seems very unlikely that there would be a ceiling at or

But that's such a, I don't know, that's such a special moment.

It's both terrifying and exciting to create a system that's beyond our intelligence.

So maybe you can step back and say, like, how does that possibility make you feel that we can create something?

It feels like there's a line beyond which it steps.

And therefore it feels like a step where we lose control.

And in fact, this is what a number of people are working towards, making sure that we could ultimately project higher levels of problem-solving ability while still making sure that they are aligned, like they are in the service of human values.

I mean, so losing control, I think, is not a given that would happen.

I mean, to some extent I've lived with this for so long, since as long as I can remember being an adult or even a teenager, it seemed to me obvious that at some point AI will succeed.

I meant, because the control problem is an interesting thing and I think the hope is, at least we should be able to maintain control over systems that are smarter than us, but we do lose our specialness.

It's sort of, we'll lose our place as the smartest, coolest thing on Earth.

And there's an ego involved with that, that humans aren't very good at dealing with.

I mean, I value my intelligence as a human being.

It seems like a big transformative step to realize there's something out there that's more intelligent.

I mean, you don't see that as such a fundamentally-

I think yes, a lot, I think it would be small.

I mean, I think there are already a lot of things out there that are, I mean, certainly if you think the universe is big, there's gonna be other civilizations that already have super-intelligences or that just naturally have brains the size of beach balls and are like completely leaving us in the dust.

We haven't come face to face, but I mean, that's an open question.

What would happen in a kind of post-human world, like how much day-to-day would these super intelligences be involved in the lives of ordinary?

I mean, You could imagine some scenario where it would be more like a background thing that would help protect against some things, but you wouldn't...

Like, there wouldn't be this intrusive kind of, like, making you feel bad by, like, making clever jokes on your experience.

Like, there's a whole set of things that maybe in the human context would feel awkward about that.

You don't want to be the dumbest kid in your class.

A lot of those things maybe you need to abstract away from if you're thinking about this context where we have infrastructure that is in some sense beyond any or all humans.

I mean, it's a little bit like say the scientific community as a whole, if you think of that as a mind, it's a little bit of metaphor, but I mean, obviously it's got to be like way more capacious than any individual.

So in some sense, there is this mind-like thing already out there that's just vastly more intelligent than a new individual is.

And we think, okay, you just accept that as a fact.

That's the basic fabric of our existence is there's a superintelligence.

I mean, there's already Google and Twitter and Facebook, these recommender systems that are the basic fabric of our, and I could see them becoming, I mean, do you think of the collective intelligence of these systems as already perhaps reaching super intelligence level?

Well, I mean, so here it comes to this, the concept of intelligence and the scale and what human level means, the kind of vagueness and indeterminacy of those concepts starts to dominate how you would answer that question.

So, say the Google search engine has a very high capacity of a certain kind, like remembering and retrieving information.

Particularly, like, text or images that are...

You have a kind of string, a word string key, like obviously superhuman at that, but a vast set of other things it can't even do at all, not just not do well.

But So you have these current AI systems that are superhuman in some limited domain and then radically subhuman in all other domains.

Same with a chess, or just a simple computer that can multiply really large numbers.

So it's gonna have this like 1 spike of super intelligence and then a kind of a 0 level of capability across all other cognitive fields.

Yeah, I don't necessarily think the generalness, I mean, I'm not so attached to it, but I could sort of, it's a gray area and it's a feeling, but to me, sort of alpha 0 is somehow much more intelligent, much, much more intelligent than Deep Blue.

And to say which, you could say, well, these are both just board games.

They're both just able to play board games.

Who cares if they're gonna do better or not?

But there's something about the learning, the self-play.

That makes it, crosses over into that land of intelligence that doesn't necessarily need to be general.

In the same way, Google is much closer to deep blue currently in terms of its search engine than it is to sort of the alpha 0.

And the moment it becomes, and the moment these recommender systems really become more like alpha 0, but being able to learn a lot without the constraints of being heavily constrained by human interaction, that seems like a special moment in time.

I mean, certainly learning ability seems to be an important facet of general intelligence.

That you can take some new domain that you haven't seen before and you weren't specifically pre-programmed for and then figure out what's going on there and eventually become really good at it.

So that's something AlphaZero has much more of than Deep Blue had.

And in fact, systems like AlphaZero can learn not just Go, but other, in fact, probably beat Deep Blue in chess and so forth, right?

So you do see this general, and it matches the intuition.

We feel it's more intelligent, and it also has more of this general purpose learning ability.

And if we get systems that have even more general purpose learning ability, it might also trigger an even stronger intuition that they are actually starting to get smart.

were to pick a future, what do you think a utopia looks like with AGI systems?

Sort of, is it the neural link, brain computer interface world, or we're kind of really closely interlinked with AI systems, is it possibly where AGI systems replace us completely while maintaining the values and the consciousness?

Is it something like it's a completely invisible fabric?

Like you mentioned, a society where it's just AIDS and a lot of stuff that we do, like curing diseases and so on.

Yeah, I mean, it's a good question and a deep and difficult 1.

I don't have all the answers yet, but I might never have, but I think there are some different observations 1 can make.

1 is if this scenario actually did come to pass, it would open up this vast space of possible modes of being.

On 1 hand, material and resource constraints would just be expanded dramatically.

So there would be a lot of, a big pie, let's say.

Also, it would enable us to do things, including to ourselves, or like that, it would just open up this much larger design space and option space than we have ever had access to in human history.

1 is that we probably would need to make a fairly fundamental rethink of what ultimately we value.

Like, think things through more from first principles.

The context would be so different from the familiar that we could have just take what we've always been doing and then like, oh, well, we have this cleaning robot that like cleans the dishes in the sink and a few other small things.

Like I think we would have to go back to first principles.

So even from the individual level, go back to the first principles of what is the meaning of life, what is happiness, what is fulfillment?

And then also connected to this large space of resources is that it would be possible, and I think something we should aim for is to do well by the lights of more than 1 value system.

That is, we wouldn't have to choose only 1 value criterion and say, we're gonna do something that scores really high on the metric of say hedonism and then is like a 0 by other criteria, like kind of wire headed brain Cinebat and it's like a lot of pleasure that's good, but then like no beauty, no achievement, like no.

Or pick it up, I think to some significant, not unlimited sense, but a significant sense, it would be possible to do very well by many criteria.

Maybe you could get 98% of the best according to several criteria at the same time, given this great expansion of the option space.

And so- So have competing value systems, competing criteria as a sort of forever, just like our Democrat versus Republican, there seems to be this always multiple parties that are useful for our progress in society, even though it might seem dysfunctional inside the moment, but having the multiple value systems seems to be beneficial for, I guess, a balance of power.

So that's, yeah, not exactly what I have in mind, that it's, well, although it may be in an indirect way, it is, But that if you had the chance to do something that scored well on several different metrics, our first instinct should be to do that rather than immediately leap to the thing, which ones of these value systems are we going to screw over?

Let's first try to do very well by all of

Then it might be that you can't get 100% of all, and you would have to then have the hard conversation about which 1 will only get 97%.

There's my cynicism that all of existence is always a trade-off.

But you say, maybe it's not such a bad trade-off.

Well, this would be a distinctive context in which at least some of the constraints would be removed.

So there's probably still be trade-offs in the end.

It's just that we should first make sure we at least take advantage of this abundance.

So in terms of thinking about this, like, yeah, 1 should think, I think in this kind of frame of mind of generosity and inclusiveness to different value systems and see how far 1 can get there first.

And I think 1 could do something that would be very good according to many different criteria.


Speaker 0: Let's talk about superintelligence, at least for a little bit. And let's start at the basics. What to you is intelligence?

Speaker 1: Yeah, I tend not to get too stuck with the definitional question. I mean, the common sense to understand, like the ability to solve complex problems, to learn from experience, to plan, to reason, some combination of things like that. Is consciousness

Speaker 2: mixed up into that or no? Is consciousness mixed up into that or is it?

Speaker 1: Well, I don't think, I think it could be fairly intelligent at least without being conscious probably.

Speaker 2: So then what is super intelligence?

Speaker 1: Yeah, that would be like something that was much more, had much more general cognitive capacity than we humans have. So if we talk about general superintelligence, it would be much faster learner be able to reason much better, make plans that are more effective at achieving its goals, say in a wide range of complex challenging environments.

Speaker 2: In terms of, as we turn our eye to the idea of sort of existential threats from superintelligence, do you think superintelligence has to exist in the physical world, or can it be digital only? Sort of, we think of our general intelligence as us humans, as an intelligence that's associated with the body that's able to interact with the world, that's able to affect the world directly with physically.

Speaker 1: I mean, digital only is perfectly fine, I think. I mean, it's physical in the sense that obviously the computers and the memories are physical.

Speaker 2: But it's capability to affect the world sort of- Could

Speaker 1: be very strong, even if it has a limited set of actuators, if it can types text on the screen or something like that, that would be, I think, ample.

Speaker 2: So in terms of the concerns of existential threat of AI, how can an AI system that's in the digital world have existential risk? Sort of, and what are the attack vectors for a digital system?

Speaker 1: Well, I mean, I guess maybe to take 1 step back, so I should emphasize that I also think there's this huge positive potential from machine intelligence, including superintelligence. And I want to stress that because some of my writing has focused on what can go wrong. And when I wrote the book, Superintelligence, at that point, I felt that there was a kind of neglect of what would happen if AI succeeds, and in particular, a need to get a more granular understanding of where the pitfalls are so we can avoid them. I think that since the book came out in 2014, there has been a much wider recognition of that. And a number of research groups are now actually working on developing, say AI alignment techniques and so on and so forth.

So I'd like, yeah, I think now it's important to make sure we bring back onto the table the upside as well.

Speaker 2: And there's a little bit of a neglect now on the upside, which is, I mean, if you look at, I was talking to a friend, if you look at the amount of information that is available, or people talking, or people being excited about the positive possibilities of general intelligence, that's not, it's far outnumbered by the negative possibilities in terms of our public discourse?

Speaker 1: Possibly, yeah. It's hard to measure it, but.

Speaker 2: What are, can you linger on that for a little bit? What are some, to you, possible big positive impacts of general intelligence, super intelligence?

Speaker 1: Well, I mean, super intelligence, because I tend to also want to distinguish these 2 different contexts of thinking about AI and AI impacts, the kind of near-term and long-term, if you want, both of which I think are legitimate things to think about, and people should discuss both of them, but they are different and they often get mixed up and then you get confusion. I think you get simultaneously, like maybe an overhyping of the near term and an underhyping of the long term. And so I think as long as we keep them apart, we can have like 2 good conversations, but, or we can mix them together and have 1 bad conversation.

Speaker 2: Can you clarify just the 2 things we're talking about, the near term and the long term. What are the distinctions? Well,

Speaker 1: it's a blurry distinction, but say the things I wrote about in this book, superintelligence, long-term, things people are worrying about today with, I don't know, algorithmic discrimination or even things, self-driving cars and drones and stuff more near term. And then, of course, you could imagine some medium term where they kind of overlap and 1 evolves into the other. But at any rate, I think both, yeah, the issues look kind of somewhat different depending on which of these contexts.

Speaker 2: So I think it would be nice if we can talk about the long-term and think about a positive impact or a better world because of the existence of the long-term superintelligence. Do you have views of such a world?

Speaker 1: Yeah, I mean, I guess it's a little hard to articulate because it seems obvious that the world has a lot of problems as it currently stands. And it's hard to think of any 1 of those which it wouldn't be useful to have like a friendly aligned super intelligence working on.

Speaker 2: So from health to the economic system to be able to sort of improve the investment and trade and foreign policy decisions, all that kind of stuff.

Speaker 1: All that kind of stuff and a lot more.

Speaker 2: I mean, what's the killer app?

Speaker 1: Well, I don't think there is 1. I think AI, especially artificial general intelligence, is really the ultimate general purpose technology. So it's not that there is this 1 problem, this 1 area where it will have a big impact, but if and when it succeeds, it will really apply across the board in all fields where human creativity and intelligence and problem solving is useful, which is pretty much all fields, right? The thing that it would do is give us a lot more control over nature. It wouldn't automatically solve the problems that arise from conflict between humans, fundamentally political problems.

Some subset of those might go away if you just had more resources and cooler tech, but some subset would require coordination that is not automatically achieved just by having more technological capability. But anything that's not of that sort, I think you just get an enormous boost with this kind of cognitive technology once it goes all the way. Now, again, that doesn't mean I'm thinking, People don't recognize what's possible with current technology and like sometimes things get overhyped, but I mean, those are perfectly consistent views to hold. The ultimate potential being enormous. And then it's a very different question of how far are we from that or what can we do with near-term technology?

Speaker 2: Yeah, so what's your intuition about the idea of intelligence explosion? So there's this, you know, when you start to think about that leap from the near term to the long term, the natural inclination, like for me, sort of building machine learning systems today, it seems like it's a lot of work to get to general intelligence. But there's some intuition of exponential growth, of exponential improvement, of intelligence explosion. Can you maybe try to elucidate, try to talk about what's your intuition about the possibility of an intelligence explosion, that it won't be this gradual, slow process, there might be a phase shift?

Speaker 1: Yeah, I think it's, we don't know how explosive it will be. I think for what it's worth, seems fairly likely to me that at some point there will be some intelligence explosion, like some period of time where progress in AI becomes extremely rapid, roughly in the area where you might say it's kind of human-ish equivalent in core cognitive faculties, that the concept of human equivalent starts to break down when you look too closely at it. And just how explosive does something have to be for it to be called an intelligence explosion? Does it have to be overnight literally, or a few years? But overall, I guess, if you plotted the opinions of different people in the world, I guess I would be somewhat more probability towards the intelligence explosion scenario than probably the average AI researcher, I guess.

Speaker 2: So, and then the other part of the intelligence explosion, or just forget explosion, just progress, is once you achieve that gray area of human level intelligence, is it obvious to you that we should be able to proceed beyond it to get to superintelligence?

Speaker 1: Yeah, that seems, I mean, as much as any of these things can be obvious, given we've never had 1, people have different views, smart people have different views. It's like There's some degree of uncertainty that always remains for any big futuristic, philosophical, grand question that just we realize humans are fallible, especially about these things. But it does seem, as far as I'm judging things based on my own impressions, it seems very unlikely that there would be a ceiling at or

Speaker 2: near human cognitive capacity. But that's such a, I don't know, that's such a special moment. It's both terrifying and exciting to create a system that's beyond our intelligence. So maybe you can step back and say, like, how does that possibility make you feel that we can create something? It feels like there's a line beyond which it steps.

It'll be able to outsmart you. And therefore it feels like a step where we lose control.

Speaker 1: Well, I don't think the latter follows. That is, you could imagine. And in fact, this is what a number of people are working towards, making sure that we could ultimately project higher levels of problem-solving ability while still making sure that they are aligned, like they are in the service of human values. I mean, so losing control, I think, is not a given that would happen. Now you asked how it makes me feel.

I mean, to some extent I've lived with this for so long, since as long as I can remember being an adult or even a teenager, it seemed to me obvious that at some point AI will succeed.

Speaker 2: And so I actually misspoke. I didn't mean control. I meant, because the control problem is an interesting thing and I think the hope is, at least we should be able to maintain control over systems that are smarter than us, but we do lose our specialness. It's sort of, we'll lose our place as the smartest, coolest thing on Earth. And there's an ego involved with that, that humans aren't very good at dealing with.

I mean, I value my intelligence as a human being. It seems like a big transformative step to realize there's something out there that's more intelligent. I mean, you don't see that as such a fundamentally-

Speaker 1: I think yes, a lot, I think it would be small. I mean, I think there are already a lot of things out there that are, I mean, certainly if you think the universe is big, there's gonna be other civilizations that already have super-intelligences or that just naturally have brains the size of beach balls and are like completely leaving us in the dust. And we

Speaker 2: haven't come face to face with them.

Speaker 1: We haven't come face to face, but I mean, that's an open question. What would happen in a kind of post-human world, like how much day-to-day would these super intelligences be involved in the lives of ordinary? I mean, You could imagine some scenario where it would be more like a background thing that would help protect against some things, but you wouldn't... Like, there wouldn't be this intrusive kind of, like, making you feel bad by, like, making clever jokes on your experience. Like, there's a whole set of things that maybe in the human context would feel awkward about that.

You don't want to be the dumbest kid in your class. A lot of those things maybe you need to abstract away from if you're thinking about this context where we have infrastructure that is in some sense beyond any or all humans. I mean, it's a little bit like say the scientific community as a whole, if you think of that as a mind, it's a little bit of metaphor, but I mean, obviously it's got to be like way more capacious than any individual. So in some sense, there is this mind-like thing already out there that's just vastly more intelligent than a new individual is. And we think, okay, you just accept that as a fact.

Speaker 2: That's the basic fabric of our existence is there's a superintelligence.

Speaker 1: Yeah, you get used to a lot of.

Speaker 2: I mean, there's already Google and Twitter and Facebook, these recommender systems that are the basic fabric of our, and I could see them becoming, I mean, do you think of the collective intelligence of these systems as already perhaps reaching super intelligence level?

Speaker 1: Well, I mean, so here it comes to this, the concept of intelligence and the scale and what human level means, the kind of vagueness and indeterminacy of those concepts starts to dominate how you would answer that question. So, say the Google search engine has a very high capacity of a certain kind, like remembering and retrieving information. Yeah. Particularly, like, text or images that are... You have a kind of string, a word string key, like obviously superhuman at that, but a vast set of other things it can't even do at all, not just not do well.

But So you have these current AI systems that are superhuman in some limited domain and then radically subhuman in all other domains. Same with a chess, or just a simple computer that can multiply really large numbers. So it's gonna have this like 1 spike of super intelligence and then a kind of a 0 level of capability across all other cognitive fields.

Speaker 2: Yeah, I don't necessarily think the generalness, I mean, I'm not so attached to it, but I could sort of, it's a gray area and it's a feeling, but to me, sort of alpha 0 is somehow much more intelligent, much, much more intelligent than Deep Blue. And to say which, you could say, well, these are both just board games. They're both just able to play board games. Who cares if they're gonna do better or not? But there's something about the learning, the self-play.

Speaker 1: The learning, yeah.

Speaker 2: That makes it, crosses over into that land of intelligence that doesn't necessarily need to be general. In the same way, Google is much closer to deep blue currently in terms of its search engine than it is to sort of the alpha 0. And the moment it becomes, and the moment these recommender systems really become more like alpha 0, but being able to learn a lot without the constraints of being heavily constrained by human interaction, that seems like a special moment in time.

Speaker 1: I mean, certainly learning ability seems to be an important facet of general intelligence. That you can take some new domain that you haven't seen before and you weren't specifically pre-programmed for and then figure out what's going on there and eventually become really good at it. So that's something AlphaZero has much more of than Deep Blue had. And in fact, systems like AlphaZero can learn not just Go, but other, in fact, probably beat Deep Blue in chess and so forth, right? So you do see this general, and it matches the intuition.

We feel it's more intelligent, and it also has more of this general purpose learning ability. And if we get systems that have even more general purpose learning ability, it might also trigger an even stronger intuition that they are actually starting to get smart. So if you

Speaker 2: were to pick a future, what do you think a utopia looks like with AGI systems? Sort of, is it the neural link, brain computer interface world, or we're kind of really closely interlinked with AI systems, is it possibly where AGI systems replace us completely while maintaining the values and the consciousness? Is it something like it's a completely invisible fabric? Like you mentioned, a society where it's just AIDS and a lot of stuff that we do, like curing diseases and so on. What is utopia, if you get to pick?

Speaker 1: Yeah, I mean, it's a good question and a deep and difficult 1. I'm quite interested in it. I don't have all the answers yet, but I might never have, but I think there are some different observations 1 can make. 1 is if this scenario actually did come to pass, it would open up this vast space of possible modes of being. On 1 hand, material and resource constraints would just be expanded dramatically.

So there would be a lot of, a big pie, let's say. Right? Also, it would enable us to do things, including to ourselves, or like that, it would just open up this much larger design space and option space than we have ever had access to in human history. So I think 2 things follow from that. 1 is that we probably would need to make a fairly fundamental rethink of what ultimately we value.

Like, think things through more from first principles. The context would be so different from the familiar that we could have just take what we've always been doing and then like, oh, well, we have this cleaning robot that like cleans the dishes in the sink and a few other small things. Like I think we would have to go back to first principles.

Speaker 2: So even from the individual level, go back to the first principles of what is the meaning of life, what is happiness, what is fulfillment? Yeah.

Speaker 1: And then also connected to this large space of resources is that it would be possible, and I think something we should aim for is to do well by the lights of more than 1 value system. That is, we wouldn't have to choose only 1 value criterion and say, we're gonna do something that scores really high on the metric of say hedonism and then is like a 0 by other criteria, like kind of wire headed brain Cinebat and it's like a lot of pleasure that's good, but then like no beauty, no achievement, like no. Or pick it up, I think to some significant, not unlimited sense, but a significant sense, it would be possible to do very well by many criteria. Maybe you could get 98% of the best according to several criteria at the same time, given this great expansion of the option space.

Speaker 2: And so- So have competing value systems, competing criteria as a sort of forever, just like our Democrat versus Republican, there seems to be this always multiple parties that are useful for our progress in society, even though it might seem dysfunctional inside the moment, but having the multiple value systems seems to be beneficial for, I guess, a balance of power.

Speaker 1: So that's, yeah, not exactly what I have in mind, that it's, well, although it may be in an indirect way, it is, But that if you had the chance to do something that scored well on several different metrics, our first instinct should be to do that rather than immediately leap to the thing, which ones of these value systems are we going to screw over? So I see. Let's first try to do very well by all of

Speaker 2: them.

Speaker 1: Then it might be that you can't get 100% of all, and you would have to then have the hard conversation about which 1 will only get 97%. There you go.

Speaker 2: There's my cynicism that all of existence is always a trade-off. But you say, maybe it's not such a bad trade-off. Let's first at

Speaker 1: least try it. Well, this would be a distinctive context in which at least some of the constraints would be removed. I'll leave you with that. So there's probably still be trade-offs in the end. It's just that we should first make sure we at least take advantage of this abundance.

So in terms of thinking about this, like, yeah, 1 should think, I think in this kind of frame of mind of generosity and inclusiveness to different value systems and see how far 1 can get there first. And I think 1 could do something that would be very good according to many different criteria.

See all Lex Fridman transcripts on Youtube

Nick Bostrom: Superintelligence | AI Podcast Clips