How should we scientifically think about the impact of AI on human civilization, and whether or not it will doom us all? In this episode, I speak with Scott Aaronson about his views on how to make progress in AI alignment, as well as his work on watermarking the output of language models, and how he moved from a background in quantum complexity theory to working on AI.

Note: this episode was recorded before  this story emerged of a man committing suicide after discussions with a language-model-based chatbot, that included discussion of the possibility of him killing himself.

Patreon: https://www.patreon.com/axrpodcast
Ko-fi: https://ko-fi.com/axrpodcast

Topics we discuss, and timestamps:
-  0:00:36 - 'Reform' AI alignment
  - 0:01:52 - Epistemology of AI risk
  - 0:20:08 - Immediate problems and existential risk
  - 0:24:35 - Aligning deceitful AI
  - 0:30:59 - Stories of AI doom
  - 0:34:27 - Language models
  - 0:43:08 - Democratic governance of AI
  - 0:59:35 - What would change Scott's mind
- 1:14:45 - Watermarking language model outputs
  - 1:41:41 - Watermark key secrecy and backdoor insertion
- 1:58:05 - Scott's transition to AI research
  - 2:03:48 - Theoretical computer science and AI alignment
  - 2:14:03 - AI alignment and formalizing philosophy
  - 2:22:04 - How Scott finds AI research
- 2:24:53 - Following Scott's research

The transcript: https://axrp.net/episode/2023/04/11/episode-20-reform-ai-alignment-scott-aaronson.html

Links to Scott's things:
- Personal website: https://scottaaronson.com/
- Book, Quantum Computing Since Democritus: https://www.amazon.com/Quantum-Computing-since-Democritus-Aaronson/dp/0521199565/
- Blog, Shtetl-Optimized: https://scottaaronson.blog/

Writings we discuss:
- Reform AI Alignment: https://scottaaronson.blog/?p=6821
- Planting Undetectable Backdoors in Machine Learning Models: https://arxiv.org/abs/2204.06974

In this episode, I'll be speaking with Scott Aronson.

Scott is a professor of computer science at UT Austin, and he is currently spending a year as a visiting scientist for OpenAI, working on the theoretical foundations of AI safety.

We'll be talking about his view of the field, as well as the work he's doing at OpenAI.

If you're interested in what we're discussing, you can just check the description of this episode, and you can read the transcripts at axrp.net.

So you recently wrote this blog post about something you called Reform AI Alignment, and basically, I don't know, your take on AI alignment that's somewhat different from what you see as a traditional view or something.

Can you tell me a little bit about, do you see AI causing or being involved in a really important way in existential risk anytime soon?

Well, I guess it depends what you mean by soon, right?

I am not a very good prognosticator, right?

I feel like, you know, even in quantum computing theory, you know, which is this sort of tiny little part of the intellectual world where I've spent 25 years of my life, I can't predict very well what's going to be discovered a few years from now in that.

And if I can't even do that, then how much less can I predict what impacts AI is going to have on human civilization over the next century?

And, of course, I can try to play the Bayesian game.

I can try to, and I even will occasionally, accept bets if I feel really strongly about something.

You know, but you know, I'm also kind of a wuss.

I'm a little bit risk averse, you know, and I like to tell people, you know, like when they, whenever they ask me, like, you know, How soon will AI take over the world?

Or, or, you know, before that it was more often, how soon will we have a full powered quantum computer?

And, you know, you know, they don't want all the considerations and, explanations that I can offer.

And I like to tell them, look, if I were good at that kind of thing, I wouldn't be a professor.

I would be an investor and I would be a multibillionaire, right?

So I feel like probably, you know, there are some people in the world, you know, who can just consistently see what is coming in decades and get it right.

I mean, there are, you know, hedge funds that are consistently successful, you know, not many, right.

But I feel like the way that science has made progress for hundreds of years has not been to trying to, you know, prognosticate the, the whole shape of the future.

It's been to look a little bit ahead, you know, look at the problems, you know, that we can see right now that could actually be solved.

And rather than predicting 10 steps ahead, the future, you just try to create the next step ahead of the future, right?

And try to sort of steer it in what looks like a good direction.

And I feel like that is what I try to do as a scientist.

And you know, I've known the rationalist community, the AI risk community, you know, since, maybe not quite since its inception, right?

But I mean, I started blogging in 2005, right.

You know, the, the heyday of Eliezer, you know, writing The sequences first on overcoming bias and then on less wrong, that started around 2006, 2007.

So, I was interacting with them since the very beginning.

A lot of the same people who read my blog also read Eliezer and Robin.

And I was aware that there are these people who think that AI existential risk is just the overwhelmingly important issue for humanity, right?

It is so important that nothing else matters by comparison.

And I was aware of the sort of whole worldview that they were sort of building up around that belief.

And I always, I would say I neither wholeheartedly endorsed it nor dismissed it.

I felt like, certainly, I see no reason to believe that the human brain represents the limit of intelligence that is possible, you know, by our laws of physics, right?

It's been limited by all sorts of mundane things, you know, the energy that's needed to supply it, you know, the width of the birth canal, right?

There is absolutely no reason why you couldn't have, you know, much more generally intelligent problem-solving entities than us, And if and when those were to arise, that would be an enormous deal.

I mean, just like, you know, it was a pretty enormous deal for all of the other animals that we share the earth with when we arose, But I feel like in science, right, it's not enough for a problem to be, you know, hugely important, even for it to be the most important problem in the world.

You know, there has to be a research program.

There has to be, you know, some way to make progress on.

And what I saw for a long time when I sort of looked at what used to be called Singularity Institute and then MURI, when I looked at what they were doing, what the people who talked about AI alignment were doing, it seemed like a lot of a priori philosophical thinking about almost a form of theology.

And I don't say that derisively, it's almost just inherent to the subject matter.

When you are trying to imagine a being that much smarter than yourself, that much more omniscient and omnipotent than yourself, the term that humanity has had for millennia for that sort of exercise has been theology.

And so there was a lot of reasoning from first principles about, assume an arbitrarily powerful intelligence, what would it do in such and such a situation?

Why would such and such approach to aligning it not work?

Why would it see so much further ahead of us that we shouldn't even bother.

And the whole exercise felt to me like I would feel bad coming in as an outsider and saying, I don't really see clear progress being made here.

But many of the leaders of AI alignment, you know, they also say that Kalsky, you know, has unfortunately, I feel bad, you know, he seemed really depressed lately.

His AGI ruin, you know, list of lethalities essay was basically saying, you know, we are duped, right?

Saying, you know, we have not had ideas that have really moved the needle, you know, this.

And so you could say, if it's really true that we're just going to undergo this step change to this being that is as far beyond us as we are, you know, beyond orangutans, and, you know, we have as much hope of controlling it or directing it to our ends as the orangutans would have of doing that for us, well then, you know, you've basically just baked into your starting postulates the futility of the exercise, right?

And it doesn't, you know, whether it's true or it's not true, you know, in science, you always have to ask a different question, which is what can I make progress on?

And I think that the general rule is that to make progress in science, you need at least 1 of 2 things.

You know, you need a clear mathematical theory or you need experiments or data, right?

But what is common to both of those is that you need something in the external, something external to yourself that can tell you when you were wrong, okay?

That can tell you, you know, when you have to back up and try a different path.

Now in quantum computing, you know, we're only just starting now to have the, you know, experiments that are, you know, on the, on a scale that is, you know, interesting to us as theorists, right?

Quantum supremacy experiments, you know, The first ones were just 3 years ago.

But that's been okay because we've had a very, very clear mathematical theory of exactly what a quantum computer would be.

And in a certain sense, We've had that since Schrodinger wrote his equation down in 1926.

But certainly we've had it since Feynman and Deutsch and Bernstein and Vazirani in the 80s and 90s wrote down the mathematical basics of quantum computation.

Okay, now in deep learning for the past decade, it's been very much the opposite, right?

They do not have a mathematical theory, you know, that explains almost any of the success that deep learning has enjoyed, but they have reams and reams of data, right?

They can try things, and they now are trying things out on an absolutely enormous scale, learning what works, right?

And with AI alignment, I felt like it was in, you know, maybe, you know, not, this is sort of not necessarily anyone's fault, you know, it's sort of, you know, inherit to the subject, but it was in the unfortunate position for decades of having neither a mathematical theory, nor the ability to get data from the real world.

And I think it's almost impossible to make scientific progress under those conditions.

A very good case study here would be string theory.

String theory has been trying to make progress in physics in the absence of both experiments that directly bear on the questions they're asking, and a clear mathematical definition of what the theory is, right?

I mean, they have some, I mean, the vertex operator algebras exist, right?

You can even write a math textbook about them.

Yeah, Yeah, no, I mean, you could say that it is amazing how much they have been able to do even in the teeth of those obstacles, right?

And partly it's because they've been able to break off little bits and pieces of a yet unknown theory where they can study it mathematically.

And ADS-CFT is a little piece that you can break off that is better defined or that can be studied independently from the whole structure.

So I think that when you're in a situation where you have neither a mathematical theory nor experiments, then you're sort of out at sea, right?

And you need to try to grab onto something.

And the case of science, that means looking for little bits and pieces of the problem that you can break off where you at least have a mathematical theory for that little piece, Or you at least have experimental data about that little piece.

And now the reason why I am excited right now about AI alignment and when OpenAI approached me last spring with the proposal that, hey, we read your blog, we'd like you to take off a year and think about the foundations of AI safety for us.

There are people who are so much more knowledgeable about AI than I am.

I studied AI in grad school for a year or 2 before I switched to quantum computing.

You know, so I had a little bit of background, you know, that was, that was in 2000, you know, that was well before the deep learning revolution, you know, although, you know, of course we, you know, all of the, the, the main ideas that have powered the revolution, you know, if a neural nets, you know, back propagation, you know, we were very familiar with all of them back then.

It's just that they hadn't yet been implemented on a big enough scale, you know, to show the amazing results that they have today.

Even then, I felt like, you know, machine learning was clearly going to be important, like it was going to impact the world on a, you know, probably shorter timescale than quantum computing would.

But I was always frustrated by the sort of inability to make clean mathematical statements that would answer the questions you really wanted to answer, whereas in quantum computing you could do that.

So I sort of fell in with the quantum crowd at some point.

So now, after 20 years out of AI, I'm sort of dipping my foot back into it.

I ultimately did decide to accept OpenAI's offer to spend a year there.

And it was partly because I've just been as bowled over as everyone else by GPT and DALI and what they've been able to do.

And I knew it was going to be an extremely exciting year for AI and it seemed like a good time to get involved.

But also I felt like AI safety is finally becoming a field where you can make clear, legible progress.

Where we actually, first of all, we have systems like GPT, you know, that fortunately, I think are not in any immediate danger of destroying the world, but they are in danger of enabling various bad actors to misuse them, to do bad things.

Maybe smallest and most obvious example is that every student on earth will be tempted to use GPT to do their homework.

And, you know, as an academic, I hear from, you know, all of my fellow academics who are extremely concerned.

But, you know, also, I fully expect that, you know, nation states, you know, and corporations will be generating propaganda and will be generating spam and hoaxes and all sorts of things like that.

Where, of course, you could do all of that before, but having an entity like GPT lets you scale it up so cheaply.

We're going to see powerful AIs let loose in the world that people are going to misuse.

And all of a sudden, you know, AI safety is now an empirical subject, right?

It is now, you know, we can now learn something from, from, from the world about what works and what doesn't work to try to mitigate these misuses, right?

And we still don't have a mathematical theory, but we can at least formulate theories and see which ones are useful, see which ones are actually giving us useful insight about how to make GPT safer, how to make DALI safer.

So now it becomes the kind of thing that science is able to act upon.

And there's a huge irony here, which is that I would say that Eliezer and I have literally switched positions about the value of AI safety research, where he spent decades saying that everyone should be, everyone who is able should be working on it.

It is the most important thing in the world.

I was sort of keeping it at arm's length.

Like, yeah, maybe we can try to die with more dignity, right?

Maybe we can try for some Hail Mary pass, but basically we're doomed.

And I'm saying, no, actually, you know, AI safety is getting interesting.

This is actually a good time to get into it.

I will say, under my understanding of Eliezer,

by like, die with dignity, he does mean like, try to solve the problem.

Like, he still is like, into people trying to solve it.

Well, yes, because he says, you know, even if it's just increasing a, you know, a 0.001 chance of survival to a 0.002% chance, then, you know, in his calculus, you know, that is as worth doing as if both of the probabilities had been much, much larger, right?

But I think that many other people who maybe lack that detachment would see how depressed he is about the whole matter and would just give up.

So am I right to kind of summarize that as he's saying, look, this whole AI thing, it seems potentially like you can see ways it could become important in the near term and there are like things you can see yourself working on and like making progress.

And it's whether or not you think that has much to do with like, you know, AI causing doom to everyone or something that's interesting enough to you that you're willing to like take a year to work on it.

Yes, well, I think that a thriving field of science, you know, usually has the full range, right?

It has like the sort of gigantic cosmic concerns, you know, that you hope will, you know, maybe be resolved in decades or centuries, right?

But then it also has immediate problems that you can make progress on that are right on the horizon.

And you can sort of see a line from the 1 to the other, right?

I think this is a characteristic of every really successful side, whether that's physics, whether that's quantum computing, whether that's the P versus NP problem.

And I do have that feeling now about AI safety, that there is the cosmic question of where are we going as a civilization?

It is now, I think, completely clear that AI is going to be a huge part of that story.

That doesn't mean that AI is going to convert us all into paperclips, right?

But I think that hardly any informed person would dispute at this point, that the story of the 21st century will in large part be a story of our relationship to AI that will become more and more powerful.

When you say you can kind of see a pathway from 1 to the other, can you tell me a bit, what connection do you see between like, okay, we like figure out how to stop, for example, students cheating on their homework with GPT?

Like yeah, how do you see that linking up to matters of cosmic concern if you do?

So in Turing's paper, Computing Machinery and Intelligence, right, in 1950, that, you know, set the terms for, you know, much of the discussion of AI that there's been in the 73 years since, right?

You know, the last sentence of that paper was, you know, We can see only a short distance ahead, but we can see much there that needs to be done.

And so I feel like, you know, a part of it is, you know, and this is a point that the Orthodox alignment people make very, very clearly as well.

But you know, you could say, if we cannot even figure out how to prevent GPT from dispensing bomb-making advice, if we don't want it to do that, or from endorsing or seeming to endorse racist or sexist views or helping people, you know, look for, you know, security vulnerabilities in code or things like that.

You know, if we can't even figure that out, then how on earth would we solve the much broader problem of, you know, aligning a superhuman intelligence with our values, right?

And so it's a lot like in theoretical computer science, let's say, right?

People might ask, you know, has there been any progress whatsoever towards solving the P versus NP problem, right?

And eventually I've written a 120-page survey article about that exact question, right?

And my answer is basically, well, you know, people have succeeded in solving a whole bunch of problems that would need to be solved along any, as far as I could tell, along any path that would eventually culminate in solving P versus NP.

So that doesn't mean that you can put any bounds on how far are we from a solution.

It just means that you're walking down a path and it seems like the right path, and you have no idea how much longer that path is going to continue.

So I feel much the same way with AI-aligned, like understanding how to make large language models safer is on the right path.

I mean, you could say if it is true at all that there is a line or a continuum from these things to a truly existentially dangerous AI, then there also ought to be a path from how to mitigate the dangers of these things to how to mitigate the dangers of that super AI, right?

If there's no line, you know, anyway, then maybe there's less to worry about in the first place.

But I tend to think that no, actually all sorts of progress is interlinked.

GPT itself builds on a lot of the progress of the past decades, it would not have existed without all of the GPUs that we have now, wouldn't have existed without all of the data that we now have on the internet that we can use to train it.

And of course, it wouldn't have existed without all of the progress in machine learning that there's been over the past decade, such as the discovery of transformer models.

So progress, even in not obviously related things, is sort of enabled GPT.

And I think that tools like GPT, you know, these are going to be stepping stones to, you know, the next progress in AI.

And I think that if we do get to AI that is just smarter than us across every domain, then we will see, we will be able to look back and see Deep Blue, AlphaGo, Watson, GPT, Dolly, yes, these were all stepping stones along a certain logical path.

I wonder, yeah, maybe this is closely related to what you were just talking about, but I think 1 thing that people who are maybe skeptical of this kind of alignment research will say is, well, the really scary problems show up in systems that look kind of different, right?

So like systems that are smart enough to anticipate what you're trying to do and potentially they can try to deceive you or like systems that are trying to do some tasks that you can't easily evaluate, right?

I'm wondering, like, so potentially your response to these criticisms is like, well, you've got to start somewhere.

Or like, it might be, you know, maybe this isn't an issue, or like, you know, it's, there's deep links here.

Yeah, well, well, well, look, I think, you know, you've got to start somewhere is, is, is true as far as it goes.

But 1 can say a little bit more than that.

1 can say if there really were to be a FOOM scenario, so if there were to be this abrupt transition where we go from AIs such as GPT and DALI, which seem to most of us, they are not endangering the physical survival of humanity, right?

Whatever smaller dangers they might present for discourse on the internet or for things like that.

If we were to just undergo a step change from an AI like that to AIs that are pretending to be like that, but that are secretly plotting against us and biding their time until they make their move.

And once they make their moves, then they just turn us all to goo in a matter of seconds.

And it's just game over for humanity and they rule the world.

If it's that kind of thing, then I would tend to agree with Eliezer and with the other AI alignment people that, yeah, it sounds like we're doomed.

What I find both more plausible and also more productive to work on is the scenario where the ability to deceive develops gradually just like every other ability that we've seen.

Where before you get an AI that is plotting to make its 1 move to take over the entire world, you get AIs that are trying to deceive us and doing a pretty bad job at it, or that succeed at deceiving 1 person, but not another person.

In some sense, we're already on that path.

You can ask GPT to try to, you know, be deceitful, right?

And you can try to train it or few shot prompt it to be a deceitful.

I don't know if you saw this example where GPT was asked to write a sorting program, but that secretly treats the number 5 differently from all the other numbers, but in a way that should not be obvious to someone inspecting the code.

And what it generates is code that has a condition called not 5, that actually is if the number is 5.

So you could say that in terms of its ability to deceive, AI has now reached parity with a preschooler or something.

So now it gets interesting because now you could imagine AI that has the deceit ability of an elementary school student.

But I mean, some people might think that it's naive to think that things are going to progress continuously in that way.

But there is some empirical evidence that things do get, I mean, if you look at the earlier iterations of GPT, they really, really struggled even just with the most basic arithmetic or the most basic math problems.

And now they do much, much better on those math problems, including high school level word problems, but they still struggle with college level math problems.

They still struggle with math competition problems or prove this interesting theorem.

So it's very much the kind of development that a human mathematician would go through.

And even the mistakes that it makes when it tries to solve a hard math problem are like the mistakes that I have seen in a thousand exams that I have graded.

They are entirely familiar kinds of mistakes to me, right down to the tone of sort of blithe self-confidence as it, you know, makes some completely unjustified step in a proof, right?

Or, you know, as it produces the proof that you requested, that there are only finitely many prime numbers, or whatever other false statement you ask it to prove, right?

It is undergoing, you know, the same kinds of mistakes that a human student makes as they're learning.

And you could even point its mistakes out to it.

You can say like, but it seems like you divided by 0 in this step.

And I'm like, oh yeah, yes, you're right.

And you know, it can correct its mistakes.

But so I think that, you know, we've now for better or for worse, you know, we've succeeded in building something that can learn in a way that is, you know, not entirely dissimilar to how we learn.

And you know, I think that it will be learning to deceive as it is learning other skills.

And you know, we will be able to watch that happen.

And so I don't find plausible this picture of AI that never even attempts to deceive until it makes its brilliant, you know, ten-dimensional chest move to take over the world.

And the relevance of this story is something like, look, AI will have like, you know, kind of stumbling, like a little bit foolish deceit attempts earlier, and we'll basically work on it then, and we'll solve the problems like quick enough that when real deceit happens, we can...

First of all, I am now working on this, putting my money where my mouth is, but I would say more generally, I am a worried person by nature.

The question for me is not sort of whether to be worried, it's which things to be most worried about.

I am worried about the future of civilization on many fronts.

You know, I'm worried, you know, about droughts that are going to become much more frequent.

And as we lose access to fresh water, what happens as weather gets more and more unpredictable, I'm worried about the resurgence of authoritarianism all over the world.

So I'm worried about geopolitical things.

I, you know, I think, you know, 80 years after the invention of nuclear weapons, you know, that continues to be, you know, a very huge thing to be worried about, as, as we were all reminded this past year by the war in Ukraine.

So I am worried about pandemics that will, you know, make COVID look like just a practice run.

And I think all of these worries kind of interact with each other, you know, to some degree, right?

Climate change is going to exacerbate, you know, all sorts of geopolitical rivalries, right?

We're, you know, we're already seeing that happen.

My way of thinking about it, AI is now 1 more ingredient that is part of the stew of worries that are going to define the century.

The US just restricted the sale of chips to China, partly because of worries about AI acceleration.

That might then, unfortunately, spur China to get more aggressive toward Taiwan.

So the AI question can't be isolated from the geopolitical questions, from all the broader questions about what's happening in civilization.

And I'm completely convinced that AI will be part of the story of, let's say, existential risks in the coming century because it's going to be part of everything that's happening in civilization.

If we come up with cheap, wonderful solutions to climate change.

See all AXRP transcripts on Youtube

20 - 'Reform' AI Alignment with Scott Aaronson