Recently, OpenAI made a splash by announcing a new "Superalignment" team. Lead by Jan Leike and Ilya Sutskever, the team would consist of top researchers, attempting to solve alignment for superintelligent AIs in four years by figuring out how to build a trustworthy human-level AI alignment researcher, and then using it to solve the rest of the problem. But what does this plan actually involve? In this episode, I talk to Jan Leike about the plan and the challenges it faces.

Patreon: patreon.com/axrpodcast
Ko-fi: ko-fi.com/axrpodcast

Art by @hamishdoodles 

Topics we discuss, and timestamps:
- 0:00:37 - The superalignment team
- 0:02:10 - What's a human-level automated alignment researcher?
  - 0:06:59 - The gap between human-level automated alignment researchers and superintelligence
  - 0:18:39 - What does it do?
  - 0:24:13 - Recursive self-improvement
- 0:26:14 - How to make the AI AI alignment researcher
  - 0:30:09 - Scalable oversight
  - 0:44:38 - Searching for bad behaviors and internals
  - 0:54:14 - Deliberately training misaligned models
- 1:02:34 - Four year deadline
  - 1:07:06 - What if it takes longer?
- 1:11:38 - The superalignment team and...
  - 1:11:38 - ... governance
  - 1:14:37 - ... other OpenAI teams
  - 1:18:17 - ... other labs
- 1:26:10 - Superalignment team logistics
- 1:29:17 - Generalization
- 1:43:44 - Complementary research
- 1:48:29 - Why is Jan optimistic?
  - 1:58:32 - Long-term agency in LLMs?
  - 2:02:44 - Do LLMs understand alignment?
- 2:06:01 - Following Jan's research

The transcript:  axrp.net/episode/2023/07/27/episode-24-superalignment-jan-leike.html

Links for Jan and OpenAI:
- OpenAI jobs: openai.com/careers
- Jan's substack: aligned.substack.com
- Jan's twitter: twitter.com/janleike 

Links to research and other writings we discuss:
- Introducing Superalignment: openai.com/blog/introducing-superalignment
- Let's Verify Step by Step (process-based feedback on math): arxiv.org/abs/2305.20050
- Planning for AGI and beyond: openai.com/blog/planning-for-agi-and-beyond
- Self-critiquing models for assisting human evaluators: arxiv.org/abs/2206.05802
- An Interpretability Illusion for BERT: arxiv.org/abs/2104.07143
- Language models can explain neurons in language models  https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html
- Our approach to alignment research: openai.com/blog/our-approach-to-alignment-research
- Training language models to follow instructions with human feedback (aka the Instruct-GPT paper): arxiv.org/abs/2203.02155

In this episode, I'll be speaking with Jan Lejka.

After working for 4 years at DeepMind on reinforcement learning from human feedback and recursive reward modeling.

In early 2021, Jan joined OpenAI, where he now co-leads the recently announced Super Alignment team.

For links to what we're discussing, you can check the description of this episode, and you can read the transcript at axrp.net.

So first of all, I guess we're going to be talking about this announcement of the super alignment team.

For people who somehow haven't heard of that or haven't read that blog post, can you recap what it is and what it's going to be doing.

So basically, we want to send us an ambitious goal of solving alignment of superintelligence within the next 4 years.

And Ilya Sutskever, the co-founder and chief scientist of OpenAI is joining the team.

And OpenAI is committing 20% of the compute secured so far to this effort or to the effort of aligning superintelligence.

And so we're staffing up the effort a lot.

In particular, we are interested in hiring machine learning researchers and engineers who haven't really worked that much on alignment before, because we think there's a lot of scope for them to contribute and have really big impact.

And yeah, we have a general overall plan of how we want to approach the problem that involves training roughly human level alignment researcher that can work automatically, and then ask that automated alignment researcher to figure out how to align superintelligence.

And so 1 of the key pieces for us to do would be to figure out how to align this automated alignment researcher.

So I think in the blog post you used the phrase human level automated alignment researcher, right?

So basically, we want to offload as many of the tasks that we do when we're doing alignment work to an automated system.

And so typically when you're using LLM or if you're building an AI system in general, the skill profiles they have isn't exactly what a human would do.

They would be vastly better at some things, like language models are now on translation or knowing facts and so on.

And then they're like, yeah, system would be significantly worse than some other tasks like language models are right now with, for example, arithmetic.

And so the question then becomes like, what are the kind of tasks that we can offload to the AI systems in which order.

As we're doing this, you'd expect humans would focus more and more on the tasks that we're not offloading.

As we go into that process, AI systems are doing a larger and larger chunk of the overall work and human researchers will basically thus be more and more effective at actually making progress.

So should I imagine something like, Instead of you replace the first OpenAI alignment team employee and then the second 1, you should imagine you replace this type of task that everyone is doing and then this type of task that everyone is doing.

And then I think in order to actually get a lot of work out of the system, you'd want to have like 90, 99 or 99.9% of the tasks being automated because then you have effectively 10x, 100x, 1000x as much research output.

What kinds of tasks are you imagining it doing?

Yeah, so broadly there's, I would throw them into 2 different buckets.

1 bucket is kind of like the tasks that look more like just traditional ML engineering research that you would do if you're just trying to make AI systems be more capable.

And then the other bucket is all the other things that we have to do on alignment.

And so in the first bucket, this is stuff like you're implementing ML experiments, running them and looking at the results.

And the second bucket, it's more like, how do you, for example, figure out what experiments you should run on to improve scalable oversight?

Or how do you make progress in interpretability?

These are really big high-level questions, but there's also just a lot more detailed questions of you have a given point that you are in research, where let's say you have just written a paper and you're like, okay, what do we do next if we continue down this route?

And so I expect that basically ML in general will get really good at the first bucket of just designing running experiments automatically.

And kind of our job of eventually accelerating alignment progress would be to figure out how to automate the second bucket.

And so you're conceiving of the second bucket as the full stack from coming up with research directions to coming up with ideas of what things might work, all the way down to what script do I run right now?

I mean, you could ask me If I think that alignment research is so similar to machine learning research, how much is that really in the second bucket?

But I think there's actually a lot in there and it's highly leveraged because alignment as a problem is still so vague and confusing.

And I think in general, it's a lot of disagreement among experts who are like the most promising directions or like what should we do next.

And so the more you can accelerate what we do there, it will actually have a really large impact.

This is basically the same pitch that you would give for a researcher to join the field, right?

We are still trying to figure out the basics.

We don't know how to rely on super intelligence or even systems that are significantly smarter than humans.

So yeah, it's like we want to recruit AI.

It's just like we want to recruit more people, I guess.

But there's something really beautiful about recruiting AI, which is it scales so much better and faster than humans do, because all you need to do is buy more GPUs and then you have more AI.

So 1 question I had is, when you said a human level alignment researcher, it seems like often in AI, it's like most things aren't exactly human level at anything, right?

So like you mentioned, like chat models, I think they're like super human and just in terms of breadth of knowledge, right?

I think it would be hard for anyone to know as many facts as GPT-4 does, but subhuman at arithmetic, at least if the human's allowed to have pen and paper.

So how important is the human level qualifier?

On these lists of tasks, if it's really superhuman at some of them, is that a problem for you or is that just so much the better?

Yeah, I think the question is really how risky is it to run that system on the task of alignment research, right?

Because if it knows a lot of facts, that isn't particularly scary.

But what we really need to figure out is if we let the system take over some amount, or ultimately almost all of our alignment research, will it lie to us?

Will it try to take the opportunity to take over?

Because now it's doing so much stuff that we can't all look at ourselves.

And so the question is, the kind of skillset that you would need to do this, how does it compare to the kind of skillset that we would need to get a lot of assistance in the line research?

And if you zoom into that question, what are actually the things that we would be worried about?

This is like, how good is the model as spinning really clear and lies or being deceptive of pretending to do something or believe 1 thing and then actually wanting another?

I think another really key capability here is self-exfiltration.

So how good would the model be at breaking the security precautions and accessing its own weights and trying to copy it somewhere else on the internet, or persuading engineer with access to the weights to download them and send them somewhere.

And so we can specifically measure how good the models are at that.

And then we can compare it to measuring how good is it at actually helping us with alignment research.

And so roughly the idea is you want the models to not be too good at these scary tasks?

So this actually relates to a critique of this line of research that basically says, okay, if I want a human level automated alignment researcher, it needs to be pretty smart, it needs to be creative, it needs to think of things that we haven't thought of yet.

It needs to be able to plan towards a goal, right?

Like, I want to get this, so I've got to do these non-obvious things in the way.

I've got to learn things about the world.

And it's also got to be really good at thinking about misalignment, right?

And so 1 might think, oh, that combination of things, that's inherently scary or dangerous.

And I guess the question almost is then, well, if you have that, if the task is you're building something, you're aligning this automated alignment researcher, do you even have any problems left for it to solve?

I think ultimately this is an empirical question.

It's kind of really difficult to know in which order, which skills get unlocked when you scale up the models.

There's a lot more work aimed at predicting emerging capabilities now.

And I think it will give us some chance of actually predicting what the next pre-trained model will be like.

But I think also, I mean, I think we can make some high level arguments.

So for example, 1 thing that is pretty clear is once you have the model as good as you can handle off, alignment research off too, wouldn't it then also be able to just improve its own capabilities?

Or it can do a bunch of ML research, so it can run experiments on improving language models, sorry, compute efficiency or something.

And then you could use that and pre-train a much more capable model shortly thereafter.

And I think this story sounds on the surface appealing, but I think in practice it will be actually a lot more complicated because you're not doing big pre-trained runs every week.

They usually take a few months, and so it would be a few months before you actually have that.

In the meantime, you still get to use the system.

I think the other thing is also there's an open question of how much low-hanging fruit there still is on actually having compute efficiency wins.

And I think ultimately, the argument here I would make is that right now the existing community of people who are trying really hard to get AI to go faster and be more capable is already quite large relative to the alignment community.

And so if you get to automate a lot of these tasks and both communities benefit equally, then I think actually alignment benefits a lot more because it's a smaller community.

And so we don't have to do these tasks anymore.

I guess my question is, given the So I took the plan to be something like, we're going to make this automated alignment researcher, and then it's going to help us with alignment.

And given that, in order to make an automated alignment researcher, it needs quite a lot of pretty smart, pretty good capabilities.

What problems does it need to solve that we won't have needed to solve in order to get it?

Maybe I can answer the second part of the previous question that you asked, which is what about long run goals?

It seems to me at least that language models, or AI in general, has proven to be on average more creative than humans, I would say.

If you look at the diffusion model images, or if you look at, you sample from a pre-trained base model or something, there's a lot of really wild stuff in there.

And there's a lot of creativity that I think you would really struggle to get out of a single human or a small group of humans just because the model has seen the whole range of everything humans have said or all the images on the internet.

And so they can sample actually from the whole distribution, whereas individuals, humans typically can't.

And then in terms of long run goals, I think this is actually not needed at all, because we can hand off pretty small, well-scoped tasks to AI systems that if they really nail those, it would actually be really useful.

And that could be really small range things like, here's the paper that we just wrote, please suggest some next steps or some new experiments to do.

If you imagine having a really A-star researcher that you can ask these questions, they don't have to pursue long run goals.

They only have to optimize over the next, I don't know, few thousand tokens.

If they do that super well, then you would get a lot of value out of them.

That seems like it's intention with this aim to automate 99.9% of alignment research, right?

Because I would kind of think that thinking what things do we need in order to get an aligned AI, I would think that's a significant chunk of the difficulty of doing a lot of research.

But I think, so what I wanted to say is, so the system's adding a lot of value, but really excelling in these tasks, right?

And then what you do is you have a whole portfolio of these tasks.

Some tasks would be like, you know, write the code that implements these experiments.

And then there's another task that's like, look at the results and tell me what you see, or suggest what to do next.

And now, once you have done these tasks, you compose them using some general recipe, as people do with RGPD or language model programs, where each of the tasks is small and self-contained, and so the system doesn't need to pursue long run goals.

Or for example, the recent work that came out from OpenAI on using process-based feedback on math, where instead of training on did the system get the right solution and just doing a roll on that, you kind of train a reward model from human feedback on every step in the proof.

And it turns out that's actually much more effective because it kind of gives the AI system a much more fine-grained way to learn and more detailed feedback.

Now is that going to be competitive with, did you enter a role on did you get the solution right in the long run?

But at the very least, you can use this kind of broken down step by step setup, get the system to do a lot of really useful things that humans would have done, and then piece it together.

So, 1 of the small tasks you mentioned was look at results and decide what to do next.

I guess I would have thought that in order to do that, you have to have the big picture in mind and think, well, what next project is most useful for my goal of solving super alignment in 4 years, right?

That's right, but you wouldn't do this in the sense that you're trying to optimize and do credit assignment for 4 years.

It's probably more like, well, you're just adding some broader goals and context into your prompt.

But when you're actually doing reinforcement learning or reinforcement learning from your feedback to improve the system, then you don't have to wait until that research project concludes to decide whether or not it's good or not.

It's just like you use the human as a reward shaping.

You're just like, does this look like a good direction that seems better than any directions I could have thought of or something?

And so I think the overall goal here is not to build the most capable automated alignment researcher that we possibly could with the tech that we have, and rather build something that is really, really useful, that we can scale up a lot.

And most importantly, that we trust is aligned enough to hand off these tasks, too.

And so if we introduce a fair amount of inefficiencies in this process, where we're essentially sandbagging the model capabilities a whole bunch by training in this way.

And we're like, well, we giving it these broken down tasks and now it would execute those.

But if we train it end to end, it would be more capable or something.

So this is typically what people call the alignment tax.

And the alignment tax matters a lot if you're, let's say, competing in a market with other companies.

I'm building a chatbot or something, and My chatbot is just, it's more aligned, but it seems a lot less capable.

That would be have a hard time competing in the market.

But if you have an automated alignment researcher, the automated alignment researcher doesn't have to compete in the market, it just has to be useful to us.

And so we can get away with paying a higher tax because we just don't have a replacement.

Or the real replacement is hiring more humans, which doesn't scale that well.

I guess another way to ask the question I was going to ask previously is, what problems will you want this automated alignment researcher to solve?

I mean, ultimately, it should solve the problem of how do we align superintelligence.

So it's the idea that we solve the problems of how to align something roughly human level, And then there's additional problems as it gets smarter and it just starts taking on those.

So I imagine that an actual solution to aligning superintelligence that we and lots of other people actually truly believe in will look quite different from what we do today, right?

If you look at how chat-GBT is aligned today, it's like a lot of reinforcement learning from human feedback.

And there's a widely shared belief that I share that that will just won't scale because it fundamentally assumes that humans really understand what the system is doing in detail.

And if the system is doing a lot of alignment research, where you're thinking of millions of virtual human equivalents or something, There's no way you'll be able to look at all of it and give detailed feedback.

And it's a really difficult task and you'll miss lots of important bugs.

But the kind of techniques we're looking at right now to scale this and align a roughly human level alignment researcher that can do these difficult tasks but won't do it crazily differently that humans would, are steps or continuations of that.

Scalable oversight, for example, is a natural continuation of reinforcement learning from human feedback.

So, scalable oversight, I would define as like generally a portfolio of ideas and techniques that allow us to leverage AI to assist human evaluation on difficult tasks.

So scalable oversight is an example of a thing that you could build off of reinforcement learning from human feedback, often called RLHF.

And so typical examples of scalable oversight are debate, recursive reward modeling, iterated distillation amplification, automated market making, and so on.

And there's a bunch of ideas floating around.

But what I actually wanted to say to get back to your original question is, I think if we actually aligning super intelligence, which is this system that is vastly smarter than humans can think a lot faster and run at much larger scale, that just introduces a whole lot of other problems, especially because it will be super general, it can do lots of tasks, and then you have to figure out how to align it, not just on the much more narrow distribution of alignment researchy tasks, but everything else.

And also you want to have a much higher degree of confidence that you've actually succeeded than you would get with, let's say, a bunch of empirical evaluations.

And so I don't really know what that would look like, and I don't think anyone does, but I think it would be really exciting if we can have some formal verification in there.

Maybe if we figured out some kind of learning algorithms that has theoretical guarantees.

I don't know what even would be possible here and theoretically feasible if you have a lot of cognitive labor that you can throw at the problem.

But all of these things are very different from the kind of things that we would do right now or that we would do next.

I also don't think that roughly human level alignment researcher would start working on those problems right away.

And instead, what they would want to do is We would want them to figure out how to better align the next iteration of itself that then can work on this problem with even more brain power and then make more progress and tackle a wider range of approaches.

And so you kind of bootstrap your way up to eventually having a system that can do very different research that will then allow us to align superintelligence.

So once you have these human level AI alignment researchers running around, do you still have a job?

Does OpenAI still have a human super alignment team?

I would be excited to be replaced by AI, to be honest.

But historically, what typically happens is like what we mentioned earlier is like the AI system does 99% or 99.9% and then we just do the rest of the work.

I think also something I'm really bullish on is making sure that humans stay in the loop or stay in control over what AI systems are actually doing, even in the long run when we're long past really being able to understand everything they're doing.

And so there'll be some humans that still have to try to understand what the high level is of what AI is trying to do.

So that wouldn't necessarily have to be the super alignment team that is at OpenAI now.

It might also require a very different skill set that we have right now.

But I think ultimately humans should always stay in the loop somehow.

So, yeah, actually, speaking of, I guess, the loop, wow, that's a bad segue, but I'm going with it.

So 1 thing that OpenAI has mentioned in a few blog posts is, so firstly, there's this idea that safety is actually pretty linked to capabilities.

You need smart models to figure out the problems of alignment.

Another thing is that there's this desire to avoid fast takeoff.

I think there's this quote from Planning for AGI and Beyond that says, it's possible that AGI capable enough to accelerate its own progress could cause major changes to happen surprisingly quickly.

Then it says, we think a slower takeoff is easier to make safe.

So 1 thing I wonder is if we make this really smart, or human level alignment researcher that we then effectively 10x or 100x or something, the size of the alignment team.

Does that end up playing into this recursive self-improvement loop?

You can't have recursive self-improvement loop without also improving your alignment a lot.

I think, I mean, I personally think fast takeoff is reasonably likely and we should definitely be prepared for it to happen.

And if it doesn't happen, I'm happy about that.

But you can draw some parallels to some other machine learning projects like AlphaGo or Dota or Starcraft where the system really improved a lot week over week.

I think, Yeah, there's obviously a lot of uncertainty over what exactly happens, but I think we should definitely plan for that possibility.

And then if that does happen, a really good way to try to keep up with it is to have the automated alignment researchers that can actually do thousands of years of equivalent work within every week.

And that's just no way that that's going to happen with humans.

Now that we have a better sense of what you mean by what we're looking at with a human level automated alignment researcher, can you give a sense of what the plan is to make 1?

1 is you need a system that is smart enough to do it, and then the second part is you need to align it to actually do it.

I'm personally not working on the first 1.

I think that will definitely happen and a lot of people are working hard at making it happen.

And there's a lot of different ways that could go, but you could just picture pre-training a much larger model and eventually it will just be smart enough.

And on the second part, that's the part I'm really interested in.

And so the question is, now I have this really smart pre-trained model, how do you get it to actually do alignment research in the way that you would want it to, or more importantly or more critically, how do you know whether you can trust enough to do that?

Because if you're handing this kind of task off to AI, there's this opportunity that a smart AI system could just try to figure out how to present results and techniques to you in a way that if you then decided to implement them would actually empower the system itself or some other related system and give it more opportunity to have more power seeking behavior in the future.

Or let's say self-exfiltrate, for example.

And so What we need to do is 1, actually align it to do the task and leverage its capabilities to do it basically as well as it could.

So that is what I would call, We need a bunch of alignment training methods.

And then we also need to be able to distinguish a model that is actually aligned on the task and tries to do alignment research really well and actually help us and be honest and be truth-seeking.

And a system that pretends to be that to us, but actually wants to take over the world or self-exfiltrate or do something else.

1 is basically the truly aligned ones, and 1 is the deceptively misaligned 1.

And an important part of our job is figure out how to distinguish these 2 systems.

And that will require a bunch of methods that I would call validation methods.

So these are methods that we wouldn't really train on in the sense that the AI systems wouldn't be directly selected on passing the validation tests.

And more, they will give us some kind of independent signal of which of 1 of these 2 versions is it.

And so examples here are interpretive research, for example.

If I can look inside the model and find some evidence that it's secretly plotting against us, that will fail the validation test, right?

Another technique that could either be training or validation, depending on how it's set up, is this easy to hard generalization problem.

Can you understand and improve the model's ability to generalize from easy problems that we can supervise to harder problems that we struggle to supervise?

And if you can do that, now you can just like, you supervise the model on the easy parts, and then you're like cross-check on the difficult ones where you'd worry that the model might secretly deceive you or try to do something else.

I guess I'd like to talk about the scalable oversight part first.

Especially as background, I think we live in a world where there's not a lot of consensus on what good alignment research is.

So yeah, How are we going to get a training signal for what counts as good alignment research?

See all AXRP transcripts on Youtube

24 - Superalignment with Jan Leike