Learn about the training pipeline of GPT assistants like ChatGPT, from tokenization to pretraining, supervised finetuning, and Reinforcement Learning from Human Feedback (RLHF). Dive deeper into practical techniques and mental models for the effective use of these models, including prompting strategies, finetuning, the rapidly growing ecosystem of tools, and their future extensions.

*Speakers:*
 * Andrej Karpathy


*Session Information:*
This video is one of many sessions delivered for the Microsoft Build 2023 event. View the full session schedule and learn more about Microsoft Build at https://build.microsoft.com 

BRK216HFS | English (US) | AI


#MSBuild

Please welcome AI researcher and founding member of OpenAI, Andrej Karpathy.

I'm happy to be here to tell you about the state of GPT and more generally about the rapidly growing ecosystem of large language models.

So I would like to partition the talk into 2 parts.

In the first part, I would like to tell you about how we train GPT assistants.

And then in the second part, we are going to take a look at how we can use these assistants effectively for your applications.

So first, let's take a look at the emerging recipe for how to train these assistants, and keep in mind that this is all very new and still rapidly evolving.

But so far, the recipe looks something like this.

Now this is kind of a complicated slide, so I'm going to go through it piece by piece.

But roughly speaking, we have 4 major stages, pre-training, supervised fine-tuning, reward modeling, reinforcement learning, and they follow each other serially.

we have a data set that powers that stage.

We have an algorithm that, for our purposes, will be an objective

And then there's some notes on the bottom.

So the first stage we're going to start with is the pre-training stage.

And this diagram is not to scale, because this stage is where all of the computational work basically happens.

This is 99% of the training compute time, and also flops.

And so this is where we are dealing with internet-scale data sets, with thousands of GPUs in the supercomputer and also months of training, potentially.

The other 3 stages are fine-tuning stages that are much more along

the lines of small few number of GPUs and hours or days.

a look at the pre-training stage to achieve a base model.

First, we're going to gather a large amount of data.

Here's an example of what we call a data mixture that comes from this paper that was released by Meta, where they released this llama base model.

a web scrape, C4, which is also Common Crawl, and then some high-quality data sets

So for example, GitHub, Wikipedia, Books, Archive, Stack Exchange, and so on.

These are all mixed up together and then they are sampled according to some given proportions and that forms the training set for

Now before we can actually train on this data, we need to go through 1 more pre-processing step and that is tokenization.

And this is basically a translation of the raw text that we scrape from the internet into sequences of integers.

Because that's the native representation over which GPTs function.

Now this is a lossless kind of translation between pieces of text and tokens and integers.

And there are a number of algorithms for this stage.

Typically, for example, you could use something like byte-pairing coding, which iteratively merges little text chunks and groups them into tokens.

And so here I'm showing some example chunks of these tokens, and then this is the raw integer sequence that will actually feed into a transformer.

Now here I'm showing 2 sort of like examples for hyperparameters that govern this stage.

So GPT-4, we did not release too much information about how

it was trained and so on, so I'm using GPT-3's numbers, but GPT-3 is a little bit old by now, about 3 years ago.

So these are roughly the orders of magnitude that we're dealing with when we're doing pre-training.

The vocabulary size is usually a couple of 10, 000 tokens.

The context length is usually something like

And this governs the maximum number of integers that the GPT will look at

when it's trying to predict the next integer in a sequence.

You can see that roughly the number of parameters is, say, 65 billion for LAMA.

Now, even though LAMA has only 65b parameters compared to GVT-3's 175 billion parameters, LAMA is a significantly more powerful model, and intuitively, That's because the model is trained for significantly longer.

In this case, 1.4 trillion tokens instead of just 300 billion tokens.

a model just by the number of parameters that

Below I'm showing some tables of rough hyperparameters that typically go into specifying the transformer neural network, so the number of heads, the dimension size, number of layers, and so on.

And on the bottom, I'm showing some training hyperparameters.

So for example, to train the 65B model, Meta used 2, 000 GPUs, roughly 21 days of training, and roughly several million dollars.

And so that's the rough orders of magnitude that you should have in mind for the pre-training stage.

Now, when we're actually pre-training, what happens?

Roughly speaking, we are going to take our tokens and we're going to lay them out into data batches.

So we have these arrays that will feed into the transformer, and these arrays are B, the batch size, and these are all independent examples stacked up in rows, And B by T, T being the maximum context length.

So in my picture, I only have 10, the context length.

So this could be 2, 000, 4, 000, et cetera.

And what we do is we take these documents and we pack them into rows.

And we delimit them with these special end of text tokens, basically telling the transformer where a new document begins.

And so here I have a few examples of documents,

and then I stretched them out into this input.

to feed all of these numbers into transformer.

And Let me just focus on a single particular cell, but the same thing will happen at every cell in this diagram.

The green cell is going to take a look at all of the tokens before it, so all of the tokens in yellow, And we're going to feed that entire context into the transformer neural network.

And the transformer is going to try to predict the next token in the sequence, in this case in red.

Now the transformer, I don't have too much time to unfortunately go into the full details of this neural network architecture.

It's just a large blob of neural net stuff for our purposes, and it's got several 10 billion parameters typically or something like that And of course as you tune these parameters you're getting slightly different predicted distributions for every single 1 of these cells and So for example if our vocabulary size is 50257 tokens then we're going to have that many numbers because we need to specify a probability distribution for what comes next.

So basically, we have a probability for whatever may follow.

Now, in this specific example for this specific cell, 513 will come next.

And so we can use this as a source of supervision to update our transformer's weights.

And so we're applying this basically on every single cell in parallel.

to make the correct predictions over what token comes next in a sequence.

So let me show you more concretely what this looks like when you train 1 of these models.

This is actually coming from the New York Times.

And they trained a small GPT on Shakespeare.

And so here's a small snippet of Shakespeare, and they trained a GPT on it.

Now, in the beginning, at initialization, the GPT starts with completely random weights.

So you're just getting completely random outputs as well.

But over time, as you train the GPT longer and longer, you are getting more and more coherent and consistent sort of samples from the model.

And the way you sample from it, of course, is you predict what comes next.

You sample from that distribution, and you keep feeding that back into the process.

And you can basically sample large sequences.

And so by the end, you see that the transformer has learned about words, and

where to put spaces, and where to put commas, and so on.

And so we're making more and more consistent predictions over time.

These are the kinds of plots that you're looking at when you're doing model pre-training.

Effectively, we're looking at the loss function over time

And low loss means that our transformer is predicting the correct, is giving a higher probability to the correct next integer in the sequence.

Now, what are we going to do with this model once we've trained it after a month?

Well, the first thing that we noticed, the field, is that these models basically, in the process of language modeling, learn very powerful general representations, and it's possible to very efficiently fine-tune them for any arbitrary downstream task you might be interested in.

So as an example, if you're interested in sentiment classification, The approach used to be that you collect a bunch of positives and negatives and then you train some kind of an NLP model for that.

But the new approach is ignore sentiment classification, go off and do large language model pre-training, train a large transformer, and then you may only have a few examples, and you can very efficiently fine tune your model for that task.

And the reason for this is that basically the transformer is forced to multitask a huge amount of tasks in the language modeling task.

Because just in terms of predicting the next token, it's forced to understand a lot about

the structure of the text and all the different concepts therein.

Now, around the time of GPT-2, people noticed that actually even better than fine tuning, you can actually prompt these models very effectively.

So these are language models, and they want to complete documents.

So you can actually trick them into performing tasks just by arranging these fake documents.

So in this example, for example, we have some passage, and then we sort of do QA, QA, QA.

And then as the transformer is trying to complete the document, it's actually answering our question.

And so this is an example of prompt engineering a base model, making it believe that it's sort of imitating a document and getting it to perform a task.

And so this picked off, I think, the era of, I would say, prompting over fine-tuning and seeing that this actually can work extremely well on a lot of problems, even without training any neural networks, fine-tuning, or so on.

Now, since then, we've seen an entire evolutionary tree of base models that everyone has trained.

For example, the GPT-4 base model was never released.

The GPT-4 model that you might be interacting with over API is not a base model.

And we're going to cover how to get those in a bit.

GPT-3 base model is available via the API under the name DaVinci.

And GPT-2 base model is available even as weights on our GitHub repo.

But currently the best available base model probably is the llama series from meta, Although it is not commercially licensed.

Now 1 thing to point out is base models are not systems.

They don't want to make answers to your questions.

So if you tell them write a poem about the bread and cheese, it will answer questions with more questions.

It's just completing what it thinks is a document.

in a specific way for base models that is more likely to work.

So as an example, here's a point about bread and cheese.

And In that case, it will autocomplete correctly.

You can even trick base models into being assistants.

And the way you would do this is you would create a specific few-shot prompt that makes it look like there's some kind of

a document between a human and assistant, and they're exchanging information.

at the bottom, you sort of put your query at

the end, and the base model will sort of condition itself into being a helpful assistant and kind of answer.

But this is not very reliable and doesn't work super well in practice, although it can be done.

So instead, we have a different path to make actual GPT assistance, not just base model document completers.

And so that takes us into supervised fine tuning.

So in the supervised fine tuning stage, we are going to collect small but high quality data sets.

case we're going to ask human contractors to gather data of the form prompt and ideal response.

And we're going to collect lots of these, typically tens of thousands or something like that.

And then we're going to still do language modeling on this data.

So it used to be internet documents, which is

a high-quantity, low-quality for basically QA prompt response kind of data.

And then after training, we get an SFD model.

And you can actually deploy these models and they are actual assistants and they work to some extent.

Let me show you what an example demonstration might look like.

So here's something that a human contractor might come up with.

Can you write a short introduction about the relevance of the term monopsony or something like that, and then the contractor also writes out an ideal response.

And when they write out these responses, they are following extensive labeling documentations, and they are being asked to be helpful, truthful, and harmless.

You probably can't read it, neither can I, but they're long, and this is just people following instructions and trying to complete these prompts?

So that's what the data set looks like, and you can train these models.

Now, you can actually continue the pipeline from here on and go into RLHF, reinforcement learning from human feedback, that consists of both reward modeling and reinforcement learning.

So let me cover that, and then I'll come back to why you

may want to go through the extra steps and how that compares to just SFD models.

the reward modeling step, what we're going to do is we're now going to shift our data collection to be of the form of comparisons.

So here's an example of what our data set will look like.

I have the same prompt, identical prompt on the top, which is asking the assistant to write a program or a function that checks if a given string is a palindrome.

And then what we do is we take the SFD model, which we've already trained, and we create multiple completions.

So in this case, we have 3 completions that the model has created.

And then we ask people to rank these completions.

a while, and by the way, these are very difficult things to do to compare some of these predictions.

And this can take people even hours for a single prompt completion pairs.

But let's say we decided that 1 of these is much better than the others, and so on.

can follow that with something that looks very much kind of like a binary classification on all the possible pairs between these completions.

So what we do now is we lay out our prompt in

rows, and the prompt is identical across all 3 rows here.

And so the yellow tokens are coming from the SFT model.

Then what we do is we append another special reward readout token at the end.

And we basically only supervise the transformer at this single green token.

And the transformer will predict some reward for how good that completion is for that prompt.

And so basically it makes a guess about the quality of each completion.

And then once it makes a guess for every 1 of them, We also have the ground truth, which is telling us the ranking of them.

And so we can actually enforce that some of these numbers should be much higher than others and so on.

We formulate this into a loss function, and

we train our model to make reward predictions that are consistent with the ground truth coming from the comparisons from all these contractors.

Once we have a reward model, we can't deploy this because this is not very useful

as an assistant by itself, but it's very useful for the reinforcement learning stage that follows now.

Because we have a reward model, we can score the quality of any arbitrary completion for any given prompt.

So what we do during reinforcement learning is we basically get, again, a large collection of prompts.

And now we do reinforcement learning with respect to the reward model.

We take a single prompt, we lay it out in rows, and now we use the SFT model.

We use basically the model we'd like to train, which is initialized at SFT model to create some completions in yellow.

And then we append the reward token again, and we read off the reward according to the reward model, which is now kept fixed.

And now the reward model tells us the quality of every single completion for these prompts.

And so what we can do is we can now just basically apply the same language modeling loss function, but we're currently training on the yellow tokens, and we are weighing the language modeling objective by the rewards indicated by the reward model.

the reward model said that This is a fairly high scoring completion.

And so all of the tokens that we happened to sample on the first row are going to get reinforced and they're going to get higher probabilities for the future.

Conversely, on the second row, the reward model really did not like this completion, negative 1.2.

so therefore, every single token that we sampled in that second row is going to

get a slightly higher probability for the future.

And we do this over and over on many prompts, on many batches.

And basically, we get a policy which creates yellow tokens here.

of the completions here will score high according to the reward model

And then at the end, you've got a model that you could deploy.

And as an example, chat GPT is an RLHF model.

But some other models that you might come across,

like for example the Kuna 13B and so on, these are SFD models.

RLHF models, and that's kind of like the state of things there.

So this comes from the InstructGBT paper.

According to these experiments a while ago now, these PPO models are RLHF.

And we see that they are basically just preferred in a lot of comparisons when we give them to humans.

So humans just prefer basically tokens that come from RLHF models compared to SFT models, compared to base model

But you might ask why, Why does it work better?

And I don't think that there's a single amazing answer that the community has really agreed on.

But I will just offer 1 reason, potentially.

to do with the asymmetry between how easy computationally it is to compare versus generate.

So let's take an example of generating a haiku.

Suppose I ask a model to write a haiku about paperclips.

If you're a contractor trying to give training data, then imagine being a contractor collecting basically data for the SFT stage.

How are you supposed to create a nice haiku for a paperclip.

But if I give you a few examples of haikus, you might be able to appreciate some of these haikus a lot more than others.

And so judging which 1 of these is good is a much easier task.

And so basically, this asymmetry makes it so that comparisons are a better way to potentially leverage yourself as

Now RLHF models are not strictly an improvement on the base models in some cases.

See all Microsoft Developer transcripts on Youtube

State of GPT | BRK216HFS