In this deeply informative video, Jeremy Howard, co-founder of fast.ai and creator of the ULMFiT approach on which all modern language models (LMs) are based, takes you on a comprehensive journey through the fascinating landscape of LMs. Starting with the foundational concepts, Jeremy introduces the architecture and mechanics that make these AI systems tick. He then delves into critical evaluations of GPT-4, illuminates practical uses of language models in code writing and data analysis, and offers hands-on tips for working with the OpenAI API. The video also provides expert guidance on technical topics such as fine-tuning, decoding tokens, and running private instances of GPT models.

As we move further into the intricacies, Jeremy unpacks advanced strategies for model testing and optimization, utilizing tools like GPTQ and Hugging Face Transformers. He also explores the potential of specialized datasets like Orca and Platypus for fine-tuning and discusses cutting-edge trends in Retrieval Augmented Generation and information retrieval. Whether you're new to the field or an established professional, this presentation offers a wealth of insights to help you navigate the ever-evolving world of language models.

(The above summary was, of course, created by an LLM!)

For the notebook used in this talk, see https://github.com/fastai/lm-hackers.

00:00:00 Introduction & Basic Ideas of Language Models
00:18:05 Limitations & Capabilities of GPT-4
00:31:28 AI Applications in Code Writing, Data Analysis & OCR
00:38:50 Practical Tips on Using OpenAI API
00:46:36 Creating a Code Interpreter with Function Calling
00:51:57 Using Local Language Models & GPU Options
00:59:33 Fine-Tuning Models & Decoding Tokens
01:05:37 Testing & Optimizing Models
01:10:32 Retrieval Augmented Generation
01:20:08 Fine-Tuning Models
01:26:00 Running Models on Macs
01:27:42 Llama.cpp & Its Cross-Platform Abilities

This is an extended version of the keynote given at posit::conf(2023). Thanks to @wolpumba4099 for chapter titles.

Hi, I am Jeremy Howard from fast.ai, and this is a hacker's guide to language models.

When I say a hacker's guide, what we're going to be looking at is a code-first approach to understanding how to use language models in practice.

So before we get started, we should probably talk about what is a language model.

I would say that this is going to make more sense if you know the kind of basics of deep learning.

If you don't, I think you'll still get plenty out of it and there'll be plenty of things you can do.

But if you do have a chance, I would recommend checking out course.fast.ai, which is a free course.

And specifically, if you could at least kind of watch, if not work through the first 5 lessons, that would get you to a point where you understand all the basic fundamentals of deep learning that will make this lesson tutorial make even more sense.

Maybe I shouldn't call this a tutorial, it's more of a quick run-through.

So I'm going to try to run through all the basic ideas of language models, how to use them, both open source ones and open AI-based ones.

And it's all going to be based using code as much as possible.

So let's start by talking about what a language model is.

And so as you might have heard before, a language model is something that knows how to predict the next word of a sentence or knows how to fill in the missing words of a sentence.

OpenAI has a language model, TextDaVinci003, and we can play with it by passing in some words and ask it to predict what the next words might be.

So if we pass in, when I arrived back at the panda breeding facility after the extraordinary reign of live frogs.

I just came up with that yesterday and I thought what might happen next.

So kind of fun for creative brainstorming.

Nat.dev lets us play with a variety of language models, and here I've selected Text Da Vinci 003, and I'll hit submit, and it starts printing stuff out.

The pandas were happily playing and eating the frogs that had fallen from the sky.

It's an amazing sight to see these animals take advantage of such a unique opportunity.

The staff took quick measures to ensure the safety of the pandas and the frogs.

So there you go, that's what happened after the extraordinary rain of live frogs at the panda breeding facility.

You'll see here that I've enabled show probabilities, which is a thing in that.dev where it shows, well let's take a look.

It's pretty likely the next word here is going to be the, and after the, since we're talking about a panda breeding facility, it's going to be pandas were.

Well, they could have been doing a few things.

They could have been doing something happily, or the pandas were having, the pandas were out, the pandas were playing.

It thought it was 20% likely it was going to be happily.

Could have been playing, hopping, eating, and so forth.

So they're eating the frogs that, and then had almost certainly.

So you can see what it's doing at each point is it's predicting the probability of a variety of possible next words.

And depending on how you set it up, it will either pick the most likely 1 every time, or you can change, muck around with things like p values and temperatures to change what comes up.

So at each time then it'll give us a different result.

Frogs perched on the heads of some of the pandas.

Now you might notice here it hasn't predicted pandas, it's predicted panned.

So you can see that it's not always predicting words.

Specifically what it's doing is predicting tokens.

Tokens are either whole words or subword units, pieces of a word, or it could even be punctuation or numbers or so forth.

So for example, we can use the actual, it's called tokenization, to create tokens from a string.

We can use the same tokenizer that GPT uses by using TickToken, and we can specifically say we want to use the same tokenizer that that model text eventually 003 uses.

And so, for example, when I earlier tried this, it talked about the frog splashing, and so I thought we'll encode they are splashing.

And what those numbers are, they're basically just lookups into a vocabulary that OpenAI in this case created.

And if you train your own models, you'll be automatically creating or your code will create.

And if I then decode those, it says, oh, these numbers are they space are space spl ashing.

And so put that all together, they are splashing.

So you can see that the start of a word is, with a space before it, is also being encoded here.

So these Language models are quite neat, that they can work at all.

But they're not of themselves really designed to do anything.

The basic idea of what chat GPT, GPT-4, BART, etc., are doing comes from a paper which describes an algorithm that I created back in 2017 called ULMfit.

And Sebastian Ruder and I wrote a paper up describing the ULMfit approach, which was the 1 that basically laid out what everybody's doing, how this system works.

But you'll see this is actually from the paper.

We actually described it as pre-training.

Now what language model pre-training does is this is the thing which predicts the next word of a sentence.

And so in the original ULM fit paper, So the algorithm I developed in 2017, and then Sebastian Ruder and I wrote it up in 2018, early 2018.

What I originally did was I trained this language model on Wikipedia.

Now what that meant is I took a neural network, and a neural network is just a function.

If you don't know what it is, it's just a mathematical function that's extremely flexible, and it's got lots and lots of parameters.

Initially it can't do anything, but using stochastic gradient descent, or SGD, you can teach it to do almost anything if you give it examples.

So I gave it lots of examples of sentences from Wikipedia.

So for example, from the Wikipedia article for the birds, the birds is a 1963 American natural horror thriller film produced and directed by Alfred, and then it would stop.

And so then the model would have to guess what the next word is.

And if it guessed Hitchcock, it would be rewarded, and if it guessed something else, it would be penalized.

And effectively, basically, it's trying to maximize those rewards.

It's trying to find a set of weights for this function that makes it more likely that it would predict Hitchcock.

And then later on in this article, it reads from Wikipedia, Any previously dated Mitch, but ended it due to Mitch's cold overbearing mother Lydia, who dislikes any woman in Mitch's.

Now you can see that filling this in actually requires being pretty thoughtful, because there's a bunch of things that like kind of logically could go there.

Like a woman could be in Mitch's closet, could be in Mitch's house.

And so, you know, you could probably guess in the Wikipedia article describing the plot of the birds, it's actually any woman in Mitch's life.

Now to do a good job of solving this problem as well as possible, of guessing the next word of sentences, The neural network is going to have to learn a lot of stuff about the world.

It's going to learn that there are things called objects, that there's a thing called time, that objects react to each other over time, That there are things called movies, that movies have directors, that there are people, that people have names, and so forth.

And that a movie director is Alfred Hitchcock, and he directed horror films, and So on and so forth.

It's going to have to learn an extraordinary amount if it's going to do a really good job of predicting the next word of sentences.

Now these neural networks specifically are deep neural networks.

And in these deep neural networks, which have, when I created this, I think it had like 100 million parameters.

Nowadays, they have billions of parameters.

It's got the ability to create a rich hierarchy of abstractions and representations, which it can build on.

And so this is really the key idea behind neural networks and language models.

It's that if it's gonna do a good job of being able to predict the next word of any sentence in any situation, it's going to have to know an awful lot about the world, it's going to have to know about how to solve math questions or figure out the next move in a chess game, or recognize poetry, and so on and so forth.

Now nobody said it's going to do a good job of that.

So it's a lot of work to find, to create and train a model that is good at that.

But if you can create 1 that's good at that, it's gonna have a lot of capabilities internally that it would have to be drawing on to be able to do this effectively.

So the key idea here for me is that this is a form of compression.

And this idea of the relationship between compression and intelligence goes back many, many decades.

And the basic idea is that, yeah, if you can guess what words are coming up next, then effectively you're compressing all that information down into a neural network.

Well, we do it because we want to pull out those capabilities.

And the way we pull out those capabilities is we take 2 more steps.

The second step is we do something called language model fine-tuning.

And in language model fine-tuning, We are no longer just giving it all of Wikipedia, or nowadays we don't just give it all of Wikipedia, but in fact a large chunk of the internet is fed to pre-training these models.

In the fine-tuning stage, we feed a set of documents a lot closer to the final task that we want the model to do.

It's still trying to predict the next word of a sentence.

After that, we then do a final classifier fine-tuning.

And then the classifier fine-tuning, this is the kind of end task we're trying to get it to do.

Now, nowadays, these 2 steps are very specific approaches are taken.

For the step 2, the step B, the language model fine tuning, people nowadays do a particular kind called instruction tuning.

The idea is that the task we want most of the time to achieve is solve problems, answer questions.

And so in the instruction tuning phase, we use datasets like this 1.

This is a great dataset called OpenOrca, created by a fantastic open source group.

And it's built on top of something called the Flan Collection.

And you can see that basically there's all kinds of different questions in here.

So there's 4 gigabytes of questions and context and so forth.

And each 1 generally has a question or an instruction or a request and then a response.

I think this is from the Flan dataset if I remember correctly.

So for instance, it could be, does the sentence in the Iron Age answer the question, the period of time from 1200 to 1000 BCE is known as what?

And then the language model is meant to write 1 or 2 as appropriate for yes or no.

Or it could be things about, I think this is from a music video, who is the girl in more than you know, answer, and then it would have to write the correct name of the, I can't remember, model or dancer or whatever, from that music video.

So fine-tuning and pre-training are kind of the same thing.

But this is more targeted now, not just to be able to fill in the missing parts of any document from the internet, but to fill in the words necessary to answer questions, to do useful things.

And then step 3, which is the classifier fine tuning.

Nowadays there's generally various approaches such as reinforcement learning from human feedback and others, which are basically giving humans or sometimes more advanced models, multiple answers to a question such as, here are some from a reinforcement learning from human feedback paper, I can't remember which 1 I got it from, list 5 ideas for how to regain enthusiasm for my career.

And so the model will spit out 2 possible answers, or it will have a less good model and a more good model, and then a human or a better model will pick which is best.

And so that's used for the final fine-tuning stage.

So all of that is to say, although you can download pure language models from the internet, they're not generally that useful on their own until you've fine-tuned them.

Now you don't necessarily need step C nowadays.

Actually people are discovering that maybe just step B might be enough.

So when we talk about a language model, where we could be talking about something that's just been pre-trained, something that's been fine-tuned, or something that's gone through something like RLHF, All of those things are generally described nowadays as language models.

So my view is that if you are going to be good at language modeling in any way, then you need to start by being a really effective user of language models.

And to be a really effective user of language models, you've got to use the best 1 that there is.

September 2023, the best 1 is by far GPT-4.

This might change sometime in the not too distant future, but this is right now, GPT-4 is the recommendation, strong, strong recommendation.

Now you can use GPT-4 by paying 20 bucks a month to open AI, and then you can use it a whole lot.

It's very hard to run out of credits, I find.

It's interesting and instructive, in my opinion, to start with the very common views you see on the internet or even in academia about what it can't do.

So for example, there was this paper you might have seen, GPT-4 can't reason, which describes a number of empirical analysis done of 25 diverse reasoning problems and found that it was not able to solve them.

So I always find you've got to be a bit careful about reading stuff like this, because I just took the first 3 that I came across in that paper, and I gave them to GPT-4.

And by the way, something very useful in GPT-4 is you can click on the share button and you'll get something that looks like this.

So here's an example of something from the paper that said GPT-4 can't do this.

And GPT-4 says, this appears to be a riddle, not a real inquiry into medical conditions.

And yeah, it sounds like Mabel was alive at noon.

This was the second 1 I tried from the paper that says GPT-4 can't do this, and I found actually GPT-4 can do this.

And it said that GPT-4 can't do this, and I found GPT-4 can do this.

Now, I mention this to say GPT-4 is probably a lot better than you would expect if you've read all this stuff on the internet about all the dumb things that it does.

Almost every time I see on the internet saying something that GPT-4 can't do, I check it and it turns out it does.

And so GPT-4 says, okay, Sally's counted as 1 sister by each of her brothers.

If each brother has 2 sisters, That means there's another sister in the picture apart from Sally.

And then this 1 I got sort of like 3 or 4 days ago.

This is a common view that language models can't track things like this.

On top of my chair in the living room is a coffee cup.

I move the chair to the bedroom, I put the coffee cup on the bed, I turn the cup upside down, then I return it upside up, place the coffee cup on the counter in the kitchen, where's my diamond?

And so GPT-4 says, yeah, okay, you turned it upside down, So probably the diamond fell out.

So therefore the diamond is in the bedroom where it fell out.

Why is it that people are claiming that GPT-4 can't do these things, and it can?

Well the reason is because I think on the whole they are not aware of how GPT-4 was trained.

GPT-4 was not trained at any point to give correct answers.

GPT-4 was trained initially to give most likely next words.

And there's an awful lot of stuff on the internet where documents are not describing things that are true.

There could be just stupid people saying dumb stuff.

So this first stage does not necessarily give you correct answers.

The second stage with the instruction tuning, also, like, it's trying to give correct answers, but part of the problem is that then in the stage where you start asking people which answer do they like better, people tended to say in these things that they prefer more confident answers, and they often were not people who were trained well enough to recognize wrong answers.

So there's lots of reasons that the, that the, you know, SGD weight updates from this process for stuff like GPT-4 don't particularly, or don't entirely reward correct answers.

But you can help it want to give you correct answers.

If you think about the LM pre-training, what are the kinds of things in a document that would suggest, oh, this is going to be high-quality information.

And so you can actually prime GPT-4 to give you high-quality information by giving it custom instructions.

And what this does is this is basically text that is prepended to all of your queries.

And so you say, like, oh, you're brilliant at reasoning.

So like, okay, that's self-assured to prime it to give good answers.

And then try to work against the fact that the RLHF folks preferred confidence.

No, tell me if there might not be a correct answer.

Also, the way that the text is generated is it literally generates the next word and then it puts all that whole lot back into the model and generates the next next word, puts that all back in the model, generates the next next next word, and so forth.

That means the more words it generates, the more computation it can do.

And so I say, first spend a few sentences explaining background context, etc.

So this custom instruction allows it to solve more challenging problems.

Here's what it looks like, for example, if I say, how do I get a count of rows grouped by value in pandas?

And it just gives me a whole lot of information, which is actually it thinking.

So I just skip over it, and then it gives me the answer.

And actually in my custom instructions, I actually say if the request begins with VV, actually make it as concise as possible.

This is the same thing, but with the V at the start.

Now in this case, it's a really simple question.

So hopefully that gives you a sense of how to get language models to give good answers.

And if you, if it's not working, it might be user error, basically.

But having said that, there's plenty of stuff that language models like GPT-4 can't do.

1 thing to think carefully about is does it know about itself?

Can you ask it What is your context length?

What transformer architecture are you based on?

At any 1 of these stages, did it have the opportunity to learn any of those things?

Well, obviously not at the pre-training stage.

Nothing on the internet existed during GPT-4's training saying how GPT-4 was trained.

Probably ditto in the instruction tuning, probably ditto in the RLHF.

So in general, you can't ask, for example, a language model about itself.

Now again, because of the RLHF, it'll want to make you happy by giving you opinionated answers, so it'll just spit out the most likely thing It thinks with great confidence.

This is just a general kind of hallucination.

So hallucinations is just this idea that the language model wants to complete the sentence, and it wants to do it in an opinionated way that's likely to make people happy.

I think a lot of them, if not all of them, pretty much were stripped out.

So if you ask it anything about, like, what's at this web page, again, It'll generally just make it up.

And it doesn't know, at least GPT-4 doesn't know anything after September 2021, because the information it was pre-trained on was from that time period, September 2021 and before, called the knowledge cutoff.

Steve Newman sent me this good example of something that it can't do.

I need to carry a cabbage, a goat, and a wolf across a river.

How do I get everything across to the other side?

Now the problem is, this looks a lot like something called the classic river crossing puzzle.

So classic in fact that it has a whole Wikipedia page about it.

And in the classic puzzle the wolf would eat the goat or the goat would eat the cabbage.

The goat would eat the cabbage and the wolf would eat the cabbage, but the wolf won't eat the goat.

Well, very interestingly, GPT-4 here is entirely overwhelmed by the language model training.

It's seen this puzzle so many times, it knows what word comes next.

So it says, oh yeah, I take the goat across the road, across the river, and leave it on the other side.

Leaving the wolf with a cabbage, But we're just told you can't leave the wolf with a cabbage.

Now the thing is though, you can encourage GPT-4 or any of these language models to try again.

So during the instruction tuning in RLHF, they're actually fine-tuned with multi-stage conversations.

So you can give it a multi-stage conversation.

Repeat back to me the constraints I listed.

My new attempt, instead of taking the goat across the river and leaving it on the other side, is I'll take the goat across the river and leave it on the other side.

Instead of taking the goat across the other side, I'll take the goat across the other side.

Well, 1 of the problems here is that not only is on the internet, it's so common to see this particular goat puzzle, that it's so confident it knows what the next word is.

See all Jeremy Howard transcripts on Youtube

A Hackers' Guide to Language Models