In April, we released our first AI Fundamentals episode: Benchmarks 101. We covered the history of benchmarks, why they exist, how they are structured, and how they influence the development of artificial intelligence.
Today we are (finally!) releasing Datasets 101! We’re really enjoying doing this series despite the work it takes - please let us know what else you want us to cover!
Stop me if you’ve heard this before: “GPT3 was trained on the entire Internet”.
Blatantly, demonstrably untrue: the GPT3 dataset is a little over 600GB, primarily on Wikipedia, Books corpuses, WebText and 2016-2019 CommonCrawl. The Macbook Air I am typing this on has more free disk space than that. In contrast, the “entire internet” is estimated to be 64 zetabytes, or 64 trillion GB. So it’s more accurate to say that GPT3 is trained on 0.0000000001% of the Internet.
Why spend $5m on GPU time training on $50 worth of data?
Simple: Garbage in, garbage out. No matter how good your algorithms, no matter how much money/compute you have, your model quality is strongly determined by the data you train it on and research scientists think we just don’t need or have that much high quality data. We spend an enormous amount of effort throwing out data to keep the quality high, and recently Web 2.0-era UGC platforms like StackOverflow, Reddit, and Twitter clamped down on APIs as they realize the goldmines they sit on.
Data is the new new oil. Time for a primer!
Show Notes
* The Token Crisis paper
* Ilya Sutskever on datasets 
* OpenAI Tokenizer
* Kaplan Scaling Laws Lecture
* Chinchilla Paper
* Sasha Rush’s Tweet
* Karpathy’s Build Conference Presentation
* LIMA Paper
* Phi-1 by Microsoft
* Washington Post Article on datasets
* Our episode with Jonathan Frankle
* Our episode with Mike Conover
* BloombergGPT
* Datasets
* HuggingFace Hub
* CommonCrawl, Overview
* C4
* List of Dirty, Naughty, Obscene, and Otherwise Bad Words
* OpenWebText
* books3
* OpenAssistant 
* The Stack
* The Pile
* LAION
* Audio:
* LibriSpeech: A dataset of audio recordings of audiobooks
* CommonVoice: A dataset of audio recordings of people speaking different languages
* Voxforge: A dataset of audio recordings of people speaking different languages​
* Switchboard: A dataset of audio recordings of telephone conversations​
* Fisher Corpus: A dataset of audio recordings of news broadcasts​
* Chinese:
* CMRC (Chinese Machine Reading Comprehension 2018)
* DuReader
* ChID
* Copyright & Privacy:
* https://stablediffusionlitigation.com/
* https://haveibeentrained.com/
* https://githubcopilotlitigation.com/
* https://twitter.com/moyix/status/1662131770463072257
* OpenAI Opt Out Process
* Check if you’re in The Stack
* Deduplication
* Deduplicating Training Data Makes Language Models Better
* Deduplicating Training Data Mitigates Privacy Risks in Language Models
* Contamination
* CodeForces example

 

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space

Hey everyone, welcome to the Latent Space podcast.

This is Alessio, partner and CTO of Residence at Decibel Partners.

I'm joined by my co-host, Swiggs, writer and editor of Latent Space.

Today, we finally, finally, we have the second episode of 101 Track.

So last time we did Benchmarks 101, I think we got a lot of mileage out of that.

We understood a lot about the benchmarks and we talked with a lot of our guests over the previous episodes about how they evaluate their models.

And today we wanted to dive into the datasets, what they are, how they're constructed, and why they matter.

I guess I should go into why we wanted to do this episode.

It's a little bit weird to separate datasets and benchmarks.

So we did benchmarks first, but a lot of the benchmarks were datasets.

So pretty much they're the 1 and the same thing, right?

Like, And I think where they start to diverge is a cause of significant interest.

But mostly, actually, I wanted to focus on datasets for 1 primary reason, which is that many people say that GPT is trained on all the internet.

So first of all, this is actually not true.

And second of all, it actually causes some potential misperceptions.

I say potential because there is some legitimate debate about this.

There's no misperceptions about us running out of data.

And we can discuss the pros and cons of whether or not we are running out of data.

It's been named the token crisis by academics and quite a lot of commentators on AI.

So in the show notes, we're going to link to a paper on to repeat or not to repeat insights from scaling LLMs under the token crisis.

And then I'm also going to link to an opposite view from OpenAI with Ilya Sutskever, talking about how they're not anywhere close to running on the data that they want to train on.

So whenever there's such an interesting divergence between practitioners and academics, I think it's a worthwhile thing to dive into.

And just in general, I think there's a lot of foundational knowledge that people skip over when they assume that everyone knows what datasets we're talking about.

Yeah, I was gonna say, I think also in terms of like the knowledge that the models have, if you say it's been trained on the whole internet, you would assume it knows everything on the internet, but it obviously doesn't.

You can go in there and ask about people that have online presences and are not actually in the knowledge base of the model.

And this also helps when thinking about what data to then use to fine tune.

So if you understand what's in the model, if you're trying to build a verticalized model for a specific use case, you can better figure out what's actually going to be meaningful versus what was already present in the first training run?

Yeah, just for some comparison, let's say the total size of the internet, some people have estimated at around 5 billion gigabytes.

Most of the datasets that we are going to talk about today are in the hundreds of gigabytes range and it's growing every single day.

You know, there's always new data being created every single day and there's always new modalities to claim that data from.

So, you know, a lot of the whisper behind OpenAI's whisper is that they're actually transcribing YouTube, which is a source of extra tokens.

And we'll have to explain what tokens are.

The first thing is about whether or not we're running out of data.

The second issue is this divergence between datasets and benchmarks.

And I wanted to dive into this specifically because they used to be essentially 1 and the same thing.

In a very standard machine learning tutorial, you would do something like the Iris data sets.

And then you would do train, test, validation, splits, and you would basically evaluate data based on samples from the data itself.

But more recently, we actually have decoupled benchmarking from the data sets they're trained on, except for the calculation of loss.

I think in our Discord, in Lanespace Discord, we've actually been doing a small paper club where we've gone over some of the foundational papers.

And we actually recently went over the BERT paper, which is the bidirectional Transformers paper that was a predecessor to T5 and is a predecessor to all the large language models today.

And in BERT, they actually invented this concept of masking, which meant that datasets could create their own training objectives, which I think is super interesting.

So basically what you have is, for example, a sentence.

And out of the sentence, you mask 1 word, and you ask the model to predict that word, and you grade the model based on whether or not it's able to predict that word.

This basically starts to go from supervised learning, where you have a data set that you're trying to train on, to sort of self-supervised learning, where you can just kind of let loose on unlimited set of data.

And so this basically lets you scale as much as you want on the data side, as much data as you have, which I think is just really interesting and foundational.

Like you don't have deep learning without self-supervised learning.

You don't have a good training objective until you have the concept of masking.

And once you have really, really good masking, then you start to find algorithms for that.

And it turns out that deep learning is the way to do it, to achieve loss unseen by any other algorithm.

And then once you have masking, you can predict, and then you can generate, and then everything follows from there.

But that is the data set and not a benchmark, right?

Because you can have all of Wikipedia as a data set, but nobody bothers to benchmark on Wikipedia because that's not a reasonable benchmark to derive any score on.

But it turns out that training on a data set leads to higher evaluation on benchmarks that we covered in the last episode, like common, what was it?

So I just think like, that's fascinating that we've just summed up, you know, now it seems so retroactively obvious, but we've just summed up maybe about 10 years of progress on deep learning.

And finally, the most important thing about datasets is 1, understanding how big they are and then using scaling loss to work yourself back into what size model you can train with them.

So we talked about the Chinchilla scaling loss and we'll cover that later.

But if you want to train a model that is 100 billion parameters, you cannot just pick any amount of data.

So understanding common crawl, how many tokens is that?

Helps you understand, okay, this is what I can get off the shelf.

And we'll dive into more of the data sets later.

But first I just wanted to do a quick explanation of what tokens actually are.

So the thing you read on the OpenAI docs is like 1 token is like 3 quarters of a word.

So when I first read it, I was like, oh, you're just doing character splitting, But that's not really how it works.

So basically 1 token is an integer that can be, you know, up to, I actually don't know what the highest number will be, but it's an integer representation of words and the same word can also have different representations based on where it is in a sentence.

For example, 1 of the big things that Transformers did to be more efficient is the space is included in the token.

So if you're doing a long sentence, instead of having 1 token for each space in it, The space is inside the token of the word.

So it really cuts down on how many you actually need to go to.

But the funny thing is that then you have different representation for the same word.

So if you take the word red, like the color, there's 1 token for lowercase red with a space in front, that's token 2266.

Then you have red with a space in front with a capital R, that's 2297.

Then you have red with a capital R, no space, that's token 7738.

So you can see that when you say a trillion tokens, for example, it's not 1000000000000 different English words.

So that's also 1 of the main things that you got to be careful of.

You can have a data set that is a lot of tokens, but has potentially a lot of repetition that is not as helpful.

And also when you're doing things like a logit bias in OpenAI, where you can deprioritize certain tokens, you have to find all the tokens for that.

So The example that they use is if you want to do a recipe for a cake with no eggs, you have to set the logged bias of both egg and like space egg, egg space tokens, not just 1.

So that's 1 thing to keep in mind if you're coming from a non-ML background, that's probably 1 of the first gotchas that you have when talking about datasets.

And just for more examples, OpenAI has a tokenizer tool that we're going to link in the show notes that you can use to see what any particular phrase translates to.

So I just plugged in, for example, latent space.

And then the last 1 is space and then SPACE.

So latent space is 3 tokens and some combination of that will form other words as well.

And it's an interesting tokenization scheme.

As far as I understand, by the way, Alessio, I think the upper limit on tokenizers is between 50, 000 to 80, 000 tokens, which is amazing.

It means that you can basically make up any sentence or language from these individual tokens.

It's actually like, you know, 5 digits of numbers, not millions and millions of tokens.

And keep in mind that they speak multiple languages, so you have to actually tokenize other languages as well as numbers, as well as symbols, emojis.

I actually am pretty amazed at the depth of tokenization.

And honestly, There was actually a well-known flaw with GPT-3 where you could actually do a quick test to see if you're talking to a bot or not.

1 of the reasons that GPT-3 is just not good at math is because they don't tokenize numbers individually, and they don't represent numbers the same way that humans do.

But if I type in into this tokenizer, GPT-3, like I type in 1 to this, that's 1 token.

If I type in 1, 2, which is 12, that's also a different token.

And if I type in 123, so it's 123, that's also another single token.

there is now 2 tokens that is made up of 12 and 34.

So it's, it's breaking up 1234, like an English word rather than a mathematical representation of, you know, 1, 1, 200, 3 tens and 4 ones.

And that's 1 of the reasons it doesn't do math very well, because it looks at things as though it's a word rather than numbers.

Yeah, and you mentioned the language thing.

Some languages are actually less token efficient than others.

For example, Spanish actually requires more tokens for the same sentence.

And they also have this issue where the lower the number of the token, like the more common the token is.

And if you tokenize a Spanish phrase and you have things, syllables like man in it, the token value of the token is very low because in the English language is used very often.

So you can have predictions that are a little weird when you use different languages.

And we'll kind of go into language-specific data sets later as well.

There's a famous article about this called, why is GPT-3 15.7 times more expensive for certain languages?

So I want a pizza, it's 4 tokens in English.

And then in French, I don't speak French, but je veux une pizza.

Don't actually know what these characters are.

All of this is represented within the token space that GPC has trained, and it's actually pre-trained.

This is 1 of the first examples of pre-training being useful, because ultimately, language models or the transformers that power the language models are transforming a set of numbers to a different set of numbers or predicting the next token ID in a sequence of numbers.

The transformers themselves don't actually know what word they're predicting.

All they know is they have, they're given some data sets of number after number after number, in fact like trillions of them, and then their job is to predict the next number given a sequence of numbers.

And then we take the tokenizers to convert those numbers into words or images or audio.

We referenced scaling laws a little bit, and basically this comes out of research around what the optimal sizing of a model should be for a given dataset.

I think in 2020, OpenAI published the first scaling law, which was the Kaplan paper, and that was estimated to be about 1.7 times tokens per parameter.

So what that means is the reason that GPT-3 is a 175 billion parameter model is because they had around about 300 billion tokens to train with.

So they worked backwards from the data set size that they had of 300 billion.

They said, okay, based on this, the largest possible model that we can train is 175 billion.

That was the state of the art at that time.

And we, as far as OpenAI is concerned, larger models were always going to be better because of understanding around emergence and capabilities and honestly just research around what AGI could be.

And just for reference, the GPT-2 was about 100 times smaller than that.

And then the year after that, Chinchilla came out of Google DeepMind, where they actually optimized for a different metric, which is compute optimal training.

So a given compute budget, a given amount of days and hours running a certain number of GPUs, so that gives you a certain compute budget in terms of flops.

If you hold that budget constant, so let's say that's roughly a few days or a few months of compute.

That is actually very material for a research team or anyone because that translates directly into dollars, right?

Like how much time are you renting on the shared GPUs?

So for a given compute budget, what is the best model that you can train given some kind of compute budget?

And the number that it came up with was 20 times tokens per parameter.

So that's actually 10x what the Kaplan laws were doing.

What DeepMind did was they had sort of replicas of GPC3 that they called Gopher.

And then they created another replica called Chinchilla and showed that despite being about 10x smaller, they were able to match or beat Gopher.

And the assertion, they would also beat GPT-3, despite being 10x smaller.

Most people call this is basically that GPT-3 was overparameterized.

And in fact, to train GPT-3 to a full 175 billion parameter model, you need 3.5 trillion tokens, not 300 billion, which I think is just fascinating and a little bit depressing.

It means like we just need a lot more data.

And it just shows you like how early this whole foundation model space is, you know, like these papers are not coming every 10 years, you know, they're coming like every 18, 24 months.

And the other thing you mentioned that with the cost of compute, you know, if you think that 1.7 X is good, you probably don't want to burn GPUs for like a much larger training scale, you know, and now the next, the next thing is the Lama optimal, which is 200 times tokens per parameter.

So an under the training GPT-3, you need 35 trillion tokens, which is like, if you take all the books published each year, like all of Kindle unlimited, all of like all of that stuff.

So to get to like 35 trillion, you need a lot of data.

We also don't know because it's not easy to find ways to train models at this size.

in a couple of years, we're probably going to say it was crazy that we were using like 20X, you know, it's actually like this audit right now.

1 thing I want to basically mention is that as far as I can tell the researchers that I'm following talking about this stuff, because we've asked this question on the podcast every opportunity that I've had.

And essentially, we're going from compute optimal, which is like at training time, to inference optimal, which is at inference time.

LLAMA was designed for an inference optimal situation where we start caring about the latency of the inferences.

And basically, that just means the size of the models have to be smaller.

They cannot be hundreds of billions of parameters.

And they're all round about this double-digit parameter size now, maybe even single-digit with the 7b and 3b-type models from MPT that we talked about with Jonathan Frankel.

It basically balances practical requirements.

It's very funny coming from a software background studying AI, because in software, performance means speed.

Whereas in machine learning, performance does not mean speed.

Performance means capabilities and evaluations on benchmarks, where inference time doesn't matter.

But now inference time does matter because AI is crossing over from research into software, into practical applications.

So now we do care about things like inference costs and memory that you need to run all these systems.

So I just think it's just fascinating that Lama Optimal is purposely overtraining Chinchilla.

DeepMind showed that Chinchilla is compute optimal, And now we're saying we just don't care.

We will purposely overtrain and be suboptimal there in order to be more optimal in inference, because inference is more important now.

If you're writing a paper, you're optimizing for training and building the proof.

If you're training a model for like production use case, the training cost is actually just a small part of overall lifetime cost of the model.

So the pendulum is kind of keep swinging depending on the application.

Maybe I'll go on to the next bit, which is LLMs as databases.

So we have this concept of tokens, and then we have the concept of training in the compute budget.

Basically what the training process is, you can be abstractly viewed as a way to compress the data sets.

For example, we have actually a nice conversion ratio between billions of parameters and the amount of data that they generate.

So just as a rule of thumb, let's say each parameter is 8 bits.

We always talk about FP16 in our podcast with George Hotz.

But let's just say 8 bits is 1 parameter.

Then 175 billion parameters is 175 gigabytes.

That is actually the definition of a gigabyte, which is 1 billion bytes.

And so therefore, a full point, floating point precision, 16 bits, means 175 billion parameters, uses 350 gigabytes to store parameters.

That's how much memory that you technically need to do that inference and just to load the model itself.

Most graphics cards will not even have that.

Even the professional grade, like A100 cards, would be like 80 gigabytes for a single 1 of them.

So to fit 350 gigabytes, you need to network all these things together with very high bandwidth.

But they're trained on 3, 000 gigabytes of data.

So that is 3, 000 gigabytes of data being compressed into 350 gigabytes of data.

And that is a form of compression, because from there, you can sort of, it's lossy compression, but it's a lossy compression in a way that learns how to decompress itself such that when you ask it to spit out some facts, it spits out something in the approximate neighborhood of what you, what you started with.

Some people don't like this idea that LLMs are databases, but it's super cool.

Another very prominent description of this I remember from last year, which was stable diffusion, which compresses all the images that it creates and all the images that it trains on into, I think something like 2 to 4 gigabytes.

Like it has knowledge of flying saucers and horses and humans and beaches and, you know, computers in images all in.

Like a downloadable file that you can host on your laptop, which is crazy.

Yeah, I think people get very surprised by how many things the model knows.

And if you were putting the raw, again, going back to how much memory you need, if you put the raw data in memory, it would be impossible to actually run.

But if you take, especially if you think about 16-bit versus 8, versus 4 and all the quantization work is like compressing the compression.

Now all of a sudden, you can put datasets and knowledge that was impossible to put on a certain device.

I can run the red pajama, like 2 billion model on my phone.

And my phone is like 64 gigabyte of storage.

If I were to put all the data that got into the training of it, it would be like running my phone way out of surge.

Instead, it's only like a few gigabytes of model.

So that's an interesting thing, especially as we think about using these models at the edge and how much stuff we can get there on time, but that's for another episode.

And for those who are again coming from our George Hotz episode, at the around the 1 hour mark, he talks about compressing humanity, a person's consciousness or knowledge or all your life experiences, how much information, how many bits of information is that?

He thinks it's 2 gigabytes, and that's probably too much compression.

You can probably be a really bad copy of yourself.

But there is a point at which you should probably be able to replicate yourself as a digital twin, which is the whole mind-uploading phenomenon that we discussed.

I think we sort of maybe knocked on that a little bit too much, in fact.

I want to basically verbally go through this chart of tokens in scale, because when we talk about, we say things like billions of tokens, trillions of tokens, billions of parameters.

It's just very, very big numbers that we don't know how to estimate.

And this chart that we had from this person, Srush on Twitter, actually was super helpful to me.

I was actually planning to make something like this, but This guy already made it, so I'm just going to quote him.

So maybe we'll just kind of go through the token chart.

So this is just memorized tokens in terms of orders of magnitude.

Being able to do token math, I think is super important.

And order of magnitude math is super important when it comes to deep learning, right?

See all Latent Space: The AI Engineer Podcast — CodeGen, Agents, Computer Vision, Data Science, AI UX and all things Software 3.0 transcripts on Applepodcasts

AI Fundamentals: Datasets 101