- The video discusses the concept of doing things that don't scale in software development.
- The example of Paul Buchheit, the inventor of Gmail, is given as someone who achieved success by focusing on getting 90% of the benefit with 10% of the work.
- The video also mentions how Gmail was initially developed by building on an existing Google product and gradually adding features based on user needs.
- Another example is given of Facebook in its early days, where they scaled by creating separate instances for each university and copying and pasting code, rather than trying to scale a single database for all users.

- The speaker discusses the challenges faced by Twitch in handling live video traffic.
- Normal websites have peaks and troughs of traffic, but live video sites have much larger peaks.
- Twitch had a hack where they could make a page static to handle high traffic.
- They also had to figure out how many video servers to use for popular streamers.
- The speaker shares similar experiences from Friendster and iMeme.
- Twitch had hacks for free peering relationships with ISPs and translation of their site.

- The speaker discusses the concept of "things that don't scale" and shares a story about Google's early struggles with indexing the web.
- The original Google algorithm was based on a paper called PageRank, but as the internet grew, it became harder to scale Google's search capabilities.
- At one point, a critical batch process to index the web failed, resulting in no new websites being added to Google's index for several months.
- This led Google to create MapReduce, a way to parallelize and break down the crawling and re-indexing process.
- The speaker emphasizes the importance of starting with a minimum viable product and fixing problems as they arise, rather than trying to perfect everything before launching.
- The concept of "things that don't scale" applies not only to physical tasks like Airbnb founders taking photos or DoorDash delivering, but also to software development.
- The speaker concludes by encouraging entrepreneurs to focus on giving people something they want and solving problems as they arise, rather than striving for perfection from the start.

The video discusses the concept of doing things that don't scale in software development. It gives examples of successful companies like Gmail and Facebook that focused on getting 90% of the benefit with 10% of the work. The challenges faced by Twitch in handling live video traffic are also discussed. The speaker shares experiences from Friendster and iMeme. The concept of "things that don't scale" is applied to Google's early struggles with indexing the web and the importance of starting with a minimum viable product. The speaker encourages entrepreneurs to focus on giving people something they want and solving problems as they arise.

Dalton Caldwell and Michael Seibel on software hacks that don't scale. Companies discussed include Google, Facebook, Twitch, and imeem. Watch the first video on doing things that don't scale here: https://youtu.be/4RMjQal_c4U

Chapters (Powered by https://bit.ly/chapterme-yc) - 
00:00 - Do things that don't scale
00:56 - 90/10 solution - Paul Buchheit
01:35 - Gmail
05:40 - Facebook
10:05 - Twitch part 1
12:07 - Friendster vs MySpace
15:03 - Imeem
16:59 - Twitch part 2
20:17 - Google
23:39 - Startup advice

We'll get a founder that's like, oh, how do I like test my product before I launch to make sure it's gonna work?

And I always come back and tell the founders the same thing.

Like, if you have a house and it's got full of pipes and you know some of the pipes are broken and they're gonna leak, You can spend a lot of time trying to check every pipe, guess whether it's broken or not and repair it, or you can turn the water on.

Like you'll know exactly the work to be done.

Hey, this is Michael Seibel with Dalton Caldwell.

Today we're gonna talk about what does it mean to do things that don't scale, the software edition.

In this episode we're going to go through a number of great software and product hacks that software companies used to figure out how to make their product work when perhaps they didn't have time to really build the right thing.

Now Dalton, probably the master of this is a person we work with, a guy named Paul Buchheit who in fronted this term, the 90-10 solution.

He always says something like, how can you get 90% of the benefit for 10% of the work?

This is what he always puts on to people when they tell him it's really hard to build something or it takes long to code it.

Would you say that's a fair assessment, Michael?

But tell the audience why it's worth listening to the guy.

Like why does he have the credibility to say that to people?

And as kind of a side project at Google, he invented something that 1.5 billion people on Earth actively use.

And he literally did it doing things that don't scale.

So I'll start the story and then please take it over.

So as I remember it, PB was pissed about the Gmail product, the Google product he was, sorry, the email product he was using.

And so Google had this newsletter product.

The first version of Gmail, he basically figured out how to put his email into this Google Groups UI.

And as he tells the story, kind of his eureka moment was when he could start reading his own e-mail in this UI.

Then from that point on, he stopped using his old e-mail client.

What I loved about this is that as he tells the story, every email feature that any human would want to use, he just started building from that point.

And so, you know, he would talk to the YC batch and he's like, and then I wanted to write an email.

And if you know PB, like he could have gone a couple of days reading emails without replying at all.

So he didn't need writing emails to start.

I remember him telling the first time he got his like coworker, like literally like his deskmate or something to try to use it and his desk mate's like this thing's pretty good it loads really fast it's really great the only problem is PB it has your email in it and I want to have my email and he was like oh shit okay well I gotta build that.

So then it started spreading throughout Google.

Oh, so he told this story where like 1 day PB came in late to work, which is, you know, knowing PB.

You know, and everyone was looking at him really weird.

And he got to his desk and someone came over to him and was like, don't you realize that Gmail's been down like all morning?

And PB was like, no, I just got to work, I didn't know.

And so he's like trying to fix it, trying to fix it.

And then his coworkers see him like grab a screwdriver and go to the server room.

And it was like, they were like, oh God, why do we trust PB with our email?

And I think he figured out like there was a corrupted hard drive and I remember that point of story he was like, he says, and that day I learned that people really think email is important and it's got to always work And like perfect that is

I think the reason I Think the reason he did it man is because he liked to run Linux on the desktop And he didn't want to run Outlook.

Like the Google suits were trying to get him to run Outlook on Windows.

And he was like, I don't really want to run Windows.

And as I recall in this final part of the story, it was hard for him to get Google to release it because they were afraid it was gonna take up too much hardware.

And so there was all these issues where there was a decent chance I think it never would have been released.

Well, this part was that everyone thought Gmail's like invite system was like some cool like growth hack.

You got, I think, 4 invites to give someone else.

It was just another version of things that don't scale.

servers, so they had to build an invite system.

There was not an option to not, but basically there was no option other than building an invite system.

It was not like genius PM growth hacking.

It was like, yeah, well, we saturated this.

The hard drives are full, so I guess we can't invite anyone else to Gmail today.

So you had another story about Facebook early days that is similar in this light.

Back when you started a startup a long time ago, you had to buy servers and put them in a data center, which is a special room that's air conditioned that just has other servers in it.

And you plug them in and they have fast internet access.

And so being a startup founder until AWS took off, part of the job was to drive to the suburbs or whatever, drive to some data center, which is an anonymous warehouse building somewhere, go in there and like plug things in.

And what was funny is when your site crashed, it wasn't just depressing that your site crashed, it actually entailed getting in your car.

Like Part of being a startup founder was waking up at 2 a.m.

And getting in your car and driving to like Santa Clara because your code wedged, you had to physically reboot the server.

And your site was down until you physically rebuilt it.

So anyway, I'm just trying to set the stage for people.

So this was what our life was like, okay?

And so my company, i.me, we had a data center in Santa Clara, and there was a bunch of other startups there as well.

And so something that I liked to do was to look at who my neighbors were, so to speak.

There was never people there, it was just their servers and there'd be a label at the top of the rack and you could see their servers and you could see the lights blinking on the switch.

And so our company was in the data center, in this data center in Santa Clara, and then 1 day there's a new tenant and, oh, new neighbors.

So I look at it and the label at the top of the cage next to ours, 3 feet away, the label said thefacebook.com.

I remember being like, yeah, I've heard of this.

I think there's maybe 8 of them when they first moved in, and they were like super cheap, they were like super micro servers, you know, like the wires were hanging out, like, you know, I'm like cool, but the lights were blinking really fast.

And so what I remember was that there was labels on every server and the labels were the name of a university.

And so at the time, 1 of them, 1 of the servers was named Stanford.

1 of them was named Harvard, you know, like, and it made sense because I was familiar with a Facebook product at the time, which was like a college social network that was at like 8 colleges.

So then I watched every time we would go back to the data center, they would have more servers in the rack with more colleges.

And it became increasingly obvious to me that the way they scaled Facebook was to have a completely separate PHP instance running for every school that they copy and pasted the code to.

They would have a separate MySQL server for every school, and they would have like a memcache instance for every school.

And so you'd see like the University of Oklahoma, you'd see the 3 servers next to each other, and the way that they managed to scale Facebook was to just keep buying these crappy servers, they would launch each school and it would only talk to a single school database.

And they never had to worry about scaling a database across all the schools at once.

Because again, at the time, hardware was bad.

If they had to scale a single database, a single users table to hundreds of millions of people, it would have been impossible.

And so their hack was the 9-10 solution, like PB used for Gmail, which is like, just don't do it.

And so at the time, if you were like a Harvard student and you wanted to log in, it was hard-coded to the URL was harvard.thefacebook.com, right, man?

And so if you try to go to stanford.thefacebook.com, it'd be like, you know, error.

And so then they wrote code so you could bounce between schools.

And it actually took them years to build a global users table, as I recall, and avoid this hack.

And so anyway, the thing they did that didn't scale is to copy and paste their code a lot and have completely separate database instances that didn't talk to each other.

And I'm sure people that work at Facebook today, I bet a lot of people don't even know the story.

But like that's what it took, that's the real story behind how you start something big like that versus what it looks like today.

So in the case of Twitch, all, if not all, like most of the examples of this came from this core problem, and it's why I tell people to not create a live video site.

A normal website, even a video site, on a normal day will basically have peaks and troughs of traffic.

And the largest peaks will be 2 to 4x the steady state traffic.

So you can engineer your whole product such that if we can support 2 to 4x the steady state traffic and our site doesn't go down, we're good.

On a live video product, our peaks were 20x.

Now, you can't even really test 20x peaks.

You just experience them and fix what happens when 20x more people than normally show up on your website because some pop star is streaming something.

And so 2 things kind of happened that were really fun about this.

So the first hack we had was if suddenly some famous person was streaming, on their channel there'd be a bunch of dynamic things that could load, like your username would load up on the page with their channel, and the view count would load up, and a whole bunch of other things that would basically hit our application servers and destroy them if 100,000 people were trying to request the page at the same time.

So we actually had a button that could make any page on Justin.tv a static page.

Your name wouldn't appear, the view count wouldn't update, like literally a static page that loaded our video player and you couldn't touch us.

We could just cache that static page and as many people as possible wanna look at it.

Now to them, certain things might not work right.

But they were watching the video, the chat worked, because that was a different system, the video worked, that was a different system, and we didn't have to figure out the harder problems until later.

Later, actually, Kyle and Emmett worked together to figure out how to cache parts of the page while make other parts of the page dynamic, but that happened way, way later.

Dude, that reminds me, let me give you a quick anecdote.

Every time you would log in, it would calculate how many people were 2 degrees of separation from you, and it would fire off a MySQL thread, where you would log in, it would look at your friends, and it would calculate your friends and friends and show you a live number of how big your extended network was.

And the founders, you know, John Abrams, he thought this was like a really important feature.

Guess what MySpaces do things that don't scale solution was?

they were in your friends list, it would say, so-and-so is in your friends list.

If it wasn't, it would say, so-and-so is in your extended network.

Friendster was trying to hire engineers and scale MySQL, and they're running into, like, too many threads on Linux issues, and, like, updating the kernels.

And Myspace was like, so-and-so's your extended network.

So, our second 1 was, it always happened with popular streamers.

So our second was, if you imagine, if someone is really popular and there's 100,000 people who wanna watch their stream, we actually need multiple video servers to serve all of those viewers.

So we basically propagate the original stream coming from the person streaming across multiple video servers until there was enough video servers to serve all the people who were viewing.

The challenge is that we never had a good way of figuring out how many video servers we should propagate this stream to.

And if a stream would slowly grow in traffic over time, we had a little algorithm that could work and like spin up more video servers and it'd be fine.

But what actually happened was that a major celebrity would announce they were going on and all their fans would descend on that page.

And so the second they started streaming, a hundred thousand people would be requesting the live stream.

And so we were trying to figure out solutions, solutions, solutions, and like how do we how do we model this?

How do we like there were all kinds of like overly complicated solutions we came up with.

And then, once again, Colin and I got together and they said, well, the video system doesn't know how many people are sitting on the website before the video stream, before it starts, starts trying to serve video.

All the website has to do is communicate that information to the video system and then it could pre-populate the stream to as many video servers as it would need to and then turn the stream on to users.

So what happened now in this setup is that some celebrity would start streaming, they would think they were live, no 1 was seeing their stream, while we were propagating their stream to all the video servers that are needed, and then suddenly the stream would appear for everyone and would look like it worked well.

But like dirty, super dirty, but it worked.

And honestly, that's gonna be kind of the theme of this whole setup, right?

You had a couple of these in iMeme, right?

So 1 of them, so at the time, again, to set the stage, the innovation of showing video in a browser without launching real player?

But it used to be to launch a video, it would launch another application in the browser that sucked and it would like crash your browser and you hated your life, okay?

So 1 of the cool innovations that YouTube, the startup YouTube had before it was acquired by Google was to play video in Flash in the browser that required no external dependencies.

Like it was a major product innovation to do that.

And so we wanted to do that for music at iMeme.

And we were looking at the tools available to do it.

And we saw all this great tooling to do it for video.

And so rather than rolling our own tools that was music-specific, we just took all of the open source video stuff and hacked the other video code that we had so that every music file played on iMeme was actually a video file, it was a .FLV back in the day.

And it was actually a flash video player.

And the entire, it was basically, we were playing video files that had like a 0 bit in the video field.

And we actually were transcoding uploads into video files.

Like the whole entire thing was, it was a video site with no video.

And I do think this is a recurring theme is a lot of the best product decisions are ones made kind of fast and kind of under duress.

In the office and the site's down, you tend to come up with good decisions on this stuff.

So we had 2 more at Twitch that were really funny.

The first 1, talking about duress, was our free peering hack.

So streaming live video is really expensive.

Back then it was really expensive and we were very bad fundraisers.

That was mostly my fault And so we were always in the situation we didn't have enough money to stream as much video and we had this global audience of people who wanted to watch content.

And so we actually hired 1 of the network ops guys from YouTube who had figured out how to kind of scale a lot of YouTube's early usage and he taught us that you could have free peering relationships with different ISPs around the world and so that you wouldn't have to pay a middleman to say serve video to folks in Sweden.

You can connect to yourself, your servers, you go I forgot what their

Yeah, it saves you money and it saves them money.

Yeah and there were these massive like switches where you could basically like run some wires to the switch and bam, you can connect to the Swedish ISP.

Now, the problem is that some ISPs wanted to do this free peering relationship where basically, you can send them traffic for free, they can send you traffic for free.

They didn't want to do that, or they weren't with it.

So I think it was Sweden, but I don't remember.

Some ISP was basically not allowing us to do free peering and we were spending so much money sending video to this country and we're generating no revenue from it.

It's like we couldn't make a dollar on advertising.

And so what we did is that after 10 minutes of people watching free live video, we just put up a big thing that blocked the video that said, your ISP is not doing a free peering relationship with us, so we can no longer serve you video.

If you'd like to call to complain, here's a phone number and email address." And that worked.

And I don't remember how fast, I just remember it worked.

And I remember thinking to myself, it's almost unbelievable.

Like we were like a website in San Francisco.

So we had this global audience and we would like call these translation companies and we'd ask them like, how much would it cost to translate our site into these like 40 different languages?

And we're like, we don't have infinite money.

And so I think we stole the solution from Reddit.

We were like, what happens if we just build a little website where our community translates everything?

And so basically it would just like serve up every string in English, and it was like, serve to anyone who came to the site who wasn't from an English speaking country.

And it was like, do you want to volunteer to translate the string in your local language?

Of course, people were like, well, what if they do a bad job translating?

I was like, well, the alternative is it's not in their language at all.

Let's not make the perfect enemy of the good.

I think we had something where we would get 3 different people to translate it and match, but that happened later.

But we basically got translation for a whole product for free.

Maybe to end, because I think this might be the funniest of them all, tell a Google story, because I think this one's the really like.

So look, for the Facebook story, that was firsthand where I personally witnessed the servers with my own eyes.

So I'm 100% confident that is what happened because that was me right.

This Google story is secondhand And so I may get some of the details wrong.

But I'll tell you, this was relayed to me by someone that was there.

So look, the original Google algorithm was based on a paper that they wrote, which you can go read PageRank.

They always didn't have enough hardware to scale it because remember, there was no cloud back then.

And so as the internet grew, it was harder and harder to scale Google.

Like there were just more web pages on the internet.

So it worked great when the web was small, but then they kept having more web pages really fast.

And so Google had to run as fast as they could to just stay in the same place.

Just to run a crawl and re-index the web was like a lot of work.

And so the way the work at the time is they weren't re-indexing the web in real time constantly.

They had to do it in 1 big batch process back in the day.

There was some critical point, this was probably in the 2001 era, again this is second hand, I don't know exactly when it was, but there was some critical point where this big batch process to index the web started failing.

And it would take 3 weeks to run the batch process.

It was like the, you know, reindex web.sh, you know, it was like 1 script that was like, do Google, you know, and it started failing.

And so they tried to fix the bug and they restarted it and then it failed again.

And so the story that I heard is that there was some point where for maybe 3 months, maybe 4 months, I don't remember the exact details, there was no new index of Google.

So any user of Google, they didn't know that, you know, the user didn't notice, but any user of Google was seeing stale results and no new websites were in the index for quite some time.

And so obviously they were freaking out inside of Google.

And this was the genesis for them to create MapReduce, which they wrote a paper about, which was a way to parallelize and break into pieces all the little bits of crawling and re-indexing the web.

There's a bunch of different software use and I would argue every big internet company now uses the descendants of this particular piece of software and then it was created under duress when Google secretly was completely broken for an extended period of time because the web grew too fast.

But I think this is the most fun part about this story.

When the index started getting stale, did Google shut down the search engine?

Again, in terms of do things that don't scale, did they build MapReduce before they had any users?

They basically made it this far by just building a monolithic product and they only dealt with this issue when they had to.

I think this is like such a common thing that comes up when we give startup advice.

We'll get a founder that's like, oh how do I like test my product before I launch to make sure it's gonna work.

Like if you have a house and it's got full of pipes and you know some of the pipes are broken and they're gonna leak, you can spend a lot of time trying to check every pipe, guess whether it's broken or not, and repair it, or you can turn the water on.

And like, you'll know, you'll know exactly the work to be done when you turn the water on.

See all Y Combinator transcripts on Youtube

Things That Don't Scale, The Software Edition – Dalton Caldwell and Michael Seibel