Cristos Goodrow is VP of Engineering at Google and head of Search and Discovery at YouTube (aka YouTube Algorithm).

This episode is presented by Cash App. Download it & use code "LexPodcast":
Cash App (App Store): https://apple.co/2sPrUHe
Cash App (Google Play): https://bit.ly/2MlvP5w

PODCAST INFO:
Podcast website:
https://lexfridman.com/podcast
Apple Podcasts:
https://apple.co/2lwqZIr
Spotify:
https://spoti.fi/2nEwCF8
RSS:
https://lexfridman.com/feed/podcast/
Full episodes playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOdP_8GztsuKi9nrraNbKKp4
Clips playlist:
https://www.youtube.com/playlist?list=PLrAXtmErZgOeciFP3CBCIEElOJeitOr41

OUTLINE:
0:00 - Introduction
3:26 - Life-long trajectory through YouTube
7:30 - Discovering new ideas on YouTube
13:33 - Managing healthy conversation
23:02 - YouTube Algorithm
38:00 - Analyzing the content of video itself
44:38 - Clickbait thumbnails and titles
47:50 - Feeling like I'm helping the YouTube algorithm get smarter
50:14 - Personalization
51:44 - What does success look like for the algorithm?
54:32 - Effect of YouTube on society
57:24 - Creators
59:33 - Burnout
1:03:27 - YouTube algorithm: heuristics, machine learning, human behavior
1:08:36 - How to make a viral video?
1:10:27 - Veritasium: Why Are 96,000,000 Black Balls on This Reservoir?
1:13:20 - Making clips from long-form podcasts
1:18:07 - Moment-by-moment signal of viewer interest
1:20:04 - Why is video understanding such a difficult AI problem?
1:21:54 - Self-supervised learning on video
1:25:44 - What does YouTube look like 10, 20, 30 years from now?

CONNECT:
- Subscribe to this YouTube channel
- Twitter: https://twitter.com/lexfridman
- LinkedIn: https://www.linkedin.com/in/lexfridman
- Facebook: https://www.facebook.com/LexFridmanPage
- Instagram: https://www.instagram.com/lexfridman
- Medium: https://medium.com/@lexfridman
- Support on Patreon: https://www.patreon.com/lexfridman

The following is a conversation with Christos Gudro, Vice President of Engineering at Google and Head of Search and Discovery at YouTube, also known as the YouTube Algorithm.

YouTube has approximately 1.9 billion users, and every day people watch over 1 billion hours of YouTube video.

It is the second most popular search engine behind Google itself.

For many people, it is not only a source of entertainment, but also how we learn new ideas from math and physics videos to podcasts, to debates, opinions, ideas, from out of the box thinkers and activists on some of the most tense, challenging, and impactful topics in the world today.

YouTube and other content platforms receive criticism from both viewers and creators, as they should, because the engineering task before them is hard, and they don't always succeed, and the impact of their work is truly world-changing.

To me, YouTube has been an incredible wellspring of knowledge.

I've watched hundreds, if not thousands of lectures that changed the way I see many fundamentals ideas in math, science, engineering, and philosophy.

mirror to ourselves and keeps the responsibility of the steps we take in each of our online educational journeys into the hands of each of us.

The YouTube algorithm has an important role in that journey of helping us find new, exciting ideas to learn about.

That's a difficult and an exciting problem for an artificial intelligence system.

As I've said in lectures and other forums, recommendation systems will be 1 of the most impactful areas of AI in the 21st century.

And YouTube is 1 of the biggest recommendation systems in the world.

This is the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, give it 5 stars on Apple Podcast, follow on Spotify, support it on Patreon, or simply connect with me on Twitter, Alex Friedman, spelled F-R-I-D-M-A-N.

I recently started doing ads at the end of the introduction.

I'll do 1 or 2 minutes after introducing the episode and never any ads in the middle that can break the flow of the conversation.

I hope that works for you and doesn't hurt the listening experience.

This show is presented by Cash App, the number 1 finance app in the App Store.

I personally use Cash App to send money to friends, but you can also use it to buy, sell, and deposit Bitcoin in just seconds.

Cash App also has a new investing feature.

You can buy fractions of a stock, say $1 worth, no matter what the stock price is.

Broker services are provided by Cash App Investing, a subsidiary of Square and member SIPC.

I'm excited to be working with Cash App to support 1 of my favorite organizations called FIRST, best known for their FIRST Robotics and Lego competitions.

They educate and inspire hundreds of thousands of students in over 110 countries and have a perfect rating at Charity Navigator, which means that donated money is used to maximum effectiveness.

When you get Cash App from the App Store or Google Play and use code LEXPODCAST, you'll get $10 and Cash App will also donate $10 to FIRST, which again is an organization that I've personally seen inspire girls and boys to dream of engineering a better world.

And now here's my conversation with Christos

YouTube is the world's second most popular search engine, behind Google of course.

We watch more than 1 billion hours of YouTube videos a day, more than Netflix and Facebook video combined.

YouTube creators upload over 500,000 hours of video every day.

Average lifespan of a human being just for comparison is about 700,000 hours.

So what's uploaded every single day is just enough for a human to watch in a lifetime.

So let me ask an absurd philosophical question.

If from birth, when I was born, and there's many people born today with the internet, I watched YouTube videos nonstop.

Do you think there are trajectories through YouTube video space that can maximize my average happiness or maybe education or my growth as a human being?

I think there are some great trajectories through YouTube videos, but I wouldn't recommend that anyone spend all of their waking hours or all of their hours watching YouTube.

I mean, I think about the fact that YouTube has been really great for my kids, for instance.

My oldest daughter, you know, she's been watching YouTube for several years.

She watches Tyler Oakley and the Vlogbrothers.

And I know that it's had a very profound and positive impact on her character.

And my younger daughter, she's a ballerina, and her teachers tell her that YouTube is a huge advantage for her because she can practice a routine and watch professional dancers do that same routine and stop it and back it up and rewind and all that stuff.

And then even my son is a sophomore in college.

He got through his linear algebra class because of a channel called 3 Blue 1 Brown, which helps you understand linear algebra, but in a way that would be very hard for anyone to do on a whiteboard or a chalkboard.

And so I think that those experiences, from my point of view, were very good.

And so I can imagine really good trajectories through YouTube, yes.

Have you looked at, do you think of broadly about that trajectory over a period?

So over a period of years, you just kind of gave a few anecdotal examples, but you know, I used to watch certain shows on YouTube.

I mean, ultimately you want people to, from YouTube's perspective to stay on YouTube, to grow as human beings on YouTube.

So you have to think not just what makes them engage today or this month, but also for a period of years.

I mean, if YouTube is going to continue to enrich people's lives, then, you know, then it has to grow with them.

And so I think we've been working on this problem, and I'll just say it broadly as like, how to introduce diversity and introduce people who are watching 1 thing to something else they might like.

We've been working on that problem all the 8 years I've been at YouTube.

It's a hard problem because, I mean, of course, it's trivial to introduce diversity that doesn't help.

could just randomly select a video from the billions that we have.

It's likely not to even be in your language.

So, the likelihood that you would watch it and develop a new interest is very, very low.

And so what you want to do when you're trying to increase diversity is find something that is not too similar to the things that you've watched, but also something that you might be likely to watch.

And that balance, finding that spot between those 2 things is quite challenging.

So the diversity of content, diversity of ideas, it's a really difficult, it's a thing like that's almost impossible to define, right?

So 2 examples is I'm a huge fan of 3Blue1Brown, say, and then 1 diversity, I wasn't even aware of a channel called Veritasium, which is a great science, physics, whatever channel.

So 1 version of diversity is showing me Derek's Veritasium's channel, which I was really excited to discover.

I actually now watch a lot of his videos.

Okay, so you're a person who's watching some math channels and you might be interested in some other science or math channels.

So like you mentioned, the first kind of diversity is just show you some things from other channels that are related, but not just, you know, not all the 3 blue, 1 brown channel, throw in a couple others.

So that's maybe the first kind of diversity that we started with many, many years ago.

Taking a bigger leap is about, I mean, the mechanisms we use for that is we basically cluster videos and channels together, mostly videos.

We do almost everything at the video level.

And so we'll make some kind of a cluster via some embedding process, and then measure what is the likelihood that users who watch 1 cluster might also watch another cluster that's very distinct.

So we may come to find that people who watch science videos also like jazz.

And so because of that relationship that we've identified through the embeddings and then the measurement of the people who watch both, we might recommend a jazz video once in a while.

So there's this clustering in the embedding space of jazz videos and science videos.

And so you kind of try to look at aggregate statistics where if a lot of people that jump from science cluster to the jazz cluster tend to remain as engaged or become more engaged, then that means those 2 are, they should hop back and forth and they'll be happy.

Right, there's a higher likelihood that a person from, who's watching science, would like jazz than the person watching science would like, I don't know, backyard railroads or something else, right?

And so we can try to measure these likelihoods and use that to make the best recommendation we can.

So, okay, so we'll talk about the machine learning of that, but I have to linger on things that neither you or anyone have an answer to.

There's gray areas of truth, which is, for example, Now I can't believe I'm going there, but politics.

It happens so that certain people believe certain things and they're very certain about them.

Let's move outside the red versus blue politics of today's world.

For example, in college I read quite a lot of Ayn Rand.

I studied, and that's a particular philosophical ideology I found interesting to explore.

I've kind of moved on from that cluster, intellectually, but it nevertheless is an interesting cluster.

There's, I was born in the Soviet Union, socialism, communism is a certain kind of political ideology that's really interesting to explore.

Again, objectively, just there's a set of beliefs about how the economy should work and so on.

And so it's hard to know what's true or not in terms of people within those communities are often advocating that this is how we achieve utopia in this world, and they're pretty certain about it.

So how do you try to manage politics in this chaotic, divisive world, not positive or any kind of ideas, in terms of filtering what people should watch next, and in terms of also not letting certain things be on YouTube.

This is exceptionally difficult responsibility.

Right, Well, the responsibility to get this right is our top priority.

And the first comes down to making sure that we have good, clear rules of the road, right?

Like, just because we have freedom of speech doesn't mean that you can literally say anything, right?

Like, we as a society have accepted certain restrictions on our freedom of speech.

There are things like libel laws and things like that.

And so where we can draw a clear line, we do, and we continue to evolve that line over time.

However, as you pointed out, wherever you draw the line there's gonna be a borderline.

And in that borderline area, we are going to maybe not remove videos, but we will try to reduce the recommendations of them or the proliferation of them by demoting them.

And then alternatively, in those situations, try to raise what we would call authoritative or credible sources of information.

So we're not trying to, I mean, you mentioned Ayn Rand and communism, you know, those are, those are 2 like valid points of view that people are going to debate and discuss.

And of course, people who believe in 1 or the other of those things are going to try to persuade other people to their point of view.

And so we're not trying to settle that or choose a side or anything like that.

What we're trying to do is make sure that the people who are expressing those point of view and offering those positions are authoritative and credible.

So let me ask a question about people I don't like, personally, you heard me, I don't care if you leave comments on this, but sometimes they're brilliantly funny, which is trolls.

So people who kind of mock, I mean the internet is full, the Reddit of mock style comedy, where people just kind of make fun of, point out that the emperor has no clothes.

And there's brilliant comedy in that, but sometimes it can get cruel and mean.

So on that, on the mean point, and sorry to linger on these things that have no good answers, but actually I totally hear you that this is really important that you're trying to solve it, but how do you reduce the meanness of people on YouTube?

I understand that anyone who uploads YouTube videos has to become resilient to a certain amount of meanness.

And we are trying in various ways, comment ranking, allowing certain features to block people, to reduce or make that meanness or that trolling behavior less effective on YouTube.

And so, I mean, It's very important, but it's something that we're gonna keep having to work on and as we improve it, maybe we'll get to a point where people don't have to suffer this sort of meanness when they upload YouTube videos.

I hope we do, but it just does seem to be something that you have to be able to deal with as a YouTube creator nowadays.

Do you have a hope that, so you mentioned 2 things that I kind of agree with.

So there's like a machine learning approach of ranking comments based on whatever, based on how much they contribute to the healthy conversation.

Then the other is almost an interface question of how do you, how does the creator filter?

So block or how do humans themselves, the users of YouTube manage their own conversation?

Do you have hope that these 2 tools will create a better society without limiting freedom of speech too much, without sort of, and even like saying that people are, what do you mean limiting, sort of curating speech?

I mean, I think that that overall is our whole project here at YouTube.

Like we fundamentally believe, and I personally believe very much that YouTube can be great.

But it's absolutely critical that we get this responsibility part right.

Susan Wojcicki, who's the CEO of YouTube, she says something that I personally find very inspiring, which is that We want to do our jobs today in a manner so that people 20 and 30 years from now will look back and say, You know, YouTube, they really figured this out.

They really found a way to strike the right balance between the openness and the value that the openness has, and also making sure that we are meeting our responsibility to users in society.

So the burden on YouTube actually is quite incredible.

And the 1 thing that people don't give enough credit to the seriousness and the magnitude of the problem, I think.

So I personally hope that you do solve it because a lot is in your hand.

A lot is riding on your success or failure.

So it's, besides of course running a successful company, you're also curating the content of the internet and the conversation on the internet.

So, 1 thing that people wonder about is how much of it can be solved with pure machine learning?

So looking at the data, studying the data, and creating algorithms that curate, the comments curate the content, and how much of it needs human intervention?

Meaning people here at YouTube in a room sitting and thinking about what is the nature of truth?

What is, what are the ideals that we should be promoting?

I mean, my own experience has demonstrated that you need both of those things.

Algorithms, I mean, you're familiar with machine learning algorithms, And the thing they need most is data.

And so, for instance, when we're building a system to try to figure out which are the videos that are misinformation or borderline policy violations, well, the first thing we need to do is get human beings to make decisions about which of those videos are in which category.

And then we use that data and basically, take that information that's determined and governed by humans and extrapolate it or apply it to the entire set of billions of YouTube videos.

And We couldn't get to all the videos on YouTube well without the humans, and we couldn't use the humans to get to all the videos of YouTube.

So there's no world in which you have only 1 or the other of these things.

And just as you said, a lot of it comes down to people at YouTube spending a lot of time trying to figure out what are the right policies, what are the outcomes based on those policies, Are they the kinds of things we want to see?

And then once we kind of get an agreement or build some consensus around what the policies are, well, then we've got to find a way to implement those policies across all of YouTube.

And that's where both the human beings, we call them evaluators or reviewers come into play to help us with that.

And then once we get a lot of training data from them, then we apply the machine learning techniques to take it even further.

Do you have a sense that these human beings have a bias in some kind of direction?

Sort of, I mean, that's an interesting question.

We do sort of, in autonomous vehicles and computer vision in general, a lot of annotation, and we rarely ask what bias do the annotators have.

Even in the sense that they're better at annotating certain things than others.

For example, people are much better at, for annotating segmentation, at segmenting cars in a scene versus segmenting bushes or trees.

You know, there's specific mechanical reasons for that, but also because the cement, it's cemented gray area, and just for a lot of reasons, people are just terrible at annotating trees.

Okay, so in the same kind of sense, do you think of, in terms of people reviewing videos or annotating the content of videos, is there some kind of bias that you're aware of or seek out in that human input?

Well, we take steps to try to overcome these kinds of biases or biases that we think would be problematic.

So for instance, we ask people to have a bias towards scientific consensus.

That's something that we instruct them to do.

We ask them to have a bias towards demonstration of expertise or credibility or authoritativeness.

But there are other biases that we want to make sure to try to remove.

And there's many techniques for doing this.

1 of them is you send the same thing to be reviewed to many people.

Another is that you make sure that the people that are doing these sorts of tasks are from different backgrounds and different areas of the United States or of the world.

But then, even with all of that, it's possible for certain kinds of, what we would call unfair biases to creep into machine learning systems, primarily as you said, because maybe the training data itself comes in, in a biased way.

And so we also have worked very hard on improving the machine learning systems to remove and reduce unfair biases when it goes against or has involved some protected class, for instance.

Thank you for exploring with me some of the more challenging things.

I'm sure there's a few more that we'll jump back to, but let me jump into the fun part, which is maybe the basics of the quote unquote YouTube algorithm.

What does the YouTube algorithm look at to make recommendation for what to watch next?

When it's from a machine learning perspective or when you search for a particular term, how does it know what to show you next?

Because it seems to, at least for me, do an incredible job of both.

Even I observe that it's improved quite a bit.

Like, when you search for something, YouTube uses the best technology we can get from Google to make sure that the YouTube search system finds what someone's looking for.

And of course, the very first things that 1 thinks about is, okay, well, does the word occur in the title, for instance?

You know, But there are much more sophisticated things where we're mostly trying to do some syntactic match or maybe a semantic match based on words that we can add to the document itself.

For instance, maybe is this video watched a lot after this query?

That's something that we can observe and then as a result, make sure that that document would be retrieved for that query.

Now, when you talk about what kind of videos would be recommended to watch next, that's something, again, we've been working on for many years.

And probably the first real attempt to do that well was to use collaborative filtering.

So you can describe what collaborative filtering is.

It's just basically what we do is we observe which videos get watched close together by the same person.

And if you observe that, and if you can imagine creating a graph where the videos that get watched close together by the most people are sort of very close to 1 another in this graph and videos that don't frequently get watched close together by the same person or the same people are far apart, then you end up with this graph that we call the related graph that basically represents videos that are very similar or related in some way.

And what's amazing about that is that it puts all the videos that are in the same language together, for instance.

And we didn't even have to think about language.

And it puts all the videos that are in the same language together, for instance.

And it puts all the videos that are about sports together, and it puts most of the music videos together, And it puts all of these sorts of videos together just because that's sort of the way the people using YouTube behave.

So that already cleans up a lot of the problem.

It takes care of the lowest hanging fruit, which happens to be a huge 1 of just managing these millions of videos.

I remember a few years ago, I was talking to someone who was, trying to propose that we do a research project concerning people who, who are bilingual and this person was, making this proposal based on the idea that YouTube could not possibly be good at recommending videos well to people who are bilingual.

And so she was telling me about this, and I said, well, can you give me an example of what problem do you think we have on YouTube with the recommendations?

And so she said, well, I'm a researcher in the US.

And when I'm looking for academic topics, I want to see them in English.

And so she searched for 1, found a video, and then looked at the Watch Next suggestions, and they were all in English.

YouTube must think that I speak only English.

And so she said, now, I'm actually originally from Turkey, and sometimes when I'm cooking, let's say I want to make some baklava, I really like to watch videos that are in Turkish.

And so she searched for a video about making the baklava and then selected it, and it was in Turkish, and the Watch Next recommendations were in Turkish.

And she just couldn't believe how this was possible.

And how is it that you know that I speak both these 2 languages and put all the videos together.

And it's just as a sort of an outcome of this related graph that's created through collaborative filtering.

So for me, 1 of my huge interests is just human psychology, right?

And that's such a powerful platform on which to utilize human psychology to discover what people, individual people wanna watch next.

You know, I've, Google search has ability to look at your own history.

And I've done that before, just what I've searched, 3 years, for many, many years, and it's a fascinating picture of who I am, actually.

And I don't think anyone's ever summarized, I personally would love that, a summary of who I am as a person on the internet to me.

Because I think it reveals, I think it puts a mirror to me or to others, you know, that's actually quite revealing and interesting, You know, just maybe the number of, it's a joke, but not really, is the number of cat videos I've watched, or videos of people falling, you know, stuff that's absurd, that kind of stuff.

And of course, it's really good for the machine learning aspect to show, to figure out what to show next, but it's interesting.

Have you just as a tangent played around with the idea of giving a map to people sort of as opposed to just using this information to show what's next, showing them, here are the clusters you've loved over the years kind of thing.

Well, we do provide the history of all the videos that you've watched.

So you can definitely search through that and look through it and search through it to see what it is that you've been watching on YouTube.

We have actually in various times experimented with this sort of cluster idea, finding ways to demonstrate or show people what topics they've been interested in or what clusters they've watched from.

It's interesting that you bring this up because in some sense, the way the recommendation system of YouTube sees a user is exactly as the history of all the videos they've watched on YouTube.

And so you can think of yourself or any user on YouTube as kind of like a DNA strand of all your videos, right?

You can also think of it as maybe a vector in the space of all the videos on YouTube.

And so, now once you think of it as a vector in the space of all the videos on YouTube, then you can start to say, OK, well, which other vectors are close to me and to my vector?

See all Lex Fridman transcripts on Youtube

Cristos Goodrow: YouTube Algorithm | Lex Fridman Podcast #68