- The video discusses the fundamentals of A-B testing and its practical applications in advertising.
- The book "Trustworthy Online Control Experiments: A Practical Guide to A-B Testing" is recommended as a valuable resource for learning about advertising.
- Epitaphs are experiments that compare a control group with a treatment group, with all variables held constant except for one being tested.
- Variants refer to different versions of a product or user experience, and A-B tests are used when comparing two variants, while ABN tests are used for multiple variants.
- The importance of running experiments is emphasized for making data-driven decisions and establishing causality.
- The major steps in running an A-B test include defining key metrics, ensuring changes are easy to make, and having enough randomization units.
- Designing the experiment involves considering factors such as the population of randomization units, sample size, and duration of the experiment.
- After running the experiment and collecting data, the results need to be checked and interpreted before making a decision on launching a change.
- Factors to consider in making the launch decision include tradeoffs between different metrics and the cost of implementing and maintaining the change.

- Practical significance is important in determining whether to launch a product
- If the cost is low, any positive change can be launched
- Statistical significance is enough to launch a change
- Long-term effects need to be monitored after launch
- Long-term effects provide insights for future improvements
- The next video will cover an end-to-end example of running an experiment in reality

Data Scientists Must Know | Data Science Interview Must Watch: A/B Testing Fundamentals. I will dive deep into practical problems in A/B testing and share things I learned from the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.

🟢Get all my free data science interview resources
https://www.emmading.com/resources
🟡 Product Case Interview Cheatsheet https://www.emmading.com/product-case-cheat-sheet 
🟠 Statistics Interview Cheatsheet https://www.emmading.com/statistics-interview-cheat-sheet 
🟣 Behavioral Interview Cheatsheet https://www.emmading.com/behavioral-interview-cheat-sheet 
🔵 Data Science Resume Checklist https://www.emmading.com/data-science-resume-checklist

✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: https://www.emmading.com/coaching 

✔️ Cracking Hypothesis Testing Problems in Data Science Interviews part 1
https://www.youtube.com/watch?v=IY7y-t30UJc

✔️ Cracking Hypothesis Testing Problems in Data Science Interviews part 2
Practical significance and statistical significance
https://www.youtube.com/watch?v=6uw0A3aKwMc

✔️ Cracking A/B Testing Problems in Data Science Interviews
https://www.youtube.com/watch?v=X8u6kr4fxXc

📚 Trustworthy online controlled experiments, a practical guide to A/B testing
https://www.amazon.com/Trustworthy-Online-Controlled-Experiments-Practical/dp/1108724264

// Comment
Got any questions? Something to add?
Write a comment below to chat.

// Let's connect on LinkedIn:
https://www.linkedin.com/in/emmading001/

====================
 Contents of this video: 
====================
0:00 Intro
1:52 What are A/B tests
3:39 Why A/B tests
5:05 Steps to run A/B tests
5:20 Experiment prerequisites
6:51 Experiment design
8:03 Running Experiments
8:22 Result to decision
10:48 Post-launching monitoring

In today's video, I want to talk about the fundamentals of A-B testing.

Different from my previous videos on A-B testing where I just share commonly asked interview questions and answers, In this video and the next few videos, I will start from the very basics and dive deep into some practical problems running advertising in reality.

So if you want to learn advertising in depth to prepare for your interview or to expand your knowledge, this video is definitely for you.

Today's video is going to be a slightly different video from what I normally do, where I just share what I already knew from my own experience.

Instead, I want to share with you not only what I already knew, but also a few new things that I was able to learn from a specific book that I have been reading over the last several weeks.

And that book is called Trustworthy Online control experiments, a practical guide to A-B testing.

It was published in 2020 and it was written by 3 industry professionals, all of them have lots of experience leading A-B testing in big tech companies.

I have recommended this book in my blog posts and videos, and it is, in my opinion, 1 of the most practical books on advertising.

So if you want to learn advertising in depth, this book is a great resource.

Since advertising covers many things, it's hard to make a single video to cover everything.

So I plan to make a couple videos from the very basics to advanced topics.

Each topic will be independent, so it's easier for you to re-watch a video to learn about that topic.

Now, let's jump right into what are epitaphs.

An epitaph is an experiment in which all elements are held constant, except for 1 variable.

Typically, it compares a control group against a treatment group.

All variables are identical between the 2 groups except for 1 factor that's being tested.

Different versions of a product or user experience are formally referred to as variants.

Variants can be as simple as colors of a button, or as complicated as different backend algorithms to display search results.

In cases, there are 2 variants, 1 control and 1 treatment group.

If there are more than 2 variants, it's called ABN test.

But in reality, ABN tests could also be used to refer to experiments with multiple variants.

I sometimes get this question, what are the differences between ABN tests and controlled experiments?

ABI tests are sometimes called ABN tests, controlled experiments, randomized controlled experiments, split tests, but they refer to the same thing.

Now let me give you an example of an AVI test.

In the book, it mentions an interesting example.

Google tested 41 gradations of blue on Google search result pages.

In each treatment group, the color is different.

Even though the test frustrated the visual design lead at that time, the result showed that color schemes significantly changed user engagement.

Appetites are widely adopted in the industry when evaluating new product ideas.

In fact, when you are browsing a website or using a mobile app, you might be part of an experiment that is running behind the scene.

Why do companies run experiments instead of simply rolling out a new feature?

The goal of running an app is to make data-driven decisions.

Only when the results are reliable and repeatable can we make the right decision.

To make the result reproducible, an important requirement is that the factor we are testing is the cause of the change in the metric, so that when launching the feature to all the traffic, the impact can be predicted from the treatment effect measured in the experiment.

For example, changes of colors could cause changes in user engagement, assuming other things stay the same.

And running epitaphs is the scientific way to do it.

In the book, the authors claim that randomized controlled experiments are the gold standard for establishing causality.

We believe online control experiments are the best scientific way to establish causality with a high probability.

Able to detect small changes that are hard to detect with other techniques, such as change over time.

Often underappreciated, but many experiments uncover surprising impacts on other metrics.

Now you know what is an ABI test, as well as the importance of running ABI tests.

Let's dive into the major steps involved in running ABI tests.

In general, there are 5 major steps involved in running a test correctly.

I have drawn this diagram to help you understand it clearly.

Before running an experiment, a few things need to be ready.

First of all, we need to define key metrics to measure the goal of an experiment.

The key metric is formally known as the Overall Evaluation Criterion, or OEC.

It should be agreed upon by different stakeholders and should be practically measured.

For example, if we want to test if changing the color of the checkup button could impact revenue, The key metric of the OEC could be revenue per user per month.

The second requirement is that changes are easy to make.

This should be obvious because we need to compare different variants and find the 1 that has the highest positive impact on the OEC.

If changes are very hard to make, it will introduce complexities to generate variants.

For example, it will be very difficult to redesign the whole website and consider that redesign as a variant.

The last requirement is to have enough randomization units to be assigned to different variants.

It's simply the who or what that is randomly allocated to different groups.

The most common user randomization unit is user.

The recommendation in the book is to have thousands of randomization units because the larger the number, the smaller the effects that can be detected.

OK, after these requirements are fulfilled, we can move forward to designing the experiment.

And the book touches a few things that need to be considered.

What population of randomization units do we want to select?

Basically, do we want to target a specific population or all the users?

Sometimes it's helpful to run experiments for a specific segment because the change only affects that segment.

For example, a new feature that is only available for users in a particular geographic region.

Another factor to consider is the size of the experiment.

We need to compute the sample size of the experiment in order to achieve the required statistical power.

Detecting a small change would need more users.

If you are interested in learning how to get the sample size, I have a video to derive the formula step by step.

The last important consideration is how long to run an experiment.

To determine the duration, we will need to consider seasonality, the day of week effect, as well as primacy and novelty effect.

All of them will influence the decision on how long we should run an experiment.

After all those decisions are made, we could run the experiment and collect the data.

In this process, typically, data scientists work with engineers to instrument logins to get logged data.

For companies that have built their own experimentation platform, this is done automatically.

After running the experiment for the required amount of time, we need to check and interpret the results and use them to make a decision.

In reality, this is where data scientists spend most time and energy on.

Once we obtain the data, the very first step is to do sanity checks to make sure the data are reliable.

We could only continue the analysis once the sanity checks are passed.

If not, we need to discard the results and look into the root cause, and we may need to rerun the experiment.

Here we will not dive into those checks, but I will explain them in detail in an upcoming video.

Once those sanity checks are passed, we could use the results to make a launch decision.

In the book, it recommends examining at least these factors.

The first 1 is the tradeoffs between different metrics.

This refers to the scenario that different metrics move to opposite directions.

For example, user engagement goes up but revenue goes down.

The other factors can be summarized as the cost of launching a change.

For example, cost for engineering maintenance after launch.

Since new code may introduce complexity and bugs to the codebase, the maintenance effort can be costly.

The time and effort we spend launching a change might not be as much as the opportunity cost of giving up a different idea.

If those costs are high, we need to ensure that the expected benefits can outweigh the costs.

In fact, that's why we typically set a practical significance boundary to reflect those costs.

And we only launch a product if the result is practically significant.

On the contrary, if the cost is low, we will choose to launch any change that is positive.

In other words, as long as the result is statistically significant, we can launch the change.

If you are not familiar with the concept of practical significance boundary, I highly recommend checking out this video which covers an analysis using both statistical and practical significance boundaries to make a launch decision.

At this point, you might think we are done with the experiment because we have made a decision.

Well, we are getting close, but we are not done yet.

If we decide to launch a new product based on the results of an experiment, we need to monitor the long-term effect after launch.

Because the short-term effect can be different from the long-term effect due to various reasons.

Also, major long-term effects have a few benefits, such as insight on knocked effects could help improve future iterations.

Alright guys, this is the first part of the tutorial on API testing.

In the next video, we will dive into an end-to-end example to talk about the whole process, running an experiment in reality.

We will talk about how to select the right metric and randomization units, how to decide how long to run an experiment.


Speaker 0: Hey guys, welcome back to my channel. In today's video, I want to talk about the fundamentals of A-B testing. Different from my previous videos on A-B testing where I just share commonly asked interview questions and answers, In this video and the next few videos, I will start from the very basics and dive deep into some practical problems running advertising in reality. So if you want to learn advertising in depth to prepare for your interview or to expand your knowledge, this video is definitely for you. Today's video is going to be a slightly different video from what I normally do, where I just share what I already knew from my own experience.

Instead, I want to share with you not only what I already knew, but also a few new things that I was able to learn from a specific book that I have been reading over the last several weeks. And that book is called Trustworthy Online control experiments, a practical guide to A-B testing. It was published in 2020 and it was written by 3 industry professionals, all of them have lots of experience leading A-B testing in big tech companies. I have recommended this book in my blog posts and videos, and it is, in my opinion, 1 of the most practical books on advertising. So if you want to learn advertising in depth, this book is a great resource.

Since advertising covers many things, it's hard to make a single video to cover everything. So I plan to make a couple videos from the very basics to advanced topics. Each topic will be independent, so it's easier for you to re-watch a video to learn about that topic. Ok, that's a long introduction. Now, let's jump right into what are epitaphs.

An epitaph is an experiment in which all elements are held constant, except for 1 variable. Typically, it compares a control group against a treatment group. All variables are identical between the 2 groups except for 1 factor that's being tested. Different versions of a product or user experience are formally referred to as variants. Variants can be as simple as colors of a button, or as complicated as different backend algorithms to display search results.

In cases, there are 2 variants, 1 control and 1 treatment group. It's called an A-B test. If there are more than 2 variants, it's called ABN test. But in reality, ABN tests could also be used to refer to experiments with multiple variants. I sometimes get this question, what are the differences between ABN tests and controlled experiments?

Well, they are the same thing. ABI tests are sometimes called ABN tests, controlled experiments, randomized controlled experiments, split tests, but they refer to the same thing. Now let me give you an example of an AVI test. In the book, it mentions an interesting example. Google tested 41 gradations of blue on Google search result pages.

In each treatment group, the color is different. Even though the test frustrated the visual design lead at that time, the result showed that color schemes significantly changed user engagement. Appetites are widely adopted in the industry when evaluating new product ideas. In fact, when you are browsing a website or using a mobile app, you might be part of an experiment that is running behind the scene. But why do we need to run experiments?

Why do companies run experiments instead of simply rolling out a new feature? The goal of running an app is to make data-driven decisions. Only when the results are reliable and repeatable can we make the right decision. To make the result reproducible, an important requirement is that the factor we are testing is the cause of the change in the metric, so that when launching the feature to all the traffic, the impact can be predicted from the treatment effect measured in the experiment. For example, changes of colors could cause changes in user engagement, assuming other things stay the same.

And running epitaphs is the scientific way to do it. In the book, the authors claim that randomized controlled experiments are the gold standard for establishing causality. We believe online control experiments are the best scientific way to establish causality with a high probability. Able to detect small changes that are hard to detect with other techniques, such as change over time. Able to detect unexpected changes.

Often underappreciated, but many experiments uncover surprising impacts on other metrics. Now you know what is an ABI test, as well as the importance of running ABI tests. Let's dive into the major steps involved in running ABI tests. In general, there are 5 major steps involved in running a test correctly. I have drawn this diagram to help you understand it clearly.

Let's go through each step 1 by 1. Before running an experiment, a few things need to be ready. First of all, we need to define key metrics to measure the goal of an experiment. The key metric is formally known as the Overall Evaluation Criterion, or OEC. It should be agreed upon by different stakeholders and should be practically measured.

For example, if we want to test if changing the color of the checkup button could impact revenue, The key metric of the OEC could be revenue per user per month. The second requirement is that changes are easy to make. This should be obvious because we need to compare different variants and find the 1 that has the highest positive impact on the OEC. If changes are very hard to make, it will introduce complexities to generate variants. For example, it will be very difficult to redesign the whole website and consider that redesign as a variant.

The last requirement is to have enough randomization units to be assigned to different variants. But what is a randomization unit? It's simply the who or what that is randomly allocated to different groups. The most common user randomization unit is user. So how much is enough?

The recommendation in the book is to have thousands of randomization units because the larger the number, the smaller the effects that can be detected. OK, after these requirements are fulfilled, we can move forward to designing the experiment. And the book touches a few things that need to be considered. What population of randomization units do we want to select? Basically, do we want to target a specific population or all the users?

Sometimes it's helpful to run experiments for a specific segment because the change only affects that segment. For example, a new feature that is only available for users in a particular geographic region. Another factor to consider is the size of the experiment. We need to compute the sample size of the experiment in order to achieve the required statistical power. Detecting a small change would need more users.

If you are interested in learning how to get the sample size, I have a video to derive the formula step by step. The last important consideration is how long to run an experiment. To determine the duration, we will need to consider seasonality, the day of week effect, as well as primacy and novelty effect. All of them will influence the decision on how long we should run an experiment. After all those decisions are made, we could run the experiment and collect the data.

In this process, typically, data scientists work with engineers to instrument logins to get logged data. For companies that have built their own experimentation platform, this is done automatically. After running the experiment for the required amount of time, we need to check and interpret the results and use them to make a decision. In reality, this is where data scientists spend most time and energy on. Once we obtain the data, the very first step is to do sanity checks to make sure the data are reliable.

We could only continue the analysis once the sanity checks are passed. If not, we need to discard the results and look into the root cause, and we may need to rerun the experiment. Here we will not dive into those checks, but I will explain them in detail in an upcoming video. Once those sanity checks are passed, we could use the results to make a launch decision. And there are many factors to consider.

In the book, it recommends examining at least these factors. The first 1 is the tradeoffs between different metrics. This refers to the scenario that different metrics move to opposite directions. For example, user engagement goes up but revenue goes down. How to make the decision?

The other factors can be summarized as the cost of launching a change. For example, cost for engineering maintenance after launch. Since new code may introduce complexity and bugs to the codebase, the maintenance effort can be costly. Also, there are opportunity costs. The time and effort we spend launching a change might not be as much as the opportunity cost of giving up a different idea.

If those costs are high, we need to ensure that the expected benefits can outweigh the costs. In fact, that's why we typically set a practical significance boundary to reflect those costs. And we only launch a product if the result is practically significant. On the contrary, if the cost is low, we will choose to launch any change that is positive. In other words, as long as the result is statistically significant, we can launch the change.

If you are not familiar with the concept of practical significance boundary, I highly recommend checking out this video which covers an analysis using both statistical and practical significance boundaries to make a launch decision. At this point, you might think we are done with the experiment because we have made a decision. Well, we are getting close, but we are not done yet. If we decide to launch a new product based on the results of an experiment, we need to monitor the long-term effect after launch. Because the short-term effect can be different from the long-term effect due to various reasons.

Also, major long-term effects have a few benefits, such as insight on knocked effects could help improve future iterations. Alright guys, this is the first part of the tutorial on API testing. In the next video, we will dive into an end-to-end example to talk about the whole process, running an experiment in reality. We will talk about how to select the right metric and randomization units, how to decide how long to run an experiment. Stay tuned for upcoming videos.

I will see you soon.

What mistakes can be done during A/B testing?

See all Emma Ding transcripts on Youtube

A/B Testing Fundamentals: What Every Data Scientist Needs to Know!