- The video discusses the importance of selecting the right metrics for running experiments and measuring success.
- It explains the concept of goal metrics, which reflect a company's long-term vision, and provides examples from Facebook.
- Driver metrics are also discussed, which are used to measure short-term progress and are more suitable for online experiments.
- The video introduces guardrail metrics, which protect the business and monitor the trustworthiness of an experiment.
- It emphasizes the need for clear definitions and attributes of good metrics, such as simplicity, clarity, and actionability.
- The requirements for metrics used in experimentation are summarized as measurability, attributability, and sensitivity.
- The video addresses the question of making launch decisions when metrics have conflicting results, highlighting the need for trade-offs and alignment among stakeholders.

- Metrics for experiments should be discussed and aligned among various stakeholders
- The book suggests combining a few target metrics into an overall evaluation criteria or choosing no more than 5 metrics
- Having too many metrics can confuse people, lead to ignoring key metrics, and affect decision-making
- The video concludes by mentioning that upcoming videos will discuss interesting topics about A-B testing.

In this video, we will take a deep dive into the fundamentals of metrics in A/B Testing. By the end of the video, you will have a solid understanding of the common types of metrics and tips for applying them in real-world A/B testing applications. Some ideas are inspired by the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. I'm sure this video will be extremely helpful for both interview preparation and day-to-day work as a DS.

Blog post by Rob and me:
https://towardsdatascience.com/the-ultimate-guide-to-cracking-business-case-interviews-for-data-scientists-part-1-cb768c37edf4

Part 1 of the tutorial: 
https://www.youtube.com/watch?v=VpTlNRUcIDo

Part 2 of the tutorial:
https://www.youtube.com/watch?v=VuKIN9S8Ivs
🟢Get all my free data science interview resources
https://www.emmading.com/resources
🟡 Product Case Interview Cheatsheet https://www.emmading.com/product-case-cheat-sheet 
🟠 Statistics Interview Cheatsheet https://www.emmading.com/statistics-interview-cheat-sheet 
🟣 Behavioral Interview Cheatsheet https://www.emmading.com/behavioral-interview-cheat-sheet 
🔵 Data Science Resume Checklist https://www.emmading.com/data-science-resume-checklist

✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: https://www.emmading.com/coaching 

// Comment
Got any questions? Something to add?
Write a comment below to chat.

// Let's connect on LinkedIn:
https://www.linkedin.com/in/emmading001/

====================
 Contents of this video: 
====================:
0:00 Intro
1:13 Goal Metric
3:13 Driver Metric
4:42 Guardrail Metric
6:14 Application of Metrics
8:17 Metrics for experiments

In this video, we will do a deep dive on metrics.

Selecting the right metrics is super important to run an epitaph in practice, because we want to be clear about the goal as well as how to measure the results before running it.

Not only these metric-related questions often appear in data science interviews, it can be a straightforward question that asks you about the pros and cons of a specific metric.

It can also be a question that asks you to formulate a metric for an online experiment.

So as a data scientist, the knowledge of metrics is fundamental.

In this video, we will start with business metrics, including goal metric, driver metric, and guardrail metric.

Then we will talk about how to formulate metrics for online experiments.

The content of this video is based on not only my knowledge on metrics, but also a few new things I've learned from reading the book Trustworthy Online Control Experiments.

I have learned quite a few things from reading that book, and I recommend it to anyone who is interested in learning about A-B testing.

There are 3 kinds of operational metrics that companies use to measure success and progress and to understand areas for improvements.

The first kind of metric is the goal metric, is also known as success metric, true North metric, North Star metric, OKR metric, and primary metric.

This kind of metric reflects an accountant's long-term vision, and it always ties to an accountant's mission.

Goal metrics are a small set of metrics that an accountant truly cares about.

How do we translate such a mission or vision to a set of metrics?

Facebook's mission is to give people the power to build a community and bring the world closer together.

And its goal metrics include advertising revenue, daily active users, and monthly active users.

While the transformation from its mission to its goal metrics isn't perfect, the goal metrics do reflect what the companies ultimately care about.

And they are simple enough to be easily communicated to different stakeholders such as investors, customers, and employees.

The goal metric should also be stable over a long period of time to allow the whole organization to work towards improving it.

While goal metrics are critical to measure the overall success of a company, they may not be suitable for online experiments, because they can be difficult to measure or may not be sensitive enough to product changes.

For example, Facebook cares about ad revenue, but not every team could use it for A-B testing.

There are teams focusing on improving user engagement, and also teams focusing on website or native app performance.

For such teams, what they do definitely contributes to the company's overall success, but they don't use those company-level goal metrics to measure performance.

Compared with goal metrics, which are about the long-term vision, We also need metrics to reflect short-term progress.

The driver metric, also known as surrogate metric, indirect or predict metric, are often used to measure short-term objectives.

They align with the goal metrics, but they are more sensitive and actionable to be able to measure short-term progress and drive teams to work on it.

That's also why they are better than the goal metrics to be used for A-B testing.

Now let me give you a concrete example of a driver metric.

A marketing team's goal is to acquire new users, and 1 of the driver metrics could be the number of new users registered per day.

The distinction between the goal metric and the driver metric is actually something new I learned from the book.

Before reading it, I thought I knew what a success metric is.

I have developed such metrics in practice and have used them to run online experiments.

But after reading the book, I realized that I was wrong.

What I thought of as success metrics were actually driver metrics.

In fact, success metric is the same as the goal metric, and it's about the long-term vision, while the driver metric is used to measure short-term progress and is more suitable for online experiments.

Check out this blog written by my friend Rob and me.

It covers several metric frameworks, which can be very helpful for you to understand what metrics are used in different business domains.

The last category of metrics is the guardrail metric.

As the name suggests, guardrail metrics guard us from harming the business and violating important experiment assumptions.

In this book, it categorizes Godreel metrics into 2 groups, which I think is very helpful to understand different roles of Godreel metrics.

The first 1 is the organizational guardrail metric.

If this kind of metric shifts to the negative direction, the business will suffer significant loss.

For example, if the loading time of a web page increases by a few million seconds, there can be a significant loss of customers and revenue.

In practice, the page loading latency is often used as a guardrail metric when new features are developed and tested through API testing.

A few other commonly used organizational gut-real metrics include errors per page and client crashes.

The other kind of gut-real metric is trustworthy-related metrics.

They are used to monitor the trustworthiness of an experiment.

That is to check if there is any violation of its assumptions.

1 of the common use metric is to check if randomization units assigned to each variant is truly random.

When the numbers in different groups are different, the authors refer this as sample ratio mismatch.

We then need to perform a t-test or a chi-square test to check if the assignment ratio matches with what was designed.

Now you know the definition of goal metrics, driver metrics, as well as guardrail metrics.

In practice, we need to be clear about the context when talking about a specific metric, because the same metric can be used differently for different teams.

1 team's driver metric can be another team's guardrail metric.

For example, 1 frontend team's goal is to improve web performance.

So reducing latency is their goal and time to interactive, TTI, can be 1 of their driver metrics.

A product team may use the same metric as a guardrail metric to make sure any product changes don't increase latency.

Next, let's look at what are the attributes of a good metric.

In the blog post I mentioned earlier, it has a few general rules to formulate metrics.

It's easy to understand and calculate, and people should be able to remember and discuss it easily.

If you cannot use 1 sentence to explain a metric, it's not simple.

Secondly, the definition of a good metric is clear and there is no ambiguity in interpretation.

Thirdly, a good metric should be actionable.

The metric can be shifted by changes in products, and it offers insights on how you can improve.

Gaming means that a metric makes you feel like you are getting results, but it offers no insights into actual business health or growth.

Short-term revenue is an example of such a metric.

Increasing prices of products may increase short-term revenue.

However, the business may lose customers in the long run.

We have talked about operational metrics, which are critical to measure a company or team's performance.

However, not all of them are suitable for online controlled experiments.

Next, let's go over what are the requirements of metrics that can be used directly for experimentation.

In the book, the authors summarize 3 attributes for metrics that are suitable for experimentation.

Measurable, we should be able to calculate those metrics with data collected during the experiment period.

Attributable, we should be able to attribute metrics values to the experiment variants.

It means that the metrics should be able to be calculated separately for different variants in the experiment.

Sensitive and Timely, experiment metrics should be sensitive enough to detect changes in a timely manner.

In online experiments, we typically select a few driver metrics as the key metrics, as well as some guardrail metrics to monitor impacts on other aspects of the business.

Now I want to share 1 question I get constantly.

Since we have multiple metrics for an experiment, how do we make the launch decision when 1 metric goes up and 1 metric goes down?

It's a very reasonable question, and this scenario happens often.

In practice, many organizations have a mental model of the tradeoffs they are willing to accept when they see any particular results.

For example, tradeoffs between user acquisition and revenue.

How should a company strike the optimal balance between revenue maximization and user acquisition?

Acquiring new users can always be done by expensive campaigns, such as providing large discounts or gifts, but it will degrade revenue.

This kind of trade-off is not something that can be determined by a single data scientist or a single data science team.

It's something that is discussed and aligned among various stakeholders, such as product teams, marketing teams, engineering teams, and the leadership team, etc.

Finally, I want to share with you 2 practical suggestions from the book on formulating metrics for experiments.

1 is to combine a few target metrics into an overall evaluation criteria, a weighted combination of the most important driver metrics, and use it as the only criteria for an experiment.

If coming up with such an OEC is difficult, the authors recommend choosing no more than 5 metrics as their target metrics.

There are 2 main disadvantages of having too many metrics.

1 is that too many metrics may confuse people and it may possibly lead to ignoring the key metrics.

The other downside is including too many metrics may affect the decision-making process and increase the chances of having false discoveries.

Alright guys, I hope you have learned something new about metrics in this video.

In the next few videos, we will continue talking about interesting topics about A-B testing.


Speaker 0: Hey guys, welcome back to my channel. In this video, we will do a deep dive on metrics. Selecting the right metrics is super important to run an epitaph in practice, because we want to be clear about the goal as well as how to measure the results before running it. Not only these metric-related questions often appear in data science interviews, it can be a straightforward question that asks you about the pros and cons of a specific metric. It can also be a question that asks you to formulate a metric for an online experiment.

So as a data scientist, the knowledge of metrics is fundamental. In this video, we will start with business metrics, including goal metric, driver metric, and guardrail metric. Then we will talk about how to formulate metrics for online experiments. The content of this video is based on not only my knowledge on metrics, but also a few new things I've learned from reading the book Trustworthy Online Control Experiments. I have learned quite a few things from reading that book, and I recommend it to anyone who is interested in learning about A-B testing.

Ok, let's get started. There are 3 kinds of operational metrics that companies use to measure success and progress and to understand areas for improvements. The first kind of metric is the goal metric, is also known as success metric, true North metric, North Star metric, OKR metric, and primary metric. This kind of metric reflects an accountant's long-term vision, and it always ties to an accountant's mission. Goal metrics are a small set of metrics that an accountant truly cares about.

I know it may sound abstract. How do we translate such a mission or vision to a set of metrics? Let me give you an example. Facebook's mission is to give people the power to build a community and bring the world closer together. And its goal metrics include advertising revenue, daily active users, and monthly active users.

While the transformation from its mission to its goal metrics isn't perfect, the goal metrics do reflect what the companies ultimately care about. And they are simple enough to be easily communicated to different stakeholders such as investors, customers, and employees. The goal metric should also be stable over a long period of time to allow the whole organization to work towards improving it. While goal metrics are critical to measure the overall success of a company, they may not be suitable for online experiments, because they can be difficult to measure or may not be sensitive enough to product changes. For example, Facebook cares about ad revenue, but not every team could use it for A-B testing.

There are teams focusing on improving user engagement, and also teams focusing on website or native app performance. For such teams, what they do definitely contributes to the company's overall success, but they don't use those company-level goal metrics to measure performance. Compared with goal metrics, which are about the long-term vision, We also need metrics to reflect short-term progress. The driver metric, also known as surrogate metric, indirect or predict metric, are often used to measure short-term objectives. They align with the goal metrics, but they are more sensitive and actionable to be able to measure short-term progress and drive teams to work on it.

That's also why they are better than the goal metrics to be used for A-B testing. Now let me give you a concrete example of a driver metric. A marketing team's goal is to acquire new users, and 1 of the driver metrics could be the number of new users registered per day. The distinction between the goal metric and the driver metric is actually something new I learned from the book. Before reading it, I thought I knew what a success metric is.

I have developed such metrics in practice and have used them to run online experiments. But after reading the book, I realized that I was wrong. What I thought of as success metrics were actually driver metrics. In fact, success metric is the same as the goal metric, and it's about the long-term vision, while the driver metric is used to measure short-term progress and is more suitable for online experiments. Check out this blog written by my friend Rob and me.

It covers several metric frameworks, which can be very helpful for you to understand what metrics are used in different business domains. The last category of metrics is the guardrail metric. As the name suggests, guardrail metrics guard us from harming the business and violating important experiment assumptions. In this book, it categorizes Godreel metrics into 2 groups, which I think is very helpful to understand different roles of Godreel metrics. The first 1 is the organizational guardrail metric.

If this kind of metric shifts to the negative direction, the business will suffer significant loss. For example, if the loading time of a web page increases by a few million seconds, there can be a significant loss of customers and revenue. In practice, the page loading latency is often used as a guardrail metric when new features are developed and tested through API testing. A few other commonly used organizational gut-real metrics include errors per page and client crashes. The other kind of gut-real metric is trustworthy-related metrics.

They are used to monitor the trustworthiness of an experiment. That is to check if there is any violation of its assumptions. 1 of the common use metric is to check if randomization units assigned to each variant is truly random. When the numbers in different groups are different, the authors refer this as sample ratio mismatch. We then need to perform a t-test or a chi-square test to check if the assignment ratio matches with what was designed.

Now you know the definition of goal metrics, driver metrics, as well as guardrail metrics. In practice, we need to be clear about the context when talking about a specific metric, because the same metric can be used differently for different teams. 1 team's driver metric can be another team's guardrail metric. For example, 1 frontend team's goal is to improve web performance. So reducing latency is their goal and time to interactive, TTI, can be 1 of their driver metrics.

A product team may use the same metric as a guardrail metric to make sure any product changes don't increase latency. Next, let's look at what are the attributes of a good metric. In the blog post I mentioned earlier, it has a few general rules to formulate metrics. A good metric should be simple. It's easy to understand and calculate, and people should be able to remember and discuss it easily.

If you cannot use 1 sentence to explain a metric, it's not simple. Secondly, the definition of a good metric is clear and there is no ambiguity in interpretation. Thirdly, a good metric should be actionable. The metric can be shifted by changes in products, and it offers insights on how you can improve. It should not be easily gamed.

Gaming means that a metric makes you feel like you are getting results, but it offers no insights into actual business health or growth. Short-term revenue is an example of such a metric. Increasing prices of products may increase short-term revenue. However, the business may lose customers in the long run. We have talked about operational metrics, which are critical to measure a company or team's performance.

However, not all of them are suitable for online controlled experiments. Next, let's go over what are the requirements of metrics that can be used directly for experimentation. In the book, the authors summarize 3 attributes for metrics that are suitable for experimentation. Measurable, we should be able to calculate those metrics with data collected during the experiment period. Attributable, we should be able to attribute metrics values to the experiment variants.

It means that the metrics should be able to be calculated separately for different variants in the experiment. Sensitive and Timely, experiment metrics should be sensitive enough to detect changes in a timely manner. In online experiments, we typically select a few driver metrics as the key metrics, as well as some guardrail metrics to monitor impacts on other aspects of the business. Now I want to share 1 question I get constantly. Since we have multiple metrics for an experiment, how do we make the launch decision when 1 metric goes up and 1 metric goes down?

It's a very reasonable question, and this scenario happens often. In practice, many organizations have a mental model of the tradeoffs they are willing to accept when they see any particular results. For example, tradeoffs between user acquisition and revenue. How should a company strike the optimal balance between revenue maximization and user acquisition? Acquiring new users can always be done by expensive campaigns, such as providing large discounts or gifts, but it will degrade revenue.

This kind of trade-off is not something that can be determined by a single data scientist or a single data science team. It's something that is discussed and aligned among various stakeholders, such as product teams, marketing teams, engineering teams, and the leadership team, etc. Finally, I want to share with you 2 practical suggestions from the book on formulating metrics for experiments. 1 is to combine a few target metrics into an overall evaluation criteria, a weighted combination of the most important driver metrics, and use it as the only criteria for an experiment. If coming up with such an OEC is difficult, the authors recommend choosing no more than 5 metrics as their target metrics.

There are 2 main disadvantages of having too many metrics. 1 is that too many metrics may confuse people and it may possibly lead to ignoring the key metrics. The other downside is including too many metrics may affect the decision-making process and increase the chances of having false discoveries. Alright guys, I hope you have learned something new about metrics in this video. In the next few videos, we will continue talking about interesting topics about A-B testing.

Stay tuned. I will see you soon.

See all Emma Ding transcripts on Youtube

A/B Testing Metrics: What You Need to Know About Success, Driver, and Guardrail Metrics!