- The speaker, Ilya, is a co-founder of Segment, an analytics API that provides information on what tools startups are using.
- Analytics is important for measuring primary and secondary metrics, testing product-market fit, and driving teams.
- The speaker will cover setting up analytics, collecting data, and the top 3 metrics for most products.
- The funnel is a sequential series of steps that users go through to get value and pay.
- Event properties are important for tracking and analyzing user behavior.
- Amplitude and Mixpanel are recommended analytics tools.
- The three metrics discussed are signups per week, retention cohorts, and revenue.
- It is important to have a data-driven team that regularly analyzes metrics and sets goals.
- Using a TV dashboard and social accountability can help drive performance and receive appropriate advice.

- The video discusses the importance of understanding the business and providing appropriate advice to advisors.
- It highlights the process of building a minimum viable product (MVP) and entering private beta.
- Communication with customers through channels like Slack and live chat is emphasized.
- Tools like Google Analytics, Amplitude, and Full Story are recommended for tracking user behavior and improving product usability.
- Email tools like Customer IO are suggested for connecting with customers.
- Data warehouse and BI tools like Google BigQuery and Mode Analytics are recommended for data analysis.
- The video advises against spending too much time on tool selection and emphasizes the need for flexibility.
- A list of recommended tools for the MVP process is provided, including Google Analytics, Amplitude, Intercom, and Full Story.
- The video concludes by mentioning that Segment is now free for early-stage startups and offers deals on various tools.

The speaker is giving a presentation to multiple companies, each with different needs. For Airbnb, it is important to have users return to their app regularly, possibly on a quarterly basis. Monitoring user activity is important for most companies.

The speaker, Ilya, discusses the importance of analytics in measuring metrics, testing product-market fit, and driving teams. They cover setting up analytics, collecting data, and highlight the top 3 metrics for most products. The video emphasizes the importance of a data-driven team, using tools like Amplitude and Mixpanel, and tracking user behavior through event properties. Communication with customers, using tools like Slack and live chat, is also emphasized. The video provides recommendations for tracking user behavior and data analysis tools. It advises against spending too much time on tool selection and highlights the need for flexibility. The video concludes by mentioning that Segment is now free for early-stage startups and offers deals on various tools. Monitoring user activity is important for most companies, with Airbnb specifically focusing on having users return to their app regularly.

Ilya Volodarsky, co-founder of Segment, presents his tactics for setting up analytics to build your MVP and measure your metrics. 

You can find the lecture slides and transcript here: https://www.startupschool.org/videos/84

It's not too late to sign up! Register now through August 4th at https://www.startupschool.org/.

I'm 1 of the co-founders at Segment, and I'm here to talk to you about how to set up analytics and the analytics foundation to build your MVP and to measure these primary and secondary metrics.

So this is gonna be a little bit more of a tactical guide around what tools are there in the analytics space and the marketing space, which ones should we actually be using, how do I set them up?

Just before I get started, a lot of you may not know what Segment is.

It's an analytics API where you can send analytics data and then toggle on different tools.

So we sit in the cool market position where we can actually see what tools startups are using and we can bring that information to you.

Secondly, we've been around for about 6 years, and so we have information about a lot of different startups as they grew and what tools they used, so we're going to be sharing a lot of those today.

So, why talk about, why even focus on analytics?

Obviously, primary and secondary metrics drive the MVP and product market fit process.

And you're using that to actually test product market fit.

You're also using it, once you get out of product market fit strict search, to actually focus the team.

So maybe there's going to be an acquisition issue in the company that's preventing your growth, or maybe the users that you're getting aren't as engaged, or maybe you're having some monetization issues.

And so the funnel actually is a forcing function to understand your business and where founders should be actually spending their time.

And then finally, all the way from, you know, 2, 3 person team to a Google with 1000000 employees, you're actually using metrics to operate and drive teams.

So eventually you have an engineering team, you have a marketing team, And so what goal do you set in front of the marketing team?

Okay, so today we're gonna cover a few different things.

So first you always start with the funnel when you're thinking about analytics.

So that's the sequential series of steps your users go through to actually get value and then pay you as well.

Then we're going to talk about collecting data for your analytics tools.

Then we're going to talk about the top 3 metrics.

So this will include primary and secondary metrics that work for most products.

And then a product market fit methodology that you can apply on top of that.

And then finally, we'll make recommendations about what tools are the best in the market right now that help the product market fit journey.

Okay, so to start, we started the funnel.

So we'll make an example funnel for Netflix, which is a company that we're all super familiar with.

Any B2B product or B2C product actually has this type of funnel where you acquire a user, you engage a user over a period of time, that loop is called retention, and then finally you monetize the user.

And then metrics, both primary and secondary, are performance indicators on top of each stage in the funnel.

So on top of acquisition, you can ask yourself, how many net new users did I get this week versus last week, and what's my growth rate there?

For engagement, you take a cohort of users.

So from Sunday to the following Monday, you have 16 people sign up and then you can track that cohort of users week over week and see what percentage of them are still using the product 4 weeks later, which is a good example of how to track retention.

And then we talk about monetization, which is how much net new revenue did I make this week versus last week?

And then you apply your own custom business funnel to this.

So if you're Netflix, we're all familiar, the user sign up for Netflix, then they play videos in a loop.

Netflix is obviously very sticky, watching it a lot.

And then finally when the trial runs out you do subscription upgraded and you get access to more content.

Okay, so how do you collect data once you have this funnel?

You basically want to say user, user 1, 2, 3 in this case, has done user sign up event and they happen to be an organic user, which means they're not invited by someone else.

Then if you're Netflix you might say the user is video played and eventually subscription upgraded.

And so this is how you instrument your tracking in your mobile app or your web app.

So imagine you're Netflix and you're holding 1 of these video played events in your hands and you're wondering questions about it.

So, you know, what video is the user actually playing?

How far did the person get inside of the video, right?

Equivalently, if you're holding a subscription upgraded event, you're gonna want to derive monetization as a North Star metric.

So if you're a subscription business, you want to send your monthly recurring revenue.

If you're a transactional business like e-commerce or retail, you want to send the actual value of the transaction.

Okay, so then you push this out into your web app, your mobile app, then you start seeing the data come in, you look at the debugger, you see, okay, user signup is here, everything looks good.

You add your first analytics tool, we'll use Amplitude as a good example here.

Amplitude and Mixpanel are pretty awesome analytics tools out in the market right now.

And then you start seeing data flow inside of 1 of these analytics tools.

You can start seeing user signups growing as soon as you launch the real mobile or web app.

Okay, so now that you have analytics set up, it's time to focus on 3 different metrics.

The first 1 is the acquisition metric, signups per week.

It's really nice if you're a B2B business to cut this by the invite type.

So you have organic users which are just signing up from coming to your website, direct sign up.

And then some users are inviting other users, so those are invite type, right?

So when you're thinking about growth, it's really important to think about the organic user in that case, right?

Another example why event properties are important.

The way you create this is you go to amplitude, you say event segmentation report, user signup, next, here's your graph.

Just website, send data, gets to amplitude, and then you can see the amount of organic users every week.

And then if you're working on the acquisition step as a secondary metric.

You can basically say, today, 218 users organically signed up in the last week.

But by the end of the month, we want that to be at 300.

And we're going to execute projects A, B, and C this month.

And then we're going to watch this graph every single day on a TV dashboard in our office or our apartment, wherever we work, and then we're gonna see if our efforts are actually driving this, right?

So that's an idea of data-driven operation of a team.

You set a metric and a goal, and then you drive towards that every day.

So someone recently asked about retention.

So with retention, you wanna think about cohorts of users.

So you wanna say, Monday to Sunday, let's say December 10th through December 17th, 16 users signed up.

And then you look at those 16 users as they use your product on week 0, which is their sign up week, week 1, week 2, week 3, and week 4.

And the general idea here is like you can convince your mom or your grandma to use your product once, but even your mom or your grandma won't continue to come back and use your product over time every single week, right?

And so if you see users that are addicted that are coming back week over week, that's a really good sign of product market fit.

So this business can see that the December 10th cohort, only 6.25% of those users are still around on week 4.

So you probably want that to be somewhere between 20 or 30 at least.

And so you can set a goal, say I'm gonna talk to a bunch of these target users and try to figure out, you know, why they're not getting value out of the product and then make some changes as well.

Okay, so what metric do you actually pick?

This is taken from 1 of Gustav's slides, pretty awesome.

You think about what value your company is giving to your users.

So Airbnb gives you value by letting you stay at different rental properties around the world, right?

And they want you to do that at least 1 time a year, otherwise they consider you a churned user.

Equivalently, Facebook gives you value by letting you look at the news feed and, you know, connect with your friends, And

they want you to do that at least 1 time a year, otherwise they consider you a churned user.

Equivalently, Facebook gives you value by letting

you look at the news feed and connect with your friends, and they want you to do that at least daily or monthly once.

And so when you think about product market fit, you basically have these 2 different curves that happen.

So we have that cohort of 16 users that signed up in 1 week and we track them over time.

And so what ends up happening is for products that don't have product market fit, they end up tending to go to 0 because people just don't care about the product, right?

And that's the definition of product market fit.

For the tools and the products that do have product market fit, you'll see some kind of natural plateau.

It should be somewhere between 20 and 30% at least.

Luckily, both Mixpanel Amplitude have really awesome reports for this.

So in Amplitude, it's a retention analysis report.

You say users enter the cohort with user signup, and then they return with the video played or the subscription upgraded event.

And then you press next and out comes this graph and then you could look at four-week retention for cohorts improve the product and watch as new cohorts that strike the four-week mark do better or worse right and that shows you whether your changes week over week are actually improving.

Okay finally revenue So this is the primary metric that you want to be thinking about.

You'll use for a subscription business, the subscription upgraded event, you'll do a property sum over new plan monthly recurring revenue, you press next and outcomes your weekly net new revenue graph.

And then you could set monthly goals on top of this to make sure you're growing at the rates that you want to be.

OK, finally, if you have a founding team, it's really good to basically put all the stuff on a dashboard and then put this dashboard on a TV in your office.

Basically this is kind of the difference between being a data driven team and not a data driven team.

A lot of founders actually set up their analytics, but then don't look at them ever again because it can be painful, right?

While a data driven team will put it on a TV and talk about projects, talk about what those projects are actually changing the metrics that they're trying to drive, and then just completely understand the business every single day, right?

And this kind of company and this kind of founder will actually scale to build better high-performance companies because the next team of employees they hire will also be looking at that same TV dashboard and be driven off those same metrics.

Next what you want to do is have some kind of social accountability around your metrics.

So if you have your friends, your parents, your advisors, your investors, package up how your business is doing into an email.

This helps you synthesize what is actually happening and then send it out to to those advisors and tell them where the business is struggling and what your plan is to fix it.

This allows the advisors to quickly understand the business and then respond back with much more appropriate advice.

So these are tools that we recommend that help this kind of tactical process of setting up these metrics.

So I'm going to talk a little bit about that MVP business workflow that Michael talked about earlier.

Segment built about 7 different MVPs before we actually found Segment.

And all of those failed and eventually we found 1 that worked.

And the process of actually building an MEP is incredibly important.

So once you have that little experiment built, you want to enter private beta, which basically just means getting like 10, 20, 30 customers to actually try this product, and then having very direct lines of communication open with them.

What Segment does nowadays, every new product we ship, we open Slack channels with each 1 of our customers and we have the product managers sit in those Slack channels and talk with the customers.

For the products that don't get product market fit, the customers just stop responding and we're asking, asking, asking, they're not responding, right?

And for the products that do have product market fit, the customers are immediately being like, oh, why don't you have this feature?

I tried inviting my team in, this is not working.

So instead of you kind of pulling at the customer, the customer starts pulling at you.

That's a good feeling of product market fit.

Okay, so at some point, the private beta is going well, you feel like people really care about this, you understand your target customer, then you want to get a larger market segment to use it.

That's the launch that we talked about earlier.

And then a launch is just more users that you get to test product market fit on.

And so if you feel product market fit there, then you can start scaling the company, right?

And you hire salespeople and you start doing paid marketing and things like that.

So different tools will guide you throughout this process.

So, as you're building an MVP, and you're about to give it to the first group of customers, install Google Analytics, install Amplitude.

Google Analytics will tell you who's coming from the internet to your website, and then amplitude will tell you which features are they using, how engaged are they with that feature set.

Unless you're able to stand over the shoulders of all of your users 100% of the time, analytics is the next best alternative for that.

We also install live chat on the page, so either Slack with your customers, or if you can't do that, then maybe have a live chat available.

In the beginning of segment, customers would ping us day and night and that's where we got the most valuable feedback from them.

So just as many open channels of communication as possible.

Basically, if you have a non-technical co-founder on your team, they'll want to ask questions around the data.

And they'll always ask the technical co-founder who will have to provide the answers.

So data warehouse kind of democratizes the data, not only for the non-technical co-founders, but for everyone else in the company that you hire after.

Company dashboards, obviously, we should probably move that to the left.

Email and push tools, so as soon as users sign up, you want to send them an email.

So at some point you'll have so much support tickets if you start feeling product market fit and if they're all going to your Gmail, 1 founder will just get overwhelmed and not be able to answer them.

So you want to have a shared inbox where multiple founders can respond.

Okay, now I'm going to go through a few different recipes of these different tools that we found really helpful in product market fit.

So the first 1 is improving product usability.

Almost every product that's launched is unusable or highly unusable for the first 3 months while you have the kinks.

And we see this with every single product, no matter how much effort we put into it ahead of time.

As soon as customers hit it, they start using it in ways that you just don't expect.

And so there's this tool called Full Story, which helps you look at sessions of customers as they use your website.

This is a new feature in Personas, which is 1 of Segment's products.

And we launched them, the metrics looked horrible.

So like customers are coming in, but they weren't actually completing it, they weren't using the product.

And we thought, oh god, this likely doesn't have product market fit, we have to go back to the drawing board.

Then 1 of the designers on our team had this amazing idea, let's look at the full story.

And so we see this user going in about to start this creation workflow.

They find this button, They clearly don't understand what the button does.

And so we saw this with a multiple, with multiple different customers coming in.

And so we're like, okay, we just have to fix that button.

And so that's why it's important to have this type of viewing.

Either stand over your customer's shoulders, which is great, what the Stripe co-founders did, or get full story, which is a more scalable way to do that.

So when we launch segment, we would wait about 43 minutes, and we would email the co email the customer and say, Hey, I'm Ilia.

Thanks so much for signing up for segment.

Your next step here is to add a source to segment.

And if you have any questions at all, please email me or call me anytime I'm available for you.

Since we launched that email in 2013, we've had hundreds of thousands of responses to it.

So it's the connection between you and the customer over email that if they get confused, they'll respond to it.

What you can use is a tool called Customer I.O.

It's a behavioral email tool, which will say, every time a user signs up, wait 30 minutes, 40 minutes, 50 minutes, whatever, and then automatically send them this content.

And then you could template the first name, the company name and so forth based off of your analytics data.

And then finally, for democratizing data access, I would say this is more advanced.

So this is after you're in your MVP stage, you're feeling good about product market fit.

You might want to install Data Warehouse, like Google BigQuery.

And then Mode Analytics is a BI tool that works on top of it.

This lets you ask questions on top of the raw data that you might not be able to do in Amplitude and Mixpanel.

Just any kind of question you can ask with SQL and then even the non-technical co-founders will eventually pick up SQL and then start asking these questions themselves.

Okay, 1 common failure mode that we see with customers is trying to pick the perfect tool.

Mixed panel or amplitude, BigQuery or Redshift, and spending way, way, way too long thinking about that.

The truth of the matter is, you shouldn't optimize for picking the right tool right now.

Both amplitude and mixed panel will give you exactly the same result at your stage.

Instead, just get through that decision as quickly as possible, but set yourself up for change in the future.

So this is an example diagram that shows a customer of Segments that used different tools over a period of about 3 years.

And so you can see that they used about 8 different tools between 2015 and 2017.

Then they either hired someone or they decided that their tools were no longer doing the job and they switched from 1 set of tools to another.

So best-in-class tools change every 2 years.

Just Be prepared for a change and don't spend too much time trying to perfect your choice right now.

Okay, this is my recommendation of what tools are the best for the MVP process.

So, Google Analytics is for understanding what users are coming to your website.

Google BigQuery is to democratize data access with a data warehouse, which is just a database of data.

Mode is the tool you use on top of that to ask questions on top of BigQuery.

Intercom is like a list of all of your customers.

They're really good CRM for early stage folks.

Full Story is for improving product usability.

And Customer IOs for emailing your customers.

Once you get big enough, you can start using Google Ads, Facebook Ads, to actually do the paid acquisition.

Okay, so I'll just jump straight to the slide.

When we were really young, when we were in 2011, 2012, we didn't have a lot of money, and so we didn't want to use all of these different tools.

And so we used Google Spreadsheet for our CRM, we didn't use Asana or Trello because they were too expensive.

And then the only things we paid for were basically GitHub and AWS to host our product.

So today, happy to say that Segment is now free for early stage startups.

So for all you folks, there's a Biddley link down here that we'll send out afterwards, and it's in the deals.

And then also Segment went out and we did a bunch of deals with these customers so that if you're early stage you can now get all of these tools for free.

So enjoy that and start using them to basically accelerate your product market fit process.

So what have you seen in terms of user base to make the numbers meaningful?

See all Y Combinator transcripts on Youtube

Ilya Volodarsky - Analytics for Startups