Building Metrics that Matter

My framework for building great product metrics

Metrics are my thing. I love thinking about, talking about, and working with metrics. I believe metrics can make or break products. When done right, they can supercharge a product and create a great user experience, but when done wrong, they can create a recipe for disaster.

Why you should care about metrics

1. Metrics help you do meaningful stuff Metrics allow us to see how our product performs. They allow us to create benchmarks and baselines, and this allows for iteration to make the product better.

2. Metrics help you make rational decisions Deciding what to build next is easy when we have an understanding of what needles it would move. Metrics allow for removing biases coming from opinions and limited world views.

3. Metrics can minimise the impact of breakages Continuously monitoring metrics helps quickly identify problems, raise alarms at the right time, and fix stuff fast.

4. Metrics help you truly understand your product Metrics challenge us to understand the reason for the existence of a product. They bring clarity around what a product accomplishes for a user, and why.

Without metrics, we would be building meaningless stuff.

The 6S framework

Over the years I've had some great successes building metrics, and some fairly bad failures. Through trial and error I came up with a 6 step framework (The 6S framework? The Sixes framework? I'm bad with names, help please!) to build the right metrics for my products. Here are the steps:

1. Understand the product goal

2. Define the North Star

3. Map the user flow and product actions

4. Define supporting metrics

5. Track funnel metrics

6. Adjust for measurability

The rest of this post elaborates these steps and gives examples for each.

Before jumping in, a disclaimer. I use 'user flows' and 'product goal' very loosely here. I've bent the definition for the purpose of metric creation. If you're looking for definitions of user flows and product goals, this is not the right place.

Here we go then.

Understand the product goal

A simple yet functional product goal should give you visibility into what a user wants to accomplish by using the product. At this stage, don't worry about making the product goal super measurable.

A handy product goal template:

Build <1>_____ for <2>______ to <3>______. This will generate impact by <4>_________.

<1> Describes the thing you're building. Is it a marketplace? An app? A platform? As you fill this in, think about why you chose this thing. Is it because it was scalable? Or because it reached a large number of users? Or something else?

<2> Describes who it's being built for. Everyone with internet access? Single parents? Millennials? As you fill this, think about why you chose this segment. How are this segment's needs different from others?

<3> Describes what the product does. Shop online? Talk to friends? Log their runs? Think about why doing this is important for the users you're building for. How do they currently do it?

<4> Describes the positive impact your product will generate for your users. Will it add convenience? Make them feel part of the community? Encourage a healthy lifestyle?

Once you're comfortable with a tentative goal, head over to the next step.

Define a north star metric

Define ONE north star metric. A good north star metric has three qualities:

1. Answers the question 'Are we making progress towards our goal?

2. Is measurable - and by this I mean you can put a counter on it. Avoid metrics like Net Promoter Scores or Customer Satisfaction Scores here.

3. Speaks to the user as well as the business. Ideally, business and user goals have a strong overlap. This makes representing both sides in your metric easy.

Here are some examples:

Click on the image to enlarge.

In my example, for an e-commerce marketplace that makes shopping more convenient for everyone, the number of transactions is probably the best primary metric. Let's check if it satisfies the criteria for a good north star metric. It adequately shows progress towards our goal - the more convenient it is to buy on our marketplace, the more the number of users, and the more the number of transactions. The number of transactions is highly measurable. It can be compared to past data, and forecasts about it can be made in the future. It takes the user view into account by ensuring the user goals are met, but also the business view - transactions and revenue have a strong correlation, and the more the number of transactions, the higher the revenue (given that we optimally scale operational costs).

A similar analogy can be drawn for all other examples in the image.

Once you have defined a north star metric, head over to the next step.

Map the user flow and product actions

A north star metric is not enough. It's a great way to see if the product as a whole is moving in the right direction, but doesn't tell us how or why this is happening. This means these is low visibility into the underlying performance. Is growth the result of a new feature we built? Are all the SEO efforts (finally) paying off? Or is it something else? A north star metric doesn't tell you why the product works. This means it's not actionable. It doesn't tell us the next step to take, in order to reach our goal.

It's important to build good secondary metrics in order to get that zoomed in view into the user journey. The first step to deriving secondary metrics is to map out the user flow. A user flow defines the broad steps a user must take to accomplish their goal. Make it high level and generic. I would encourage you to ignore the details. A good user flow:

1. Answers 'What are the steps a user must take to accomplish their goal with the product?'

2. Answers 'What steps lead us to our North Star?'

3. Highlights only essential steps - detailed enough for useful insights, but not so detailed that it distracts from the core of the product.

4. Considers the happy as well as unhappy path.

From here, derive the product actions you can take to help the user at each step. A product action is one change that you can make in one step of the user flow in order to make it easier for the user to accomplish their goal. While you can take multiple actions, confine yourself to one or two large actions that you can take at this step.

And there you have it. You have mapped out the user flow and the product actions for your product. Iterate a few times and make sure you feel comfortable with what you have.

Now for the fun part.

Derive secondary metrics

Once you have the product actions in place, start to think about how you would measure them. In our example, if we have made the marketplace more discoverable, either through ad campaigns or through affiliates or referrals, or any other methods, then we should see the number of new users on our platform rise. So, the supporting metric for the product action 'make marketplace more discoverable' becomes 'new users per day'. The process for getting from product action to supporting metric is exactly the same as the process for getting from product goal to north star metric! Think of the product action as your goal, and you will have a solid supporting metric.

The following example illustrates this point:

Easy, right? This is very useful for us to understand changes to our north star metric. It's also super useful for us to make changes to our product for impacting our north star metric. In my example, I know that if a user gets their delivery on time, they are more likely to make a second purchase. If therefore I make my deliveries faster, I can increase the number of transactions per day!

So far, we have a good goal for our product. Based on the goal, we have the north star metric we want to work with. Then, we have some supporting metrics that help us understand the user journey and come up with reasons for changes in the north star metric. All we need now is for some way to debug a newly built feature, identify friction, and iterate. Let's move on to the next step then.

Practical tip: Consider launching the product to all your users only if the north star metric moves in the direction you want it to, AND all supporting metrics reflect either no change or positive change.

Track funnel metrics

Funnel metrics allow for finer iterations in specific features you are building. We often work on one or two features at a time rather than the whole funnel Sometimes, product teams have limited span of control. In these scenarios, funnel metrics help debug and iterate on smaller features.

The process is the same as for deriving secondary metrics, except we look at one feature now instead of the entire product. Let's imagine again you're working on the e-commerce marketplace. Your primary metric of transactions per day is flatlining despite new users coming in. Your secondary metrics tell you that you need to increase the transaction frequency per user. So you decide to build a loyalty program for your users. Your hypothesis is, by having a loyalty program, users will be more likely to make a second and third purchase. This will increase your transaction frequency, and that in turn will increase the number of transactions per day.

When launching this new loyalty program you want to be able to see the finer details of what users are doing. So we repeat the steps for secondary metrics, but with a more detailed funnel.

So here we have our funnel metrics. This way, if the loyalty program is not successful, you can easily check if it was because not enough users see it (not enough new exposures per day), or not enough users buy it (low checkout initiations per day) or something else.

The exercise for deriving the product goal, north star metric, and the supporting metric, need not happen too frequently. Typically it happens once a year or so, depending upon your velocity and changes in business. Funnel metrics, however, need to be derived each time you build something new.

And we're done deriving our metrics! Let's see what we have for our e-commerce loyalty program example:

Goal: Build a marketplace for everyone to shop online. This will make shopping more convenient.

North Star Metric: Number of Transactions

Secondary (Supporting) Metrics:

1. New users per day

2. Product Category Selections

3. Cart adds

4. Drop offs at payments

5. On time deliveries

6. Number of returns

7. Transaction frequency per user

Funnel metrics:

1. New exposures to loyalty program per day

2. Page visits to loyalty program per day

3. Checkout initiations for loyalty program purchase per day

4. New signups to loyalty program per day

Now for the final step.

Adjust for measurability

First, eliminate from the list any metrics that you have zero influence over. If you're working on the loyalty program, you probably have little influence on 'Dropoffs at payments', so remove it from the list of supporting metrics. This makes your list more manageable and the data easier to interpret.

Second, tweak your north star metric depending on your situation. For example, if you're working to make deliveries faster and more efficient for the e-commerce platform, because you are so far removed from the conversion funnel, and because customers will have to get a few deliveries fast before they begin to see a pattern, you will have to wait a while before you see the impact of your product on the number of transactions. In such cases, pick the secondary metric you can best impact to be your personal north-star. We call this the primary metric. In our example, it will be 'number of on-time deliveries'. Once you have done this, add the north star back to your list of supporting metrics.

In this case, your final list of metrics should look like this:

Goal: Build a marketplace for everyone to shop online. This will make shopping more convenient.

Primary Metric: On time deliveries

Secondary (Supporting) Metrics:

1. Number of Transactions

2. New users per day

3. Number of returns

4. Transaction frequency per user

Funnel metrics:

1. New exposures to loyalty program per day

2. Page visits to loyalty program per day

3. Checkout initiations for loyalty program purchase per day

4. New signups to loyalty program per day

And that's done! Remember that building metrics is an iterative process that demands patience. You will have to do it a few times before getting it right, and even then, you may have to keep iterating. Just as anything else with product management, stuff is best learnt on the job!

I made a worksheet out of framework. It looks like the picture below, with pre-filled cells. It makes creating metrics 10x faster.

If you've subscribed to my newsletter, it will be in your mailbox on Wednesday. If you haven't, subscribe to get your copy!