Choosing, refining, and tracking product metrics
Why do smart product teams often build products with mediocre or no impact? Often, the answer is found in the discipline with which a product team measures the product.
Choosing your metrics: Metrics categories
When selecting your metrics, you’ll want to consider a few different categories. These categories cover multiple granularities and perspectives that will help you make rigorous product decisions. In general, the following categories address questions that will give you a well-rounded understanding of the product’s value for its users and your business.
Note, a given metric can belong to more than one category. At the same time, it may make sense to skip a certain category altogether for your product, depending on the product’s unique context and its maturity.
1. Health metrics
It’s vital that your team understands how well the product is performing, and is aware of potential issues.
Key question addressed: Is the product available and performing in the manner that users would reasonably expect?
Common examples that fall within this category include:
- Latency: The delay between a user’s action and the application’s response.
- Initial load time: The time it takes for the page to appear to the user.
- Uptime: How long or how reliably a system has been running.
- Data loss rate: The intentional or unintentional loss of data caused by forces within or outside an organization. This could happen through normal processes like data migration or through malicious attacks.
- HTTP errors: The number of requests that ended in an error for the end-user.
2. Usage metrics
The success of a product largely depends on how users engage with it, as well as how often (and if) they come back for more. Usage metrics should almost always be among the most vital inputs for product decisions.
Key question addressed: How are users using the product?
Examples include:
- Time-of-day/day-of-week trends: When is traffic highest and lowest?
- Top N actions: The main actions that users perform in your product.
- Funnel metrics: Volume, velocity, and conversion metrics related to the funnel.
- Help docs usage: How users engage with help resources.
- Password reset rate: Friction associated with passwords can reduce conversions and significantly affect business. It’s important to balance security with ease of shopping or signing-in.
3. Adoption metrics
Adoption metrics give you a view into the user experience, but from a different perspective than usage metrics. They focus on how your users adopt the product and how they react to a new feature or service.
Key question addressed: Is the product (along with its key features) being used as much as we’d hope, and in the ways that we’d like?
Teams often track the following adoption metrics:
- Active users: The number of users engaging with the product. This is often measured as daily active users (DAUs), weekly active users (WAUs), and monthly active users (MAUs).
- N of M day usage: A measure of how many days/week someone used the product (e.g. 3 out of 7 days).
- Strategic feature adoption trends: Answer questions like, "Which features have the lowest and highest adoption rate?", "Which features are rarely or never used?", and "What percentage of DAUs adopted the new feature?"
- Free-to-paid conversions: The number of free customers that become paid customers after a defined time period.
4. Satisfaction metrics
Satisfied customers are most likely to spread the word about your product. Customer satisfaction impacts not only your other key metrics but also your company’s brand.
Key question addressed: What is our customers’ overall sentiment towards the product or its main features?
Examples include:
- Overall CSAT: This is customers’ overall satisfaction with your product or service.
- New feature CSAT: The customers’ satisfaction score of a newly launched feature.
- Support CSAT: The customer satisfaction score for the support services offered with your product.
5. Ecosystem metrics
A product doesn’t operate in isolation. Tracking a product’s position within its macro-environment can help teams keep their eyes on bigger opportunities and long-term planning.
Key question addressed: What is the macro state of the product within the domain in which it operates?
Common metrics to regularly track are:
- Share of wallet: This represents the number of dollars a customer spends on the product rather than on competing products.
- Third-party integrations: Third-party integrations make a product more “sticky” and valuable for customers. Customers can qualify or disqualify products based on integrations with other applications.
- Industry rank: A comparison to competitors based on factors such as estimated market value, number of customers, customer reviews, potential for growth and other factors.
- Market share within target segments: The percentage of the addressable market your product has penetrated.
6. Outcome metrics
"For some companies, the product is the business. For others, it delivers business results."
Key question addressed: What overall business results are we seeing from this product?
The metrics chosen are unique to each business, but often include:
- Revenue: Companies track how a product impacts the overall revenue.
- Gross margin: The difference between total revenue and Cost of Goods Sold divided by total revenue.
- Active users: The number of users who have interacted with the product within a defined time period.
- % of Fortune 1000 covered: The number of Fortune 1000 companies that have adopted the product (used by companies selling into large enterprises).
For any product, you’ll likely track dozens of metrics (especially in the usage metrics category). At the same time, you don’t want to be inundated with data to the point where it's difficult to see the big picture.
Refining your metrics: Key metrics and leading indicator metrics
I recommend defining a small set of metrics (3-5) that'll be faithful indicators of your product’s progress towards its strategy and goals. These are your Key Metrics (KMs).
Key metrics
At the highest level, a company typically chooses a north star metric that’s responsive to product changes, and is a holistic representation of the product’s value for users.
However, a north star doesn’t live alone. Rather, metrics measure a complex product with different dimensions. As Reforge CEO Brian Balfour says, “most products should have four high-level metrics: an acquisition metric, retention metric, engagement metric, and monetization metric. These four act as a system where one can influence the other.”
By capturing a different dimension within each product, product managers can better understand the context behind the numbers, and find a balance that will move the product forward.
When it comes to the criteria for choosing your key metrics, there isn’t a set of hard rules. The criteria I use for key metrics are:
- It must be responsive to product changes.
- It’s an aggregate measure of the product’s value for its users.
- It can be readily tied to business value.
- It’s expected to be long lasting (at least two to three years).
KMs give a great bird’s-eye view of how the product is doing. You’ll typically create annual goals and targets for your KMs. Your company’s executives and board will also likely look at these metrics to assess the product’s progress.
Some teams avoid using a certain indicator as a KM because there are outside factors that impact the metric. For example, a team will say: “for this, I depend on sales, so I can't use it as a KM." It’s true that sales impacts some KMs you might choose, but I don’t worry about having 100% control. There's no such thing as 100% control anyway.
"In healthy companies with a strong sense of individual ownership, the thinking goes: If sales can’t or doesn’t sell my product, that’s still my problem."
In certain cases (e.g. B2B SaaS with long adoption cycles), KMs are lagging indicators of work done (e.g. revenue generation). With long sales cycles, it’s hard to confirm how features or performance is impacting revenue.
That’s why teams also need a small set of leading indicators, which I call Leading Metrics (LMs).
Leading metrics
LMs help you understand activity, changes, and experiments within a shorter time frame.
For example, over the next two to three quarters, you might have an LM for adoption of a particular strategic feature (which over time will move your KMs), an LM for country availability, or an LM for self-serve funnel success.
For companies that choose KMs that are lagging indicators, I recommend also defining 3-5 LMs at a time. LMs can change more frequently as they measure overall product progress and business growth.
Best practices for tracking metrics
Now that you’ve chosen and refined your metrics, it’s time to put them to use. Some tactics I’ve found useful for using metrics:
- Treat your metrics dashboard like a product.
- Actually track its usage (meta!).
- Set a Chrome pinned tab for the dash.
- Review your pinned tab on set days (I do it every morning).
- Set a weekly email report (people love emailed metrics).
Words of caution: False precision, decision support, and excuses
When setting metrics, beware of false precision. Often, when it comes time to project our metrics for next year, we put in a lot of work to pretend that we’re being highly analytical. The reality is that there’s no way to accurately predict certain numbers. Too much energy spent on precise predictions is a waste for most early- and mid-stage products.
When in doubt, set qualitative goals that are the closest possible proxy to the numbers you want to achieve.
Closing thoughts
To sum up my key advice for those leading their team in choosing the right product metrics:
- Categorization is effective: Using the 6 metrics categories above will help ensure you’re thinking holistically about user value and business value.
- Choose a limited group of KMs and LMs: 3-5 key metrics and 3-5 leading metrics have worked quite well for me on most products, across both B2B products and consumer products.
- Remember to be metrics-informed, not metrics-driven: If your team does not understand the “why” behind a given metric and yet proceeds to impact that metric, it’s likely that it's being driven by metrics rather than being informed by them.
The ability to break the rules—and to know when—is often what distinguishes great PMs from the others. A great PM will diligently track metrics, but will rely on them as one source of input among many.