Success metrics vs. counter metrics: Why you need both
I’m willing to say it: The advent of metrics-based product development is a miracle of the software age. Metrics allow us to define what success looks like and then measure our progress until our goals are definitively reached.
But, sometimes selling all out to pursue the prosperity of one or a few success metrics can wind up having unexpected effects on the business.
Counter metrics make sure that doesn’t happen. Here’s how.
Success metrics guide our efforts
Product development metrics come in a wide variety of forms, each with a variety of names depending on who you ask. However, most emphasis is placed on the metrics that measure a product’s success.
These “success metrics” or “goal metrics” are the metrics we use to decide which features and enhancements we should prioritize in our products. They are data-driven measures of what we believe success looks like expressed quantitatively so that success can be easily measured. They are the fundamental mechanism by which we know we’re on the right track, and they are essential to keeping our operations lean and effective.
For a given product, we might have piles of metrics that indicate success, and/or we might choose to prioritize a single “North Star metric” for maximal clarity and optimize all our efforts around that.
For example, let’s say we have a fitness app, and our mission is to make our users as physically fit as possible. There are lots of ways we might measure this, but we decide we want to cut through all the noise with a single North Star metric so our team isn’t distracted by too many paths forward. After a lot of deliberating, we decide that “time spent exercising (with fitness content in our app)” is what we want to pursue above all else. So we add the appropriate events and start roadmapping features likely to increase this metric.
One feature might include the ability to cast workout videos to your TV or other displays to make viewing exercises in the home easier. We might also track heart rate via Apple Watch and add motivational prompts to workout videos every time we detect heart rate has gone below what we’d expect of someone properly engaged in an exercise. Whatever we choose, the feature should increase time spent working out in a material way.
Seems reasonable, right? But what if it all backfires?
Prioritizing metrics always has consequences
When we decide on our success metrics (particularly North Star metrics), we make a commitment to pursuing activities that drive results for those metrics. That means we necessarily focus on some activities at the expense of others, and it also means the way we get our results may have unintended side effects.
Returning to the example of our fitness app, there’s lots of stuff we might have to say “no” to in order to prioritize “time spent working out.” We might deprioritize informational content in favor of more workout content. We might bolster our push notification strategy in lieu of accessibility features, or we might feature only the most popular workout instructors over those that appeal to more niche audiences.
Whatever the case, we can’t do it all, so we choose features that optimize for the success metrics that we decided have the highest return on investment for the business.
And that’s fine. In fact, it’s highly strategic for reaching our goals in a lean and sustainable way. The problem occurs when our assumptions don’t fully consider the big picture.
So let’s say we’ve succeeded in materially increasing our North Star metric. Time spent working out in the app is through the roof, and we’ve proven this definitively for the majority of our users across multiple demographics.
This might be a huge win, or it might be a disaster.
While time spent exercising may have gone up, it’s entirely possible that overall app engagement may go down as a result. Maybe our users really enjoy watching informational content between exercises. And maybe deprioritizing it will slash our overall engagement numbers in a way that just can’t be made up by exercise. After all, a person can only work out so many hours in a given day, but they can’t watch passive content with far fewer limitations.
Likewise, the over-emphasis on time spent exercising could give the impression that the app only fits for hardcore fitness enthusiasts and not the average person. That means users may be less likely to recommend the app to others, thereby negatively impacting user acquisition.
And that’s not all. Really any number of things can go wrong when we focus on one metric at the expense of others. And from a strategy standpoint, we may be totally fine with temporarily diminished user acquisition or lower overall engagement in favor or more exercise. If our goal is to win an app competition for most impactful on exercise habits for $1 million in funding, that might be all we care about until we secure the cash.
It can be dangerous to go in blind by focusing exclusively on success metrics, since they rarely provide visibility into peripheral consequences that we may very much like to avoid. What we need are additional metrics to make sure we don’t fall off a cliff as we gaze upon our North Star.
Counter metrics measure and avoid negative consequences
Counter metrics (also referred to as “guardrail metrics” or “health metrics”) are metrics that measure aspects of a product or business that may be impacted by optimizing around your success metrics. They’re an essential part of defining your success metrics because they prevent things from going terribly wrong by providing visibility that you would otherwise lack.
As such, for every single success metric, you should have one (or multiple) corresponding counter metrics that act as an ongoing sanity check. Lucky for you, deciding on your counter metrics simply involves a little creative exploration around possible consequences of pursuing your success metrics.
In fact, we already discovered two for our fitness app! Time spent consuming informational content is a counter metric to time spent exercising, because it ensures we avoid slashing our overall app engagement and therefore our funding. Net promoter score (aka NPS or the likelihood someone will recommend our app to others) is also a valuable counter metric to time spent exercising because it ensures we aren’t over-indexing on too small a subset of our users at the expense of others we may be interested in pursuing.
That said, it’s important to understand that counter metrics need to be weighed with the priorities of your business. It may be strategic to focus only on a small subset of your potential users before expanding to others. Alternatively, you may need to sacrifice overall engagement for something like time spent exercising in part or in whole. So when you’re deciding which counter metrics to choose, you should identify possible side effects, but also question the degree to which those side effects are worth measuring for your goals.
If you’re struggling to discover counter metrics, a great place to start is with metrics that generally evaluate the health of the product/business and work back from there. For example, high-level app engagement is typically a priority for most apps, so to arrive at the counter metric of “time spent consuming informational content” we simply need to ask ourselves, “Aside from fitness content, what other factors impact overall engagement?”
Keep the overall product in mind at all times
One way to remember how important it can be to balance your metrics is to understand that the dance of success metrics vs. counter metrics is really just an applied version of data thinking vs. product thinking. If you keep the bigger picture of your product’s health in mind at all times, it’s harder to be seduced by the flashy success of one metric or another. When in doubt, zoom out!
About Joseph Pacheco
Joseph is the founder of App Boss, a knowledge source for idea people (with little or no tech background) to turn their apps into viable businesses. He’s been developing apps for almost as long as the App Store has existed—wearing every hat from full-time engineer to product manager, UX designer, founder, content creator, and technical co-founder. He’s also given technical interviews to 1,400 software engineers who have gone on to accept roles at Apple, Dropbox, Yelp, and other major Bay Area firms.
Gain insights into how best to convert, engage, and retain your users with Mixpanel’s powerful product analytics. Try it free.
