Is your product on the right track?

What do you do when you just know your plan is working, but can’t prove it? Whether you’re trying to win a football game–or your market–the answer is to dig deeper into the data.

Back in 1985, now-legendary San Francisco 49ers coach Bill Walsh was brought in to turn around the worst team in the National Football League (NFL). By many accounts, it was a hopeless situation, and after his first two seasons, it seemed as if very little progress had been made.

Walsh needed additional facts to prove to the 49ers’ owner, and himself, that the team was actually on the cusp of success. He outlined the approach in his book, The Score Takes Care of Itself:

“Often it takes a keen eye and a strong stomach to dig through the “ruins” of your results for meaningful facts. A season’s won-lost record (or your market share, sales figures, stock price) may not—will not—tell you what you need to know to be fully informed about the strength of your organization. Thus, I looked for clues that might indicate whether we were moving in the right direction at the right speed and, if not, what we needed to do to address the problems.”

Walsh’s analysis of how and why the 49ers won and lost revealed some critical gaps in the team’s roster, but also some encouraging signs. Most of their losses had been by only a few points. And they had won three of their final five games, meaning the team was buying into his philosophy and hadn’t given up on what was already a failed season. 

Following his honest assessment of the organization, Walsh made some shrewd roster moves and strategy adjustments. The next year, the 49ers came “out of nowhere” to win the Super Bowl, then won four more over the following 12 years. Without unearthing the leading indicators of success, the organization may have scrapped a plan that was working.

What is your data telling you about your product’s trajectory and where you can improve?

For most products, the ultimate “success” is some form of board or investor metric that directly creates value for the business, like revenue or number of subscribers – the equivalent of the win-loss record in football. But identifying the factors that move those numbers up and down is the only way to know your true trajectory.

Let’s look at two quick examples based on a subscription video service:

1. Your subscriber numbers are well beneath your targets, but retention is high.

This means your product is well-liked, but you have an acquisition problem. An analysis of your power users will show the types of people to target, and some savvy marketing campaigns should bring in new users who value your product, stick around, and generate recurring revenue for the business.

2. Subscriber growth is high but engagement is low.

It appears you have a powerful acquisition engine but a flawed product experience. It might be that the content is subpar, or that the recommendation engine lacks precision. Either way, there is cause for concern and you need to dig deeper into why users aren’t engaging to prevent the bottom from dropping out.

This interplay of metrics is why you need a nuanced view of product health, and we suggest having a “constellation of complementary metrics” rather than a single “north star.”

Are you sure you’ve looked deep enough and found all the critical information?

Bill Walsh didn’t win five Super Bowls by doing things like every other coach. He was meticulous in evaluating his organization’s strengths and weaknesses, and never passed up an opportunity to make his team better.

“My observation is that two leaders—coaches—looking at the same information will not see the same thing. The one who’s a more skilled analyst, who digs deeper and wider, will benefit more. It is an endeavor to which I allocated as much energy as my preparation for every game and opponent.”

Digging deep into product data is all about segmentation. Just as Walsh broke down the 49ers’ performance by offense, defense, and each position within those groups, you need to understand your product at a granular level. Here are some common examples:

• How does engagement vary by platform, region, time of day, and operating system?
• Why did that sharp dip in usage on <insert date> happen?
• Where in the onboarding flow did newly-acquired users drop off? Why?
• Which users are close to the goal and should be nudged?
• What is the tipping point of engagement that usually gets free users to purchase a paid plan?
• Who are the power users and what behaviors do they have in common?

As Walsh indicated, “more is more” when it comes to digging into your strengths and weaknesses. If you’re hitting a wall when trying to answer any of the questions above, you may need an analytics tool with more flexibility. (Hint: Mixpanel can help with all of them.)

More information doesn’t have to mean more time spent on analysis

Getting the full value out of your data almost always involves human interpretation, and is the difference between being data-driven and data-informed. But there are tools to speed up insight generation.

For example, Mixpanel can:

• Search across all your dashboards and saved reports to find anomalies you may not have known about.
• Identify correlations between user behaviors and long-term retention, so you know how to nudge your new users.
• Automatically segment your funnel by dozens of properties, at your request, and surface the factors driving conversion up or down. 
• Tell you which launches and experiments worked, while accounting for statistical significance.

The score will take care of itself

To bring this full circle, Bill Walsh, like any savvy product owner, was successful because he looked beyond the obvious facts that everyone else could see. He always knew how his team was doing, saw the hidden signs of trouble (and success), and quickly acted to fix any areas of weakness. Whether in sports or products, the best are always looking for ways to improve, and assume success will follow. In his own words:

“It all comes down to intelligently and relentlessly seeking solutions that will increase your chance of prevailing. When you do that, the score will take care of itself.”