How to use feedback to create a roadmap

One of the best things I learned at Amazon was that customers are always “beautifully, wonderfully dissatisfied,” and no matter what you give them, they’ll tell you why it doesn’t work for them. These customer requests or complaints—we call them “product gaps”—are a true gift if you effectively capture and prioritize them.

At Mixpanel, we’ve developed a system where everybody in a position to talk to a customer can log a product gap. This system helps us capture and categorize product gaps based on the impact and severity of the problem. As a result, we are able to collect more customer feedback and build a better product.

Below, I’ll outline the Mixpanel approach to product gaps and how you can use it to inform your product roadmap.

A product gap emerges when a customer tells you that something doesn’t work for them. Often, they will tell you a solution (“I just need a feature that does XYZ”)—but if you’re going to discover a product gap with potential to become a useful product feature, you have to find the actual problem this customer is trying to solve.

This approach works better for two reasons. First, different customers might be experiencing the same problem, yet they all imagine a different solution. So if you capture just the suggested solution, you’ll end up building a dozen different solutions that all have one underlying problem. Second, as Steve Jobs put it, “A lot of times, people don’t know what they want until you show it to them.” The very hard and important task of a product manager is to anticipate people’s needs and wants.

I’ll give you an example: a customer may tell you that they need CSV export. But if you ask “why?” you might discover that what they actually need is the ability to use data in a third-party tool— something that could be achieved through an integration. The example below shows the two ways you could document this product gap: as a solution or as a need.

While the CSV export solves the immediate problem of this customer, it might not be the best way to satisfy the real customer need.

To capture customer feedback effectively and at scale, we’ve made documenting product gaps part of the new hire onboarding. Every new customer-facing employee at Mixpanel learns how to identify product gaps; how to write a comprehensive description; and how to correctly record the reason the client needs this problem to be solved—or what question the client is trying to answer.

We’ve included “Product Gaps” as a custom field in Salesforce and added a Zendesk connector so our support team can quickly turn any support ticket into a product gap. Once logged, product gaps are routed automatically into a Google spreadsheet, and the team gets notified via Slack.

The spreadsheet can be used to sort product gaps by features, severity, and ARR (Annual Recurring Revenue), which speeds up prioritization and new feature planning. Any change in the spreadsheet is automatically reflected in Salesforce records. When a product gap gets fixed, our support team is notified and follows up with the person who reported the gap, closing the customer feedback loop.

Product gaps inform our roadmap. As much as we’d like to solve all of our customers’ problems, we have to prioritize what needs to be done based on urgency and impact. Very simply, our prioritization comes down to the following factors:

The severity of the problem, the number of customers affected, and business impact can be measured using Weighted ARR. Weighted ARR starts with the total ARR of the customer that reported the gap divided by the number of gaps that this customer has reported, to measure the business impact of each gap.

But all gaps are not equal in how much they matter to the customer. When product gaps are logged, they get assigned an impact ranking on a scale from “nice-to-have” to “deal breaker,” where adding a new color scheme, for example, would rank lower than, say, a usability flaw that causes people to spend too much time on critical tasks. Gaps with a higher impact are allocated a higher proportion of the customer’s ARR; gaps with a lower impact are allocated a lower proportion of the customer’s ARR. For each gap, we total up the weighted ARRs from all the customers that have reported that gap.

Last but not least, we also look at the strategic fit. We ask ourselves: Is the gap aligned with our vision of providing the most powerful self-serve user analysis tools to the teams behind the world’s leading digital products? A gap that aligns with our overall product direction will get a higher priority. A gap that is outside of the scope of our product vision will likely get the lowest priority

Our customers said that they wanted to compare funnels broken down by user segment. But, as we dug deeper to identify the underlying problem, we found that what they really wanted was to know which segments drove conversion. So we created On Demand Analysis for Funnels, which automatically finds the segments that have the biggest impact on conversion. Now, without having to compare different funnel breakdowns, our customers can quickly get the insights they need.

Customer feedback can be incredibly valuable if you apply it correctly. Our approach is to look for the “why?” behind customer requests to uncover the larger needs that customers may not be aware of. To find such needs, we make it a priority to document all feedback, enabling our employees with training and tools to identify product gaps and share them with the team. Finally, we have a system for prioritization that helps us focus our time on things that matter the most.