Why we decided to retire our most popular report
Today, Mixpanel says goodbye to Segmentation. Segmentation has been our most heavily used report since its release in late 2011, and until recently, it was the report that greeted users when they logged in. It answered literally millions (35.3 million, to be more precise) of queries for our customers. Retiring the report is a bittersweet moment for many of us here, but a necessary step to deliver more advanced analytics capabilities to our customers.
The Insights report, which replaces Segmentation as the go-to report for data exploration in Mixpanel, can answer any question Segmentation could, plus many more. As the product manager responsible for Insights and Segmentation, I felt I should share some context on the evolution of Mixpanel’s data exploration reports, the logic behind our choice to invest further in Insights, our process for transitioning from Segmentation to Insights smoothly, and the lessons we’ve learned along the way.
A brief Mixpanel history lesson
In the beginning, there was Trends. Trends was Mixpanel’s very first report, dating back to 2009, and the original ancestor of Segmentation and Insights.
Trends kicked off the party by allowing users to view event counts over time, and… that’s about it. Our early customers clamored for a means to dig deeper into their data, and in 2011, Segmentation was the answer. It added the ability to filter and break down metrics by properties, along with new bar chart, pie chart, and table visualizations.
Segmentation could answer the majority of questions a product manager or data analyst might want to answer, but ultimately Segmentation is limited to only one of Mixpanel’s two datasets. Segmentation supports Events (the set of actions taken by users), but not People (the profiles for those users, which include actions and characteristics like account status).
This became a problem in in 2015 when Mixpanel introduced support for queries utilizing both the Events and People datasets. Segmentation could run an Events query, filtered or grouped by a People property, but could not run a query on People profiles themselves. For example, you would not be able to answer the question “what is the distribution of my user base across the world’s countries?”
When Insights was released in 2017, we knew it would eventually replace Segmentation. Right off the bat, Insights could analyze both the Events and People datasets in Mixpanel. You could, to recall the earlier example, determine the worldwide distribution of your user base in just a few clicks. And unlike Segmentation, it was built with the future in mind.
That said, when we debuted Insights, it still fell short of Segmentation in some areas. There were some queries Segmentation was better equipped to handle, and some that Insights was better equipped to handle. But in the past two years, we’ve not only brought the two reports to functional parity, but added heaps of new functionality in Insights.
Why Insights over Segmentation?
Now, Insights can do everything Segmentation can, and much more. In Insights, we’ve been able to build myriad features Segmentation would struggle to support, including:
- Cohort analysis to understand how specific groups of users behave
- Formulas to calculate custom metrics and KPIs
- Time range comparisons to see how trends change month over month, year over year, etc.
- Automatic alerts with explanations, so you can act quickly on surprising anomalies
Not only is Insights the more capable report, but it’s more flexible to build upon for the future as well. For example, in the Segmentation era, we had to build an entire separate report, cleverly named Formulas, to calculate… formulas. With Insights, we could build formulas right in. It’s the perfect platform upon which to build even more advanced analytical tools, many of which you’ll see released in 2019.
Of course, we wouldn’t be Mixpanel if we were not data-informed in this decision as well. We took a close look at engagement and retention metrics for the two reports, and noted that those metrics were eerily similar. Since there was no significant quantitative difference in the two reports’ impact on our business, we could be more confident in our qualitative reasoning to move forward with Insights alone.
Why not keep both reports?
Ultimately, Segmentation and Insights fulfill the same purpose – enabling our customers to quickly and easily explore their data. For a Mixpanel user, the presence of two reports that could answer the same question is confusing, so downsizing to one data exploration report makes Mixpanel simpler to use. Similarly, both reports require our engineering team’s hard work to improve, or even maintain. By removing redundancy, our team can move faster to deliver new, high-value features to our customers.
Decision made. But what about the execution?
Naturally, removing Mixpanel’s most-used report, even when replacing it with a superior alternative, is an action that should be taken carefully. Our top priorities were as follows:
- Ensure the transition is smooth and unsurprising for customers
- Ensure the transition does not sacrifice any functionality
Our process began with an early message to customers. We notified all Segmentation users about the upcoming change a full five months in advance, leaving them plenty of time to make adjustments, like converting Segmentation reports to their equivalent Insights reports.
We removed as much friction from that conversion process as possible by building an automatic conversion tool – a user need merely click one button to get their most important Segmentation reports in Insights instead.
Following the announcement, we reached out to our customers to ensure that their needs were fully met in Insights. They helped us prioritize features such as pie charts and annotations, and alerted us to some edge cases where one report was more capable than the other. Having built those features into Insights (thank you, flexibility of the Insights platform!), we can be confident that the transition will retain all functionality.
De-risking the change
Despite the reassurance from our customers, we wanted to be positive that the change would not have an adverse effect on engagement with Mixpanel overall. To get an early indication of how the change would impact our metrics, we made Insights replace Segmentation as the default landing report.
Specifically, we monitored daily report views and queries made, DAU / WAU, and 1-week retention for both new and existing users, comparing the numbers before and after changing the default report to Insights. We even used Insights’ time comparison feature to easily view the 30 days before and after the change was made.
Happily, the metrics told a positive story—Insights usage almost doubled that of Segmentation, with no loss to retention. With that data in hand, we could confidently announce the final day of support for Segmentation, giving our customers another chance to make the switch, and finally, to pull the trigger on retiring Segmentation today.
What’s coming next?
Segmentation can still be found in Mixpanel’s “Applications” folder, where it will remain while we continue to monitor our metrics to be sure we made the right decision.
As for Insights, you can look forward to some exciting new enhancements, enabling more powerful calculated metrics, simple report sharing and presenting, and more. We’re excited for this next chapter in Mixpanel’s data exploration report saga, and we can’t wait to build an even more powerful Insights tool.