How often should I use product analytics?
Frequent analysis is essential to making data-informed decisions about your product. But what does “frequent” really mean? To a certain extent, the frequency with which you should be analyzing your data depends on your role and your product’s unique value moment. At Mixpanel, we define value moments as users saving, exporting, or adding a report to a dashboard. But in our experience, once teams discover how easy it is to run reports and see their data in action, it quickly becomes a regular part of their workday.
How regular? Well, the median Data-Curious user runs 7 queries per day, while Data-Informed users run more than twice as many. Data-Informed users are also more spontaneous, deliberate, and curious about their data, as evidenced by the fact that they view, create and save queries 3x more than Data-Curious users do. In addition, Data-Informed users engage with their product data more often, answering questions as they arise rather than time-bucketing their analysis in larger blocks. Even though they make up just 10% of the users we analyzed for the report, Data-Informed users run over 40% of all queries.
Data-Informed users are intentional about incorporating data into their work. They’ve developed the habit of asking questions of their data, and rely on that data to guide product decisions. Just like some folks find it more effective to keep a room clean by tidying on a consistent basis, the same goes for users that depend on data for iterative product development.
Data-Curious users engage less often and have shorter sessions. While they definitely monitor their data, they don’t spend as much time identifying why the numbers have changed as Data-Informed users do. Data-Curious users tend to keep a pulse on their data and are satisfied with knowing the status of a metric—e.g., whether logins are up or down—but aren’t as likely to explore why those numbers change as Data-Informed users.
Resource from Mind the Product
Jorge Rodriguez-Ramos considers how you can best incorporate data-driven decisions into product management.
How long should I spend doing analysis?
Session lengths vary depending on what you’re trying to learn from your data or what question you want to answer. We discovered that analysis doesn’t necessarily mean digging into data all day. In fact, most teams use analysis to be informed on high-level product health and overall performance—our study reveals that the median session length for Data-Curious users of product analysis is just under 5 minutes.
For the top 10% of product analysts, however, five minutes isn’t nearly enough time to satisfy their curiosity. Data-Informed users’ median session length is over 10 minutes, 2x longer than that of Data-Curious users. What’s more, median Data-Informed users also have 5x the number of daily sessions of median Data-Curious users. If our experience advising product teams is any indicator, one question often leads to another in an attempt to understand what their users are doing.
Try it yourself
See how a media company breaks down total video views by campaign name, video title, user location, or even video genre helps pinpoint where the spike is coming from.Try it with sample data
Across all users, the average time to value is under 4 minutes, an indicator of how quickly reports can be created, and insights about your product can be uncovered with the right producy analytics tool.
While Data-Curious users achieve value in 3.5 minutes, Data-Informed users take a little longer (4.3 minutes), likely because they need extra time to build more complex analyses that include additional filters, breakdowns, or a custom date range.
Though Data-Informed users’ time to value is a little bit longer than that of Data-Curious users, Data-Informed users achieve value moments more often. That could be because they’re specifically tasked with creating reports for others to consume, or it could be that they’re more adept at advanced analyses.
The key difference between these groups, as we noted above, is that while Data-Curious users monitor their data, Data-Informed users “tidy as they go”, actively digging into their data to answer questions as they arise—a habit that’s crucial to iterative product development.
While Data-Informed users are found across all company stages, those at Startups run, save, and view queries the most often. Why? Startups have the most to learn. They have to make quick decisions and rapidly experiment with their product to achieve product-market fit, often with limited resources and in highly competitive markets. As a result, they stand to benefit the most by using data to guide their business.
Tech Giants, on the other hand, spend the longest time per session, taking an average of 9+ minutes to explore their data, presumably because they have more established products that require greater depth of analysis. Although Startups do the most with their data, their sessions average under 4 minutes—the shortest of all company types. That could be because they have less data to navigate or because they’re focused on quick, iterative questions that don’t require extensive investigation.
You can get a lot out of your data in a fairly short amount of time. Spending 5 minutes per session is enough to keep up with most users, so queue up “Clocks” by Coldplay and get cracking. It’s the perfect song—and amount of time!—for you to cozy up with your data and learn something new.
But the real key to Data-Informed product development is consistency. Regularly spending time with your data (even in short bursts) will help you become more comfortable with the numbers and help you identify important trends. Stripe Lead PM, Shreyas Doshi even suggests setting yourself up to track the usage of your key dashboards (meta!).
Resource from Mind the Product
Matt LeMay explains what data means to product management and how it should be used to ensure that we’re building products designed to achieve meaningful outcomes.
How frequently should I run analysis?
Data-Informed users are in their product analytics tool day in and day out spending 5 days a week with their data. In contrast, Data-Curious users jump into product analytics only 3 days a week, which means they could miss key insights.
Most people do the bulk of their analysis on Monday and Tuesday—perhaps to catch up on weekend trends or prep for weekly team check-ins—and ease up slightly as the week progresses to focus on other activities (e.g., devising new features and planning A/B tests).
Product analysis by industry mirrors the patterns of users in those industries. In other words, analysts are attuned to the habits of their users and are eager to see how their highest activity days impact key metrics in real-time.
For example, Tech, Retail, and Media all run the most reports on Monday. In contrast, Financial Services companies fall into a Tuesday-centric pattern of analysis—the same day their own products receive the most activity. Gaming, however, runs the most analyses on Wednesdays, which makes us think that video games are a favorite hump day pastime.
If you want to move beyond being Data-Curious and become Data-Informed, product analytics should be part of your daily routine. Instead of only logging in when there are spikes in usage, aim for regular check-ins to keep up with your data and to stay on top of your core metrics with custom alerts.
But that doesn’t mean you have to dedicate hours to your analytics every time you log in (the value of self-serve analysis is in the time saved, after all). As we noted above, short bursts of analysis can go a long way toward ensuring that you’re always up to date, regardless of whether you’re a Data-Curious user or a Data-Informed user.