Frequently Asked Queries - Chapter 02 - Mixpanel
Frequency of analysis

How often should I use product analytics?

Frequent analysis is essential to making data-informed decisions about your product. But what does “frequent” really mean? To a certain extent, the frequency with which you should be analyzing your data depends on your role and your product’s unique value moment. At Mixpanel, we define value moments as users saving, exporting, or adding a report to a dashboard. But in our experience, once teams discover how easy it is to run reports and see their data in action, it quickly becomes a regular part of their workday.

How regular? Well, the median Data-Curious user runs 7 queries per day, while Data-Informed users run more than twice as many. Data-Informed users are also more spontaneous, deliberate, and curious about their data, as evidenced by the fact that they view, create and save queries 3x more than Data-Curious users do. In addition, Data-Informed users engage with their product data more often, answering questions as they arise rather than time-bucketing their analysis in larger blocks. Even though they make up just 10% of the users we analyzed for the report, Data-Informed users run over 40% of all queries.

The takeaway

Data-Informed users are intentional about incorporating data into their work. They’ve developed the habit of asking questions of their data, and rely on that data to guide product decisions. Just like some folks find it more effective to keep a room clean by tidying on a consistent basis, the same goes for users that depend on data for iterative product development.

Data-Curious users engage less often and have shorter sessions. While they definitely monitor their data, they don’t spend as much time identifying why the numbers have changed as Data-Informed users do. Data-Curious users tend to keep a pulse on their data and are satisfied with knowing the status of a metric—e.g., whether logins are up or down—but aren’t as likely to explore why those numbers change as Data-Informed users.

Resource from Mind the Product

Data-Driven Blunders and how to Avoid Them  by Jorge Rodriguez-Ramos 

Jorge Rodriguez-Ramos considers how you can best incorporate data-driven decisions into product management.

Frequency of analysis

How long should I spend doing analysis?

Session lengths vary depending on what you’re trying to learn from your data or what question you want to answer. We discovered that analysis doesn’t necessarily mean digging into data all day. In fact, most teams use analysis to be informed on high-level product health and overall performance—our study reveals that the median session length for Data-Curious users of product analysis is just under 5 minutes.

For the top 10% of product analysts, however, five minutes isn’t nearly enough time to satisfy their curiosity. Data-Informed users’ median session length is over 10 minutes, 2x longer than that of Data-Curious users. What’s more, median Data-Informed users also have 5x the number of daily sessions of median Data-Curious users. If our experience advising product teams is any indicator, one question often leads to another in an attempt to understand what their users are doing.

“In my experience, more often than not, the key insights come from the data. So it pays big dividends to spend real time studying it. And in fact, good product leaders often immerse themselves in their data, trying to decide where the real potential is and which of their hunches are worth pursuing.”

Marty Cagan Founder of Silicon Valley Product Group

“In observing fast-growing companies, Sean Ellis (author of Hacking Growth) observed that their product teams employed high-tempo testing. It’s a process of testing potential growth drivers ranging from new channels to new engagement features deep within a product. The more tests you run, the more you learn about how to grow your business. This is only possible if product and data teams spontaneously pull the analysis they need to iterate quickly. If rapid growth is a priority for your business, self-service product analysis needs to be at the top of your list to be able to run high-tempo testing.”

Ramli John Managing Director at ProductLed Institute

Try it yourself

Explore Mixpanel

See how a media company breaks down total video views by campaign name, video title, user location, or even video genre helps pinpoint where the spike is coming from.

Try it with sample data

Across all users, the average time to value is under 4 minutes, an indicator of how quickly reports can be created, and insights about your product can be uncovered with the right producy analytics tool.  

While Data-Curious users achieve value in 3.5 minutes, Data-Informed users take a little longer (4.3 minutes), likely because they need extra time to build more complex analyses that include additional filters, breakdowns, or a custom date range. 

Though Data-Informed users’ time to value is a little bit longer than that of Data-Curious users, Data-Informed users achieve value moments more often. That could be because they’re specifically tasked with creating reports for others to consume, or it could be that they’re more adept at advanced analyses. 

The key difference between these groups, as we noted above, is that while Data-Curious users monitor their data, Data-Informed users “tidy as they go”, actively digging into their data to answer questions as they arise—a habit that’s crucial to iterative product development. 

“Being able to get at the data yourself is hugely beneficial. You never know when inspiration is going to strike in the form of a burning question that can only be answered with a look at the data, and having something self-serve at hand means you can prove or disprove your initial hypotheses yourself and continue forming deeper thoughts and experiments. The more hypotheses you can form, the more you can experiment, which leads to more and faster product iteration.”

Janna Bastow Founder at ProdPad & Co-founder at Mind the Product

“You don’t grow a pig by measuring it. You can measure the pig every week and the pig will get bigger, but unless you’re measuring the food that goes into the pig, you have no understanding as to the why.”

James Mayes CEO & Co-Founder at Mind the Product

Segment differences

While Data-Informed users are found across all company stages, those at Startups run, save, and view queries the most often. Why? Startups have the most to learn. They have to make quick decisions and rapidly experiment with their product to achieve product-market fit, often with limited resources and in highly competitive markets. As a result, they stand to benefit the most by using data to guide their business.

Tech Giants, on the other hand, spend the longest time per session, taking an average of 9+ minutes to explore their data, presumably because they have more established products that require greater depth of analysis. Although Startups do the most with their data, their sessions average under 4 minutes—the shortest of all company types. That could be because they have less data to navigate or because they’re focused on quick, iterative questions that don’t require extensive investigation.

“The questions you need to answer differ depending on the stage your product is in, and the cycle time required to get to those answers may vary. It is still very important for established products to use data to drive their decisions, but the timeframe in which that data is collected may just be longer than for a startup.

Startups may also be more adept at moving quickly and running small, iterative tests to learn fast. Other segments might benefit from using data more to guide their business decisions, but just don’t have the evidence-driven culture built into their team DNA.“

Emily Tate Managing Director at Mind the Product

The takeaway

You can get a lot out of your data in a fairly short amount of time. Spending 5 minutes per session is enough to keep up with most users, so queue up Clocks” by Coldplay and get cracking. It’s the perfect song—and amount of time!—for you to cozy up with your data and learn something new. 

But the real key to Data-Informed product development is consistency. Regularly spending time with your data (even in short bursts) will help you become more comfortable with the numbers and help you identify important trends. Stripe Lead PM, Shreyas Doshi even suggests setting yourself up to track the usage of your key dashboards (meta!).

Resource from Mind the Product

Data-Driven Product Management by Matt LeMay

Matt LeMay explains what data means to product management and how it should be used to ensure that we’re building products designed to achieve meaningful outcomes.

Frequency of analysis

How frequently should I run analysis?

Data-Informed users are in their product analytics tool day in and day out spending 5 days a week with their data. In contrast, Data-Curious users jump into product analytics only 3 days a week, which means they could miss key insights.

Most people do the bulk of their analysis on Monday and Tuesday—perhaps to catch up on weekend trends or prep for weekly team check-ins—and ease up slightly as the week progresses to focus on other activities (e.g., devising new features and planning A/B tests).

“Think about how you can include your product analytics tool in the day-to-day and week-to-week operating of your business. It will make your team’s analytics and experimentation muscles that much stronger.”

Vince Maniago VP of Product Management at Personal Capital

Industry differences

Product analysis by industry mirrors the patterns of users in those industries. In other words, analysts are attuned to the habits of their users and are eager to see how their highest activity days impact key metrics in real-time.

For example, Tech, Retail, and Media all run the most reports on Monday. In contrast, Financial Services companies fall into a Tuesday-centric pattern of analysis—the same day their own products receive the most activity. Gaming, however, runs the most analyses on Wednesdays, which makes us think that video games are a favorite hump day pastime.

The takeaway

If you want to move beyond being Data-Curious and become Data-Informed, product analytics should be part of your daily routine. Instead of only logging in when there are spikes in usage, aim for regular check-ins to keep up with your data and to stay on top of your core metrics with custom alerts.

But that doesn’t mean you have to dedicate hours to your analytics every time you log in (the value of self-serve analysis is in the time saved, after all). As we noted above, short bursts of analysis can go a long way toward ensuring that you’re always up to date, regardless of whether you’re a Data-Curious user or a Data-Informed user.