A primer on retention analytics for product leaders

Test yourself with this word problem: Imagine you’re a product manager responsible for your company’s new “business messaging and collaboration” product. This quarter, you’re tasked with improving user retention in the first 7 days, by 15%. A lofty goal! Your team suggests two ideas:

1. Change onboarding so it nudges new users to invite their team to collaborate right away. (Because Michael on the product team thinks that people are more likely to retain if they add their co-workers to a new project the week they join.)
2. Add a pop-up after registration prompting new users to change the UI from default to custom. (Because Michelle on the design team thinks that people are more likely to retain if they have the option to switch to dark mode.)

Which one will you choose? Both options require similar effort and resources, and you can pick only one. Michael (idea #1) has already mocked up the updated flow and started talking to engineers. (What?!) Michelle is finishing the top nav redesign and has the bandwidth to help with the pop-up design and implementation on Monday. (Now or never!)

There is, of course, no simple answer. You need more data: What’s the baseline for improvement? How do you define retention for this product? What business outcome are you trying to drive?

Let’s unpack retention analysis for product managers, from high-level goals (what to prioritize) to actionable tips for retention analysis (how to segment retention data) in Mixpanel.

Prioritizing retention

As the old adage goes, acquiring a new customer can cost five times more than retaining an existing customer. But the impact of an uptick in your retention metric goes far beyond money savings. 

An improvement in your retention rate means your customer lifetime value goes up. That means you can invest in growth more aggressively, grow faster than your competition, and seize the market. It also means your typical user journey gets longer, which automatically creates more opportunities and touchpoints for monetization.

Very simply: the better you retain users, the faster you grow your business.

Steps for analyzing retention

1. Define what retention means for your product

A common misconception about defining your retention metric is that it’s tied to revenue (i.e., a paying customer is a retained customer). This logic is flawed because it makes it difficult to predict the likelihood of churn and do something about it. Just because somebody is paying for a product right now doesn’t mean they will next month, or next year. You’ve probably seen many business subscriptions getting canceled after an email goes out to your team (“Anybody still using this tool?”).

And historically, such a definition made sense, because without access to other data, companies had to wait for Finance to close the books before knowing whether their product was doing well. But with product analytics, you can learn and iterate much faster by looking at how people get value from your product, and whether they come back.

A retention metric tied to product usage (depth and frequency) is a better predictor of churn, and a more helpful KPI for teams to act on. It’s important to track an event or action that represents that your users are getting value from your product, again and again. This event or action is something meaningful that your users do after they log in, such as “Complete a ride” in a ridesharing app, or “Watch a video” in a media streaming service. The depth and frequency of engagement will depend on the typical product usage interval; for example, you’ll expect users to come back more frequently to a messaging app (daily), compared to, say, a digital healthcare app (quarterly, or twice a year).

To bring it all together with a definition, retention is the percent of users who stay with your product and continue using it (i.e., get value) in a given time period. To calculate retention, you just take the number of your active users during that period divided by the number of active users at the beginning of the period, and multiply that number by 100 to convert it to a percentage. This way, “Active” will represent only those users who complete a meaningful action (vs. counting repeat logins, which don’t signal value).

2. Establish a baseline for improvement

To improve retention, you need a baseline—that’s obvious. There are two ways you can get this data in Mixpanel:

• Using Mixpanel’s Retention report, you can calculate how many people “did A and came back on a specific day and did B” (N day retention).
• Or, you can calculate how many people “did A and came back on a specific day (or week, or month, etc. or any time unit afterwords) and did B” (Unbounded retention).

You have the numbers…now what? You can use segmentation to more deeply analyze your retention rate and understand what makes people stick around. Below are some ideas to try. Follow along by registering a free account and playing around with our sample data sets.

3. Find out what moves your retention metric

Your retention rate shows you an average across all users. By digging into behavioral and demographic trends beneath the metric, or by looking at specific groups (cohorts), you can identify what makes your users stick around, and which of them are likely to become loyal customers. 

With Mixpanel, you can group users based on any attribute, such as whether or not they use feature X, and compare how these groups retain over time. So let’s go ahead and validate Michael’s idea.

To compare the difference in retention rate among users who perform a specific product action (such as “Invite team”) and those who don’t, we defined and saved two cohorts in Mixpanel. We then went into the Retention report and selected “Send Message” as our event A and B, so we can track how many people come back to perform a meaningful event over time.

As you can see, those who invite team members retain better. You can adjust the date range to get the specific day you want to benchmark. (Try it here!)

Now, on to validating Michelle’s idea (adding a pop-up after registration prompting new users to change the UI from default to custom). To compare two user paths—those who “adjust settings” and those who don’t prior to performing the key “Send Message” action—we built a funnel with the following steps:

• Open app
• Adjust settings
• Send message

We saved this cohort directly from the Funnels report, then followed the same process to build a funnel and group together those who sent a message without adjusting settings.

Our report shows that users who spent time adjusting their settings retain longer than those who don’t. Check out the difference in this demo data set. Which idea are you going to prioritize now?

Interpreting retention data

So far, we’ve covered the three essential steps for retention analysis: defining what retention means for your product, measuring your retention rate over time to get a baseline for improvement, and using cohort analysis to break down your retention data and understand what causes churn.

Note what’s not here: qualitative data. By adding that into the mix, you will enhance your retention analysis and better interpret your quantitative data.

• Separate “happy churn” from churn. As Akio Bandle, Lead PM at ZipRecruter outlined in the Guide to Product Analytics, “especially with transactional services, some users churn because your service works as intended, which is a good thing.”
• Look at the full breadth of behavior for your best, most retained users. You want to spend more time understanding your best customers, rather than obsessing about getting back those who churned. Use analytics to see what features they use the most, but also interview them: send them an email, or offer a gift card for 15 minutes of their time. Building the best, most valuable product for your customers is the most straightforward way to boost retention (and prevent churn in the first place).