Debugging with product analytics
As a product manager, deep-diving into problems like high churn, bugs, and crashes is part of the job.
Over the years, I’ve learned that finding solutions to these kinds of problems is a lot like debugging software code: You always want to start with trying to isolate the issue as much as possible by breaking it down into smaller parts. This quickly shrinks down the biggest and most time-consuming questions such as “Where could this be?” or “Who is affected by this?”
I really love the CS50 lecture on the Phone Book Problem that explains exactly why this philosophy makes a lot of sense.
In a startup I worked with, we once saw a very big drop (40%) in the signup process (Signup form > Signup successful).
My way of breaking the problem down here was to add a simple event “Signup button clicked” to see if was an issue with the fields or something deeper than that.
The event gave me a better picture! Signup form > Signup button clicked was a 16% drop. That was a bummer and something that could be reduced by removing fields (but that’s a different post), but more to my immediate interest was that I was able to uncover that Signup button clicked > Signup successful was the larger and more problematic 24% drop.
From there, it was an easier problem to check and then an even easier fix that got us back on track.
To show what this kind of debugging looks like in the tools, let’s walk through a scenario in Mixpanel. Here’s an occurrence of high churn that we want to find the cause of.
- I build a funnel of my basic user flow impacting my Focus metric (using the Funnel Trends view).
- As we can clearly see, we have a major drop on Dec 15th. Now what? Breakdowns! Did we have a release? Is this a specific OS? Browser? App/OS Version?
- So I change my funnel date to the 15th and start adding the different properties to the breakdowns. Usually, I start with the big ones:
– Operating system (Mac/Windows or iOS/Android)
– Browser (Safari, Firefox, Chrome, Edge)
– Versions (Chrome V96.0.4664.110)
What I find are cases in which browser version is impacting our flow greatly. - Breaking things down further shows that the biggest issue is Firefox iOS. From here, I can go on to basic tests of looking at that specific case and not get lost with all the different guess work.
The good news about this process is, most of the time, you can find low-hanging fruit very fast (the example above took me five minutes) and then get right into fixes that impact your KPIs drastically. However, for an explainer on going deeper into Funnel analysis in Mixpanel, check out this great blog.
So, take my advice and let product analytics debugging be your first line of attack for cutting through the 🤔 of solving issues in your product.
PS: I know you’re all anxious to know what the problem was in my real-world startup problem above. It was a validation issue that gave users a wrong message on the password field. Thank you, product analytics debugging!