On skepticism and storytelling: How PMs can become more data-informed

For today’s post, we spoke with Rohit Gossain, who has held multiple product management and data science roles over the last decade with technology companies such as Shutterfly, OpenTable, Adobe, and Expedia Group. 

Here’s what you’ll learn: 

• How to become more comfortable with analyzing data 
• How to be sure that what you’re doing is actually moving the needle
• Common pitfalls when analyzing product data and how to avoid them

Mixpanel: How have you seen product analytics evolve over the course of your career? 

Rohit Gossain: I started off my career in 2010 as a technical analyst working with a product manager. At the time, product analytics software didn’t really exist—we were very much relying on Google Analytics. My team was doing more work around BI and reporting, so, in general, there wasn’t nearly as much knowledge around user behavior analysis. It was much more common for people to make decisions based on hypotheses, not data. 

After that, I moved to Expedia’s analytics team, which is when I noticed the space starting to evolve. Google Analytics’ usefulness became more limited in terms of what we were trying to achieve; it was hard to merge data from different sources, conduct deeper analyses, and understand different points along the customer journey. At the time, we were conducting A/B tests, but the metrics were not behavioral, which presented an issue. For example, we were trying to understand how our conversion rates looked based on launching a homepage test. Unfortunately, you can’t measure bottom-of-the-funnel conversions based on changes you make at the top of the funnel. The concept of “product analytics” did not exist at the time. The metrics were more marketing-based–revenue, conversions, leads–and not behavior-based. 

The space really started to mature with the rise of big data and cloud computing. We saw a new Growth PM role emerge, where you have to make decisions based on a wide variety of data and on understanding the customer journey. PMs were making more and more decisions based on data. 

2015 was when I saw a distinction emerge between real-time customer data and BI/analytics data. With this kind of data, PMs could answer questions more pertinent to their focus areas: Who are the users using my feature? Who are the users looking for these features? What features make sense for them? Which customers stopped using our products and switched to a competitor? 

And it has continued to evolve into where we are today: where PMs are expected to make decisions based on user behavioral data and there are several product analytics solutions, like Mixpanel, available to assist. 

Mixpanel: How can companies be sure that what they’re doing is moving the needle? And that there are not other factors affecting the numbers? 

Rohit: I’m a huge advocate for making this process as cross-functional as possible. If you’re a PM, I’d advise that you have discussions with various teams (marketing, data, operations, finance, etc.) because if you’re running a test, you need to ensure that those teams are not changing anything on their end that will end up impacting the numbers. At that point, you can truly say that your experiment ended up moving the needle. 

Many companies don’t do this well; they don’t share information properly. There should be a team meeting where you’re telling everyone, for example, “We’re running these five tests in the next 1-2 weeks that might impact revenue.” What you’re trying to find out is: are there initiatives that other teams are launching at the same time that will affect your results? Distribution of information is very important in this scenario. It’s a harder challenge in larger companies because not everyone will take note of this. And that’s why a lot of companies are actually failing to do this properly. 

It’s tough to do, but I consider that to be a key part of any PM’s role: reaching out to all the stakeholders and ensuring that they’re in the loop. 

Mixpanel: How has your data science background affected your work as a PM? 

Rohit: Due to my data science background, my mind is now structured in such a way that I’m always looking at the data and trying to find the facts. I never trust my gut. I’ve become so much more skeptical about everything as a result. 

That’s where solutions like Mixpanel come in—it’s so much easier now than it used to be to do this sort of analysis. Previously, you had to know SQL to even pull reports in the first place and had to know exactly what you were looking for. 

Once I have some data for corroboration, I can then move forward with creating the product strategy and convince my boss that this is something worth building. If you have data, it’s easy to convince people because you have proof. I think about it as storytelling; data helps bring the right story to your stakeholders. 

Mixpanel: What advice do you have for PMs around incorporating data into their decisions? 

Rohit: Here are a few tips:

• You need to be always skeptical. Don’t just rely on your gut because there’s so much information around us and it’s easy to be biased even if you’re not aware of it. 
• I recommend that you start with a null hypothesis. Assume that your hypothesis won’t work, then try to utilize data to help you provide some evidence that it will work. 
• You don’t need a data background to be data-informed. People get stuck in thinking that data is reports and charts, but data is actually available everywhere in all sorts of forms: NPS scores, customer emails, talking to other members of your team. 
• Learning how to be more data-oriented is not easy. But it’s much easier than it used to be. There are so many good tools right now—you don’t need to know how to write SQL! You can use a platform like Mixpanel, select a few things and a report is automatically generated. 
• A great resource who has really shaped my thinking on this is Avinash Kaushik, who writes the Occam’s Razor blog. He writes a lot about the intersection of marketing and analytics. Highly recommended. 

Mixpanel: How would you say that product analytics solutions like Mixpanel have been helpful in your previous roles? 

Rohit: Data can help product teams prioritize where they should invest resources to grow the business. For example, when I worked in ecommerce, Mixpanel helped us understand things like which categories are performing better, which customers are buying, which customers want to buy specific designs, how much time it takes for them to purchase a product, and how the overall journey looks. Based on the analysis, we tried to figure out which category we should focus on, what problems customers were experiencing, and how we could solve them. And then we measured again by looking at the data. 

On top of that, Mixpanel enables PMs to make decisions more quickly. PMs have a lot on their plates, so it’s crucial for them to have instantaneous access to the data so they can continue iterating. They don’t have to be software engineers or data scientists to pull these insights; Mixpanel has removed that layer of complexity. 

This has created an environment where PMs require less support from data teams. This is a win-win because it allows PMs to spend less time pulling reports and more time driving value for customers. 

You have a really interesting background, in that you’ve worked in both data and product roles. Do you have any advice on how PMs can work better with data teams? 

Rohit: If you’re working for a company that has limited data resources, then you need to start rolling up your sleeves and digging into the data yourself. As a PM, your role is to make your product successful.

If you’re lucky enough to be partnered with an analyst, then work closely with them to create the right prioritization and make sure you’re asking the right questions. Set up the right expectations as to when you can have that analysis on your desk. 

What are some of the pitfalls that you’ve seen around companies analyzing their product data? Do you have any tips on how these can be avoided? 

Rohit: Here are some of my tips: 

• Make sure you’re asking the right questions. It’s easy to get excited about specific ideas to move the needle on KPIs, but it’s best to hold off. At Shutterfly, before jumping into a spec and brainstorming ideas, we conduct a lot of analysis. We try to ask the right questions: does this initiative make sense? Are we solving the right problems? The hardest part of the equation is asking the right question. If you have the right question, it’s easier to find the right answer. 
• Collect perspectives from different teams. Typically, every team wants to optimize their own metrics, which creates a very narrow focus. However, if you bring in people from a variety of teams for a brainstorm, you can channel all of those perspectives towards solving the same problem. I often find that this results in the right question. 
• You may already be tracking the right metrics. Once you find the right question, you’ll find the right metrics to match. There’s a tendency to want to add more and more metrics, but there may not be a need. The right metric may already exist, and it may already be tracked.  
• Get the implementation right from the get-go. There can be gaps in engineering teams’ knowledge around implementing these analytics tools, so it’s necessary to ensure they’re set up correctly from the outset. Yes, it’s fantastic to collect the data, but at the end of the day, if you haven’t implemented correctly, the result is bad data—and that affects everything. 
• Never underestimate the importance of data governance. Once you have your tool up and running, the biggest challenge is data governance. In the beginning, you’re likely not tracking that many events or properties, so it’s easy for everyone to be aligned. What can often happen later is that new employees come in, they have no idea what events even exist, or if they’re good or bad. They keep trying to add more and more without checking to see what already exists, which results in confusion and lost trust in the platform. 

If you’re really serious about data quality, it’s important to invest the resources in maintaining it. Companies that do this well have a standalone data governance team dedicated to preventing quality from becoming an issue. If you don’t have that team, it doesn’t matter what platform you use–you won’t have good data. Yes, there are tools that automate these things, but take it from me: you cannot automate everything. There needs to be a human involved who is ensuring that the right data is going into the system; everyone will be much happier in the long run.