How to use betas to launch killer products

I’m Prachi Wadekar, a Product Manager on Mixpanel’s data infrastructure team. In this blog post I’ll share why we launch features in phases and make customer feedback the primary input for determining readiness.

Amazon’s customer obsession principle

Leaders start with the customer and work backwards. They work vigorously to earn and keep customer trust. Although leaders pay attention to competitors, they obsess over customers.

Many Product Managers struggle to prioritize their time each day because there is so much they could do. I’ve often faced that dilemma, but have found clarity by making customer problem solving my singular focus. The goals of any business and their customers are achieved simultaneously – when customers feel heard and find value in your product, they’ll want to keep using it and will be happy to pay a fair price for what they’re getting.

The key lies in prioritizing challenging problems that create business and customer value and then designing a minimum viable product you can launch to address a pain point for the customer.

This process should be a repeatable loop where you continuously collect, analyze and then take action on the data you gather from your customers.

Creating opportunities

At Mixpanel, PMs do a few things. We use Mixpanel to track how customers are using the features we focus on (example below).

We also use Salesforce where our go-to-market teams log “product gaps.” These are things customers tell us about the problems they believe Mixpanel can help them address beyond what we already offer in the product. Most importantly, we get ourselves in front of customers and talk to them about how we can drive their product innovation and solve hard customer problems. Mixpanel PMs solve these challenging problems by launching great products. And I want to double click here on the word “launch” because I cannot stress enough how critical it is to have a good process when you set out to build a winning solution.

Setting the context

The product manager’s role itself has evolved significantly since I started in the field. PMs used to bring order to engineering tasks, and now we are expected to take more ownership as the mini-CEO of our products. That means making hard decisions about the product’s vision and roadmap.

Developing a product now is more about its evolution than a revolution. It’s becoming more and more important for us to get a minimum viable product in front of customers before we can establish a problem-solution fit. New-age product development is sprint-driven where we spin up a minimum viable solution, bring that in front of customers, and seek validation from them. Once the first pass looks good, we start attacking more problems in priority order and continue to iterate on the product in what we call a “beta” process. We make the product “generally available” only after we have established a problem-solution fit.

Diving deeper

Let’s be clear – the purpose of a beta is NOT to have customers do QA for your product. It’s to validate that your solution solves their problem and to get feedback on potential improvements.  The best practice is to have a clearly-defined set of questions that you want to answer with the beta.

Beta testers are representatives of the target customer for the feature you are testing. They’ve either expressed a direct interest in the problem you are trying to solve or their usage of the product indicates they have a need for it. Customer-facing teams such as sales, support, and customer success can be good sources of candidates. The easiest way to fill a beta is to ask for volunteers for a limited number of seats and let customers nominate themselves.

What PMs should do in addition is build a pool of your product’s power users that will be interested in providing product feedback. And as you build this pool of users, make sure you have every persona covered so that you know you will be gathering feedback from every possible user type. At Mixpanel, we ensure we have customers from all company sizes and plan types participating in betas. These customers can participate in design research, help you test your product and give you regular feedback – both during and after beta. Here is an example from a recent beta I ran for our Data Pipelines product.

The good thing with betas is the change management is fairly straightforward. If you learn through a beta that the solution you built does not solve the customer problem, you can always go back to the whiteboard with your team to identify better approaches.  

Incentivize vs. create value

Hopefully it’s now clear why feedback and betas are important. But how do you incentivize target customers to join your betas?

In my experience, and from what most good PMs have told me, you don’t need to incentivize customers to join betas. If you have been listening to them consistently and have picked a problem to solve that will offer value to them, they’ll be happy to participate.

With that said, I’ve found a gesture of appreciation goes a long way. At Mixpanel, that could mean offering add-on features at no cost until their next renewal. We also continue to sync with customers and give them a direct channel for feedback where they can speak with the PMs and designers to share opinions and influence the product roadmap. I cannot stress enough how valuable betas have been to me in forming great relationships with customers.

Public vs. Private betas

Once you’ve decided to run a beta, you need to think about how big you want it to be. How much time do you have to meet one-on-one and gather feedback? The idea of a private vs. public beta might come in handy here. Private beta, also known as a closed beta, is where you give access to a restricted audience to play with your feature. This lets you go deeper with each customer involved. You can ask open-ended questions about features, UX, and ideas for addressing even more of their pain points.

The other option is a public beta, also known as an open beta. Any customer can join, which means you get a larger sample size, but also can’t have in-depth conversations with all of them.  You’ll need to lean more on quantitative feedback, a.k.a the metrics that you might be tracking in tools like Mixpanel. Examples include metrics like number of sign ups, how the feature impacted overall DAU, and changes in support ticket volume.

Ensuring quality

Designating a feature as “beta” will also help to set correct expectations with customers. They’ll know it may not work as expected, it may be missing functionality, and the UX may change. Having said that, betas aren’t an excuse for shipping low-quality software. You should test features internally first, and only push them out to customers once they’re in a place where they’ll be valuable.

Support during betas

Features you launch in beta are new and it’s crucial your customers have sufficient resources to navigate and learn the product. PMs, Engineers, Designers and Support teams can usually own this. Apart from good documentation, make sure you establish direct feedback channels where responses and solutions are delivered quickly. These are your opportunities to show customers that their overall sentiment about your product matters. Here is an example of how we establish these direct channels with Mixpanel customers every time we launch a feature in beta.

Stay data driven

The most important step, as you gather customer feedback, is measuring the outcomes of the beta. Anytime you are solving a problem for the customer, make sure you write it down as a goal for the team. You should be able to put down what success metrics you want to track and how you have been progressing towards your targets. Using Mixpanel can help you track and gather the quantitative feedback. Couple that with conversations with your customers, because no amount of tooling can replicate the type of data you get from talking to people.

The real goal here is to move quickly. You want to continuously act on what your customers are telling you and fine tune the product so that you can get it in front of more customers.

How long should a product stay in beta?

Ideally, a beta should not last longer than 4-5 weeks. However, the best practice is to set clear beta exit criteria and stay in beta until those conditions are met. These can be product-based (when x, y and z are complete), qualitative (positive feedback), or quantitative (usage metrics).

“Achieving product/market fit requires at least 40% of users saying they would be “very disappointed” without your product.”
– Sean Ellis (GrowthHackers.com)

By the time you are ready to take your product off beta you are confident you have established a solution for the problem you set to solve and you’re ready to get it in front of every customer. Ensure you put together a comprehensive go-to-market plan as the product takes center stage and launches as generally available to all customers.

Closing thoughts

The most important goal of a PM remains staying customer-focused and picking high priority problems that can create value for both your customers and business. To do this:

• Run through the beta process
• Gather feedback
• Listen to your customers
• Deliver fast solutions
• Keep this iterative innovation loop going until you’ve established a problem-solution fit

Once you establish that fit and make your product generally available, move on to the next problem!