4 key phases in product testing (and why not every feature should make it to beta)

When you’re releasing a new product or feature, the step that gets the most attention is usually the beta testing phase.

But getting a feature to a market-ready state is a long process involving months of work before you get to beta testing. Sometimes, features don’t even make it to the beta stage—and that’s often not a bad thing when you consider the alternative is overspending time and resources to force a feature that isn’t worth it.

As a product manager at Mixpanel, I spend a lot of my time testing and evaluating new features and changes for our product. In this blog, I’ll walk you through the four key phases of product testing I follow—with one being the most important to get right for large launches.

In the first phase, you’re simply testing an idea—no more, no less. Your goal here is to talk to your customers and figure out whether or not there’s something worth pursuing here. Is there excitement for this potential feature?

Even though you’re talking to customers, this isn’t really a research group. You don’t have designs or prototypes yet. The focus is just on gauging interest in the idea. We usually try to have seven or eight calls with customers to validate whether there is enough traction for the idea, but you could get creative with how you get that qualitative feedback, especially if this might not be a risky feature.

For example, you could get your teammates on the sales or customer success team to bring it up casually on calls. (“By the way, we were thinking of building XYZ feature, what do you think of that?”) This is a great low-effort way to get a sense of whether the feature you want to build is something that your customers want, and it will give you a lot more confidence to pursue that path.

It’s early, but at this stage, it’s still helpful to get a rough sense of the size and complexity of the problem. What workarounds or alternatives are your users resorting to (that your potential feature could solve)? We use Mixpanel’s Flows report to look at all of these substitute paths that users are taking to get an early understanding of the journey that a potential feature could improve:

After the initial idea testing, if you do end up deciding to build this feature, you can move on to the next phase.

Now that you know the idea is solid, it’s time to move on to concept testing. This is a very important phase, especially for heavy engineering investment projects, because it determines whether this idea as designed merits upcoming investment or whether the design needs modifications to make it usable.

Again, you’ll be doing a few calls with customers in this phase, only this time you’ll have some wireframes and prototypes to show them and help them visualize what the feature will look like. (There are many tools that you can use to do this—like Figma, Maze, and Sprig—which have prototype and survey building features.)

The goal in this phase is to see which design concepts are landing and which ones aren’t. If your users respond very positively to one of your concepts, it’s usually a good sign to move it forward.

Next is alpha testing (sometimes called “running pilots” in the B2C world).

Before I go further, I should note that this phase tends to be reserved for larger feature and product launches. If you’re making an iterative improvement or releasing a smaller feature, you don’t have to go through an alpha—or even a beta, the next testing phase. It’s can be impractical, with astronomical costs that aren’t worth it.

Having said that, once you’ve done POC (proof-of-concept) work with your alpha-worthy feature that shows high-level resonation with users, it’s time to spin up a true version and let a very small group of customers and your internal teams have at testing it.

What’s discovered in the alpha phase can sometimes be stirring, though. Your feature, now that it’s actually built, may not work as expected. It’s important that teams hitting friction at this point understand the alpha phase is a totally reasonable time to scrap a feature.

I point this out because it’s common for teams to feel too invested in a feature to pull the plug. But if the warning signs are there—very low (or no) usage or customers not finding it as useful as they expected it to be during the prior two stages—you may be better off cutting losses, still relatively early in the process, and moving on to new work.

You have to know when to let go, especially with expensive launches that are large enough to warrant an alpha. Sure, it may have taken several months of work to get to this point, but it’s better to eat that cost than to sink another several months (or more) into a feature that your customers have already communicated that they don’t want or find useful.

The other risk at this phase is going overboard with data and analytics. Because your build is likely missing some things and there aren’t many customers using it yet, there aren’t as many meaningful things to track and monitor to draw conclusions. Overbalancing on testing just doesn’t make sense here.

Finally, you’re at the beta testing stage. At this point, it’s definitely happening—you’ve committed to launching a feature. This is usually the most exciting phase because up until now, the feature has barely seen the light of day.

Now, you can get, at a meaningful scale, a read on how customers are interacting with a near-finished version of your feature. Are they using it as you expected? Does it affect other features or interdependencies within the product? When you’re choosing beta customers, make sure to get a good mix of customers from different regions and industries. It’s often tempting to think, “Oh, we just want any customers who are interested,” but this is an important opportunity to test whether there are differences in how your customers use your feature depending on the industry, persona, and so on. Choosing the right customers is a big and critical part of the beta testing period.

It’s also important to keep the beta testing time window tight—generally between six and 10 weeks. It shouldn’t go much longer than that because you don’t want customers to get used to a feature that will undergo further (and sometimes drastic) changes.

We’ve seen teams just leave a beta test running for a year, which isn’t a great user experience. By the end, the feature will have changed quite a bit and you’ll have to move all your customers in the beta over to the new feature, which can cause a lot of confusion and friction for them.

This is also the phase where you should be most heavily monitoring usage data and analytics. We use our dashboards in Mixpanel to see which testers are using the feature, how they’re using it, how frequently they’re using it, and so on. Like in the feature concepting stage, Mixpanel Flows visualizations are also very useful here. We create a report for each customer in beta, which lets us understand their usage and also helps us ask better questions when we discuss their feedback about the feature.

Making time for each key phase of product testing is essential for taking a new feature to market. Each stage provides valuable insights into product viability, functionality, and user experience, and they all play a role in your feature’s success.

A good product testing strategy should give you the opportunity to mitigate risks and the space to scrap an idea if it’s not working so that you can maximize the chances that every feature and product you deliver resonates with your customers and users.