What is A/B testing?

A/B testing injects a healthy dose of reality into your products. It’s all too easy for product designers, developers, and marketers to fall in love with their own creations and A/B testing shows them what users really think. For example, when designing a website, how can you determine the optimal location for each button? Or which graphics maximize sign-ups? Or what version of your navigation bar do users prefer? Only A/B testing can tell for certain. Here are some benefits of A/B testing your digital product:

Testing takes you right to the heart of users’ desires for your product. More often than not, you and your users share a goal: you want to help them solve a problem, enjoy an experience, make a purchase, or get help. By testing and optimizing each stage of their journey, you can speed up their travels and both get more of what they want. Plus, A/B testing is non-invasive and doesn’t disrupt their experience because most of it occurs without their active knowledge. After repeated tests, the product slowly evolves into something that’s intuitive, useful, and which people love.

With the right A/B testing tools, you can test almost every element of your site, app, or message. Here are a few common examples of A/B testing goals for several verticals:

And here are a few common site or app elements teams can test to achieve these goals:

Teams can test each of these elements for size, placement, appearance, and even the timing or circumstance under which they appear.

To learn more, download Mixpanel’s Benchmarks Report.

To anyone who sat through a grade school science class, the A/B testing process will sound familiar: it mirrors the scientific method. Testers develop a hypothesis for how to improve performance and create variants of the text, imagery, videos, or timing. Then they run, analyze, and repeat, implementing results if they’re meaningful.

What’s your end goal? Are you interested in conversions, purchases, or usage? Whichever it is, measure your metrics before testing so you have a baseline.

If you’re analyzing a website, do you have high bounce rates? If an app, do you have low usage rates? If email, low click-through rates? Examine your customer journey and where they seem to drop off most often within the conversion funnel. If you find clear leaks before places like forms or CTAs, start your testing there. Consider incorporating data from:

Before beginning, are there any learnings you can draw upon from past tests? Are there certain features, colors, or placements that users have reacted favorably to before? Use this data to draft a hypothesis for what will improve results. For example, “Making the CTA button more prominent will drive 20% more clicks.” Now, create a variant that reflects your hypothesis. This can be as simple as changing text or as complicated as building a new form, page, or graphic. In the example above, we would build a second CTA button. Remember, never test more than one variable one at a time. If you commit the error of testing two things at once, say, by changing the color and the position of the CTA, you’ll never know which of the two changes influenced your results.

If using a customer analytics platform, upload your variant and run a test. There are two big factors to running empirically accurate tests:

Based on the results and the statistical significance, can you declare a clear winner? If so, implement that change. If not, the control, also known as your “null hypothesis,” always wins, and you don’t implement the change.

The science cycle never ends. Implement any learnings and continue to test. Each time, return to your conversion funnel and use A/B testing to identify and remove blockages to users’ progress. In doing so, you’ll increase the utility of site or app.

A/B testing against large populations with simple tests is great, but to hone your product faster with better data, you can deploy:

Take the time now to inventory your product marketing tools. Do they permit you to run A/B tests? If so, and there are places in your conversion funnel where users find friction, A/B test away.