Community Tip: Optimizing for goals with A/B tests

Mixpanel can help you take advantage of A/B testing for your app, website, or notification content to drive user action and engagement. This Community Tip will walk you through three awesome, real life A/B testing examples.

How can Mixpanel help you A/B test?

For your iOS app, A/B testing will allow you to experiment and tweak your app without app store resubmission. You can also track A/B tests for web by utilizing super properties (note that the “mpmetrics” call in this post has changed to “mixpanel”). Finally, you can run A/B tests with Mixpanel notifications from the Mixpanel web interface.

When doing any A/B testing, remember to tweak only one variable per variant, so as to keep “all else equal” and make it easier for you to pinpoint exactly which change made an impact on your users’ behavior.

A/B testing for mobile apps

There are two ways you can A/B test on mobile: one is through the Mixpanel web applications simple, visual A/B test building tool:

mobile A/B testing

… and the other one is using Developer Tweaks. Let’s walk through an example for both.

A/B tests created through your Mixpanel dashboard allow you to modify your app’s visual elements without any coding. Additionally, you can also measure whether any changes you make have a significant impact on your users’ behavior. For example: What were to happen if you changed the text of a key Call To Action (CTA) button? Or color, size, or location of the CTA button? Would making an in-app purchase button in a video game a brighter color increase the number of users who decide to make a purchase? Would changing the background color of a shopping app encourage your users to browse more often and for longer periods of time, thus increasing their average cart size and potentially their purchase frequency? How about a food ordering app: would having a larger, more visible navigation bar make a user’s experience more seamless, thus encouraging them to re-use your app and make more purchases? All of these tests can be done using Mixpanel’s visual experiment builder – no app-store submission or coding required!

visual a/b test

The second way to perform A/B tests on mobile is using Tweaks. Tweaks are small bits of code in your app which allow you to control variables in your app right through your Mixpanel dashboard. Tweaks’ functionality is well exemplified through a video game. Let’s assume we have an adventure game where our character is running around a vast land, solving puzzles and beating bad guys, to save the world. Now, what if we hypothesized that making the game slightly more difficult would make it more fun? What if we made the enemies faster? How about if we had more enemies in the game overall? Now, what if we assumed that our game was too difficult instead? We could make our character lighter so that they could jump higher. We could alternatively make them faster to be able to speed past some distracting enemies. With Tweaks, we can change any integer, float, bool, or string variable in our game and experiment as much as we’d like!

tweaks test

You can view the results of your A/B test right in the A/B Testing tab of your Mixpanel dashboard. Simply click into the Analytics button for a specific A/B test, and you can see your results instantly.

mobile a/b test

mobile a/b test

A/B Testing on Web

The first interaction between you and your users is one of the most crucial, which is why it’s important to create a positive, friction-free environment for your users on your landing page. There are many things that you can test on your landing page, but the ones outlined below are arguably the most important:

Entry points to key workflows (subscribe, signup, purchase, etc.): Your users likely enter a key workflow by clicking or touching a button, and they should be able to identify and enter these workflows frictionlessly. Why not test whether having your sign up button follow your users as they scroll through your landing page makes a difference in signups? Why not try to make your CTA buttons larger? Brighter? Regionalizing the location? Any and all of these tests might make a difference in how your users interact with your platform.

Language: Choice of words and tone are key to any customer interaction. Why not try to change all of your possessive determiners from “yours” to “my”? Try out “create my account” or “start my free trial” instead of “create your account” or “start your free trial.” A different test might entail you focusing on the benefits of your product. Perhaps you might say “create my account to (reap some benefits)” rather than simply “create my account.” Try out as many things as you think would make a material difference in user engagement, and let the data tell you what’s optimal for you.

Once you’ve started your A/B test, a powerful way to analyze your results is by building funnels to see if your users performed the Call To Action you intended them to.

web variants

web variants

A/B Testing with Notifications

In Mixpanel, you can even test your notifications! For this example, let’s test email variants. Have you ever wondered if your users would be more likely to open an email if the subject line was worded differently? In a time where multiple apps and websites are fighting for you users’ attention, people are inclined to only scan the subject lines of their crowded, private inboxes.

Let’s say your business was offering something for free. For a gaming company, this could be free virtual money. For an ecommerce company, this might be free shipping. For a SaaS company, this might be a timed trial that hasn’t yet been activated. Whichever way you look at it, all of these have a clear CTA. Now, you want to notify your users of this freebie and want to make sure your email’s subject line catches their attention. But … you don’t know whether “Special offer just for you!” or “You’re awesome – here’s something free!” would be a better subject line. With Mixpanel, we can easily A/B test which email variant gives us the best click-through results!

To do this in Mixpanel, we would first click into our “Notifications” tab to create an email notification. We would then create our variants in the Mixpanel Notification Builder:

email variant 1

email variant 2

When deciding the percentage of our users who would get each variant, we can even decide to add a control group of people who don’t receive the email. This would answer the question: does your email have any impact at all?

control group

From there on out, all you need to do is target your user group, and click send! Because Mixpanel tracks Notification Sent and Notification Open events, you can see your conversion rates between variants, and even for your control group!

notification conversions

We have the data … now what?

What can we do with our A/B testing data once we have it? You can click straight into the A/B testing or Notifications tabs to see the data behind your A/B test.

a/b testing analytics

Alternatively, you can choose to view your experiment data in other Mixpanel reports. The Funnels report is a great way to see if your A/B test increased your conversion rate through longer, multistep flows or among specific cohorts (e.g. new vs returning users) or user types (free vs premium):

funnels analysis

Final thoughts and pro tips

Remember to always focus on the goal of your A/B test. This is typically how to make it easier and faster for your customers to perform the CTAs you intend them to. To that end, it’s great to always start an A/B test with a hypothesis that will outline a change-and-effect theory you have about your test. This can be as easy as using this simple framework: “changing X into Y will Z.” For example, changing my email subject line from “Special offer just for you!” into “You’re awesome – here’s something free!” will drive more email opens and thus more successful calls to action. Finally, remember to have fun! A/B testing allows you to get creative and while having your data do the hard work for you.

Have any questions about A/B testing? Reach out to support@mixpanel.com to speak with someone smart, quickly.

Get the latest from Mixpanel
This field is required.