Mixpanel Tip #153: Executing A/B tests - Mixpanel
Blog Post

Mixpanel Tip #153: Executing A/B tests

Last edited: Dec 10, 2017
Suhail Doshi

A/B testing is a powerful way to optimize your website for better user conversion to a specific goal you may have. There are a lot of tools out there that specialize in providing different ways to A/B test – some are pretty complex and others are relatively simple. The beauty about A/B testing with Mixpanel is that you don’t have to learn anything new in order to do it; all you need to know is standard event tracking with properties attached.

The Test

At its heart, A/B testing is just creating 1 variable with various values and running simultaneous tests to compare outcomes towards a specific goal. This standard way to A/B test can be easily done with Mixpanel using our Funnel Analysis, and segmenting with Super Properties. Let’s take a look at a simple example of a website with 2 different home pages and how they affect users through a 3 step signup funnel.



Above are the 2 different home pages. For the most part, they consist of the same content just organized differently. Each page has an event in Mixpanel called “viewed home page”, and depending on the version a user sees there will be a super property assigned with V1 or V2 as the value. A super property is extra information that you can attached to an event, just like a regular property. We use super properties here because, unlike regular properties, they only have to be registered once but will follow a user around wherever they go on your website. Once a super property is registered, every event that user does will have that super property attached to it. Here are a few lines of Mixpanel code that can be used to do this:

mpmetrics.register_once({"home page": "V1"});
mpmetrics.track("viewed home page");

Note that here we are using the “register_once” method instead of “register”to call the super property. We only want each user to see one version of the home page no matter how many times they view it, and we only want to track this on their first visit. In the second line we’re tracking an event called “viewed home page” – this will be the first step in our funnel. Now all we have to do is create similar events for the rest of the steps in the funnel and then use the Funnel Builder to create it. Here it is, with a little bit of fudged data to show:


This is a fairly standard funnel. We can see the drop-off rates and the end completion rate. To us the A/B testing functionality, simply click on the “Property” (not shown) and select “Home Page”. This will show all of the values of the home page, V1 and V2 for this example, and the individual conversion rates for each:


This gives us a breakdown on the performance of each home page down the entire funnel. The conversion percentage as well as the total number of people for each step is listed, and at the end the overall conversion for the home page. Not only does this give you outcome for the end goal, but you can see how each home page individually affects user interaction down every step of the funnel.

This is a very simple example of how to A/B test with Mixpanel. We could have added as many iterations of the home page as we wanted, and also had more or less steps in the funnel too.

Understanding Global Implications of A/B Testing

A/B testing is a great way to measure one part of your website against one outcome. However, have you ever thought about how changing one part of your website, like the home page, affects the rest of your site altogether -not just sign ups? What if sign ups went up, but overall game playing went down?

Most A/B tests are very narrowly focused and don’t show you how conducting an A/B test globally affects your website. Mixpanel has developed a tool that can help you better understand how changing one aspect of your website affects user interaction across all of the events associated with it. We call it Tests.

Below is a screenshot of the Tests page from the above example. It shows the results for every event that the “Home Page” super property is attached to (“sign up” has been excluded here since users only do it once):


We can see in the middle column that the unique visitors for the events “Play song” and “Viewed home page” match up with the corresponding steps in the above funnel breakdown. What we can also now see is how many times, on average, visitors performed each event and the total number of events done – all in one simple page.

Here’s a closer look at the two left columns:


We can see that while V2 of the home page has better conversion rates down the entire sign up funnel, the average number of times users play and complete songs is higher for users that have viewed V1 of the home page – V1 appears to be associated with a higher overall user interaction than V2.

It’s interesting to see results like this with Tests, but keep in mind that there are definitely more experiments that should be done to really understand what is causing users in one group to behave differently than the other. Tests is meant to be a tool that can help you check your results across your entire website and uncover any surprising things that often do happen when running website experiments.

Get the latest from Mixpanel
This field is required.