A/B testing to increase your conversions the Eric Ries way - Mixpanel

A/B testing to increase your conversions the Eric Ries way

Suhail Doshi

There are a lot of articles on the internet related on how to
A/B test and there are numerous open source
that are available to do it but nobody
does a great job doing macro level A/B testing which we think is extremely

I personally learned about this idea from Eric
when he talked about a case where
IMVU A/B tested the text of something only to find it was able to
increase the conversions of something like revenue and they had no idea it had
this effect on their users. It goes to show that an A/B test may increase
conversions of something desirable but adversely effect something else or do
completely the opposite of that scenario which would be profound.

When we designed Mixpanel, we thought about this idea and made sure it would
be feasible to do this in the future so I’d like to show you how we’re able to
do this for you today, you can literally create an
and try it yourself or
follow along in some manner. We think being able to do these macro level A/B
tests really differentiates ourself from current tools available today.

Basic understanding

With Mixpanel, you’re always logging events with a track() function:

mpmetrics.track("played a song", {"genre" : "hip-hop"});
mpmetrics.track("added to playlist", {"genre" : "pop"});

By Mixpanel standards, this is pretty basic API usage, this lets our customers
track when a user plays a song or adds a song to their playlist but we take it
one step further by letting them segment those events by the genre of that
song which has its own merits in terms analysis.

Regardless, understanding properties ({“genre” : “pop”}) is the cornerstone in
your understanding in how we can macro-level A/B test. These are user-defined
and you can add more than one.

Creating the test

Any A/B testing article will generally explain that you can split users using
the modulo operator on some user id:


function pick_background_color(user_id) {
var test_id = user_id % 2;

if (test_id) {
    return "red";
} else {
    return "blue";



This is basic code that chooses the background color to use based on a user_id
provided to us so that the user always sees the same background color
consistently. You should be able to setup a basic test to do almost anything
you want just using the modulo operator and some basic programming logic.

Logging the data

Once you have the test created, what you can then use Mixpanel’s properties as
a way to log the test data. For example, lets say we have a music site and
want to see how a red versus blue background effects our users behavior on the

So what we do, when we decide the background color we’ll use in the CSS using
the pick_background_color() function. In that function, we’ll modifiy and add
some Mixpanel code:


function pick_background_color(user_id) {
var test_id = user_id % 2;

if (test_id) {
    var color = "red";
} else {
    var color = "blue";

mpmetrics.register({"test-background" : color});

return color;



What _mpmetrics.register _does is it binds that property to every event along
with any other events also bound to that event. This allows us to see how a
red vs. blue split in background color effect every event on our music site.
So we’ll be able to see whether a blue or red background increases overall
engagement: songs being played and songs added to a playlist (or all the other
events being logged).

That’s all it takes extra: mpmetrics.register();

On our reports, you can click on the “View” button and then click properties
to get to your “test-background” property to see its effect and the split of
red vs blue for that specific event.

Wrap up

It’s not enough to just test whether a blue background color increases song
plays, you might find it decreases songs being added to playlists–this is
important to know because you may reconsider your product decisions if you’re
a site like Muxtape who thrived off users creating new playlists that would be
shared around increasing their overall traffic.

Imagine if these metrics were related to your bottom line, you better know how
certain things effect everything else otherwise your product could be headed
down an unexpected and undesirable path.

Get the latest from Mixpanel
This field is required.