Multivariate Tests

You’ve analyzed your user data for a landing page that’s doing well and believe a few changes could boost your conversion rate a little higher. But how can you know for sure which changes will have the biggest impact? Multivariate tests are one of the best methods of testing an analytics-based hypothesis that involves a combination of variables.


What Are Multivariate Tests?

Multivariate tests are used when you want to change numerous elements at one time in order to optimize a web page and discover interaction effects between elements. You are testing variations of each element and combining them to create different versions of the web page for comparison. 

A multivariate test can include page elements such as:

  • Call to action (CTA)
  • Images
  • Colors
  • Text
  • Page Layout
  • Headlines
  • Form length
  • Buttons

The goal of a multivariate test is to determine which combination of page elements yields the best results. You also learn about the interaction between variables.

For example, you can test two CTAs and two button colors. In this scenario, you would test four versions of the web page to discover which CTA and color combo yielded the highest click-through rate.

Test Page 1 – CTA 1 + Button Color 1

Test Page 2 – CTA 1 + Button Color 2

Test Page 3 – CTA 2 + Button Color 1

Test Page 4 – CTA 2 + Button Color 2

A multivariate test provides assurance that a web page is fully optimized down to the finer details. It’s a powerful analytical tool that can direct multiple changes simultaneously.  

How does your product stack up to the competition?

Get the 2019 Product Benchmarks report to learn more.

Download Report

How to Perform Multivariate Testing

Multivariate testing isn’t a single test. Because multiple variables are involved, multivariate testing includes comparing multiple pages that combine the variables in different ways to find the best performing combination. 

There are also different types of multivariate tests depending on how traffic is divided up among the pages being tested. 

Full factorial testing is the most common type of multivariate test. The traffic is split evenly between the different versions of the web page. This type of multivariate test is best for finding the highest performing variable combination. 

Fractional factorial testing, also known as partial factorial testing, drives traffic to only some of the pages. The results for the remaining pages are guesstimated based on the results of the pages that received traffic. It requires less traffic, but the results are less exact and rely on mathematical equations that can be complex.

Regardless of the type of test you run, there are several essential steps for performing multivariate testing:

Select an Analytics Tool for Testing

In order to run a multivariate test, you’ll need an analytics tool that supports testing. Many analytics platforms support A/B testing but not all of them are capable of running multivariate tests. 

Get a Baseline Measurement

Before testing variables, you should take a look at your user data analytics to measure the landing page as it currently exists. This will be the baseline for comparing how the variant combinations perform. It will also help you determine a hypothesis for why the conversion rate isn’t as high as you want it to be.

Identify the Variables

Next, identify the variables that you want to change. This should be based on intel gathered through analytics and which components are most closely connected to the conversion goal. 

Create Different Versions of the Web Page

You’ll need a different web page for each variable combination. If you’re testing 3 headlines with 2 text blocks you would have 6 test pages total.

3 headlines X 2 text blocks = 6 variable combinations

Split the Traffic Between the Web Page Versions

For accuracy, we recommend using the full fractional testing method. To see which web page version performs the best you’ll need to split traffic evenly between each of the pages. The more variable combinations you have the more traffic is needed to conduct the test. It can help to use a testing duration calculator to estimate how much traffic is needed before starting the test.

Collect the Data

Once the pages are viewable you’ll need to start collecting data for analysis. Here again, your analytics platform will play a vital role in automatically gathering data on user behavior and converting it into reports that help you decipher the results of the multivariate testing.

How to Determine Multivariate Test Results

Once the multivariate tests are complete you’ll need to have a statistically significant result. When a result is statistically significant there is a high level of confidence that the outcome from the test group likely applies to all users. 

To get a statistically significant result you’ll need to have a certain number of page views for each variation of the page. Since it can be difficult to determine exactly how many views you’ll need Mixpanel has a feature that will show you the statistical significance of results so you know for sure. 

With a multivariate test, you can also determine exactly which page element produced the biggest effect. This is known as the impact factor. Many analytics tools provide impact factor metrics that tell you which variable influenced conversions the most. 

It’s possible that one variable significantly improved page performance while another tested variable had no effect at all. When you identify an element that matters most on a page you can then drill down and test out variations of that particular element.

ANOVA (analysis of variance) can be used to determine which element is the most influential. It’s a statistical model that’s often used to compare the mathematical influence of an element relative to the other elements when there are more than two samples.

Knowing how individual elements impact conversion rate is beneficial beyond optimizing the test page. That knowledge can be used by the design team in the future to create better-optimized landing pages right from the start. 

When to do Multivariate Testing Instead of A/B Testing

There are times when another type of optimization testing known as A/B tests (split tests) is more appropriate than multivariate testing and vice versa. When you’re weighing whether to perform multivariate tests vs A/B tests it largely comes down to the number of variables that are changing and traffic.

A/B testing is used to gauge how one overall version of a page affects user behavior, conversions, etc. compared to another version. The versions are usually distinctly different all around. If you plan to change multiple elements of a landing page and want to know which specific elements affect behavior that’s when it can be beneficial to do multivariate testing instead of A/B testing. Because the page versions are so dramatically different in A/B testing it’s virtually impossible to attribute improved performance to a specific element.

Multivariate testing can also eliminate the need to do numerous A/B tests that look at variants one-by-one. It can be a faster process when you know you want to change more than one page element and analyze the performance of each one. However, the page will need to have enough traffic. If traffic is low a multivariate test can take a while and an A/B test could provide more accurate results.

While A/B tests are performed more often, typically require fewer resources and can yield bigger results, multivariate testing provides deep insights that allow you to optimize the finer details of a web page. You’ll also have better insight into which elements improve the conversion rate.

Learn how to pick the metrics that matter for your business

Download the free Guide to Product Metrics today.

Get the Guide