A/B Tests vs Multivariate Tests

A/B Test

An A/B test, also known as a split test, compares two versions of a web page (sometimes more) to determine which one is better at meeting a specific conversion goal. Typically, there are significant differences between the two pages and they could be altogether different. This method of testing can also be used to compare two versions of an email, app interface or ad. 

Multivariate Test

A multivariate test is used to gauge how variations in numerous page sections or elements perform when combined. Very similar yet unique pages are made for each combination of variants to determine which one has the best conversion rate. This type of test can tell you which elements have the biggest impact on user engagement and help to optimize individual elements of a web page. 

Examples:

An A/B test compares two pages with completely different headlines, text, CTAs and images. 

A multivariate test compares pages that are identical except the text is in a different font and the CTA is a different size. Four pages are created so that all the possible variant combinations are compared (Font 1/CTA 1, Font 1/CTA 2, Font 2/CTA 1, Font 2/CTA2).

Although some of the same principles and tools are involved, A/B tests and multivariate tests are distinctly different in a number of ways. 

Combinations of Variations

Many people consider multivariate tests to be a more complex version of an A/B test because there are more combinations of variations. It’s also not an either/or scenario like an A/B test. There can be dozens of variable combinations to compare and contrast. You are also testing how variables interact with one another on the page, which isn’t the case with an A/B test. 

Number of Test Pages

An A/B test is only going to involve two versions of a web page (sometimes three or four). A multivariate test, on the other hand, can include dozens of different versions of the web page because numerous variable combinations are being tested. 

Traffic Needs

The traffic needs to be evenly split between the web pages in both A/B tests and multivariate tests. That means to get a statistically significant result with a multivariate test that can have many page versions you’ll need more traffic than an A/B test that only has two page versions. 

For example, if your landing page gets 1,000 views in a week each page in an A/B test would get 500 views. However, if you were to do a multivariate test that had 12 page versions each one would only get around 83 views. 

Global vs Local Optimum

A/B tests tend to be used to find the best overall page whereas multivariate tests are used to find the best version of various elements on the page. This is sometimes referred to as the global optimum vs the local optimum.

Big Changes vs Subtle Changes

An A/B test tends to compare dramatically different pages with big changes. In a multivariate test,the differences aren’t as stark because the variations tend to be subtle. Essentially, with an A/B test you’re comparing two entirely different versions of a web page and with a multivariate test you’re comparing different elements within a web page.

Deciphering the Results

The results of A/B tests are usually easier to decipher because the test pages are so different and there are less of them. The subtlety of the variations and number of different pages can sometimes make multivariate test results less clear cut.  

Time Needed for Results

An A/B test will provide results much faster since there are only two options being compared that are clearly different. Depending on traffic levels and the number of variables, it could take months to complete a multivariate test. 

A/B tests and multivariate tests are similar in how they’re conducted. The main difference is that A/B tests look at the performance of just one variable at a time or the overall page whereas multivariate tests are testing multiple variables at once. 

A/B Split Test

With an A/B split test you’ll serve up two versions of a web page (email, ad, etc.) and split the traffic between them 50/50 to determine which one performs better. Whichever landing page is deemed the best may then be tested against another version of the web page. 

For example:

Test 1 – Current Control Page vs Page A 

Analysis shows that Page A performed better than the Control Page.

Test 2 – Page A vs Page B

Analysis shows that Page A again performs the best. 

Test 3 – Page A vs Page C

Analysis shows that Page A is once again the best option. Given the outcomes of all three A/B split tests, it’s determined that the landing page should be the Page A version. 

A/B/n testing is an alternative way to conduct an A/B test. Instead of running multiple tests with just two pages, you simply compare numerous versions of the page at the same time and split the traffic evenly. 

For example:

Current Control Page – 25% of traffic

Page A – 25% of traffic

Page B – 25% of traffic

Page C – 25% of traffic

Image Source: Conversion XL

Multivariate Test

Before running a multivariate test you’ll need to identify your key performance metrics (KPIs) and the page elements that are more likely to affect them. You then need to decide what variations of the elements you want to test. A multivariate test involves many more page versions than an A/B test because each possible variant combination needs to be compared. 

For Example:

3 Headlines X 2 CTAs X 2 Images = 12 Combinations/Page Versions

Page 1 – Headline A + CTA A + Image A 

Page 2 – Headline A + CTA B + Image A

Page 3 – Headline A + CTA A + Image B

Page 4 – Headline A + CTA B + Image B

Page 5 – Headline B + CTA A + Image A

Page 6 – Headline B + CTA B + Image A

Page 7 – Headline B + CTA A + Image B

Page 8 – Headline B + CTA B + Image B

Page 9 – Headline C + CTA A + Image A

Page 10 – Headline C + CTA B + Image A

Page 11 – Headline C + CTA A + Image B

Page 12 – Headline C + CTA B + Image B

Like the A/B split test, traffic should be divided up evenly for each of the pages. The scenario above is a good example of why a multivariate test requires much more traffic. Many multivariate tests involve 8-25 combinations that must be tested. 

Image Source: Search Engine Land

A/B tests tend to be the default starting point for testing. They are simpler, require less traffic and are usually less time-intensive. The fact that they compare pages with significant changes that have a big impact on conversion rate is also a factor.

It’s probably best to use an A/B test when: 

Multivariate tests aren’t used as much as A/B tests, but they provide insight that can’t be gained through split testing. 

It’s probably best to use a multivariate test when:

A multivariate test is particularly useful when an element is used across various pages, such as a universal CTA, navigation or footer. Testing on one page will apply to all the other pages without the need for additional tests.

Many analytics and web design experts will agree the best option in some cases is to use both A/B tests and multivariate tests. The A/B test can be used first to determine which layout or landing page design converts the best. Once the landing page is getting a good amount of traffic multivariate tests can then be used to fine-tune the page and gain insights for future development. 

Using a combination of the two tests can help you get the best conversion rate possible. 

At the end of the day, both A/B and multivariate testing are beneficial methodologies to help you test hypotheses and make decisions based on data rather than assumptions. They both have their pros and cons, which dictates when they are most useful. However, using them together can be the best way to fully optimize a specific web page as well as your entire website, app, email marketing or ad campaign.