How Multivariate Testing can help you optimise your website

September 15, 2016 - 5 minutes read

To maximise customer retention on a site, customers need to be satisfied with the content they see. The chances of customers leaving a website is high if the initial content that meets their eye is not what they want to see. Testing allows for businesses to present their customers with exactly what they want by running multiple tests over time which analyse customer behaviour and whose results can be used to determine how satisfied the customer is with the website. Optimising customer satisfaction translates directly into more sales for the business.


We previously talked about A/B testing and how its a great tool for running tests with two variables. Multivariate testing is much more flexible and makes use of inputs from multiple variables to make decisions. Different combinations of variations can be pitted against one another to figure out which combination gives back the best results. Based on these results the tests can be routinely modified for a better user experience.

How is it better than A/B testing?

With the ability to change multiple variables, your testing will give back larger data sets of information relating to customer preferences. It also allows for testing multiple content simultaneously on your website.

If your only goal is to check which sign-up button receives higher responses, then A/B testing is the go to process to make  your decision. But what if you also want to test different background images that go along with the sign-up button. This is where multivariate testing comes to your aid.

The example below shows how the Obama 2008 campaign used multivariate testing to optimise their sign-up page:

According to Dan Siroker, CEO of optimisation software firm Optimizely, the campaign team tested all the combinations of four sign-up buttons and six different media (3 images and 3 videos).

The images below show the variations to the media sections which were tested.




The four different variations of sign-up buttons which were tested were: “JOIN US NOW“, “LEARN MORE“, “SIGN UP NOW” and “SIGN UP

That would make the total number of combinations in this multivariate test to be:

6 variations of  image * 4 variations of sign-up button =  24 total combinations

Results showed that the “Learn More” button coupled with the “family image” tested the highest.



While the original page had a sign-up rate of 8.26%, the leading combination showed a sign-up rate of 11.6%. That is a 40.6% increase in sign-up’s which was achieved through multivariate testing.

Are there any cons to multivariate testing?

Things can get complicated

Like I said, multivariate testing gives you large data sets of feedback that need to be analysed to optimise the decision making process. Although having more data makes for better decisions, things can get much more complicated than your usual A/B test. There’s a higher chance that mistakes or errors occur when reporting results. Get ready to handle the data load if you’re doing a multivariate test.

You need more traffic

An A/B test cuts down your testing base into two. Even with an average amount of traffic A/B tests can give meaningful results. But, multivariate testing keeps cutting into your traffic the more combinations you run. Your website needs large enough traffic for each individual test for the results to be meaningful.

Optimising your website should be high on the list of priorities for any business. Running a successful business means providing your customers with the best possible experience. Multivariate testing provides you with the information you need to tailor your website to present the best and most attractive face of your business to your customers. While intuitions can go wrong, data always speaks the truth.

Tags: , , , ,