Skip to main content

Difference between A/B testing and Multivariate testing

It’s been a while, so in the meantime I’ve found this quite useful as we are look to start introducing this within my streams of work!

Many people seem to get confused about the terms A/B testing, split testing and multivariate testing. Essentially, A/B testing and split testing are the same concepts but multivariate testing is different. So for all practical concepts, consider A/B Testing = Split Testing. The difference between A/B testing (or split testing) and Multivariate Testing:

A/B Testing

In A/B testing you split traffic amongst two or more completely different versions of a webpage (landing page, home page, etc.) The variations of your original page can differ in any manner. You can either just change the headline; or you can even change entire design, layout, offer and what not in the variations. What you change in A/B test is only limited by your creativity. For example, if you are A/B testing on your landing page you may want to create one version with 15% discount, the other with free shipping and one with the same offer but different page design.

Really useful for testing radically different ideas for conversion rate optimization.

Multivariate Testing

In multivariate test, you identify a few key areas/sections of a page and then create variations for those sections specifically (as opposed to creating variations of whole page in an A/B split test). So for example, in multivariate test you can choose to create different variations for 2 different sections: headline and image. A multivariate testing will combine all these section specific variations to generate unique versions of page to be tested and then simply split traffic amongst those versions. That is, you will get these variations of the page: Headline1_Image1, Headline1_Image2, Headline2_Image1, Headline2_Image2.

Really useful for optimizing and refining an existing landing page or homepage without doing significant investment in redesign.

I hope that was a good comparison between A/B testing and multivariate testing. If you think I left any point, please let me know. I will update the comparison table accordingly.

Blog post from:


Popular posts from this blog

My Testbash Brighton 2017 Notes

This was my second Testbash... if you ever get the chance in the future these conferences are a must!
I took copious amounts of notes from the 9 talks and tried to highlight my key takeaways here... hope they make sense but please comment if you have any questions :-)
Amy Phillips - Continuous Delivery
A survival guide to joining a fast paced environment/project…
Where does testing fit within Continuous delivery:

As highlighted, basically from start to finish…
There are lots of things we can do when joining a project that is using Continuous Delivery but one of the main points from this talk was to do your research! There should be an element of "Continuous" in every aspect of the project.
·Learning the ling, what's the difference Continuous Delivery, Continuous Deployment, Continuous Integration, Continuous design, Continuous Improvements etc.? ·Understand what your role in the project is going to be ·Understand the teams values, What's going well and maybe what the team p…

The way we work shouldn’t be ‘Alien’ anymore, Should it!?!?

It’s been a good while since I have written anything (Mainly because I’m not that good a writer) but given my 16 months in the public sector I thought it was time… 
For a good number of years now the DevOps trend has given rise of buzzwords and methods aiming to speed delivery and it’s now an assumption of mine that most organisations are au fait with the term ‘Continuous Delivery’.
I was wrong…
Within the organisation that I am currently placed I believe I have a high-functioning collaborative team who work flexibly and in continuous cycle. We also involve the users which is a huge benefit to the development teams, but that’s not saying we get everything right 100% of the time! Teams must continue to learn and evolve to remain productive.
Our team work in one of several delivery groups and with this comes multiple challenges; Silos No single environment strategy lack of communicationsVaried ways of working.  And thats just to name a few, which is interesting when all delivery groups …

Leeds Testing Atelier - April 2018