Why A/B/Multivariate test?

There’s a lot of love about online marketing from a strategic standpoint.  Possibly the biggest differentiator between online marketing and other marketing methods is the degree to which it can be tracked.  After launching a website, social media presence, or campaign, the next logical step is to understand and optimize the performance you’re receiving.

While marketers often rely on industry best practices, past performance, and trends to optimize performance, one method of optimization seems to often get swept under the rug: A/B and/or multivariate testing.  A/B and multi-what?  Let’s start with what Wikipedia says and bring it down a notch:

A/B Testing

A/B testing, split testing, or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates.

In layman’s terms, two (or several) different versions of an ad, email, or web page are tested and measured to see which one performs best.  Pretty straightforward.  The two promotional banners below could be tested to see which encouraged more sales.

Multivariate Testing

This method is different than multivariate testing which applies statistical modeling which allows a tester to try multiple variables within the samples distributed.

Again, in (hopefully more) simple English, multivariate testing allows you to identify several items on a page, for example,  that for each, you wish to test different one or several different options.  An example would be an email where you want to test three different headlines, two alternate images, as well as three variations of the call-to-action to see which combination is the most successful.

Sounds great. Nah, I’m not interested

The goal of these testing methods is to serve the best possible option to the end-user and in doing so to optimize performance.  So, why doesn’t everyone run tests like this?

Here are three common reasons that A/B Multivariate testing isn’t done:

1. “It costs too much money”
   Unless budgets are incredibly constrained, this shouldn’t be an issue.  Testing doesn’t have to be expensive, especially with the tools available today, including free one’s like Google’s Website Optimizer (note that Website Optimizer is for just that-websites, not email, ads or other media).  And, if you’re working with an agency, they likely have tools that they’ve already licensed.  The cost is then limited to concepting the different versions and putting the test in place.
2. “We tried it once and didn’t see much of a difference between versions”
   You definitely won’t hit a home run every time.  Just because testing on one occasion revealed that between the options tested there wasn’t much of a difference, that doesn’t mean that nothing should be tested.
3. “We know better”
   Clients often simply think that they understand their offering, brand and customer so well that testing is just a waste of time and money.  While I can understand (and respect) a client’s knowledge and intimacy with their brand, what sometimes comes to light is that what performs best can defy common logic or perceived wisdom.  Sometimes the consumer just sees it differently from their side of the equation.

What you don’t know can cost you money

To put it as bluntly as possible, if you don’t test something you simply won’t know if it could have performed better.

Yes, you can run tests and find that other options offer marginable, offer only negligible gains, or offer even worse results.  But, you may also find out that there’s an option that performs significantly better, sometimes by a long shot, pushing the impact of your marketing dollars far further.

If your budget allows for it, I’d recommend testing when it makes sense.

Some final advice

First of all, you certainly don’t need to test everything.  That tagline that the new guy brought up in the marketing meeting that clearly sucked will still suck no matter how many ways you test it.

A great way to get started is to test something that will effect the return on your marketing investment.  Possibly an email that is going out that has a call-to-action to redeem an offer or sign up for a seminar, or a landing page on your website where you drive paid marketing traffic and hope for a conversion.

To find a great candidate for testing, identify something that has created some uncertainty internally.  Are you struggling with two subject lines that both state the same thing, just with a different tone?  Are you considering changing the placement or styling of a form?  Wondering if you might capture more leads if less information were required of the visitor?  Each of these situations would create a great testing opportunity.

A few A/B/Multivariate testing resources that might help you:

• Google Website Optimizer
  Google Website Optimizer is designed for testing landing pages for Google AdWords campaigns.  The tool is made available for free by Google.
• Google Website Optimizer Blog
  The blog for Website Optimizer has great ideas and results from real-world tests.
• Which Site Won
  This site is really enlightening if you’re considering testing. Marketing agency consultant Jay Baer of Convince and Convert recently tweeted about it and I wish I would have stumbled across it long ago.  The site shows results of A/B tests, and even lets you play along and see if you can pick which version performed better prior to seeing the actual results.
• Marketing Experiments
  They’ve been running tests for quite some time over at Marketing Experiments, and they provide their results in succinct, easy to understand terms.
• 15+ Free A/B Split Testing Resources
  There’s a good round-up of free resources available on the Internet for A/B testing on this Wingify blog post.  Definitely worth a read.  If you like the post, browse the other blog entries on Wingify… They’re all about testing.

Good luck!

Tagged as: A/B Testing, Analytics, Conversion, Google Website Optimizer, Multivariate Testing, Online Marketing/Strategy