While marketers often rely on industry best practices, past performance, and trends to optimize performance, one method of optimization seems to often get swept under the rug: A/B and/or multivariate testing.  A/B and multi-what?  Let’s start with what Wikipedia says and bring it down a notch:

A/B Testing

A/B testing, split testing, or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates.

In layman’s terms, two (or several) different versions of an ad, email, or web page are tested and measured to see which one performs best.  Pretty straightforward.  The two promotional banners below could be tested to see which encouraged more sales.

Multivariate Testing

This method is different than multivariate testing which applies statistical modeling which allows a tester to try multiple variables within the samples distributed.

Again, in (hopefully more) simple English, multivariate testing allows you to identify several items on a page, for example,  that for each, you wish to test different one or several different options.  An example would be an email where you want to test three different headlines, two alternate images, as well as three variations of the call-to-action to see which combination is the most successful.

Sounds great. Nah, I’m not interested

The goal of these testing methods is to serve the best possible option to the end-user and in doing so to optimize performance.  So, why doesn’t everyone run tests like this?

Here are three common reasons that A/B Multivariate testing isn’t done:

1. “It costs too much money”
   Unless budgets are incredibly constrained, this shouldn’t be an issue.  Testing doesn’t have to be expensive, especially with the tools available today, including free one’s like Google’s Website Optimizer (note that Website Optimizer is for just that-websites, not email, ads or other media).  And, if you’re working with an agency, they likely have tools that they’ve already licensed.  The cost is then limited to concepting the different versions and putting the test in place.
2. “We tried it once and didn’t see much of a difference between versions”
   You definitely won’t hit a home run every time.  Just because testing on one occasion revealed that between the options tested there wasn’t much of a difference, that doesn’t mean that nothing should be tested.
3. “We know better”
   Clients often simply think that they understand their offering, brand and customer so well that testing is just a waste of time and money.  While I can understand (and respect) a client’s knowledge and intimacy with their brand, what sometimes comes to light is that what performs best can defy common logic or perceived wisdom.  Sometimes the consumer just sees it differently from their side of the equation.

What you don’t know can cost you money

To put it as bluntly as possible, if you don’t test something you simply won’t know if it could have performed better.

Yes, you can run tests and find that other options offer marginable, offer only negligible gains, or offer even worse results.  But, you may also find out that there’s an option that performs significantly better, sometimes by a long shot, pushing the impact of your marketing dollars far further.

If your budget allows for it, I’d recommend testing when it makes sense.

Some final advice

First of all, you certainly don’t need to test everything.  That tagline that the new guy brought up in the marketing meeting that clearly sucked will still suck no matter how many ways you test it.

A great way to get started is to test something that will effect the return on your marketing investment.  Possibly an email that is going out that has a call-to-action to redeem an offer or sign up for a seminar, or a landing page on your website where you drive paid marketing traffic and hope for a conversion.

To find a great candidate for testing, identify something that has created some uncertainty internally.  Are you struggling with two subject lines that both state the same thing, just with a different tone?  Are you considering changing the placement or styling of a form?  Wondering if you might capture more leads if less information were required of the visitor?  Each of these situations would create a great testing opportunity.

A few A/B/Multivariate testing resources that might help you:

• Google Website Optimizer
  Google Website Optimizer is designed for testing landing pages for Google AdWords campaigns.  The tool is made available for free by Google.
• Google Website Optimizer Blog
  The blog for Website Optimizer has great ideas and results from real-world tests.
• Which Site Won
  This site is really enlightening if you’re considering testing. Marketing agency consultant Jay Baer of Convince and Convert recently tweeted about it and I wish I would have stumbled across it long ago.  The site shows results of A/B tests, and even lets you play along and see if you can pick which version performed better prior to seeing the actual results.
• Marketing Experiments
  They’ve been running tests for quite some time over at Marketing Experiments, and they provide their results in succinct, easy to understand terms.
• 15+ Free A/B Split Testing Resources
  There’s a good round-up of free resources available on the Internet for A/B testing on this Wingify blog post.  Definitely worth a read.  If you like the post, browse the other blog entries on Wingify… They’re all about testing.

Good luck!

Related posts:

1. Free Google multivariate testing tool now public
2. Study: Professional marketers missing the basics
3. Google pay-per-action for AdWords

Call to Action: Secret Formulas to Improve Online Results
Somehow Call to Action escaped my reading list for years. I learned about it from someone I recently interviewed. The book is based on Bryan and Jeffrey Eisenberg’s concept of Persuasion Architecture—a methodology which, at a high level, combines consumer psychology, business goals and website design with an overall goal of maximizing conversions.

Web Analytics: An Hour a Day
I’m one of those people who can’t get enough of web analytics. Written by respected blogger and Google Evangelist Avinash Kaushik, this book demystifies web analytics in an easy to read, enjoyable manner. The practice of web analytics isn’t as simple as it may appear at a glance. Avinash’s book provides new perspectives in a concise manner.

Related posts:

1. Web analytics: averages mask the truth [video]
2. Google pay-per-action for AdWords
3. Why A/B/Multivariate test?

The cost for Website Optimizer? It’s free. The catch? Depends on how you look at it, and more importantly, what your needs are. Things to consider:

1. It appears, although nothing in the documentation I read says it explicitly, that you can only run experiments on a single page (as opposed to a flow which consists of multiple pages-like a checkout process made up of 4 screens, for example).
2. I could not find any documentation that ensured me that users who were viewing a specific version of a page would see that exact same version of the page if they returned during the testing period.
3. Website Optimizer does not allow variable content to be a part of the test. You can apply Website Optimizer ‘experiments’ to variable content pages, however, the variable content itself cannot be tested against.

If you’re looking to test a single landing page which is well trafficked, or a part of a marketing campaign, Website Optimizer is definitely priced right. And, there is no doubt that multivariate testing is a more effective way to perform these types of tests. If you’re looking to perform more involved testing, you’ll likely be looking to other providers such as Offermatica, Optimost, or SiteSpect. Be warned: those tools are decidedly NOT free.

Related posts:

1. Why A/B/Multivariate test?
2. AdWords testing increased relevance for advertisers
3. Page 1 Google rankings in hours (easily)

The new ads are for Google AdSense for content sites. AdSense publishers can decide if they want to participate in the program or not. Participants will be able to review details of the advertisements which will appear on their site and decide if the ads are appropriate for their site or not.

One would assume that the amount paid for an actual conversion will be far higher than what one gets for a click under the current PPC model-especially as the amount will be able to be determined by the Advertiser, setting up a affiliate-like relationship, brokered by Google.

The program introduces new challenges for those managing a web site with the ads. With PPC, the user had little control over what went on their site and only had to rely on ad copy driving a click. Now, they’ll have to consider if the web site the user will visit as a result of a click has the potential of closing the deal.

Update: Google’s Inside AdWords blog has a post specifically about the Pay-Per-Action beta.

Related posts:

1. AdWords testing increased relevance for advertisers
2. Live cashback a desperate move?
3. Making money how?