While marketers often rely on industry best practices, past performance, and trends to optimize performance, one method of optimization seems to often get swept under the rug: A/B and/or multivariate testing.  A/B and multi-what?  Let’s start with what Wikipedia says and bring it down a notch:

A/B Testing

A/B testing, split testing, or bucket testing is a method of marketing testing by which a baseline control sample is compared to a variety of single-variable test samples in order to improve response rates.

In layman’s terms, two (or several) different versions of an ad, email, or web page are tested and measured to see which one performs best.  Pretty straightforward.  The two promotional banners below could be tested to see which encouraged more sales.

Multivariate Testing

This method is different than multivariate testing which applies statistical modeling which allows a tester to try multiple variables within the samples distributed.

Again, in (hopefully more) simple English, multivariate testing allows you to identify several items on a page, for example,  that for each, you wish to test different one or several different options.  An example would be an email where you want to test three different headlines, two alternate images, as well as three variations of the call-to-action to see which combination is the most successful.

Sounds great. Nah, I’m not interested

The goal of these testing methods is to serve the best possible option to the end-user and in doing so to optimize performance.  So, why doesn’t everyone run tests like this?

Here are three common reasons that A/B Multivariate testing isn’t done:

1. “It costs too much money”
   Unless budgets are incredibly constrained, this shouldn’t be an issue.  Testing doesn’t have to be expensive, especially with the tools available today, including free one’s like Google’s Website Optimizer (note that Website Optimizer is for just that-websites, not email, ads or other media).  And, if you’re working with an agency, they likely have tools that they’ve already licensed.  The cost is then limited to concepting the different versions and putting the test in place.
2. “We tried it once and didn’t see much of a difference between versions”
   You definitely won’t hit a home run every time.  Just because testing on one occasion revealed that between the options tested there wasn’t much of a difference, that doesn’t mean that nothing should be tested.
3. “We know better”
   Clients often simply think that they understand their offering, brand and customer so well that testing is just a waste of time and money.  While I can understand (and respect) a client’s knowledge and intimacy with their brand, what sometimes comes to light is that what performs best can defy common logic or perceived wisdom.  Sometimes the consumer just sees it differently from their side of the equation.

What you don’t know can cost you money

To put it as bluntly as possible, if you don’t test something you simply won’t know if it could have performed better.

Yes, you can run tests and find that other options offer marginable, offer only negligible gains, or offer even worse results.  But, you may also find out that there’s an option that performs significantly better, sometimes by a long shot, pushing the impact of your marketing dollars far further.

If your budget allows for it, I’d recommend testing when it makes sense.

Some final advice

First of all, you certainly don’t need to test everything.  That tagline that the new guy brought up in the marketing meeting that clearly sucked will still suck no matter how many ways you test it.

A great way to get started is to test something that will effect the return on your marketing investment.  Possibly an email that is going out that has a call-to-action to redeem an offer or sign up for a seminar, or a landing page on your website where you drive paid marketing traffic and hope for a conversion.

To find a great candidate for testing, identify something that has created some uncertainty internally.  Are you struggling with two subject lines that both state the same thing, just with a different tone?  Are you considering changing the placement or styling of a form?  Wondering if you might capture more leads if less information were required of the visitor?  Each of these situations would create a great testing opportunity.

A few A/B/Multivariate testing resources that might help you:

• Google Website Optimizer
  Google Website Optimizer is designed for testing landing pages for Google AdWords campaigns.  The tool is made available for free by Google.
• Google Website Optimizer Blog
  The blog for Website Optimizer has great ideas and results from real-world tests.
• Which Site Won
  This site is really enlightening if you’re considering testing. Marketing agency consultant Jay Baer of Convince and Convert recently tweeted about it and I wish I would have stumbled across it long ago.  The site shows results of A/B tests, and even lets you play along and see if you can pick which version performed better prior to seeing the actual results.
• Marketing Experiments
  They’ve been running tests for quite some time over at Marketing Experiments, and they provide their results in succinct, easy to understand terms.
• 15+ Free A/B Split Testing Resources
  There’s a good round-up of free resources available on the Internet for A/B testing on this Wingify blog post.  Definitely worth a read.  If you like the post, browse the other blog entries on Wingify… They’re all about testing.

Good luck!

Related posts:

1. Free Google multivariate testing tool now public
2. Study: Professional marketers missing the basics
3. Google pay-per-action for AdWords

1. Copy the entire bit.ly link
2. Paste it into the address bar in any web browser
3. Add a ‘+’ after the bit.ly link
4. Hit return

You’ll be directed to a bit.ly page that shows the statistics for that link.  Nice.

One word of caution: bit.ly does occasionally start offering up the same shortened bit.ly URL when a strong demand for the exact same link comes up.  Do click on the “Referrers” link on the bit.ly statistics page and you’ll get a better idea if this is the link you created or one that bit.ly is giving to a lot of others also.

Credit goes to Jeff Widman for the tip (@JeffWidman, BrandGlue).  Thanks Jeff!

No related posts.

Surprisingly, my group is the smaller, not larger one.

A recent study by Alterian reveals that less than half of professional marketers use analtyics to measure their campaigns.  Is employing analytics really all that difficult?  Apparently it is.  A press release regarding the study noted that nearly a third “…of those surveyed cite analyzing results as the hardest part of any campaign.”

Not only is apathy for performance measurement en vogue, how a website fits into the marketing mix is askew:

One-fifth of respondents claimed that their website was only ‘basic’ and not at the core of its marketing activity, but with increased investment predicted in online marketing channels, companies need to better understand the importance of the website in effectively underpinning its online presence.  This demonstrates a need for companies to better understand the channels available to them before investing their marketing funds.

Time to back up and get the basics in check, especially in a time when every marketing dollar counts.

Related posts:

1. Where’s the online marketing community?
2. Yahoo takes a play from Google’s aquisition book
3. Why A/B/Multivariate test?

In the book The Search author John Battelle describes Google as “a database of intentions.” If you prescribe to that classification, the Google tools I’m going to employ will provide some interesting insight into shopper intentions and how that information can be leveraged.  The first tool I’m going to use is Google Insights for Search.  Insights for Search is another Google Experiment, this time one that allows one to peer inside the volume of keyword (or phrase) searches over time and geographical regions.  Let’s see what we can learn.

What can Google tell us about Valentine’s Day shopping?

Our hypothesis will be that people are still shopping online, but they’re likely looking to get a bargain.  The hypothesis ins’t a stroke of genius, but the devil is in the details.  If we’re going to launch a campaign or put some copy on our homepage regarding our holiday deals, let’s use words that resonate with online shoppers.  Further, since we’re theoretically doing this exercise the week before the holiday, Pay-per-Click is a definite medium we would target.  We’d want to know not only what keywords we might want to focus on for our ads to display, but also what keywords to place in generalized ads to get Internet searcher’s attention.

Let’s take a look at the frequency of terms being searched upon in the United States over the past couple of years using Insights for Search.  The search phrases I’m going to take a look at are:

• Valentine’s Day
• Valentine’s Day roses
• Cheap Valentine’s Day gift
• Valentine’s Day specials
• Valentine’s deals
• Valentine’s discount

Search volume for “Valentine’s Day”

Let’s start by looking up a generalized term about the holiday to use as a baseline for interest overall around the topic over time.  The graph below shows search volume for “Valentine’s Day” from January of 2007 until February of 2009 in the United States.  To be sure we’re focused on shopping, we’ve restricted our inquiry to the “Shopping” category.   The February ‘09 data for this year, term and region is reported to the current week, but we’ll focus on the week before the holiday for all of our analysis.

• 2007: 60
• 2008: 48
• 2009: 53
  

Before we move forward, let’s be clear about what these numbers mean. The numbers on the graph do not allow us to understand the total number of searches for this term in the United States. I’ll let Google tell you what the numbers mean as opposed to paraphrasing:

The numbers on the graph reflect how many searches have been done for a particular term, relative to the total number of searches done on Google over time. They don’t represent absolute search volume numbers, because the data is normalized and presented on a scale from 0-100; each point on the graph is divided by the highest point, or 100. The numbers next to the search terms above the graph are summaries, or totals.

If I check the points on the graph, last week’s number (the week before Valentine’s Day) registers 60. The week before Valentine’s Day in ‘08 is lower, at 48, and that week in ‘07 clocked in at 53.  This shows an increase in the volume of shopping related searches for “Valentine’s Day” for the week before Valentine’s Day this year in the United States as compare to the previous two years.

For the purpose of this exercise, I’m also going to scale data to a 100 point scale when our results fail to do so to make comparisons easier.  Here’s what that will look like for the aforementioned data:

• 2007: 60 (100)
• 2008: 48 (80)
• 2009: 53 (88)

As above, when data is extrapolated to a 100 point scale I’ll include it in parenthesis.  Graph data will not be changed.

Not following the trend

So, we’ve established that the volume of searches for Valentine’s Day gifts the week before the holiday has increased considerably in comparison to the two previous years.  To help illustrate our point regarding the discount terms, I’m going to run an example non-discounted term: “Valentine’s Day roses.”

Search volume for “Valentine’s Day roses”

Below is our result.  Search volume is fairly close to 2008 levels, both of which years are lower than 2007.

• 2007: 62 (100)
• 2008: 43 (69)
• 2009: 42 (68)

So, what about the thrifty shopper?

Moving on to bargain hunting, let’s run the same time period but try some phrases that would show people searching with the intent of finding discounted Valentine’s Day gifts. After trying a few phrases and seeing volume increases that were generally aligned with the increase in searches for “Valentine’s Day,” I found several that also showed strong upticks, specifically around the idea of getting a bargain.

Search volume for “Cheap Valentine’s Day gift”

This first graph is for “Cheap Valentine’s Day gift.” For the weeks preceding Valentine’s day, 2007’s interest level is at 40, 2008’s is 36, and 2009 holds the top end (100). So, while shopping-related searches have increased for the term “Valentine’s Day,” searches for those looking for a cheap Valentine’s Day gift have also increased, but to a much more significant degree.

• 2007: 40
• 2008: 36
• 2009: 100

Let’s look for more terms and see if we can find higher variances while still reflecting intent.

Search volume: “Valentine’s Day specials”

Another term I thought that might indicate an intention to find a deal was the phrase “Valentine’s Day specials.” Again, the result shows the same trend.

• 2007: 31 (62)
• 2008: 28 (56)
• 2009: 50 (100)

Search volume: “Valentine’s deals”

Bingo. “Deals” seems to be a word that people are using far more when shopping for Valentine’s Day this year and looking for a bargain.  Do note that I dropped “Days” from the phrase as I wasn’t getting results, but felt “Valentine’s” would be enough of a qualifier.  Take a look:

• 2007: 19
• 2008: 29
• 2009: 100

Search volume: “Valentine’s discount”

“Valentine’s discount,” the last phrase I’m including, follows suit with a considerable uptick.

• 2007: 41
• 2008: 42
• 2009: 100

Do we have any actionable data yet?

Not really.  We still need more information to paint the full picture.  If we were going to use this information to understand linguistics and what words were being used more frequently when shopping online for Valentine’s Day compared to previous years, this data is helpful.  But, we really need to get a better idea of how much search volume we’re talking about.

If a phrase was searched on three times during the week before Valentine’s Day in 2008 and 10 times the week before Valentine’s Day this year, we’d see a huge volume increase year-over-year, but creating a marketing campaign based upon that information wouldn’t be the wisest move.

In my next post I’ll take what I’ve learned so far and leverage another of Google’s tools to show how this information could be used to make online marketing decisions.

Related posts:

1. Google AdWords keyword tool now providing search volume
2. Have you been getting Google keyword search volume right?
3. Android and SEO/SEM? Dream on.

Non e-commerce web sites often measure their success based on the averages analytics tools provide.  Does average page views per visit or average time spent on site sound familiar?  If so, the video below is worth your time.  In it web analytics professional Avinash Kaushik lectures on how averages can blind us from seeing the valuable data that can drive effective decision making.

If you enjoy this video, consider Avinash’s book Web Analytics and Hour a Day, and add his web analytics blog to your feed reader.

Related posts:

1. Two recommended books for the online marketer
2. Yahoo takes a play from Google’s aquisition book
3. Study: Professional marketers missing the basics

Today, there are other options, and ones that go far beyond a relative rankings system or even traffic. For the price (free), no one provides more information than Quantcast. Before we dive in, here’s how Quantcast defines themselves:

Quantcast is a new media measurement service that lets advertisers view audience reports on millions of websites and services. Only Quantcast combines directly measured audience data with panel-based estimates to deliver accurate third-party metrics and easy-to-read profiles on digital media properties.

Here’s a quick overview of some of what you can learn about your competitors on Quantcast’s web site:

• Visits per month
• People per month (individual visitors)
• Gender, age, household income, ethnicity and more of domestic traffic

That’s far more than you’ll find over at Alexa. But, there’s more. Beyond traffic and demographic information, Quantcast also provides:

• Brand and site affinities
• Other sites with similar audiences

Let’s take a look at some of that data. Here’s some sample Quantcast data for John Battelle’s Searchblog (http://battellemedia.com)

Here’s some information Quantcast provides:

• People – Global: 36,884
• People – U.S.: 23,461
• Average Page Views/Visit – US: 1.33 US
• Average Page Views/Visit – Global: 1.31
• 62% of traffic is Male, 38% Female
• The household income for 64% of visitors is between $30,000 an $100,000
• The majority of visitors, 70%, are 35 or over
• Visitors have brand & site affinities for telegraph.co.uk and the Washington Post

How Quantcast arrives at the information it provides is through the combination of panel data and directly measured data. Not all sites on Quantcast are providing directly measurable data, however. John Battelle’s Searchblog is, and you can tell that because on the report page for the web site, there is a “Quantified Publisher” icon toward the top of the page. This indicates that this web site has placed a tracking script on each page of the web site, allowing Quantcast to directly measure (and share) site usage information. If this icon isn’t present, the data Quantcast provides is not nearly as accurate.

Just to be clear, providing competitive metrics isn’t why Quantcast exists. That said, by the nature of the information it provides Quantcast can be used for competitive metrics. If you leverage it for competitive research be sure you understand how Quantcast collects information. If you’re making decisions based on what Quantcast (or any other data provider offers) the first step is understanding the strengths and weaknesses of your data source.

Related posts:

1. Alexa rethinks rankings
2. Google Analytics: 00:00:00 Avg. Time on Site?
3. Google Labs provides popularity of keywords, sites

My gooruze page: smcandrew.gooruze.com
My Sphinn profile: smcandrew on Sphinn

Related posts:

1. 5 Reliable (social media) sources
2. Study: Professional marketers missing the basics
3. Math and… better math on the State of the Union

There are some caveats.  Current IndexTools customers will be required to sign a standard Yahoo! agreement, and at this time the freebie won’t be for everyone.  The masses will have to wait until a later release to take advantage of the tool.  During that time Yahoo will likely be tightly integrating IndexTools with their complimentary offerings.

So here’s the analytics landscape by search provider:

• Google’s analytics tool: Google Analytics
• Microsoft’s analytics tool: adCenter Analytics (previously “Gatineau”)
• Yahoo’s analytics tool: IndexTools (for now)
• Ask’s analytics tool: Nothing to speak of. They do have “the algorithm” for whatever that’s worth.

Although I’ve never used Index Tools but I’m more than excited to see another compelling offering at no charge.

More on Yahoo and IndexTools:

• IndexTools web site
• IndexTools (Yahoo!) Web Analytics goes FREE! – VisualRevenue

Related posts:

1. Yahoo! Search Pad for research online
2. Marketers and Developers: There’s (finally) a Google Analytics API
3. Analytics means you analyze

…uploaders can see how often their videos are viewed in different geographic regions, as well as how popular they are relative to all videos in that market over a given period of time. You can also delve deeper into the lifecycle of your videos, like how long it takes for a video to become popular, and what happens to video views as popularity peaks.

The blog posting on YouTube’s site indicated that new features will be added to the reporting capabilities quickly. That’s good news—the initial roll out features are welcomed, but aren’t all that robust or useful—especially after the drought of information previously available. YouTube, here’s a few useful metrics to add to your list:

• Average duration of video view in minutes:seconds
• How many times the video was shared
• Views on syndication sites by syndication URL
• Viewer demographic information

YouTube: if you need more ideas on what your users really want, contact me, or better, look at your user feedback over the past few years.

Oh, and one more thing. I’ve got enough places to track data. Why can’t we just get this data inside Google Analytics?

Also see:

• The original post on YouTube’s blog
• YouTube Insight blog post on The Official Google Blog
• YouTube Insight: Underwhelming – Conde Nast Portfolio.com

Related posts:

1. Google introduces Search Options and Snippets to SERPs
2. Page 1 Google rankings in hours (easily)
3. Marketers and Developers: There’s (finally) a Google Analytics API

Time’s visualization went with a view which scaled the size of the words in proportion to their frequency.  Think tag cloud.  The basic visualization technique was posted under the heading “State of the Union by the Numbers”.  Yawn.

With a simple explanation of how the technique works, a teenager could whip this out with a steno pad and a pencil.  The hardest part would be making it through the whole speech.  Reporting.  Not analysis.

Slate Magazine took a very different approach.  And they knew their results were in a different neighborhood than what Time and others were up to.  Slate’s work-up relied on the slick natural-language processing algorithms baked into Crawdad Technologies’ software.  A bit different than a simple counting exercise.  An excerpt from the Slate article (A Computer Reads the State of the Union):

By natural-language-processing standards, simply counting the frequency of words is unthinkably primitive. As computers get better at understanding the structure of sentences—think about that corrugated green line under a sentence in Microsoft Word reminding you to avoid the passive voice—they can do a much more thorough job reading political speeches and offering their insights.

Needless to say, Slate’s experiment is just a bit more interesting.  Looking at not only last night’s State of the Union, but all of Bush’s Presidential addresses, the software identified “four distinct voices” during his time in the White House:

1. “Domestic Bush” – 2001
2. “Security Bush” – 2002, 2003
3. “Visionary Bush” – 2004, 2005, 2006, 2007
4. “Legacy Bush” – 2008

The table below provides further insight.

For another example of their technology in action, visit the Wonkosphere, currently tracking buzz online surrounding the 2008 Presidential race.  Learn more about Crawdad Technologies and their software at their web site, crawdadtech.com.

What does this have to do with online marketing?  Good natural-language processing technology is a Swiss Army knife for measuring buzz—and its tone—online.  Before stumbling across Crawdad I had planned to post about a few companies I’ve been interested in whose offerings cater to online marketers.  As with Crawdad’s Wonkosphere, these solutions focus on monitoring social media outlets.  A few of those companies: Nielsen (BuzzMetrics), Radian6 and Relevant Noise.

Related posts:

1. Google Buzz Poll: Force or Flop?
2. Online reputation management and the power of consumer opinion
3. Study: Professional marketers missing the basics