A/B Testing and Multivariate Testing of Marketing Campaigns

I get asked frequently if Market2Lead supports A/B testing of email campaigns and the answer is “yes, of course,” but sometimes I wonder if the people have thought through the process.

You can dissect the average email campaign down into 6 broad areas that can influence the outcome as shown in the graphic above. In my August 28, 2009 post on effective campaigns I discussed the value of a high quality list as the primary determinant of the effectiveness of a campaign, so we won’t repeat that here. Instead I want to discuss how one would best go about A/B testing campaigns for the 5 remaining items.

Select the A/B Testing Outcome To Test

In B2B with long sales cycles, and multiple touch points, it would be silly to A/B test email campaigns for closed won opportunities, but in B2C this might be the perfect answer. In B2B email campaigns you are probably more likely to constrain yourself to opens, clicks, and offer downloads. In B2C with short sales cycles, you might consider “added to basket”, or purchases as the outcome you are testing.

A/B Testing Requires Random List Segmentation

To be an effective A/B test, the other factors beyond the one you are testing must be held constant or, in the case of segmented lists of people, be made random. So the first question should be, how does one produce random lists:

1. Allow the marketing automation system to randomly pick people from your segmented list for either the A or B treatment. To the best of my knowledge no marketing automation vendor offers this today.
2. Manually produce two lists (A and B) using some random mechanism; you could use the odd numbered contact ID numbers for the A list for instance. Trying to create random lists by selecting letters found in names or emails is not as random as you might think!  But if the contact ID numbers are assigned randomly when new leads are created or imported, it is a great way to go.

A/B Testing with One Program and Two Lists?

Some folks believe that it is simpler to use one program with two random lists, and have the functionality built in to apply the appropriate A/B subject line, or A/B email body, or A/B offer, or A/B Landing Page, or A/B Form to the A or B target list. In the reporting, the system would then have to be smart enough to produce two different reports on clicks, opens, downloads etc based on the A/B variable that was chosen. I find this too complicated.

I much prefer to create the program for A, copy it, point it to list B, change my one variable (subject line, or email body, or offer, or landing page, or form) and launch both programs. I can do this in two minutes and they each have their own results, it is easy to interpret, and easy to copy these programs for future tests.

You Don’t Need Dedicated Functionality for A/B Testing

The bottom line is that you can do some very creative A/B testing or multivariate testing as long as you have a means to create random lists. Everything else is easily done by copying the program and changing your one variable, and the reporting will make much more sense to you and everybody else.

So, an exercise for the reader: How would you do A/B/C testing since 3 does not divide evenly into 10, so our trick of using odd versus even contact IDs won't help...

-Kevin

Comments

[Rex Dixon]

Do you have any A/B Tests you can upload/share with the testing community? Would be great to see some results/data if you have some time to do that.

[drip marketing]

@Rex Dixon you can find a lot of examples of A/B Testing with results and data by searching through Google. It's proven itself effective.

[Natalie Myers]

@Rex Dixon

We have 64 A/B tests and Multivariate tests (case studies) with creative samples and results at WhichTestWon.com.

Natalie Myers
Senior Reporter
WhichTestWon.com

[Kaitlynn]

I wonder whether it is legal to http://www.torrentbasket.com/filles-entre-elles download films by means of torrent files

[Database Marketing]

There was a start-up launched in Indy recently called ninjabutton (ninjabutton.com) that A/B split tests call-to-action buttons on a website and tells you which is more effective in real-time data. It's pretty awesome.

[Omniture Test and Target]

Here's another nice explanation, a little more simplified.

----
Sam Edwards
http://metamarketer.com/web-analytics-optimization/conversion-optimization/