Marketing Automation and Lead Management Blog

A/B Testing and Multivariate Testing of Marketing Campaigns

I get asked frequently if Market2Lead supports A/B testing of email campaigns and the answer is “yes, of course,” but sometimes I wonder if the people have thought through the process.

You can dissect the average email campaign down into 6 broad areas that can influence the outcome as shown in the graphic above. In my August 28, 2009 post on effective campaigns I discussed the value of a high quality list as the primary determinant of the effectiveness of a campaign, so we won’t repeat that here. Instead I want to discuss how one would best go about A/B testing campaigns for the 5 remaining items.

Select the A/B Testing Outcome To Test

In B2B with long sales cycles, and multiple touch points, it would be silly to A/B test email campaigns for closed won opportunities, but in B2C this might be the perfect answer. In B2B email campaigns you are probably more likely to constrain yourself to opens, clicks, and offer downloads. In B2C with short sales cycles, you might consider “added to basket”, or purchases as the outcome you are testing.

A/B Testing Requires Random List Segmentation

To be an effective A/B test, the other factors beyond the one you are testing must be held constant or, in the case of segmented lists of people, be made random. So the first question should be, how does one produce random lists:

1. Allow the marketing automation system to randomly pick people from your segmented list for either the A or B treatment. To the best of my knowledge no marketing automation vendor offers this today.
2. Manually produce two lists (A and B) using some random mechanism; you could use the odd numbered contact ID numbers for the A list for instance. Trying to create random lists by selecting letters found in names or emails is not as random as you might think!  But if the contact ID numbers are assigned randomly when new leads are created or imported, it is a great way to go.

A/B Testing with One Program and Two Lists?

Some folks believe that it is simpler to use one program with two random lists, and have the functionality built in to apply the appropriate A/B subject line, or A/B email body, or A/B offer, or A/B Landing Page, or A/B Form to the A or B target list. In the reporting, the system would then have to be smart enough to produce two different reports on clicks, opens, downloads etc based on the A/B variable that was chosen. I find this too complicated.

I much prefer to create the program for A, copy it, point it to list B, change my one variable (subject line, or email body, or offer, or landing page, or form) and launch both programs. I can do this in two minutes and they each have their own results, it is easy to interpret, and easy to copy these programs for future tests.

You Don’t Need Dedicated Functionality for A/B Testing

The bottom line is that you can do some very creative A/B testing or multivariate testing as long as you have a means to create random lists. Everything else is easily done by copying the program and changing your one variable, and the reporting will make much more sense to you and everybody else.

So, an exercise for the reader: How would you do A/B/C testing since 3 does not divide evenly into 10, so our trick of using odd versus even contact IDs won't help...

-Kevin

Are your marketing assets generating demand?

Key Performance Indicators (KPI) are measurements that indicate the health of the business. We use them in dashboards to give status at a glance. As a continuation of my series of posts on Key Performance Indicators for Marketing, lets talk about measuring the effectiveness of Marketing assets and offers.

Marketing Offer Types for Demand Generation and Lead Nurturing

We invest a lot of our budget in creating them, we use them as bait for lead generation, educational fodder for lead nurturing campaigns, and ongoing loyalty marketing to our installed base. The assets we invest in generally include:

• Website pages
• Whitepapers
• Analyst reports
• Flash demos
• Data sheets and brochures
• Case studies, success stories
• Press Releases (maybe)
• Webcasts and podcasts
• Free trial offer
• Surveys
• Newsletters
• Training guides
• Best practices documents
• Online demo

To accurately report on them you will want to tag them with an "Offer Type", similar to the list I describe above. Next we need to decide the dimensions that we want to measure, some obvious, some perhaps not so obvious.

Measurements for marketing offer performance

1. Usage in a period of time
2. Form abandonment/completion rate by asset (if they have a form)
3. Age
4. Time to next interaction by prospects who registered for this offer

Usage is a no brainer. You will want to trend this out, and of course as you move the asset around on your website and offer it through other media, it will go up and down in popularity.

Age is important because too many asset choices on your website or in your delivery channel, will result in folks not finding the great assets and perhaps not choosing any asset at all. If you subscribe to the theory that too much choice results in less decisions to choose anything, read this article on the paradox of choice that throws doubt on that theory. Bottom line for me is that I would rather have 10 great offers, than hide the 10 in the middle of 40 mediocre offers.

I also have a fear that some assets are innoculating prospects. They read them, and never come back to your website. Hence the 4th metric, how long after an asset is downloaded until they interact with you again. This metric could help you weed out the weak offers by something other than usage and age.

There are a few other considerations in Asset performance measurement:

• Separate out "offers downloaded results" by net new leads versus existing prospects in your database versus installed base customers.
• Separate out proactive out reach of offers (email offers) from the results of folks finding your offers for themselves either on your website or through a third party.
• Your offers should have unique IDs so that they can be reused in multiple different programs/campaigns, and you can run the report based on the offer ID. If you change the form, fit, or function of the offer, give it a new ID.
• Also consider tracking all offers regardless if they have a gated form or not. This is an asset report, not a gated asset report! But in your reports you may want to highlight the difference.

What is the best KPI for Marketing offers?

So we have defined the types of offers, we have defined 4 dimensions of measurement, what would a single KPI for assets look like? We want a KPI that will set alarm bells going if it trends in the wrong direction, and we want that change to be obvious soon after it happens, not months later. Occam's razor tells us we should reach for the obvious one: look at total asset downloads per week or per month, and watch the trend. If you have 100 assets, at $3K each, isn't it worth keeping an eye on this $300K to see if it is working for you or against you?

-Kevin

Measuring Marketing Effectiveness

I read the recent post in the blog of Morgan Stewart, Director of Research and Strategy at ExactTarget, (a Market2Lead partner and all around good guys!) They just released a fascinating report they created with Econsultancy, on "Marketing Budgets 2010: Effectiveness, Measurement & Allocation". Great stuff and worth looking at the snapshot of the slides on Morgan's blog.  One slide in particular caught my attention:

So in their survey, 79% said increased Sales was a key metric for measuring marketing effectiveness, followed by traffic to the website at 71%. Wow. Yes I know increased sales is a goal of marketing, but it is a measure of the effectiveness of the entire company. Every other department could be doing awful, marketing is knocking the ball out of the park and gets a bloody nose because sales are down! And website visits - a measure of marketing effectiveness? So the increase in visits to Bank of America's website because of their notoriety over paying large bonuses after the bailout means Marketing is doing a good job (I hope they got bonuses for that).   My point is that an effective metric has to demonstrate a strong causal relationship to the thing it supposed to measure. Sales revenue is a good measure of the effectiveness of the channel, and the product/service. Website visits are a good measure of the brand awareness (both good and bad). Neither are good measures of overall marketing effectiveness.

Some good measures of marketing effectiveness

So what are some good measures of marketing effectiveness. Remember effectiveness is defined as doing the right things, and efficiency is doing things right. You combine both to get greater productivity. Let's measure effectiveness based on where you might allocate your marketing budget:

net new lead generation: Effectiveness measure is cost per "Sales Accepted" marketing lead

Notice that I am less interested in cost per "new" lead because anyone can drag in a bunch of garbage leads at a low price. A "sales accepted" marketing lead means that Sales is telling you that it is acceptable quality.

Lead nurturing: Effectiveness measure is velocity of leads through the funnel, or number of new opportunities from marketing leads. You can also look at your on going response rates to campaigns, how many open, clicked downloaded, although you have to be careful not to confuse this with the asset performance - A/B testing!

Awareness: Effectiveness measure could be brand recognition, website visits, social media mentions, traditional media mentions, etc

Loyalty marketing: Customer retention, satisfaction as mentioned in the report

Asset performance: Web pages per visit, asset downloads, form completion rates, event attendance.

A single measure of Marketing effectiveness?

I don't believe there is a single good measure of marketing effectiveness. All the ones I mentioned measure specific areas of marketing. Marketing ROI you say? So everything marketing does fits into nice little programs that are time bound, produce revenue in the near future and can be measured easily...not! Marketing ROI is good for specific campaigns.  So, anyone out there have any ideas for a good single measure of marketing effectiveness?

-Kevin