A/B testing in marketing is nothing new. As a matter of fact, the concept of A/B testing has been around for a long time: almost 100 years. The principles of A/B testing were first discovered and formalized in the 1920s by statistician and biologist Ronald Fisher. He used it in agricultural experiments trying to figure out, for example, if using more fertilizer or less would yield more of a particular crop.
From there, the concept continued to evolve, being used in clinical trials in the 1950s. In the 1960s and 1970s, marketers adapted the approach to determine, for example, which produced more sales in a direct marketing campaign, postcards or letters?
By the turn of the millennium, online marketers were using A/B testing for everything from landing pages to website design. Politicians use it to test which landing page will convert users into donors, and companies use it to see which ad is better at getting people to buy a product.
Which brings us to today, and a discussion of why it’s important to use A/B testing in your email marketing campaigns.
A/B Testing: What exactly is it?
A/B Testing is pretty simple: you have a variable, and you want to see which value (A or B) works better at producing your desired result. The variable can be anything from “which subject line will cause people to open this email?” to “which featured image will cause more readers to click through to my website?”
How does A/B Testing work?
Once your variables are chosen, you set a time limit and send that email to a portion of your audience. So let’s say you have an email list with 10,000 email addresses, and you want to see which subject line leads more people to open your email. You choose a portion of that list, say half (5000). Of these 5000 email addresses, 2500 (half of the half, if you will) will receive an email with Subject Line A, while the other 2500 will receive an email with Subject Line B. You determine an amount of time for the test to run, let’s say 4 hours. After 4 hours, whichever subject line produces the most open emails is then sent to the remaining 5000 email addresses. What you’ve done is optimized which email is likely to be opened based on the subject line and sent that the biggest part of your audience.
It’s pretty simple, and there are tools out there to help, like MailChimp and Constant Contact.
Why don’t we take a look at a sample?
What it looks like in practice
In this example, I’m setting up an email newsletter for a client, and I want to test which subject line will lead to more email opens, using MailChimp. First, after creating my campaign and selecting “email,” I click A/B Test:
After naming my campaign I click “Begin.” Then it’s time to choose which elements I want to test. In this case, I choose Subject Line. As you can see, I’m going to test 2 subject lines on 50% of my email list for a period of 4 hours. Whichever subject line wins will be sent to the remaining 50% of my list.
Finally, I type the subject lines. Half of my test segment (25% of my total email list) will receive subject line A, while the other half will receive subject line B. The body of the email, in this case, is the same for both emails.
Once the test is done, MailChimp provides me with the results. Here’s an example of a campaign run for a client:
Have you used A/B testing in any of your email marketing campaigns? How did it go?
Further Reading:
A Refresher on A/B Testing – Harvard Business Review
The Importance of A/B Testing – Paul Olyslager
The A/B Test: Inside the Technology That’s Changing the Rules of Business – Wired
