3 Steps to Success with A/B Testing

January 26, 2017

Get your lab coats on. It’s time for some experimentation. You probably already know the basic premise of A/B testing. If not, here it is: A/B testing, also referred to as “split testing,” is when an email marketer creates two slightly different versions of the same message and sends them both off to separate test groups. A good A/B test will reveal usable data to help you determine what’s working in your marketing emails and what’s not.

You may be reading this because you’ve tried your hand at A/B testing, and, well, nothing really happened. Email A didn’t outperform Email B in any noteworthy way; the reverse wasn’t true either. And you’re left to wonder, “What was the point?” Right?

It can definitely get frustrating, but don’t make the mistake of giving up. It’s a process, and it takes a while to warm up to, but, when implemented properly, A/B testing can produce an increase in your email open and click-through rates.

Here’s a simple three step approach that will help you get great results from A/B testing:


Step 1: Select Enough Subscribers to Yield Meaningful Results

Critical Impact’s A/B Test tool will automatically create two randomly selected test lists. But what percentage of your list should you select for the A/B Test? You should make sure that you have enough subscribers in your A and B lists to yield meaningful results. If you prefer hard numbers to “winging it,” then use a free “A/B Split Test Significance Calculator,” such as this one to determine how large your A and B pools need to be in order to produce results that are statistically significant.


Step 2: Test Time

Now that you’ve selected the size for an A and B list you can work with, it’s time to find out what truly makes your subscribers tick.

From the time-of-delivery to the all-important subject line, there are several factors you can tinker with when defining your A/B split test parameters. You can try different presentation styles for your call-to-action, different button colors, and different images to feature. You can test longer emails vs. shorter ones, different writing styles, and different headlines. Wow! Where should you begin?

Here’s the rub. If you want the best chance at intelligible, actionable results, then you can’t just randomly pick a test parameter. You should instead form a hypothesis. “My subscribers are more likely to respond favorably to my emails if…” Your hypotheses will be more on-point if you understand how your subscribers interact with your emails. Observe real subscribers interacting with your email campaign—to help you form the best possible hypotheses.


Step 3: Select a Winner!

One of the great things about the Critical Impact email marketing platform is that it can easily be configured to take instant and automatic action after running A/B tests. As soon as the A/B test data shows a preference for email A or email B, the “winning” email is used for the remainder of the campaign.

These “A/B instant impact” campaigns work by allowing the user to add a “remainder” list to their A and B lists. The A and B campaigns are dispatched first, then the “winning” campaign is identified and sent to the “remainder” list.

You have two options to select the “winning” email: select either the one that has the higher open rates or higher click rates. You’ll want to select this option based on the type of test you are performing.

Select to send the winner to the higher open rates when running a test for subject line or from name, since these are the variables people see before choosing to open the message. If you’re comparing two messages, it’s better to select the winner based on click rates. If you’re running a Send Time comparison, you could use either metric, depending on which is more important to you.

With this knowledge in hand, you should be ready to increase your click or open rates using Critical Impact’s industry leading A/B Test technology.