The Insiders’ Guide to A/B Testing

It doesn’t matter if your list is large or small, one of the best ways to increase opens and clicks is to run A/B tests. Compare two different options at a time, to see what works best for your unique group of subscribers. Our A/B test tool shows how different subject lines, message content, from names, and delivery times affect subscriber engagement rates. Let’s take a closer look at these powerful testing options in this insiders’ guide to A/B testing.

How A/B Testing Works

The goal of an A/B test is to see which options are the most effective and generate the most opens or clicks. The A/B test sends two variations of a message campaign to a small percentage of your total recipients. Half of the test group receives Test A and the other half receives Test B. Whichever test produces the most opens or clicks will be named the winner! Then the A/B Testing tool will automatically send that winning version to the remainder of your list.

Critical Impact’s A/B Test tool makes this testing process easy to set up by automatically selecting the random test groups, calculating the winning send, and automatically sending the winning version to the rest of your subscribers.

What You Can Test:

1. Message: Compare two messages (change the wording, layout, images, etc)
2. Message and Subject Line: Send two different messages and subject lines
3. Subject Line: Compare two subject lines
4. From Name: Send the tests with different from addresses
5. Send Date/Time: Schedule the tests to be sent at two different times, to see how the time of day effects the open or click rates

Strategies to Run the Most Effective A/B Test

Here are a few strategic tips to help you get the most out of your A/B testing:

1. Define Your Goals

Define a goal for your upcoming email campaigns. Remember, goals should be specific, measurable, attainable, realistic, and timely. Here are some examples of email marketing goals that can be achieved using an A/B test:

• Increase open rate by 5% over the next 3 months by testing send times and subject lines.
• Find the best time of day to send the newsletter, that generates the most opens.
• Compare personalized vs. generic email messages to see which leads to more clicks.
• Find out if shorter or longer subject lines get more opens.
• Test various call to action buttons and text to find the combination that leads to the highest click-through rates.

2. Prioritize

Try to prioritize the A/B tests based on the impact you expect the A/B tests to have, while keeping in mind the energy it will take to create the two tests. Some A/B tests will take significantly less time and energy to implement. For example, writing two subject lines will take significantly less time than creating two different email layouts.

3. Make a Plan to Achieve Your Goal

Now that you have a goal, it’s time to make a plan! If your goal is to find out if shorter or longer subject lines get more opens, you should create a step by step plan like the following:

1. Make a list of all of the email campaigns going out next week.
2. Write two subject lines for each message: one under 6 words in length, and the other over 7 words in length.
3. Run A/B tests for each message sent out next week, comparing the subject lines and keeping all of the other variables constant (such as send time, message content, from address, etc).
4. After the week of testing, review your results. Determine if the shorter or longer subjects generated the highest open rate.

4. Learn and Grow

Learn from failures and successes to create awesome message campaigns. Take the time to look over your test results to see if any patterns emerge that can help you with future message sends. Sometimes your A/B tests will cause an increase in opens, and sometimes it won’t make any noticeable effect all. The key is to keep learning from each A/B test and use that information to create better message campaigns in the future.