You may be reading this because you’ve tried your hand at A/B testing, and, well, nothing really happened. Email A didn’t outperform Email B in any noteworthy way; the reverse wasn’t true either.
It can get frustrating, but don’t make the mistake of giving up. It’s a process, and it takes a while to warm up to, but, when implemented properly, A/B testing can produce an increase in your email open and click-through rates.
A/B testing is the process of sending one variation of your email to a subset of your subscribers and a different variation to another subset of subscribers and then looking at results to see which one performed better. This can vary depending on the brand or industry.
Results may vary from one brand or industry to another. For example, a short copy might get a better click-to-open rate in eCommerce but might perform poorly with infomercial products.
Instead of relying on generic research data, test with your audience and product to draw the correct conclusions.
Three steps to A/B testing success
Step 1: Select Enough Subscribers to Yield Meaningful Results
Critical Impact’s A/B Test tool will automatically create two randomly selected test lists. But what percentage of your list should you select for the A/B Test? You should make sure that you have enough subscribers in your A and B lists to yield meaningful results.
If you prefer hard numbers to “winging it,” then use a free “A/B Split Test Significance Calculator,” such as this one to determine how large your A and B pools need to be in order to produce results that are statistically significant.
Step 2: Test Time
Now that you’ve selected the size for an A and B list you can work with, it’s time to find out what truly makes your subscribers tick.
From the time-of-delivery to the all-important subject line, there are several factors you can tinker with when defining your A/B split test parameters. You can try different presentation styles for your call-to-action, different button colors, and different images to feature.
You can test longer emails vs. shorter ones, different writing styles, and different headlines. Wow! Where should you begin?
If you want the best chance at intelligible, actionable results, then you can’t just randomly pick a test parameter. You should instead form a hypothesis. “My subscribers are more likely to respond favorably to my emails if…” Your hypotheses will be more on-point if you understand how your subscribers interact with your emails. Observe real subscribers interacting with your email campaign—to help you form the best possible hypotheses.
Step 3: Select a Winner!
One of the great things about the Critical Impact email marketing platform is that it can easily be configured to take instant and automatic action after running A/B tests. As soon as the A/B test data shows a preference for email A or email B, the “winning” email is used for the remainder of the campaign.
These “A/B instant impact” campaigns work by allowing the user to add a “remainder” list to their A and B lists. The A and B campaigns are dispatched first, then the “winning” campaign is identified and sent to the “remainder” list.
You have two options to select the “winning” email: select either the one that has the higher open rates or higher click rates. You’ll want to select this option based on the type of test you are performing.
Select to send the winner to the higher open rates when running a test for subject line or from name, since these are the variables, people see before choosing to open the message. If you’re comparing two messages, it’s better to select the winner based on click rates. If you’re running a Send Time comparison, you could use either metric, depending on which is more important to you.
With this knowledge in hand, you should be ready to increase your click or open rates using Critical Impact’s A/B Test technology.
Set up the parameters
- Test length: Do not be too eager to get the results. Good results takes good data and that takes time. How long might depend on the response rate for the lowest number reaching your set threshold, or you could determine yourself based on previous data.
- Segmentation: Determine what part of your list will be participating. You should have a list that is large enough to be statistically significant. OUT ICON
- Testing split size. Once you know which segments will receive the test, you have to decide how to split the send. You could do a 50/50 split where half gets the control, and the other receives the variable version. Or, you can send control version A to 20% and test version B to another 20%, then wait and send the winner to the remaining 60%.
- Metrics: Determine what metrics you are trying to test the data for.
Test and test one thing at a time
This might sound obvious but testing too many things leaves you unable to determine how successful your test was for each variable. Initially go with the practices that should get you the best results, but when testing start with one by one to get the best result for that item. When you have a result, you are satisfied with, move on to testing the next variable.
This way you should be able to hone your message and format for a successful campaign. You can try to do many tests if your list is large enough, or you can do it over different campaigns though you will have different messaging in the copy and subject line, but mostly the format, tone and direction of content should be similar.
Let’s look at some items to test. In this case we are going to frame this in terms of a retail sale.
(Don’t forget though to test by time of day or day or week as well)
Subject line
- Question – use a question rather than a statement.
- Number – Use a number for a percent discount rather than just saying sale
- Emojis – see if an emoji such as a plant for a plant sale works
- Capitalization – Use sentence case than just the first word as capitalization
- 1st vs 2nd person – talk about You Save, not just Save
Email Copy
- Plain vs. HTML – some respond well to just text. Especially if it is a short message
- Personalization – include the first name, place etc in the body copy
- 1st vs 2nd person – again use the You Save, not just Save.
- Urgency – like the weekly flyers -let them know they only have a certain time to take advantage of a sale
- Copy Length – Try a few words or a paragraph or two to describe what the sale is about
- Placement of CTA – at the top, bottom, within or where? Test it
- Layout – sometimes it’s just all about the design for ease of reading
CTA
- Button vs text – you could have a graphic or a CSS design text button
- Color – what is the text or button call to action? Research is still out on the best colors to use, so refine for your market
- Copy – Buy now etc. See below
- Capitalization – use all caps such as BUY NOW rather than Buy Now
- Image – try a graphic vs the text for your CTA
CTA text ideas
Here are a few text ideas for your testing on text:
[A]
Buy Now
Check Out
Add to Cart
[B]
Let’s Shop!
I’m Ready to Go!
Let’s Check Out Now!
Common examples of additional copy include:
- Customer testimonials
- A longer list of all the features or benefits associated with your product or service
- Stories that explain, in detail, how your product or service helped people overcome a problem or achieve their goal
- Stats and data that reinforce the need for your product or service
- Frequently asked questions (with answers)
Conclusion
Of course, there is more to cold call emailing, but these points will help you with a good head start. We don’t want to write a whole sales course outline here, but hope this article helps.
Until next time, keep those emails rolling!
– Jim
Jim Gibbs, Senior Account Executive at Critical Impact
Jim Gibbs is Critical Impact’s Growth Channel team lead and has been selling and closing for a long time. Jim is known to be able to sell bottled water to fish.