Email A/B Testing: Elements to Test + Mistakes to Avoid

An email marketing strategy without A/B testing is like steering a boat without purpose. A/B tests not only offer you direction but also improve your subscribers’ experience. 

Email generates $42 for every $1 spent making it one of the most effective options for eCommerce brands. 

But not every email will give you consistent wins. This is where we come in. We’ve prepared a comprehensive A/B testing guide that will take your email marketing game to the next level. 

Take a look at what elements contribute to different email conversions rates.

What Components to A/B Test in Email Marketing?

Generally speaking, the more subscribers that open your emails, the higher your click-through and purchase rates will be. So focus on maximizing open rates by testing the right components. 

1) From Name

The sender name is probably the first thing a recipient looks at. 

Some eCommerce stores use first names to attach a human factor. For instance, seeing ‘Leo from Etsy’ in my inbox has established brand recall. Nowadays, if I view my inbox and see from ‘Leo’, I know the email is from Etsy.    

However, it’s not necessary for all eCommerce stores. You can experiment with just the brand name as well. For example, Target simply states the sender as ‘Target’. Perhaps, they have achieved better open rates through their brand name.   

Elements to A/B Test:

• Does the first name vs full name add value?
• Does a person’s name vs brand name make an impact? 
• Do different email addresses affect open rates?
  

ConvertCart team recommends:

a) For transactional emails, such as order confirmation emails, use “Orders from Brand Name”. For newsletters, “Leo from Brand Name”. For ad campaigns, send from the “Brand Name”.

b) If you are using a person’s name, then opt for a simple and friendly name. Usually, it’s best to state only a first name with the brand name.    

c) Match the "From" address to your "From" name. For newsletters, send from a personal email address such as ‘leo@brandname.com’. For other emails such as support or updates, then ‘support@brandname.com’ works well. 

💡Gmail truncates addresses at 20 characters, while Yahoo will vary on the size of a browser, showing as few as 14 characters.

Metric to measure: Open rate‍

Bonus Read - Shopify Email Marketing Guide

2) Subject Lines

Subject lines often act as the elevator pitch to your email. In fact, 64% of customers decide to open emails based on subject lines. The above data shows how adding a product name into the subject line helped an eCommerce store increase the open rate.  

Elements to A/B Test:

• Does the length of the subject line matter? Test short vs long copy. 
• Does including the subscriber's name make a difference? 
• Does mentioning offers in the subject line matter?
• Do emojis in the subject line increase open rates?
• Do brackets like [Sale!] or [Special Offer] increase open rates?
• Will open rates increase if certain words are CAPITALIZED? 
• Does using the recipient’s first name increase open rates?
• Do question-based subject lines increase open rates?
• Do numbers or statistics create more impact?
• How do ambiguous vs specific subject lines work?
• Does personalization in subject lines work?
• Does a sense of urgency increase open rates?
  

ConvertCart team recommends:

a) Avoid misleading subject lines. It only creates mistrust and customers might think twice before opening your next email. 

b) Based on your email type, you can create unique subject lines. For transactional and ad email campaigns, try to be creative. However, while sending emails for services and updates, it’s better to be straightforward.     

💡The ideal length of a subject line should be around 7 words and 41 characters. Also, try to limit punctuation marks.  

Metric to measure: Open rate 

3) Preheader

The preheader is the third thing a recipient reads after the sender and subject line. It’s a contributing factor to open rates. With the subject line, the preheader acts as the trailer to your movie (email, in this case). 

Most email service providers pick up the starting text of an email body as the preheader if you don’t add any custom text. While this works if your email body gets straight to the point. However, if the text is irrelevant, it might end up confusing the recipient. 

Elements to A/B Test:

• What kind of text makes the most impact?
• Does length matter?
• What works better, ambiguous vs specific?
• Does personalization increase open rates?
• Do emojis in preheaders contribute to open rates?
  

ConvertCart team recommends:

a) Make sure it connects with the subject line. It sets the tone that carries throughout the email.     

b) There’s no set length for preheader text. However, email service providers offer compact messaging within 30-40 characters due to screen size, especially on mobile devices. 

💡“Hey Alexa, read my new emails”. Amazon Alexa can read aloud emails and preheaders will offer context and not have Alexa say “Click here to read more”.     

Metric to measure: Open rate

4) Time of Day and Week

The above data shows some industry benchmarks on optimal times to send emails. While these are a great place to start, every eCommerce sector is different and only A/B testing can help you to set timing that converts more.   

Elements to A/B Test:

• If emails triggered by specific behavior increase click-to-open rates?
• Does timing around the trigger event make a difference? 
• What frequency works - daily, weekly, monthly or quarterly?  
  

ConvertCart team recommends:

Choose an email marketing tool that analyzes user behavior from past emails to predict the optimal send time. If you send emails when your customers are most active, there is a higher probability of better open rates. 

Metric to measure: Open rate, click-to-open rate & CTR.

5) Personalization 

Nowadays, everyone gets emails from brands. Customers know when eCommerce brands blast emails without context. You can segment customers according to user behavior and create personalized emails to strengthen shopping experiences. 

Elements to A/B Test:

• What layout works better? 
• Does short or long copy maximize conversions?
• Does adding images make a difference? 
• Do customer testimonials increase conversions?
• How many features or benefits help customers to decide? 
• Do statistics and data matter in behavior?
• Do FAQ-based content increase click-through rates? 
  

ConvertCart team recommends:

Write to your target customers. To do that, you need to understand them. 

If you are an eCommerce store selling products that are usually used by teenagers and young adults, then you have to find a common ground. In fact, Gen X consumers account for 92% of all email users. Get their attention through pop-culture references, trending memes, and offer them a space for community building.  

💡 If you want to reach a customer’s ‘Primary’ inbox and skip the ‘Promotions’ tab, then write more text-based emails. Emails with HTML elements are analyzed and labeled by ESPs as promotional emails.   

Metric to measure: CTR

Bonus Read: Email Personalization Templates

6) CTAs

Achieving sales is the end goal of eCommerce email marketing campaigns. That’s why the placement of the call-to-action matters.   

Elements to A/B Test:

• Which text copy gets the most clicks?
• How does placement change CTR?
• Does the CTA button color make a difference?
  

ConvertCart team recommends:

Keep in mind - 

Use low commitment CTAs. Emails are invites, landing pages are parties

~ Matt Byrd

💡 Your customers click on the shiny CTA button. But then they leave your website without purchasing. Analyze landing pages to see if they are optimized for email messaging and create a smooth shopping experience.     

Metric to measure: CTR

7) Mobile-Friendly Emails

Many people check their personal emails such as promotions from eCommerce brands on smartphones. If a large number of your email list view emails on their mobile and don’t convert, then it’s important to build mobile screen-friendly emails.   

Elements to A/B Test:

• Which layout on mobile devices works best? 
• Is the email content - subject line, colors, text, and CTAs formatted for the mobile screen size?  
  

ConvertCart team recommends:

Don’t forget the 3-second rule. Customers viewing emails on mobile screens skim rather than read. Put all your important information in the first fold or highlight it    

Metric to measure: Open rate, click-to-open rate & CTR

As an eCommerce brand, is everyone telling you to A/B test emails? And, perhaps, you have A/B tested emails and it didn’t work? Maybe you saw some results at first, but now, it’s not so exciting? Now that you know which elements to test, let’s understand what you might be missing out on in the A/B testing process.

How to Avoid Email A/B Testing Mistakes & Maximize Conversions 

1) Determine Statistical Significance

Most email marketing tools will promise you extraordinary results. 

For example, email Variant A is shown to small sample size from your customer list and Variant B to another sample size. Variant A results in 10.5% and Variant B 11.25% open rate respectively. Now, most email marketing tools will consider Variant B as the winning email. However, when you deploy the Variant B to the rest of the email list, you find it tough to get the same result.

So how can you be sure an email version will work or not?

Statistical significance helps quantify whether a result is likely due to chance or to some factor of interest

~ Tom Redman, Author 

To calculate statistical significance, you’ll need to first create a null hypothesis (default assumption of a statistical test) and an alternative hypothesis (this contradicts your null hypothesis). Next, determine the significance level (usually, the p-value is 0.05-5%) that will reject the null hypothesis.

The next steps include performing a power analysis to understand your sample size for the test. After the test is run, marketers can use an A/B testing tool to determine if the campaign result is statistically significant or not. 

2) Identify a Clear Hypothesis 

Before you start A/B testing, identify the hypothesis. A hypothesis prevents guesswork and offers a clear direction to your email campaign.     

For example, a hypothesis could be that “Subject lines with questions get more open rates”. This means you’ll be creating emails with two subject line variations and test if the result will support your hypothesis.       

Here are some important things to consider to build a compelling hypothesis:

• How much data do you need to collect
• Run the two variations at the same time
• Mobile vs desktop
• Time and frequency

3) Establish a Campaign Goal 

Once you have built a hypothesis, it’s time to decide victory metrics for your email campaigns. The ROI metric for every email campaign should be something tangible.     

Make sure to test different types of email campaigns. Often eCommerce store owners only test transactional or promotional emails. However, you should also test emails such as welcome, surveys, and company announcements.

Here are some goals to give you a head start:

• Will mobile-friendly design increase open rates?
• Do pop culture references make a difference?
• Will giving offers reactivate stagnant subscribers?    

4) Segment Customers Based on Shopping Behavior

Batch and blast emails are so outdated. Your customers are unique and receive many emails from eCommerce brands. A templatized version just gets lost in the crowd. Segmenting customers through their shopping behavior lets you build super-relevant emails. Here are some factors to consider when configuring segmentation:

• Segment according to similar subscribers
• Segment according to active subscribers
• Segment customers to test and get statistically significant results 
  

Furthermore, revisit your email list every month and cull the stagnant subscribers to get better results.

5) Test One Thing at a Time 

It’s best to test one variable at a time. If you test too many variables, then it’ll be impossible to know exactly what worked and what didn’t. 

So if you are wondering how to increase open rates, then change subject lines and test the variants. However, if you want to know if different preheaders might affect open rates, then you should run two separate tests.

First, test two different subject lines with the same preheader.

Then, run a second test using the winning subject line with two different preheader texts. Once you’ve tested both variables on their own, you can combine the winning subject line and preheader for the best results.

6) Set a Duration for Testing

How long should an A/B test run? Many eCommerce owners we work with ask this question. There’s no ideal answer. Some experts recommend that running a test for a minimum of one to two weeks. While this may work for some, it might not for others. 

The duration of the tests depends on many factors:

• size of subscriber base 
• high waiting time
• delivery time
• number of devices  

7) Plan a Continuous A/B Test 

A/B testing isn’t really about being “done”.
So, you didn’t get the results you wanted. This doesn’t mean you should stop testing. 

The same goes if you did get positive results. After rolling out a successful email campaign, you might keep using the same elements to replicate the success. However, after a while, that might not work too.     

You can’t get stagnant with A/B testing. Once you have your results, positive or otherwise, plan your next test. Make sure to test:

• Different elements with the same subscriber base 
• Same elements with a different subscriber base, or   
• Mix-match of previous and different elements with the same subscriber base. 

Improve Your A/B Testing Strategy Today

This guide covers everything to help you design an A/B testing strategy that offers consistent wins. And if it still sounds overwhelming, connect with us to get your email marketing A/B testing on the right path.