11 Tested Ways to Speed Up your WooCommerce Store
How to Speed Up Your Shopify Store - 17 Practical Ideas
17 Proven Ways to Speed Up your BigCommerce Store
32 Holiday Marketing Ideas for eCommerce Stores (2023 Update)
The Best Quiz Funnel Examples for your eCommerce Store
Top 20 eCommerce Thank You Email Examples That Convert
How To Get More Sales on Shopify: 31 Proven Hacks
11 Proven Ways To Reduce Shopify Bounce Rate
Add to Cart Button: 14 Ways To Get People To Click Them
How to Increase Add-to-Cart Conversion Rate: 22 Brilliant Ideas
Build a High-Converting Squeeze Page: 13 Conversion Tips (+ Great Examples)
eCommerce Referral Email: 10 Examples That Really Work
Re-engagement Email Examples That (Actually) Win Back Subscribers
13 Proven Ways to Reduce Email Unsubscribe Rate (eCommerce)
How to Use Targeted Email Marketing to Drive More Sales (eCommerce)
25 Ways to Grow Your Email Click Through Rate Quickly (eCommerce)
How often should eCommerce stores send marketing emails?
How To Improve Email Marketing "Click-to-Open" Rate in eCommerce
28 No-BS Ways To Get More Email Subscribers in eCommerce
What is a Good Open Rate for eCommerce Emails?
eCommerce Email Design: 25 Beautiful Examples (& Why They Drive Sales)
Getting Traffic But No Sales? 21 Reasons Why (+ How To Solve)
11 High-Converting Win Back Email Examples (+ Templates)
23 Thanksgiving Email Examples For 2023 (eCommerce)
The Right Way to Calculate Email Marketing ROI in eCommerce
13 Solid Ways To Improve Marketing Email Delivery Rate (eCommerce)
4 eCommerce Brands Nailing Augmented Reality (And How You Can Do It Too)
Deliver An Amazing Post Purchase Experience: 17 Ideas (& Examples)
50+ eCommerce Email Marketing Statistics (2023 Data)
eCommerce Newsletter: 20 Ways To Stand Out And Actually Drive Sales
52 Ways eCommerce Businesses Can Use ChatGPT
eCommerce Inbound Marketing: 17 Ideas That Still Work
Shopify Upselling Ideas for Product page, Cart, Checkout
How to Market to Gen Alpha
5 Powerful Ways to Drive Conversions via Microsites (eCommerce)
Why Are Shoppers Dropping Off My Checkout Flow?
eCommerce Gift Card Marketing: 8 Creative Ways to Drive Max ROI
153 A/B Testing Ideas for eCommerce (Homepage, PDP, Cart, Checkout)
Optimize your Shopify Store for Mobile: Proven ideas + Examples
[2023] Cart Abandonment Rate Statistics
Product Page Statistics Every eCommerce Pro Should Know (2023 Update)
eCommerce Splash Page: 17 Attention-Grabbing Examples
Top Book Recommendations for eCommerce Business Owners
How to Increase Average Session Time on Your Shopify Store
The Right Way to Calculate eCommerce Conversion Rate (& How to Improve it)
11 Ways to Create a Killer "Coming Soon" Page (With Brilliant Examples)
6 Ways to Make Your Products Look Exclusive (& Real-world Examples)
Mobile Commerce: Examples, Trends, Best Practices (For 2023)
eCommerce Tiered Discount: 10 Awesome Examples You'd Want To Copy
eCommerce Affiliate Marketing: 7 Common Mistakes to Avoid (+ Things that Actually Work)
Boost Conversions this Father's Day: 15 Brilliant Tips (2023 update)
21 High-Converting Mobile Landing Page Examples to Inspire Yours
Flash Sales Guide: 10 Proven Ideas & Amazing Examples
Labor Day Marketing: 15 Amazing Ideas and Examples (2023)
Juneteenth—Marketing Handbook for eCommerce
Memorial Day Marketing: 10 Fresh Ideas and Examples (2023)
Next-Level 4th of July Marketing Ideas (+ Examples)
How to avoid "No results page" on your eCommerce store
14 Common CRO Mistakes eCommerce Brands Make
eCommerce BNPL: Who should DO it—Who should AVOID
16 Timeless Lead Generation Strategies for eCommerce (2023 Update)
eCommerce Subscriptions: 15 Amazing Examples (+ Ways to Increase Subscription Sales)
Cart Abandonment Pop-Up: 14 Amazing Examples (That Actually Work)
10 eCommerce brands winning at Social Commerce (+ Lessons we can learn from them)
13 eCommerce Brands that use AI like Champs
11 Next-Level Reciprocity Marketing Examples (eCommerce)
14 Ways Online Jewelry Stores Can Boost Conversions
Selling Luxury Products Online: 7 Unique Strategies to Boost Conversions
eCommerce conversion rate by industry (2023 update)
Prime Day Sale Marketing Ideas (Even if you are NOT Selling on Amazon)
Mother's Day Emails: 20 Amazing Examples (eCommerce)
18 brilliant ways to use microcopy to boost eCommerce conversions
Does infinite scrolling kill conversions? Here's what we found
BigCommerce Product Page: 46 Ways to Prevent Drop-offs
30 Amazing eCommerce Email Templates (from 6 industries)
Our Favorite Hero Image Examples in eCommerce (2023)
Improve BigCommerce conversion rate: 20 proven ideas
Promote your online store in the USA - 15 unique ideas
17 Underrated Easter and Spring Marketing Ideas (eCommerce)
International Women’s Day - 22 Marketing Ideas for your eCommerce store
100 Conversion Rate Hacks for eCommerce Stores
Top eCommerce Events in India (2023)
32 Amazing Valentine’s Day Email Examples (eCommerce)
eCommerce migration: 9 challenges for online stores
10 Boxing Day Marketing Ideas (+ Examples) for eCommerce
2023 eCommerce Calendar (& Expert Marketing Advice for Businesses)
35 Amazing Countdown Timer Examples (eCommerce)
23 Christmas Email Examples & Templates for eCommerce (2023)
11 Guaranteed Ways To Boost Revenue per Visitor (eCommerce)
3 eCommerce SEO mistakes to avoid and get more conversions
Prevent Shopify Cart and Checkout Abandonment: 24 Tested Ideas
Top 20 Product Page examples from the UK (eCommerce)
11 Ways to Improve Inventory Turnover Ratio in eCommerce
Top 20 Product Page examples from Australia (eCommerce)
How to go Omnichannel on a Budget (Strategies + Case Studies)
eCommerce copywriting: 23 inspiring examples from the US
6 eCommerce Conversion Rate Case Studies from the US
Surviving a Downturn: 26 Ideas for eCommerce Stores (Inspired from 3 Global Brands)
Product Detail Page: High-converting Templates (eCommerce)
22 Ways to Use Visual Cues to Drive More Conversions in eCommerce
No result fofund
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Email Marketing

Email A/B Testing: Elements to Test + Mistakes to Avoid

Stagnant email conversions? Here's an A/B test planning guide to improve your eCommerce store sales and achieve consistent wins.

Email A/B Testing: Elements to Test + Mistakes to Avoid

An email marketing strategy without A/B testing is like steering a boat without purpose. A/B tests not only offer you direction but also improve your subscribers’ experience. 

Email A/B Testing: Elements to Test + Mistakes to Avoid (eCommerce)

Email generates $42 for every $1 spent making it one of the most effective options for eCommerce brands. 

But not every email will give you consistent wins. This is where we come in. We’ve prepared a comprehensive A/B testing guide that will take your email marketing game to the next level. 

Take a look at what elements contribute to different email conversions rates.

What Components to A/B Test in Email Marketing?

Generally speaking, the more subscribers that open your emails, the higher your click-through and purchase rates will be. So focus on maximizing open rates by testing the right components. 

1) From Name

The sender name is probably the first thing a recipient looks at. 

Some eCommerce stores use first names to attach a human factor. For instance, seeing ‘Leo from Etsy’ in my inbox has established brand recall. Nowadays, if I view my inbox and see from ‘Leo’, I know the email is from Etsy.    

However, it’s not necessary for all eCommerce stores. You can experiment with just the brand name as well. For example, Target simply states the sender as ‘Target’. Perhaps, they have achieved better open rates through their brand name.   

Elements to A/B Test:

  • Does the first name vs full name add value?
  • Does a person’s name vs brand name make an impact? 
  • Do different email addresses affect open rates?

ConvertCart team recommends:

a) For transactional emails, such as order confirmation emails, use “Orders from Brand Name”. For newsletters, “Leo from Brand Name”. For ad campaigns, send from the “Brand Name”.

b) If you are using a person’s name, then opt for a simple and friendly name. Usually, it’s best to state only a first name with the brand name.    

c) Match the "From" address to your "From" name. For newsletters, send from a personal email address such as ‘leo@brandname.com’. For other emails such as support or updates, then ‘support@brandname.com’ works well. 

💡Gmail truncates addresses at 20 characters, while Yahoo will vary on the size of a browser, showing as few as 14 characters.

Metric to measure: Open rate

Bonus Read - Shopify Email Marketing Guide

2) Subject Lines

Email A/B Testing: Elements to Test + Mistakes to Avoid

Subject lines often act as the elevator pitch to your email. In fact, 64% of customers decide to open emails based on subject lines. The above data shows how adding a product name into the subject line helped an eCommerce store increase the open rate.  

Elements to A/B Test:

  • Does the length of the subject line matter? Test short vs long copy. 
  • Does including the subscriber's name make a difference? 
  • Does mentioning offers in the subject line matter?
  • Do emojis in the subject line increase open rates?
  • Do brackets like [Sale!] or [Special Offer] increase open rates?
  • Will open rates increase if certain words are CAPITALIZED? 
  • Does using the recipient’s first name increase open rates?
  • Do question-based subject lines increase open rates?
  • Do numbers or statistics create more impact?
  • How do ambiguous vs specific subject lines work?
  • Does personalization in subject lines work?
  • Does a sense of urgency increase open rates?

ConvertCart team recommends:

a) Avoid misleading subject lines. It only creates mistrust and customers might think twice before opening your next email. 

b) Based on your email type, you can create unique subject lines. For transactional and ad email campaigns, try to be creative. However, while sending emails for services and updates, it’s better to be straightforward.     

💡The ideal length of a subject line should be around 7 words and 41 characters. Also, try to limit punctuation marks.  

Metric to measure: Open rate 

3) Preheader

The preheader is the third thing a recipient reads after the sender and subject line. It’s a contributing factor to open rates. With the subject line, the preheader acts as the trailer to your movie (email, in this case). 

Most email service providers pick up the starting text of an email body as the preheader if you don’t add any custom text. While this works if your email body gets straight to the point. However, if the text is irrelevant, it might end up confusing the recipient. 

Elements to A/B Test:

  • What kind of text makes the most impact?
  • Does length matter?
  • What works better, ambiguous vs specific?
  • Does personalization increase open rates?
  • Do emojis in preheaders contribute to open rates?

ConvertCart team recommends:

a) Make sure it connects with the subject line. It sets the tone that carries throughout the email.     

b) There’s no set length for preheader text. However, email service providers offer compact messaging within 30-40 characters due to screen size, especially on mobile devices. 

💡“Hey Alexa, read my new emails”. Amazon Alexa can read aloud emails and preheaders will offer context and not have Alexa say “Click here to read more”.     

Metric to measure: Open rate

4) Time of Day and Week

Email A/B Testing: Elements to Test + Mistakes to Avoid

The above data shows some industry benchmarks on optimal times to send emails. While these are a great place to start, every eCommerce sector is different and only A/B testing can help you to set timing that converts more.   

Elements to A/B Test:

  • If emails triggered by specific behavior increase click-to-open rates?
  • Does timing around the trigger event make a difference? 
  • What frequency works - daily, weekly, monthly or quarterly?  

ConvertCart team recommends:

Time of Day and Week - Email A/B Testing: Elements to Test + Mistakes to Avoid

Choose an email marketing tool that analyzes user behavior from past emails to predict the optimal send time. If you send emails when your customers are most active, there is a higher probability of better open rates. 

Metric to measure: Open rate, click-to-open rate & CTR.

5) Personalization 

Email A/B Testing: Elements to Test + Mistakes to Avoid

Nowadays, everyone gets emails from brands. Customers know when eCommerce brands blast emails without context. You can segment customers according to user behavior and create personalized emails to strengthen shopping experiences. 

Elements to A/B Test:

  • What layout works better? 
  • Does short or long copy maximize conversions?
  • Does adding images make a difference? 
  • Do customer testimonials increase conversions?
  • How many features or benefits help customers to decide? 
  • Do statistics and data matter in behavior?
  • Do FAQ-based content increase click-through rates? 

ConvertCart team recommends:

Write to your target customers. To do that, you need to understand them. 

If you are an eCommerce store selling products that are usually used by teenagers and young adults, then you have to find a common ground. In fact, Gen X consumers account for 92% of all email users. Get their attention through pop-culture references, trending memes, and offer them a space for community building.  

💡 If you want to reach a customer’s ‘Primary’ inbox and skip the ‘Promotions’ tab, then write more text-based emails. Emails with HTML elements are analyzed and labeled by ESPs as promotional emails.   

Metric to measure: CTR

Bonus Read: Email Personalization Templates

6) CTAs

Email A/B Testing: Elements to Test + Mistakes to Avoid

Achieving sales is the end goal of eCommerce email marketing campaigns. That’s why the placement of the call-to-action matters.   

Elements to A/B Test:

  • Which text copy gets the most clicks?
  • How does placement change CTR?
  • Does the CTA button color make a difference?

ConvertCart team recommends:

Keep in mind - 

Use low commitment CTAs. Emails are invites, landing pages are parties
~ Matt Byrd

💡 Your customers click on the shiny CTA button. But then they leave your website without purchasing. Analyze landing pages to see if they are optimized for email messaging and create a smooth shopping experience.     

Metric to measure: CTR

7) Mobile-Friendly Emails

Email A/B Testing: Elements to Test + Mistakes to Avoid

Many people check their personal emails such as promotions from eCommerce brands on smartphones. If a large number of your email list view emails on their mobile and don’t convert, then it’s important to build mobile screen-friendly emails.   

Elements to A/B Test:

  • Which layout on mobile devices works best? 
  • Is the email content - subject line, colors, text, and CTAs formatted for the mobile screen size?  

ConvertCart team recommends:

Don’t forget the 3-second rule. Customers viewing emails on mobile screens skim rather than read. Put all your important information in the first fold or highlight it    

Metric to measure: Open rate, click-to-open rate & CTR

As an eCommerce brand, is everyone telling you to A/B test emails? And, perhaps, you have A/B tested emails and it didn’t work? Maybe you saw some results at first, but now, it’s not so exciting? Now that you know which elements to test, let’s understand what you might be missing out on in the A/B testing process.

How to Avoid Email A/B Testing Mistakes & Maximize Conversions 

How to Avoid Email A/B Testing Mistakes & Maximize Conversions 

1) Determine Statistical Significance

Most email marketing tools will promise you extraordinary results. 

For example, email Variant A is shown to small sample size from your customer list and Variant B to another sample size. Variant A results in 10.5% and Variant B 11.25% open rate respectively. Now, most email marketing tools will consider Variant B as the winning email. However, when you deploy the Variant B to the rest of the email list, you find it tough to get the same result.

So how can you be sure an email version will work or not?

Statistical significance helps quantify whether a result is likely due to chance or to some factor of interest
~ Tom Redman, Author 

To calculate statistical significance, you’ll need to first create a null hypothesis (default assumption of a statistical test) and an alternative hypothesis (this contradicts your null hypothesis). Next, determine the significance level (usually, the p-value is 0.05-5%) that will reject the null hypothesis.

The next steps include performing a power analysis to understand your sample size for the test. After the test is run, marketers can use an A/B testing tool to determine if the campaign result is statistically significant or not. 

2) Identify a Clear Hypothesis 

Before you start A/B testing, identify the hypothesis. A hypothesis prevents guesswork and offers a clear direction to your email campaign.     

For example, a hypothesis could be that “Subject lines with questions get more open rates”. This means you’ll be creating emails with two subject line variations and test if the result will support your hypothesis.       

Here are some important things to consider to build a compelling hypothesis:

  • How much data do you need to collect
  • Run the two variations at the same time
  • Mobile vs desktop
  • Time and frequency

3) Establish a Campaign Goal 

Once you have built a hypothesis, it’s time to decide victory metrics for your email campaigns. The ROI metric for every email campaign should be something tangible.     

Make sure to test different types of email campaigns. Often eCommerce store owners only test transactional or promotional emails. However, you should also test emails such as welcome, surveys, and company announcements.

Here are some goals to give you a head start:

  • Will mobile-friendly design increase open rates?
  • Do pop culture references make a difference?
  • Will giving offers reactivate stagnant subscribers?    

4) Segment Customers Based on Shopping Behavior

Email A/B Testing: Elements to Test + Mistakes to Avoid

Batch and blast emails are so outdated. Your customers are unique and receive many emails from eCommerce brands. A templatized version just gets lost in the crowd. Segmenting customers through their shopping behavior lets you build super-relevant emails. Here are some factors to consider when configuring segmentation:

  • Segment according to similar subscribers
  • Segment according to active subscribers
  • Segment customers to test and get statistically significant results 

Furthermore, revisit your email list every month and cull the stagnant subscribers to get better results.

5) Test One Thing at a Time 

It’s best to test one variable at a time. If you test too many variables, then it’ll be impossible to know exactly what worked and what didn’t. 

So if you are wondering how to increase open rates, then change subject lines and test the variants. However, if you want to know if different preheaders might affect open rates, then you should run two separate tests.


First, test two different subject lines with the same preheader.

Then, run a second test using the winning subject line with two different preheader texts. Once you’ve tested both variables on their own, you can combine the winning subject line and preheader for the best results.

6) Set a Duration for Testing

How long should an A/B test run? Many eCommerce owners we work with ask this question. There’s no ideal answer. Some experts recommend that running a test for a minimum of one to two weeks. While this may work for some, it might not for others. 

The duration of the tests depends on many factors:

  • size of subscriber base 
  • high waiting time
  • delivery time
  • number of devices  

7) Plan a Continuous A/B Test 

A/B testing isn’t really about being “done”.
So, you didn’t get the results you wanted. This doesn’t mean you should stop testing. 

The same goes if you did get positive results. After rolling out a successful email campaign, you might keep using the same elements to replicate the success. However, after a while, that might not work too.     

You can’t get stagnant with A/B testing. Once you have your results, positive or otherwise, plan your next test. Make sure to test:

  • Different elements with the same subscriber base 
  • Same elements with a different subscriber base, or   
  • Mix-match of previous and different elements with the same subscriber base. 

Improve Your A/B Testing Strategy Today

This guide covers everything to help you design an A/B testing strategy that offers consistent wins. And if it still sounds overwhelming, connect with us to get your email marketing A/B testing on the right path. 

Conversion rate optimization
Thank you.
We'll be in your inbox soon 🙂
Oops! Something went wrong while submitting the form.
X
Conversion rate optimization
x
[Free Guide]

50 Things you need to Learn from Amazon.com’s UX

DownloadGET A PRODUCT PAGE AUDIT