As marketers and product managers, we often have to make decisions without data. We have to trust our gut, try something new and…sometimes fail.
E-mail pop-ups are one such case in point where it’s difficult to find guidance. The available data about their performance is scarce. Case studies are often promotional and look biased. And to my knowledge, it’s difficult to find the results of A/B tests involving overlays and modals online.
That’s why I felt lucky to have access to the pop-up stats of 50 top e-commerce retailers. This gave me the chance to review real-life data and learn the secrets of effective pop-ups.
In this article, I’ll share our findings.
Get ready for a whole lot of stats and A/B tests’ results!
Offers Matter (a Lot)
We mentioned stats. So let’s start with metrics about the impact of offers on the performance of opt-in pop-ups.
How Much Do Offers Matter?
Here are the results for our sample.
The upper row aggregates the data of pop-ups that included an offer. By offer, we mean everything that might convince the visitor to subscribe (from a coupon, to a chance to win something, to an ebook, etc.).
The second row aggregates data from campaigns that did not include an offer. Their only pitch was a request to join an email list, subscribe to the website’s newsletter or receive exclusive offers.
When looking at these results, it’s no wonder why a majority of the retailers in our sample (42 out of 50 retailers) included an offer in their pop-up. The retailers that include an offer have a conversion rate that’s almost 2 times higher than those who don’t.
Do Offers Always Matter?
If you’re still doubting these stats, here’s a real-life illustration at the website level.
This skateboard shoe retailer alternates between a pop-up inviting her visitors to join its email list and a pop-up offering a chance to win a free pair of shoes.
When the sweepstakes are up, their pop-up can convert up to 23{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} of their visitors into email subscribers (!!). When the offer is removed, their conversion rate goes down to less than 2{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760}.
Let’s review one last example.
What Kind of Offers Work?
Offers do work. But what kind of offers? This A/B test gives us an additional hint.
This client tested two versions of his campaign. The first version mentions a $75 coupon. It had a conversion rate of 7.6{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760}.
This second version is more vague. It mentions “offers” without detailing what’s behind this wording. It had a conversion rate of 5.8{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} (a 31{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} gap with version 1).
So not only are potential subscribers looking for offers – it looks like they prefer to know exactly what the offer is. The more specific you can be about your offer, the better.
Timing Is a Double-Edged Sword
When setting up overlays on a website, one of the most difficult questions to answer right off the bat is: When should I display the pop-up?
Why Displaying a Pop-Up on Landing Can Be a Good Idea
Last year, we reviewed 300 ecommerce websites. 76{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} of the stores using pop-ups were displaying them as soon as the visitors landed on the website.
We were quite shocked by this statistic.
Indeed, as a visitor or a CRO specialist, it could sound weird. Why bombard your visitors with a modal right away? It doesn’t give them a chance to learn about your brand, services, etc.
Well, stats reveal that this strategy is not as dumb as it looks.
Here are the results of an AB test run by a high-end audio system retailer (I took this one as an example, but we’ve reproduced these results more than once):
As you can see:
- The first version where the pop-up is displayed right away has a lower conversion rate.
- But, it ended up collecting 173{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} more emails than the version displayed after 2 pages.
Why? Because the longer you wait, the fewer people will see your campaign. And even if the subscription rate is higher, it doesn’t compensate the smaller size of your sample.
The same applies if you choose to add a delay before your pop-up:
Here our client, an adult product retailer, tested a 10-second and a 20-second delay.
Again, the conversion of the version that’s displayed later is slightly better. But it ends up collecting -9{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} fewer emails.
Should All Pop-Ups Be Displayed Right Away?
Does this mean that the best email pop-ups are displayed as soon as a user opens a web page?
Hell, no!
Having a higher conversion rate means that you’re annoying a smaller percentage of your users. In the first AB test we mentioned, the difference between version 1 and version 2 is that only 92{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} of the visitors who saw the second version chose to close it, while more than 96{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} of the users who saw the second version chose to dismiss it. Put differently, the second version annoyed two times fewer users.
We also assume that pop-ups displayed on landing work better if the user already knows the brand or has visited the website before. Since she already knows what she can find on the website, she’s more eager to share her email right away if it can get her a good deal than if she didn’t know what she would find on the store. Unfortunately, we’ve never had a chance to test this hypothesis so far.
All in all, timing is a double-edged sword. If you want to focus on the number of emails you can collect, then it’s best to display your pop-up sooner. If you want to preserve the UX, it’s better to wait. Personally, I lean towards the second option.
Creating Mobile-Friendly Pop-Ups Won’t Help
Google Guidelines for Mobile Pop-Ups
In August 2016, Google declared war on “instrusive interstitials.”
Webmasters had no choice but to adapt their campaigns to the new policy. It meant either creating a mobile campaign that displays well on a desktop or separating campaigns altogether.
We thought this change would benefit all sides: from visitors who would be happier about their navigation experience to marketers who would collect more leads with improved forms.
It turns out, things didn’t really go that way.
The Impact of the Guidelines on Conversion Rate
The first takeaway of our study is that a significant proportion of retailers haven’t adapted their strategy to these new guidelines. 17 out of the 50 retailers that we reviewed are still displaying the exact same non-mobile-friendly pop-ups on mobiles and desktops.
And it looks like it doesn’t impact their pop-ups’ conversion rate. These retailers have an average conversion rate of 4.41{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760}, 27{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} higher than the stats of their counterparts who chose to comply.
What Lesson Can We Draw from These Stats?
Sometimes, it’s worth sacrificing conversions if it means that your users have a better experience on your website.
And in this case, I would not recommend risking your SEO traffic for the sake of collecting a few more emails.
Don’t Forget to Optimize Your Copy
Until now, we’ve reviewed pop-ups’ fundamental elements: the offer, the targeting, and the adaptation to mobile devices.
To be honest, I’m the kind of guy who likes to focus on the main elements of a marketing strategy. I’ve always felt uncomfortable when someone brought up the idea of testing the impact of using a different picture to illustrate a home page, checking the impact of font types or A/B testing a wording.
In other words, I usually don’t focus too much on the copy of my pop-up campaigns.
But again, it turns out the copy is worth testing. In this section, I’m going to share an A/B test that changed my mind about the importance of wording for pop-ups.
This watch store tested two headlines. As you can see, the two mention exactly the same prize: 20 watches, worth $2000 in total.
The first version insists on the total amount.
While the second focuses on the number of watches you could win.
All things being equal, the first headline (13.40{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} conversion rate) achieved 2x the subscription rate of the second variant (6.70{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760}).
The results of this test show that:
1) The words you choose to describe your offer or content do matter. And they can have a huge impact.
2) In general, all the details that make a pop-up matter are worth testing: from the targeting, to the offer, to the design and the wording that you use.
(Although to be honest, I’m still not convinced that the color of a CTA alone can have a huge impact).
Before You Invest in Pop-Ups
These pop-up stats taught us a few things.
First, experiments can have a huge impact. Most of the tests mentioned had a dramatic impact on conversions and UX.
Secondly, you have to find the delicate balance between UX and immediate conversions to achieve your best opt-in pop-up results. A smarter pop-up doesn’t always collect more emails. But it will certainly annoy fewer people.
Originally published March 13, 2019 – Updated January 18, 2022
Mobile reading?
