Trigger
Warning:
We know there is a ton of debate raging around
statistical significance, the 95{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} mythical (or not?) mark, testing and traffic,
and more.
If you are an optimizer who feels strongly
about this stuff, you might get triggered. Well, the comments are open and we
can have a really enlightening conversation.
Let’s begin!
Testing
doesn’t work for low-traffic websites.
That’s for sure.
Because to derive statistically significant
insights, you need to show your experiment’s variants to (at least) thousands
of users.
In fact, at Convert Experiences, we often
recommend our users run A/B tests — which are the simplest CRO experiments —
only if they can send about 10K visitors to each variant.
Also, we suggest recording around a 1000
conversions to yield conclusive evidence of the superiority of the control or
variant.
This kind of explains why CRO isn’t a million customer kind of market as there are just a few 100k websites (or businesses) with the footfall to test with any success and the budgets for the pricey (rapidly going “enterprise”) optimization solutions!
Should Low-Traffic Sites Rule Out Testing?
Given that an average small business only gets
a few hundred visitors a day to its website, A/B test experiments don’t make
sense for them.
But:
That doesn’t mean a small business can’t run
experiments at all.
In fact, there are many experiments that even
a small business can run.
And there are many benefits to running them.
For example, if you own a small business,
running experiments will help you move
away from gut-based toward data-driven decision-making. It will force you
to look at data … and with in-your-face numbers, you’ll stop defaulting to *I think* strategies. This data-backed
approach alone will transform how you build and grow your business.
Running experiments will also teach you to learn by trying, rather than
accepting the status quo. It will inspire you to see if the principles-believed-to-work actually
work for you.
Finally, optimizing will prepare you for some “real” CRO when you have “enough” traffic as it helps develop the right mindset for testing — a prerequisite for effective CRO.
So start running whatever experiments you can over whatever marketing channels you use. Here are some of your options.
Testing with Email
Surprisingly, email marketing is one channel
that lets you experiment with different ideas, without needing tons of
subscribers (or “contacts”).
“Says
who?”
Says every email marketing software provider
right from MailChimp, ConstantContact, and Campaign Monitor to HubSpot (that
supports email marketing in its marketing automation software).
Let’s start with HubSpot.
So if you want to run an A/B test using HubSpot’s
email platform, all you’ll need are 1,000 contacts: 1,000 contacts is also our benchmark for running A/B tests on samples
of email sends. — HubSpot
Campaign Monitor also says that you can run A/B tests even with small emails lists. Here’s the math behind it:
Now, you
may be thinking, “My list size is only 500 subscribers, how do you expect me to
run a successful test?” My answer is that you’ll need to set your sights on a
larger MDE [Minimum Detectable Effect (MDE)]. When you increase the MDE, your
required sample size decreases. So instead of needing 592 subscribers per
variation to detect a 20{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} relative effect, you would only need 94 subscribers
per variation to detect a 50{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} relative effect. Note that the baseline
conversion rate also plays a part in your sample size – the lower your
conversion rate (in this case, open {c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760}), the more subscribers you’ll need to run
an accurate test. — Campaign Monitor
So if you’ve about 500 or 1000 subscribers,
you’re good to test, provided you have a responsive database that is interested
in what you have to say.
However, a few email marketing providers
recommend a slightly higher contact base.
For instance, email marketing solution
provider ConstantContact recommends a higher sample size at twice the minimum
size that HubSpot allows. It suggests that for meaningful A/B testing, the
sample size must be at least 1,000 contacts. So, if you’ve 2000 subscribers,
you should be good to test (with your A and B versions each getting 50{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} of the
total contacts as test traffic).
For
subject line testing, we recommend testing each line on a sample size of at
least 1,000 contacts if possible. 1,000 users for each version is typically a
good benchmark for any A/B test, but if you don’t have enough contacts or
visitors to hit those numbers, it’s still worth running the test. — ConstantContact
MailChimp also recommends somewhat larger
lists to run A/B tests. It suggests that you must have at least 5,000 contacts
to run tests. But because so many small business contact bases are much smaller
than that, it recommends testing ideas across the full list:
We recommend
you send combinations to at least 5,000 subscribed contacts to get the most
useful data from your test. If the size of your list or segment doesn’t allow
for 5,000 subscribers in each combination, consider testing your entire list
and use the campaign results to inform future campaign content decisions. — MailChimp
While it’s true that having a huge sample size
will help you reach statistical significance much sooner (and with much more
confidence), but you can start testing with your small list as well!
Here are 150+ email split testing ideas for
your email campaigns.
Also, here’s a quick guide on email A/B testing.
Social media is another channel that lets you
test your ideas, no matter how small your social following might be. Besides,
in a way, on social media, nothing actually limits your reach.
So for example, at Convert Experiences, even
we don’t have tens of thousands of followers, yet our awesome Social Seller
keeps experimenting with elements of our social media copy such as its
hashtags, images, and more.
And her tests bring some great wins for us. For instance, her experiment using a certain set of hashtags once won us a retweet from Google Analytics!
Now, that’s some social media exposure, isn’t
it?
You might say that it’s easy to dismiss quick
wins as (maybe) flukes BUT when you see a set of hashtags routinely earn more
exposure than the others you test it against, you’ve more than a test result …
what you have is a trend.
You can run a host of social media A/B tests to uncover such trends and boost your social media performance. We’ve published a great primer on social media A/B testing here. Read it to learn how you can start with social media A/B testing, the different social media A/B tests you can run, and the top social media A/B testing best practices that you must keep in mind.
A/B Testing with Paid Ads (PPC)
Just like email and social media marketing
channels, the PPC marketing channel also lets you experiment with your ad
copies and all the other elements of your ad campaigns.
Whether it’s Google Adwords or Facebook ads,
most paid networks support experiments so you can get the highest ROI.
But to get statistically significant results
from your PPC experiments, you need a few thousand impressions of each ad copy.
Google recommends testing ad groups that get at least 5000 impressions/week.
So how much do a few thousand ad impressions
cost?
It depends a lot on the platform and the
industry (among other factors), but there’s data to help you do the budgeting.
For example, AdStage analyzed over 110 million ad impressions (on Google Adwords) in
Q1 2018 and found that on an average, you need $116.91 to get a thousand
impressions (CPM) on Google Ads. Last year, it was about 13{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} cheaper.
Social media networks like Facebook also offer
accessible CPM options.
Data from Statista research states that the average CPM (cost-per-mille or cost per 1,000
impressions) for Facebook is USD 1.26. Other Facebook CPM
average cost estimates from AdExpresso
and WordStream fall in the $7-10 range.
The thing is that you can start testing ad copies even on a limited budget.
An (Almost) “Traffic-Proof” Way of Experimenting: Personalization
Personalization or personalizing a visitor’s
website experience based on what you know about them is another common form of
experimentation.
The best part about personalization is that
even low-traffic websites can work with them as the traffic constraints that
apply to CRO tests don’t apply to personalization.
The reason behind this is that with
personalization, the end goal isn’t to optimize conversion rates. Instead, its
goal is to offer a more relevant user experience that leads to more conversions
or higher customer satisfaction and retention.
For example, you could create different
versions of your product page to show to your customers and non-customers (or
leads).
On your product page for your customers, you
could show content such as product guides and videos, quick links to access
support and more, so that they get a more rewarding product experience. Whereas
the version you show your leads — for example, those who signed up for a trial
— could focus on your product’s features or case studies to help them convert.
About 93{c87e2df4b343d0515d304e127afe4653a549475791ab451641a18e09bd64e760} of companies see better conversions with
personalization.
And you can start with personalization right
away.
Plus, with a tool like Convert Experiences, you can set up personalized website experiences within minutes. Just set the targeting rules and you’ll be able to show personalized messaging to your different audience segments — prompting each segment to move to the next stage of their buying journey.
Wrapping it up …
While low website traffic can restrict you
from running substantive CRO experiments, it can’t stop you from
“learning.”
And that’s what’s important — because once you shift your focus to learning by trying rather than accepting the norm, you’ll start developing a culture of experimentation, a prerequisite for effective CRO.
It doesn’t matter what growth stage you’re at, you can invest in developing a culture of testing. It will come in handy when you actually have enough traffic to your website to go for hardcore, definitive CRO experiments.
Originally published April 25, 2019 – Updated June 20, 2024
Mobile reading?
Authors
