Will A/B Testing Impact Your Site Speed? Find Out Here

When an optimization agency researched how A/B
testing tools impacted a website’s speed, it found
lags. While most seemed perceptible to the end users, a few posed potent
threats to a website’s user experience (at ~ 4 seconds!).

We, too, participated in the analysis — competing head to head with the likes of Optimizely & VWO among others — and made it to the top three. (We ranked better than these two actually!)

In the years following that research, we’ve
worked on making fast faster. In
fact, speed directs our development efforts as even minuscule improvements in
our code’s performance impact the loading times of the thousands of websites
that use us for serving experiments.

As an optimizer, you, too, should care about
how you’re affecting your website performance by adding experimentation to your
optimization mix.

Here are a few essentials to keep in mind.

A/B Testing Can Cost a Few Milliseconds

In an ideal world, running an A/B test or an
experiment wouldn’t impact your website’s speed at all.

But that’s not the case.

Because when you use a third-party A/B testing solution provider like us, and when your users request pages you’re experimenting with, part of the request is served by our servers. This back and forth causes some lag.

Depending on the element you’re testing, this
lag may be more or less perceivable to your users.

Let’s take two scenarios to understand this:

Suppose you’re testing your homepage’s headline
or UVP.

This headline can be called a “hero” element
as its loading is critical to your user’s experience with your website. The
time it takes to show up directly impacts your user’s perceived “loading
experience.”

It also adds to your website’s “Time to first
paint” metric as it’s the time your website is taking before the main content
appears.

If the experiment tool you use takes time to
deliver this headline, your users will have a noticeably slower loading experience
with your website.

Now, contrast this with an experiment you’re
running on an element somewhere in the middle of your homepage. In this case,
any lag caused due to delivering this experience will be barely discernible
because it will most likely be available when the user goes through the hero
area and scrolls down to that part of the page.

Perceptible or not, Convert Experiences or ANY OTHER A/B testing tool you use adds to your website’s loading speed.

That said, if your website is optimized for speed, going with a fast A/B testing solution like Convert Experiences and setting it up right will hardly make a difference to your website’s loading time. Which is to say, you WILL maintain your speediness and can continue working toward the “load under two seconds” target or any other ambitious speed goals.

Understanding the Real Impact of A/B Testing on Speed…

Singling out and understanding the impact of
an A/B testing tool on speed can be tricky.

This is so because most website speed
diagnostic tools like the Google Page Speed Test use different ways of
measuring a website’s performance.

In short, it’s never as straightforward as:

X seconds before Convert Experiences and X.5
seconds after Convert Experiences.

Why?

Because we may relay data between your website
and our servers and the integrations you use. For example, when a user
completes a goal of your experiment, we’ll send that data to your Google
Analytics account, if you use the Convert Experiences + Google Analytics integration
— as your website loads. But the time we take for these doesn’t actually add to your website’s loading time. This is just
us working in the background and not getting in the way of your users and your
website.

A tool’s support team, and not diagnostic speed test tools, can best answer questions about a testing tool’s speed impact.  Besides, you can get really close to realizing zero impact testing if you do a few things right.

Getting as Close to Zero Impact Testing as You Can

One thing that’s settled about speed as it relates to conversions, user experience, and SEO, is this:

Every millisecond counts.

You want to ensure that as you add yet another
script to your website, you save even a millisecond’s worth of speed
optimization.

Your choice of an A/B testing and experiences tool is critical here.

Going with an A/B testing solution that’s “present everywhere” 🌐

Because your experimentation service provider
serves some part of the page you’re experimenting with, it’s important that it
hosts the content it needs to serve close to where your users are.

This means your experimentation service
provider’s servers that host this content must be close to your users.

Otherwise latency — or the time it takes a
visitor to request your website and the server to serve it — can add many
seconds to your website’s loading time.

Convert Experiences, for example, is powered by the Akamai CDN network. And via Akamai’s 260,000+ servers in more than a hundred and thirty countries, we’re close to where your users are. And that lets us serve your content to your users blazing fast, with the least latency.

Going with an A/B testing solution that prioritizes speed 🚀

No matter how fast a service technology like
experimentation is, it can always do
better. For instance, we weren’t always on the Akamai network. We
re-engineered our solution to move to it (from Amazon Cloudfront) because we
wanted to serve your experiments faster.

By switching to Akamai, we reduced a large number of DNS lookups when serving tests and saved tens of milliseconds of speed on each experience view:

Going with an A/B testing solution that supports an optimal setup ✔️

An optimal experimentation setup means your experiments always use lean snippets (that are placed just right), a limited number of goals, clean code, and logical execution of the elements being experimented with among other things.

Your solution, too, needs to be maintained.
For example, routinely archiving non-active experiments (so that unnecessary
server calls are avoided), ensuring proper caching settings (that enable faster
delivery after the first load), managing media well (for speedy delivery) and
more.

These things contribute a big deal to your loading time and your user’s experience and can spiral out of control easily.

That’s why we at Convert have our support and success teams on standby to help you get your experiments and tool configuration right so that your experiments deliver faster. We also have a lot of documentation on this.

The Cost to Pay for Breakout Success (and Continued Growth)

Given that few tactics are as effective as
experimentation to increase your conversion rate (and keep improving it), you
can’t avoid going for it because of speed concerns.

Besides, if you do it right, A/B testing or
any form of experimentation won’t make your website slow. Sure, a small side
effect exists… but that’s all it is.

Lightweight experimentation solutions like Convert Experiences help you further mitigate it by delivering your experiments in a flash, without adding multiple seconds to your website’s loading time and risking your SEO channel or frustrating your users. Ready to try it out? Sign up for a 15-day free trial.