What is a Hypothesis?
A proposed solution to a problem, whose validity requires evaluation.
Why is a Hypothesis Critical to Your Optimization Program?
The innate desire for improvement is what
drives an optimization program, but testing just for the sake of testing is a
setup for failure. An experiment needs a primary goal that determines success.
Formulating a hypothesis forces us to think about what that goal is, how we should try to achieve it, and what results to look for. A good hypothesis provides structure.
A poor hypothesis creates confusion; garbage in, garbage out. The hypothesis is the foundation of an experiment, it contains all of the information necessary for building a split test.
How do you Construct a Split Test Hypothesis?
There are three main components of a split
test hypothesis: Comprehension,
Response, and Outcome.
To make components easy to remember, they
follow the CRO acronym that all of us are very familiar with.
Each component is comprised of two to four short written snippets describing data collected via research, design, and measurement. Assemble the snippets together into sentences and you’ve built a solid hypothesis. Here’s an example, with each data snippet marked in bold:
- Comprehension: We have observed a reduction in multi-item purchases by comparing the last 6 months of purchase
data.
- Response: We want to promote paired products with an inline upsell at the cart page on mobile phones for returning
users with an item already in their cart.
- Outcome: This should lead to single item purchasers more easily finding and purchasing complementary products, which will be measured by average order value (AOV) and backed up by average order size, multi-item purchase count, order conversion, and revenue.
You can see how each snippet is a description
of an important piece of data that was either collected from research, proposed
via design, or outlines measurement criteria.
Assembled together, they read as a complete hypothesis. Now, let’s dive into each component.
Comprehension
Comprehension comes from research; it is identifying and understanding a problem you want to capitalize on. For example: “90{1652eb1ffa4184925f6a63a9c04ea6b421acb7a78117241e7d4325cdca8339fa} of users are abandoning from the signup form after the 3rd input.” Comprehension is composed of two snippets: the observation, and the observation method.
Observation
To identify a problem, first you must observe it. Dig into some data and get your hands dirty. Find something about your site that is not performing as well as you want it to. The two main avenues for research are analytics and feedback. Analytics is numeric, quantitative data on a large scale; typically numeric, statistical data that’s passively collected for later aggregation. Feedback is descriptive, anecdotal data that’s actively collected and representative of user behavior.
Observation Method
The tool, technique, or process you used to identify the problem. Incorporating the observation method into the hypothesis helps prove that you did your research. Examples of observation methods for each type of research include:
Analytics
Feedback
- Techniques: surveys, usability testing, product reviews
- Tools: Hotjar, Qualaroo, TurnTo
Response
Response comes from design; it is proposing
and targeting the change you want to make. Often, this can be approached by
starting with a question, like “what will
happen if we increase the size of product images in the category grid display?”
The response is the “meat and potatoes” of your experiment; the core that will be configured in the split testing software. The snippets that form a response are: variation, location, and audience.
Variation
What is being proposed as an improvement. This is the theoretical part of an experiment. Whatever change you think will outperform the current state of the site, big or small: stronger call-to-action text on a button, larger images, reduction of form inputs, redesigned header navigation. You get the picture; a short description of your idea for outperforming the baseline.
Location
Where the experiment will take place on your website; types of pages, URL targeting, device scope and/or screen breakpoints.
Audience
Who should see the experiment; specific
details regarding user’s status in relation to providing meaningful behavioral
feedback.
The snippets of a response are how you scope
an experiment and fine tune the signal-to-noise ratio of the result data.
For example, if the variation is changing the position of the estimated order total of an eCom site, then the targeting might be the cart page and then the audience would only be users who have an item in their cart. You would not want to collect data from users that interact with an empty cart page, since they wouldn’t see the UI elements that display a cart total anyway.
Looking for an A/B testing tool that lets you get very sophisticated with your targeting and audience building? Check out what Convert Experiences is capable of.
Outcome
Outcome comes from measurement; it is how success is defined and evaluated. The snippets of an outcome are: desired behavioral change, the primary KPI, and secondary KPIs.
Desired behavioral change
The change in user behavior that you want to be caused by the response; either encouraging a positive action or discouraging a negative action.
Primary KPI
The metric that determines an experiment’s success or failure. Measurement of a user action that the experiment is trying to affect. An experiment’s raison d’être.
Secondary KPIs
Metrics that back up the Primary KPI. These
help you flesh out the story being told by the primary KPI, and/or check for
unexpected user behavior. Secondary KPIs are often conversions at each stage of
the funnel between the targeted area and primary KPI endpoint.
For example, if your primary KPI is revenue from orders and the experiment targets a product page, then the secondary KPIs would be add-to-carts, cart pageviews, checkout pageviews, and order conversions.
Hypothesis Builder
Over at Corvus CRO we’re all about building
little tools to help make the lives of CRO experts easier. To that end, we’ve
constructed a hypothesis
builder to help simplify and standardize the construction
of split test hypotheses. It is a web form that takes text input for each of
the necessary snippets and assembles them into a full hypothesis. You can use
the tool to proof out and build your own hypotheses. There’s even a “copy to
clipboard” button so you can easily grab your constructed hypothesis and paste
it into your project management tool of choice. We hope you get as much use out
of the tool as we do!
Collecting each snippet can be a time consuming process, but the payoff is solidly constructed experiments that will provide more overall value in the long run.
An ounce of prevention is worth a pound of cure; a little more preparatory work up front significantly reduces later overhead.
