Making Money On-line: How to Form an Effective Hypothesis for Optimized Testing

A strong hypothesis is the cornerstone to any meaningful and effective test. Every hypothesis that is proven or rejected provides your business with valuable insights behind visitor behavior and continues to drive your optimization plan while supporting company goals and objectives.

Remind me, what exactly is a hypothesis?

Before we dive into the components and best practices you should keep in mind when crafting your hypothesis, let’s first discuss what exactly a hypothesis is. If you type “hypothesis” into Google, you’ll likely receive a definition similar to the below:

“Hypothesis: a supposition or proposed explanation made on the basis of limited evidence as a starting point for further investigation.”

In simpler terms, a hypothesis is a prediction you make before running an experiment or test. It is a statement that addresses a specific problem or question while also providing a suggested solution.

Hypotheses should be informed by quantitative data (web analytics, past test data, campaign insights, audience insights, heat maps) and/or qualitative data (user testing, focus groups, customer support feedback). The hypothesis created based on this information will clearly state what is being altered, what the predicted outcome of this alteration will be, and the rationale behind that prediction.

Ultimately, the outcome of the test will either prove or disprove your hypothesis.

So, how do I create my own hypothesis?

When beginning to craft your hypothesis, you’ll want to first ask yourself what the problem, question or reason for testing is. Once that is defined, you can then outline a clear description of what is being changed to address this reason, ending with the results you expect to see from this change.

For instance, if currently the click-through rate on your product details page is not at the rate that aligns with your business goals, you may want to perform a test that helps to address the underlying problem. An example hypothesis that may kick-off this experiment could be as follows:

Changing the product list on the search results page from a single column format to a dual column format will increase the number of product options visible and boost click-through to the product details page.

To ensure you establish a clear and meaningful business hypothesis, there are three components you will always want to include:

The change you are testing
The results you expect to see from this change
The specific audience you expect the change to impact

If you want to take your hypothesis a step further, it’s beneficial to include the following additional components:

By how much of an impact you expect your change to have
After how much time

What exactly do each of these components entail?

The change you are testing: This is the part of your hypothesis where you will clearly state exactly what change you will be making to your site and describe what will be tested. It is important to be as specific as possible here. Simply stating that you will implement a redesign is too broad – knowing exactly what elements make up the redesign will allow for a clear conclusion as to whether your hypothesis was proven or disproven.

Don’t: Redesigning the search results page will increase click-through to the product details page.

Do: Changing the product list on the search results page from a single column format to a dual column format will increase the number of product options visible and boost click-through to the product details page.

The results you expect to see from this change: Here you will want to clearly state the impact you expect to see from this change, as well as what will be used to determine success. You want to be able to answer the question of how you know if your change is truly successful – will it increase purchase conversions? Increase form submissions? Reduce time spent before reaching the next page? The goal here is to ensure your hypothesis is specific and measurable.

Who will be impacted: It is important to define the audience that will be affected by your change. We don’t want to assume what you are testing will impact everyone that comes to your site, or will even be shown to everyone. This may in fact be the case, but often times it is not. It’s critical the audience you choose to include in your test makes sense depending on what it is you are testing, and this is a significant factor when it comes to getting clean data and results. That being said, we will want to define this audience within our hypothesis as well.

Example: Displaying a pop-up modal on the homepage, to unsubscribed visitors, with messaging that encourages visitors to sign-up for our newsletter will increase newsletter subscriptions.

The above example clearly tells us that the experiment will be tested on unsubscribed visitors to the site, and will allow us to better conclude if the experiment was successful for these group of visitors when analyzing the results.

By how much of an impact you expect your change to have: Including this component in your hypothesis will better allow you to assess whether or not your hypothesis passes or fails. For instance, rather than just stating that your change will increase purchase conversion rate, it is extremely effective to also include by how much you expect this change to increase your purchase conversion rate. This will often be an estimate, but it is still helpful to include as you will have a clear measurement for whether or not the change you are testing is successful. This will also help you estimate for how long you should run your test for.

Example: Changing the form fields within the checkout funnel from a single column format to a dual column format for all visitors will increase the purchase conversion rate from 80% to 83%.

After how much time: Including an estimated duration of the experiment into your hypothesis will allow anyone who is analyzing the results to understand that the data is not valid until the set duration has passed. This does not necessarily have to be set based off of time, but can also be defined by other measurements such as reaching a certain number of sign-ups, purchases, page views, or after a marketing promotion may be over. Sound CXO systems and service teams should also have a sample size calculator which will assist with this calculation effort.

If you’ve included all of the above components in your hypothesis, you’re on your way to designing a meaningful and effective test. If you want to get the most out of your experiment and the results, it’s all going to start here.

Want to learn more about how the Oracle Maxymiser Consulting team can take your optimization program to new heights? The Oracle Maxymiser Consulting team is made up of passionate strategists, designers, developers, quality assurance professionals, trainers, analysts, and platform experts. We’re excited to understand your business needs and work with you to drive ROI.

Contact us here or find out more information about Oracle Maxymiser (and the rest of the Oracle Marketing Cloud suite) at our home on the web.