“I believe that if we change the design of the landing page, it will lead to an improvement in signups”. In the regular, “normal” vocabulary of the natural world, the opening quote sentence is a passable hypothesis. However, in the world of product discovery, it is a terrible one. And the product discovery process will see a Product Manager formulate and make decisions based on hypothesis on an iterative basis. This is why we must conduct hypothesis testing the right way.

But what is Hypothesis Testing?

Simply put, Hypothesis Testing is a technique in product management that allows a product manager to validate their ideas about a product in the Product Discovery process.

A breakdown of the major “parts” of a hypothesis.

In hypothesis testing, after formulating a hypothesis, data gathering is done to test it. There are two types of hypotheses: null and alternative. The null hypothesis states that there is no difference or relationship between the two variables, while the alternative hypothesis states that there is a relationship or difference between the variables.

At the beginning of this article, we introduced a hypothesis that we said was terrible. A correct hypothesis concerning the same scenario would be:

A breakdown of a correct hypothesis for Hypothesis Testing during Product Discovery.

So, let us break down the most important things to note when conducting Hypothesis Testing.

Be Specific

This is probably the most essential thing to note about hypothesis testing. For instance, the first thing to note in the first “bad” hypothesis we introduced was that the landing page redesign was loosely defined. What aspect of the landing page is being changed? The colors? The button placement?

Also note that in the good hypothesis, the “impact” question of the hypothesis was practical and specific. According to Product expert Teresa Torres, saying a design change will “increase usability” is not specific enough. Why? Because it is not measurable. The same goes for hypothesizing an increase in engagement. Engagement, though measurable, is still not specific enough. Will it increase the time spent on the site? The number of button interactions? The email signups?

Product Managers should also note that targeting your hypothesis to a specific group of people is the only way to truly narrow it down to a measurable metric. Like the example in the diagram above, simply saying “design change x should…increase conversion of users” is not enough. What type of users are you targeting with this design change? Are you targeting seasoned experts? Or power users? Or first-time users? Is a user already utilizing a competitor’s product?

Being specific in hypothesis testing also involves measuring the best-guess degree of improvement the design change could provide for your product. This is often not more than guesswork, but if done right, it could make a world of difference between what design changes are thrown out and which ones are kept. For instance, if the degree of improvement expected from the hypothesis being tested is a 10 percent increase in conversion rate, then a 9 percent increase should denote a failure. This might seem extreme, but it helps protect your product from biases and mediocrity and might even inform your future estimates of what an acceptable expectation of improvement should be.

Finally, we should define the duration of the hypothesis being tested. This protects the product team from losing track of the data or identifying false positives where there are none. The hypothesis should have a finite timeline that lets the product team come back to the drawing board and compare ideas again.

Determine the Appropriate Sample Size

Sample size is another essential factor in hypothesis testing. A sample size that is too small can lead to inaccurate results, while a sample size that is too large can lead to a waste of resources. It is essential to determine the appropriate sample size when conducting hypothesis testing to ensure accurate results. A larger sample size increases the chances of obtaining accurate data and decreases the chances of making mistakes when analyzing the data.

Conduct Continuous Testing

Continuous testing is crucial in hypothesis testing. It enables product managers to keep testing their hypotheses throughout the product development process to ensure they are on the right track. Continuous testing helps product managers to identify and address any issues early before they become significant problems. It also enables product managers to adjust their strategies in response to changing circumstances.

Use the Right Statistical Tools

Product managers should use the right statistical tools when conducting hypothesis testing. Statistical tools enable product managers to analyze data and draw conclusions from it. The choice of statistical tools depends on the type of hypothesis being tested and the sample size. Product managers should seek the guidance of statistical experts when choosing the right tools.

Collaborate with Other Teams

Hypothesis testing is a collaborative process that involves different teams in an organization. Product managers should work closely with teams such as marketing, engineering, and design to conduct successful hypothesis testing. Collaboration helps to ensure that all teams are aligned in terms of goals, objectives, and timelines. It also helps to ensure that all teams have a stake in the product’s success.

Love the article? Read more about Product Discovery Basics For Building Better Products