Statistical relevance meaning and purpose
In A/B testing solutions, there exists a key metric called Statistical Relevance. Different tools have different terminology for this metric.
Omniconvert calls this Chance to Win. Other names might include statistical significance, confidence level, etc
The purpose of this metric is twofold:
- Determine the confidence level that the uplift/downlift is not caused by random chance
- Determine how likely it is, that given the same external factors, the current internal factors will lead to the same outcome.
In the context of AB testing experiments, the chance to win will determine that both the conclusion is valid and how likely it is that the difference between your experiment’s control version and test version isn’t due to error or random chance.
For example, if you run a test with a 95% chance to win, you can be 95% confident that the differences are real.
It’s commonly used in business to observe how your experiments affect your business’s conversion rates. In surveys, the chance to win is usually used as a way to ensure you can be confident in your survey results. For example, if you asked people whether they preferred ad concept A or ad concept B in a survey, you’d want to make sure the difference in their results was statistically relevant before deciding which one to use.
How do I know when my experiment has reached statistical relevance
At Omniconvert, in order for an experiment to reach statistical relevance, we recommend to let it run until these conditions are met simultaneously:
- there are at least 2500 unique users per variation
- there are at least 200 conversions on the main goal
- the experiment has run for at least 1.5 buying cycles for that specific type of product or industry
- the chance to win and the Bayesian probability are above 95% for a winning variation. If all three conditions listed above are met, but the chance to win in below 95%, then the conclusion is that the idea tested in the variation should not be implemented.
What are the Frequentist and Bayesian algorithms
The chance to win and the Bayesian probability are the results of two different algorithms that calculate the statistical relevance: Frequentist and Bayesian.
You can opt to show them both in your Reports or to chose only one of them when you first add your website, or you can change the satting later, using the Edit website option in the main menu: