To help you better understand the progress and reliability of your A/B testing experiments: Sample Size Calculator.
The feature gives real-time visibility into how your A/B test is progressing toward statistical significance.
It helps you answer critical questions such as:
- Is my experiment collecting enough data?
- How long until I can trust the results?
- Is the test ready to conclude?
This removes the guesswork and helps ensure that your test results are based on sufficient traffic and conversions.
How it works
The Sample Size calculator feature uses this as a basis, but automates the process so you don’t have to manually have to input the data every time you want to check how your experiment is going.
The Sample Size Calculator guides you through three clear stages.
1. Collecting Data
This is the initial phase of your experiment.
The calculator is in “Collecting Data” mode until each variation has:
- At least 500 users
- At least 10 conversions on the main goal (usually the Sale goal)
During this time, Omniconvert is gathering enough data to begin estimating your experiment’s duration and outcome.
2. Running Test
Once the minimum thresholds are reached, the calculator begins to project the number of days needed for your experiment to reach maturity.
The system continues to monitor how data is evolving and updates the projection accordingly.
3. Results ready
This final stage indicates that your experiment has gathered enough data to be statistically relevant.
You’re now able to review the results and make confident decisions based on them.
You will also see different messages depending on how the experiment went.
1. Experiment was a success, Chance to win was high enough.
2. Experiment was close to being a success. You can try making a few changes and running it again.
If your experiment is Paused, Expired, or has 0% traffic allocated to the control group, the calculator will show a message letting you know that sample size cannot be calculated until the experiment is updated.
