Terms and columns meaning
First, in order to interpret your experiment results, you must know what each column represents:
- This column shows you the Name of the Variation (‘Control’, ‘Variation #1’, ‘Variation #2’, and so on) on which you can click to see a preview of what will look like on your website.
- The amount of traffic set for each Variation/Control – Traffic percentage that you can change by editing the experiment and going on the ‘Traffic Allocation and Goals’ tab.
- The ID which is composed of the Experiment ID (the first number) and the Variation/Control ID (which is unique)
- This column shows you the number of unique visitors who saw the experiment.
- This column shows you the total amount of times the variation/control was seen by the users included in this experiment.
- Engagement -> is counted when a visitor stays more than 180 seconds on the website and has at least 3 page views in the current session.
- Bounce rate -> shows how many people left the page that is included in an experiment where the goal has been added, without interacting with it.
- Sale -> is a custom goal. Click here to see more information.
Reporting for A/B testing, personalizations and overlays
The first thing you need to do in order to see the results for a particular experiment is to filter the data, using the period filter, the device type filter, and the type of conversions filter (assisted or linear).
You can also see an experiment summary, which includes data such as: the device type and the URL(s) on which the experiment is set to run, the traffic allocation, the segmentation, and the dates when it was created/paused/expired:
Below you will find the KPI’s Timeline section, where you can generate a graph that shows the evolution for certain KPIs of your interest.
You can see the evolution for the following KPIs: conversion rate, conversions number, average goal value, revenue per visitor, total revenue, number of views, number of unique sessions, and number of users.
Using the button on the right you can download the report as PNG/JPEG/PDF/SVG.
You can also show the graphical evolution for a certain goal, by using this filter:
In the Performance section, you will see the results for the goals that you are tracking (default goals and goals that you have implemented) and you can filter the results by conversions. This means that you can see how did the users who converted on a certain goal also converted on the other goals.
The Reports column shows the statistic information about Goals, Conversion Rate, Conversions, etc.
- Goals – shows the names of the goals that have been added to the experiment to be counted.
- Conversion Rate – shows the percentage of users that converted, which in this case is the number of conversions divided by the number of users x 100.
- Conversions – shows the number of visitors who reached that goal.
- Revenue – shows the total amount of money received from the users that have been influenced/included in the experiment and reached the thank you page where the Sale goal was implemented.
- Revenue / Visitor – shows the value of a user, which in this case is calculated by dividing the total amount (Revenue) by the number of users.
- Average goal value – shows the average amount of money that the experiment brought in.
- Chance to win – or Statistical relevance shows how much, in percentage, of a chance to win has the variation over Control.
Mathematically, the conversion rate is represented by a binomial random variable, which is a fancy way of saying that it can have two possible values: conversion or non-conversion.
Let’s take an example for a better understanding:
Let’s call this variable ‘p’.
Our job is to estimate the value of ‘p’ (the conversion rate) and, for that, we do ‘n’ trials (or observe a number of ‘n’ visits to the website). After observing those ‘n’ visits, we calculate how many visits resulted in a conversion. That percentage value is the conversion rate of your website.
Please, keep in mind that every Variation relates to Control. As you can see, the uplift in results is highlighted with green, whilst the decrease with red, and it is shown in percentage relative to the Control version.
The Chance to win and Bayesian probability shown at the end of the Performance section are the results calculated with the Frequentist and Bayesian algorithms. A good result, which shows that you should implement the modification made within a variation should be above 95% on the main goal.
To download a .csv report containing all this data or one containing the details for each goal conversion, you can use the option at the top of the page:
Reporting for surveys
In the reporting page for a survey, you will find the same filters available for your data and the experiment summary, as in any other experiment (see above).
In the KPIs Timeline section, you can see a graphical representation and the value for the total number of visitors who were invited to take the survey, the number of respondents, the response rate (calculated as percentage of respondents from the total number of users) and the average completion time.
Below, you can see the actual responses that were given to each question in the survey, you can filter the results based on certain answers or unique answer questions:
A detailed response contains the exact response written or chosen by the user, the date and time when he submitted that response, the OS and also the device type and screen resolution:
To download a .csv report containing the responses or the captured leads, you can use the option at the top of the page:
For any type of experiment, you also have the option to generate a PDF report and also view/download/edit the previous ones: