This month’s common question we are answering is “How does Geeklab calculate the campaign’s confidence level?” Well, let’s take a deep dive into how Geeklab calculates the confidence level of each test in order to produce reliable results.
Learning phase
The learning phase per variant is a minimum of 3 days. Additionally, it requires at least 30 page views for each. This means that the learning phase can be over in 3 days if the minimum amount of page views was achieved each day. On the other hand, it can also be for example 5 days if 30 page views weren’t met on the 2 days in between. A learning phase helps take into account changes in user behavior during the start of the campaign.
Collecting data
While the campaign is collecting data, calculates cumulatively per day how long until it presumably reaches a stable significance level. For this Geeklab uses chi-squared. Each day has its own target threshold of significance, which is based on the performance of the previous day. The days the significance threshold is passed will take the campaign one day closer to stable significance. However, days that don’t hit the threshold add a day to the counter of days left. Therefore the days left counter is dynamic and varies based on performance.
Campaign conclusions
The ultimate goal of the campaign is to find a winning variant or two winning variants. A winner will be a variant with a significance rating of over 80%. If two variants have a combined probability of 90%, both will be declared winners. On the other hand, a variant with less than 2% winning probability will be declared a losing variant.
The campaign can be ended with confidence if the campaign results do not yield a significance level and investing more is not justified.
Fun fact: We named our system Maggie, inspired by Margaret Hamilton one of the software developers behind the Apollo 11 moon landing.
