What are the odds?

How likely is it that Variant B really converts better than Control?

The data. standard_ab_test_v2.csv, the published sample file for A/B test: 4,795 rows; columns User ID, Assigned Group, Converted, Order Value, Signup Date, Device. Sample data, not a customer's.

The calculation. Python, standard library only. This exact code ran against the file when this page was built, and printed the output below it.

import csv
from math import erf, sqrt
rows = list(csv.DictReader(open("standard_ab_test_v2.csv")))
n, won = {}, {}
for r in rows:
    g, c = r["Assigned Group"], r["Converted"]
    if g and c in ("Yes", "No"):
        n[g] = n.get(g, 0) + 1
        won[g] = won.get(g, 0) + (c == "Yes")
post = {}  # Beta(1, 1) prior -> Beta(1 + won, 1 + lost)
for g in n:
    a, b = 1 + won[g], 1 + n[g] - won[g]
    post[g] = (a / (a + b), a * b / ((a + b) ** 2 * (a + b + 1)))
for g in sorted(post):
    print(g, "posterior mean", f"{post[g][0]:.2%}")
(mb, vb), (mc, vc) = post["Variant B"], post["Control"]
z = (mb - mc) / sqrt(vb + vc)  # normal approximation
print("P(Variant B beats Control):",
      round(0.5 * (1 + erf(z / sqrt(2))), 4))

Control posterior mean 4.93%
Variant B posterior mean 7.38%
Variant C posterior mean 5.39%
P(Variant B beats Control): 0.998

What it says. Starting from a flat prior, the data leaves a probability of about 0.998 that Variant B's true conversion rate is higher than Control's. That is the statement people usually want from a test, and it is not what a p-value says: a p-value is the chance of data this extreme if there were no difference.

What it does not say. The answer leans on the prior, which is harmless here with about 1,600 users a group and matters a lot with a few dozen. The normal approximation to the two Beta distributions is close at these sizes and poor when a group has only a handful of conversions.

Run A/B test on this file or your own, or open its stored sample report. For this question on your own data, use Bayesian A/B Test.