The Analysis Library

Did either new checkout design convert better than the current one?

The data. standard_ab_test_v2.csv, the published sample file for A/B test: 4,795 rows; columns User ID, Assigned Group, Converted, Order Value, Signup Date, Device. Sample data, not a customer's.

The calculation. Python, standard library only. This exact code ran against the file when this page was built, and printed the output below it.

import csv
rows = list(csv.DictReader(open("standard_ab_test_v2.csv")))
n, won = {}, {}
for r in rows:
    g, c = r["Assigned Group"], r["Converted"]
    if not g or c not in ("Yes", "No"):
        continue  # no group or no outcome recorded
    n[g] = n.get(g, 0) + 1
    won[g] = won.get(g, 0) + (c == "Yes")
rate = {g: won[g] / n[g] for g in n}
for g in sorted(n):
    print(g, n[g], "users,", won[g], "converted,", f"{rate[g]:.2%}")
best = max((g for g in n if g != "Control"), key=rate.get)
d = rate[best] - rate["Control"]
se = (rate[best] * (1 - rate[best]) / n[best]
      + rate["Control"] * (1 - rate["Control"]) / n["Control"]) ** 0.5
print(best, "minus Control:", f"{100 * d:+.2f}", "points, 95% interval",
      f"{100 * (d - 1.96 * se):+.2f} to {100 * (d + 1.96 * se):+.2f}")
print("skipped", len(rows) - sum(n.values()), "incomplete rows")

Control 1601 users, 78 converted, 4.87%
Variant B 1584 users, 116 converted, 7.32%
Variant C 1595 users, 85 converted, 5.33%
Variant B minus Control: +2.45 points, 95% interval +0.79 to +4.11
skipped 15 incomplete rows

What it says. Variant B converted 7.32% of users against 4.87% for Control, a difference of 2.45 percentage points. The 95% interval (0.79 to 4.11) stays above zero, so in this sample the lift is unlikely to be noise alone.

What it does not say. 15 rows had no group or no outcome and were skipped, not guessed. This is the simple two-proportion interval for the better of two variants, picked after looking, so it is a little optimistic; the library analysis adjusts every comparison with the Holm method and also checks that traffic was split as planned.

Run A/B test on this file or your own, or open its stored sample report.