What does all the evidence say?

If each device were a separate study, what is the pooled effect, and do the studies agree?

The data. standard_ab_test_v2.csv, the published sample file for A/B test: 4,795 rows; columns User ID, Assigned Group, Converted, Order Value, Signup Date, Device. Sample data, not a customer's.

The calculation. Python, standard library only. This exact code ran against the file when this page was built, and printed the output below it.

import csv
rows = list(csv.DictReader(open("standard_ab_test_v2.csv")))
cell = {}
for r in rows:
    g, c = r["Assigned Group"], r["Converted"]
    if g in ("Variant B", "Control") and c in ("Yes", "No"):
        k = cell.setdefault((r["Device"], g), [0, 0])
        k[0] += 1
        k[1] += c == "Yes"
est = []  # one "study" per device: B minus Control, and its variance
for dev in sorted({d for d, _ in cell}):
    (nb, yb), (nc, yc) = cell[(dev, "Variant B")], cell[(dev, "Control")]
    pb, pc = yb / nb, yc / nc
    est.append((pb - pc, pb * (1 - pb) / nb + pc * (1 - pc) / nc))
    print(dev, f"{100 * (pb - pc):+.2f} points")
w = [1 / v for _, v in est]
fixed = sum(wi * d for wi, (d, _) in zip(w, est)) / sum(w)
q = sum(wi * (d - fixed) ** 2 for wi, (d, _) in zip(w, est))
df = len(est) - 1
i2 = max(0.0, (q - df) / q)
tau2 = max(0.0, (q - df) / (sum(w) - sum(x * x for x in w) / sum(w)))
wr = [1 / (v + tau2) for _, v in est]  # DerSimonian-Laird
rand = sum(wi * d for wi, (d, _) in zip(wr, est)) / sum(wr)
se = (1 / sum(wr)) ** 0.5
print("fixed effect:", f"{100 * fixed:+.2f}", " I squared:", f"{i2:.0%}")
print("random effects:", f"{100 * rand:+.2f}", "points, 95% interval",
      f"{100 * (rand - 1.96 * se):+.2f} to {100 * (rand + 1.96 * se):+.2f}")

Desktop +3.48 points
Mobile +4.57 points
Tablet -0.59 points
fixed effect: +2.47  I squared: 72%
random effects: +2.48 points, 95% interval -0.64 to +5.60

What it says. The three device groups disagree more than chance alone would explain (I squared 72%), so the random-effects pool is the honest summary: +2.48 points, with an interval of -0.64 to +5.60, wider than the fixed-effect answer because it allows the true effect to differ by device, and wide enough now to include zero.

What it does not say. Three studies are too few to estimate between-study spread well, and these are subgroups of one experiment, not independent studies; the example only shows the arithmetic. For real studies, the meta-analysis in the library pools effect sizes across them and draws the forest plot.

Run A/B test on this file or your own, or open its stored sample report. For this question on your own data, use Meta-Analysis & Forest Plot.