By MCP Analytics Team | Published , updated
Compare two groups? Find what drives an outcome? Spot a trend? The question decides which statistics fit, so it is the first thing to get right.
The analyses filed under this question. Each card opens the analysis page, where you can run it on a file.
Can I stop this experiment early? Sequential testing with approximately always-valid inference (asymptotic), so peeking does not inflate false…
Did any variant beat the control, and by how much?
Do your groups differ once a baseline is adjusted for?
Are two categorical columns related?
Are these two things practically the SAME within a margin?
Does the outcome differ between groups, by how much, and exactly which groups differ, with t-tests or ANOVA and the nonparametric check beside them.
Which results survive correction for multiple testing?
Did the same subjects change between two measurements, and by how much?
Did the treatment work, or were the treated already different?
Did the same subjects change across three or more conditions?
The data. standard_group_comparison_v2.csv, the published sample file for Group comparison: 242 rows; columns customer_id, region, satisfaction_score, tenure_months, signup_month. Sample data, not a customer's.
The calculation. Python, standard library only. This exact code ran against the file when this page was built, and printed the output below it.
import csv, statistics as st
rows = list(csv.DictReader(open("standard_group_comparison_v2.csv")))
groups = {}
for r in rows:
score = float(r["satisfaction_score"])
groups.setdefault(r["region"], []).append(score)
for g, v in sorted(groups.items()):
print(g, len(v), "rows, mean", round(st.mean(v), 1))
big = {g: v for g, v in groups.items() if len(v) >= 30}
hi = max(big, key=lambda g: st.mean(big[g]))
lo = min(big, key=lambda g: st.mean(big[g]))
d = st.mean(big[hi]) - st.mean(big[lo])
se = (st.variance(big[hi]) / len(big[hi])
+ st.variance(big[lo]) / len(big[lo])) ** 0.5 # Welch
print(hi, "minus", lo, "=", round(d, 1), "points, 95% interval",
round(d - 1.96 * se, 1), "to", round(d + 1.96 * se, 1))
North 80 rows, mean 100.3
Pilot 2 rows, mean 91.4
South 80 rows, mean 113.3
West 80 rows, mean 131.2
West minus North = 30.9 points, 95% interval 27.3 to 34.4
What it says. Among the regions with at least 30 rows, West scores highest and North lowest. The gap is 30.9 points, and the interval of 27.3 to 34.4 is the range of gaps this sample is consistent with, using Welch's standard error, which does not assume the regions vary equally.
What it does not say. The file also holds Pilot (2 rows): it is listed but left out, because a mean of a handful of rows says almost nothing. Choosing the highest and lowest group after looking makes a single interval optimistic; the library analysis runs ANOVA across all groups, reports every pairwise difference with its interval and effect size, and adds the nonparametric checks.
Run Group comparison on this file or your own, or open its stored sample report.
Further reading on the method in the worked example. Each reference was looked up and its authors, title, year and venue confirmed on 27 October 2026.
The stored sample report for Sequential A/B Test (Peek With an Approximate Guarantee). Each analysis has its own report layout; open its page to see what it returns.
Tell us what went wrong, in your own words. We capture the page you're on automatically, so no need to describe where you are.