ds@mcpanalytics.aiEmail your data and your question and get an answer about your own data, with the method shown.Email your data and question. Get an answer.
Is the difference real? Group differences analyses | MCP Analytics
Question

Is the difference real?

By MCP Analytics Team | Published , updated

Compare two groups? Find what drives an outcome? Spot a trend? The question decides which statistics fit, so it is the first thing to get right.

See a worked example

How the analyses here work

Worked example

Do customers in different regions report different satisfaction, and by how much?

The data. standard_group_comparison_v2.csv, the published sample file for Group comparison: 242 rows; columns customer_id, region, satisfaction_score, tenure_months, signup_month. Sample data, not a customer's.

The calculation. Python, standard library only. This exact code ran against the file when this page was built, and printed the output below it.

import csv, statistics as st
rows = list(csv.DictReader(open("standard_group_comparison_v2.csv")))
groups = {}
for r in rows:
    score = float(r["satisfaction_score"])
    groups.setdefault(r["region"], []).append(score)
for g, v in sorted(groups.items()):
    print(g, len(v), "rows, mean", round(st.mean(v), 1))
big = {g: v for g, v in groups.items() if len(v) >= 30}
hi = max(big, key=lambda g: st.mean(big[g]))
lo = min(big, key=lambda g: st.mean(big[g]))
d = st.mean(big[hi]) - st.mean(big[lo])
se = (st.variance(big[hi]) / len(big[hi])
      + st.variance(big[lo]) / len(big[lo])) ** 0.5  # Welch
print(hi, "minus", lo, "=", round(d, 1), "points, 95% interval",
      round(d - 1.96 * se, 1), "to", round(d + 1.96 * se, 1))
North 80 rows, mean 100.3
Pilot 2 rows, mean 91.4
South 80 rows, mean 113.3
West 80 rows, mean 131.2
West minus North = 30.9 points, 95% interval 27.3 to 34.4

What it says. Among the regions with at least 30 rows, West scores highest and North lowest. The gap is 30.9 points, and the interval of 27.3 to 34.4 is the range of gaps this sample is consistent with, using Welch's standard error, which does not assume the regions vary equally.

What it does not say. The file also holds Pilot (2 rows): it is listed but left out, because a mean of a handful of rows says almost nothing. Choosing the highest and lowest group after looking makes a single interval optimistic; the library analysis runs ANOVA across all groups, reports every pairwise difference with its interval and effect size, and adds the nonparametric checks.

Run Group comparison on this file or your own, or open its stored sample report.

Sources

Further reading on the method in the worked example. Each reference was looked up and its authors, title, year and venue confirmed on 27 October 2026.

  1. Welch B (1947). The generalization of 'Student's' problem when several different population variances are involved. Biometrika. https://doi.org/10.1093/biomet/34.1-2.28
  2. Welch B (1951). On the Comparison of Several Mean Values: An Alternative Approach. Biometrika. https://doi.org/10.2307/2332579
  3. Tukey J (1949). Comparing Individual Means in the Analysis of Variance. Biometrics. https://doi.org/10.2307/3001913

By MCP Analytics Team. Updated November 2026. Analysis list from the library catalog.

A delivered report from this list

Sequential A/B Test (Peek With an Approximate Guarantee): sample report Open full report →

The stored sample report for Sequential A/B Test (Peek With an Approximate Guarantee). Each analysis has its own report layout; open its page to see what it returns.