Statistical Tests: Which Hypothesis Test Should You Use?

You have data and a question. Revenue went up after a campaign: was it real or noise? Two customer segments show different churn rates: is the gap statistically meaningful? Choosing the wrong test can produce a confident-looking p-value that means nothing, or miss a genuine effect hiding in your data. This guide maps every common scenario to the right statistical test so you stop guessing and start deciding.

Worked Example: Did a Simulated Variant Change the Mean?

Question. Did a simulated variant change the mean outcome from a control group?

Inputs. Simulated data, seed 2026. The script makes two independent groups of outcome values, 12 values each. The control draw is centered on 10.0 and the variant draw on 11.0, both with spread 2.0, so a gap of 1.0 was built in.

Method. Welch's two-sample t test (Welch 1947). It compares group means and does not require equal group spread. This choice fits a question about a mean in two independent groups. The test assumes that observations are independent and that each group's values are not strongly shaped by extreme outliers.

import numpy as np
from scipy import stats

seed = 2026
rng = np.random.default_rng(seed)
control = rng.normal(loc=10.0, scale=2.0, size=12)
variant = rng.normal(loc=11.0, scale=2.0, size=12)

res = stats.ttest_ind(variant, control, equal_var=False)
gap = variant.mean() - control.mean()
print(f"Simulated data, seed {seed}")
print("Built in: control 10.0, variant 11.0, spread 2.0")
print("Values per group: 12")
print("Control mean:", round(float(control.mean()), 2))
print("Variant mean:", round(float(variant.mean()), 2))
print("Gap (variant minus control):", round(float(gap), 2))
print("Welch t statistic:", round(float(res.statistic), 3))
print("Two-sided p-value:", round(float(res.pvalue), 4))

Simulated data, seed 2026
Built in: control 10.0, variant 11.0, spread 2.0
Values per group: 12
Control mean: 10.0
Variant mean: 10.64
Gap (variant minus control): 0.63
Welch t statistic: 1.102
Two-sided p-value: 0.2845

Result. The control mean is 10.0 and the variant mean is 10.64. The estimated gap is 0.63 (differences are taken before rounding). The two-sided p-value is 0.2845. With 12 values per group and a built-in gap of 1.0 against a spread of 2.0, this test finds the gap about one run in five, so p = 0.28 is the usual outcome here and is not evidence of no difference. The result does not show that the groups are the same.

Limit. These are simulated values, not customer data. The test also does not show that a real-world change caused an outcome shift.

Why Choosing the Right Test Matters

Every hypothesis test balances two risks. A Type I error (false positive) occurs when you declare a difference real when it is actually random variation: you roll out a pricing change that had no true effect. A Type II error (false negative) happens when you miss a real difference: you abandon a winning campaign because the test lacked statistical power to detect its impact.

The test you choose determines how well you control these risks. Parametric tests like the t-test assume your data follows a normal distribution and offer maximum statistical power when those assumptions hold. Use them on non-normal data, however, and with small samples or strong skew your p-values can become unreliable, so you may see effects that are not there or miss ones that are. Nonparametric alternatives like the Mann-Whitney U test make fewer assumptions about the shape of the data. They can have less power when the parametric assumptions do hold, and they have assumptions of their own.

Matching the test to the data also matters for the number of groups you compare. A t-test handles two groups. Repeated pair tests can raise the chance of false positives. Use a correction such as Bonferroni (Dunn 1961) when you test a set of pairs. ANOVA and its nonparametric counterparts exist precisely to solve this problem.

Quick Comparison: Every Test at a Glance

When to Use Each Test

T-Test

Use Welch's t-test to compare the mean outcome in two independent groups when the data support a mean comparison and observations are independent. See the worked example above. The independent-samples t-test compares two separate groups (treatment vs. control), while the paired t-test compares the same subjects measured twice (before vs. after). Welch's t-test is the safer default because it does not assume equal variances. With large samples, the t-test is less sensitive to moderate non-normality. Full t-test guide →

ANOVA

Use one-way ANOVA when comparing means across three or more independent groups, for example testing whether average order value differs across four landing page variants. ANOVA tells you whether at least one group differs; follow up with post-hoc tests (Tukey HSD, Bonferroni) to find which pairs. Two-way ANOVA adds a second factor and tests for interaction effects. If normality or equal-variance assumptions fail, switch to Kruskal-Wallis. Full ANOVA guide →

Chi-Square Test

Use the Chi-Square test of independence when both your variables are categorical and you want to know if they are associated. Common applications include testing whether customer segment (new vs. returning) is related to purchase category, or whether conversion rates differ across traffic sources. Requires large enough expected cell counts; if some cells are small, use Fisher's Exact Test instead. Full Chi-Square guide →

Mann-Whitney U Test

Use the Mann-Whitney U test as the nonparametric alternative to the independent-samples t-test. It compares the rank distributions of two independent groups and works well with ordinal data, skewed distributions, or when outliers make the t-test unreliable. It tests whether one group tends to have larger values than the other. Common in A/B testing on revenue data (which is almost always right-skewed) and customer satisfaction scores. Full Mann-Whitney guide →

Wilcoxon Signed-Rank Test

Use the Wilcoxon signed-rank test as the nonparametric alternative to the paired t-test. It compares two related measurements, such as before and after an intervention or matched pairs, when the differences are not normally distributed. Particularly useful for small samples where normality is hard to verify, or when working with ordinal rating scales. Full Wilcoxon guide →

Kruskal-Wallis Test

Use the Kruskal-Wallis test as the nonparametric alternative to one-way ANOVA. It compares rank distributions across three or more independent groups when normality assumptions are violated. Follow up a significant result with Dunn's post-hoc test to identify which pairs differ. Common in comparing customer satisfaction scores, response times, or order values across multiple segments. Full Kruskal-Wallis guide →

Fisher's Exact Test

Use Fisher's Exact Test for 2x2 contingency tables when sample sizes are small (small expected cell counts). Unlike the Chi-Square test, Fisher's computes the exact probability rather than relying on an approximation, so it is always valid regardless of sample size. Use it for rare events, small A/B tests, or medical/safety data where precision matters more than computational convenience. Full Fisher's Exact guide →

McNemar's Test

Use McNemar's test when you have paired binary outcomes: the same subjects measured on a yes/no variable at two time points. It tests whether the proportion of "yes" changed between measurements. Common applications include before/after surveys ("Did you purchase? Yes/No"), diagnostic test comparisons, and matched case-control studies. Full McNemar's guide →

Kolmogorov-Smirnov Test

Use the Kolmogorov-Smirnov (KS) test to compare an observed distribution against a theoretical distribution (one-sample) or to compare two observed distributions (two-sample). It is sensitive to differences in both location and shape. In business analytics, the KS test is widely used to check model calibration, detect data drift, and check normality assumptions before applying parametric tests. Full KS test guide →

Power Analysis

Use power analysis before you run any test to determine how many observations you need to detect a meaningful effect. It links four quantities: sample size, effect size, significance level (alpha), and statistical power (1 - beta). A study may give an unclear result if it lacks enough data to detect an effect that matters. Run power analysis during experiment design, not after. Full power analysis guide →

Decision Flowchart: Choosing the Right Test

Start at the top and follow the branches based on your data characteristics.

Multiple Comparisons: When and How to Correct

Every time you run a hypothesis test at alpha = 0.05, you accept a 5% chance of a false positive. Run 20 tests on the same dataset and you expect one spurious "significant" result even if no real effects exist. Multiple comparison corrections control this inflation so your findings remain trustworthy.

When You Must Correct

Some form of correction is usually needed when you test multiple hypotheses on the same dataset and report any result as significant; how you define the family of tests matters. This includes: post-hoc pairwise comparisons after ANOVA, testing multiple outcome metrics in a single experiment, subgroup analyses that were not pre-specified, and running the same test across many segments (e.g., testing conversion lift in each of 15 countries).

Bonferroni Correction

The simplest and most conservative approach: divide your significance threshold by the number of tests. With 10 comparisons at alpha = 0.05, each individual test must reach p < 0.005 to be declared significant. Easy to implement and explain, but overly strict when you have many tests: it dramatically increases the risk of missing real effects. Best for small numbers of planned comparisons where false positives are costly. Full Bonferroni guide →

Holm-Bonferroni Method

A step-down procedure that is uniformly more powerful than Bonferroni while still controlling the family-wise error rate. Sort p-values from smallest to largest and compare each to a progressively less strict threshold. It rejects more true effects than Bonferroni with the same Type I error guarantee, making it the recommended default for most applications. There is no reason to use plain Bonferroni when Holm-Bonferroni is available. Full Holm-Bonferroni guide →

Benjamini-Hochberg Procedure

Controls the false discovery rate (FDR) rather than the family-wise error rate, making it less conservative and more suitable for exploratory analyses with many tests. Instead of guaranteeing that no false positive slips through, it controls the expected proportion of false positives among rejected hypotheses. Use it when you are screening many variables for potential effects (e.g., which of 200 product features predict churn) and can tolerate some false leads in exchange for detecting more real effects. Full Benjamini-Hochberg guide →

Beyond Frequentist Testing

Classical hypothesis tests answer a narrow question: "Is this result unlikely under the null hypothesis?" Two complementary approaches provide richer answers for business decision-making.

Bayesian A/B Testing

Instead of a binary significant/not-significant verdict, Bayesian A/B testing gives you a probability that one variant beats another, for example "there is a 94% probability that the new checkout flow increases revenue." This maps directly to business risk decisions and does not require fixed sample sizes, making it ideal for continuous experimentation programs. Full Bayesian A/B testing guide →

Causal Impact Analysis

When you cannot run a randomized experiment (a policy change affects all customers simultaneously, or you need to measure the impact of an external event), Causal Impact uses Bayesian structural time series to construct a synthetic counterfactual. It estimates what would have happened without the intervention and quantifies the causal effect with credible intervals. Essential for measuring marketing campaigns, product launches, and operational changes that lack a clean control group.

Hypothesis Testing for Marketing & Business

A/B tests, campaign comparisons, and conversion rate optimization all rely on the statistical tests above. Whether you're testing a new checkout flow or comparing ad creatives, hypothesis testing tells you if the difference is real or just noise.

• Marketing Analytics, Analyze campaign A/B tests, measure statistical significance of conversion differences, and compare channel performance with confidence.