t-Test vs Mann-Whitney U: Which Should You Use?

Picture a made-up case. An analyst runs a Shapiro-Wilk test on A/B test data, gets a small p-value, and switches from a t-test to the Mann-Whitney U. The result changes from significant to not significant, and the analyst concludes the effect is not real. The analyst may have made the wrong call by mechanically following the "test for normality first" recipe, because the t-test can still work with non-normal data and a failed normality test alone does not settle the choice.

Choosing between the t-test and the Mann-Whitney U is one of the most common decisions in applied statistics, and it is frequently made poorly. The textbook rule, "use Mann-Whitney when data is non-normal", oversimplifies the decision. This guide explains what each test compares, when a t-test may work with non-normal data, and how to choose a test that fits the question.

Worked Example (Simulated): Do Checkout Times Differ Between Two Flows?

Question: Do checkout times differ between two flows?

Inputs: Simulated data, seed 20261029. The script makes two independent groups of 30 checkout times in seconds. It uses a lognormal draw to make the times right-skewed. The gap is built in: the log-means are 4.0 for group A and 3.8 for group B. Group A has a higher mean and median in this run.

Method: Compare means with Welch's t-test. Compare rank order with the two-sided Mann-Whitney U test. The tie-adjusted chance is the share of all A and B pairs where A is higher, with ties counted as half.

import numpy as np
from scipy import stats

SEED = 20261029
rng = np.random.default_rng(SEED)
# Simulated checkout time in seconds for two independent flow groups.
a = rng.lognormal(mean=4.0, sigma=0.65, size=30)
b = rng.lognormal(mean=3.8, sigma=0.65, size=30)

welch = stats.ttest_ind(a, b, equal_var=False)
u = stats.mannwhitneyu(a, b, alternative='two-sided', method='auto')
# Tie-adjusted probability that a randomly picked A value exceeds B.
diffs = a[:, None] - b[None, :]
p_super = (np.sum(diffs > 0) + 0.5 * np.sum(diffs == 0)) / diffs.size

print(f'Simulated data, seed {SEED}')
print(f'Each group: n={len(a)} independent sessions; unit=seconds')
print(f'A mean={np.mean(a):.2f}; median={np.median(a):.2f}')
print(f'B mean={np.mean(b):.2f}; median={np.median(b):.2f}')
print(f"Welch t={welch.statistic:.3f}; p={welch.pvalue:.4f}")
print(f'Mann-Whitney U={u.statistic:.1f}; two-sided p={u.pvalue:.4f}')
print(f'Tie-adjusted P(A > B)={p_super:.3f}')

Printed output:

Simulated data, seed 20261029
Each group: n=30 independent sessions; unit=seconds
A mean=68.42; median=54.55
B mean=51.51; median=45.07
Welch t=1.727; p=0.0899
Mann-Whitney U=552.0; two-sided p=0.1335
Tie-adjusted P(A > B)=0.613

Result: In this run, the Welch test gives p=0.0899 and the Mann-Whitney test gives p=0.1335. The two flows differ by design, and with 30 sessions each neither test finds it at the usual 0.05 cut-off, which is a convention the analyst picks. That is a missed gap, not proof of no gap. The pair chance is 0.613, so A is higher in more than half of the pairs in this sample.

Limit: These are made-up times and one sample. They do not show how real users would act, and they do not show that a flow caused a change.

Sources

Each source is cited for which test is which. No number on this page rests on them.

• Student (1908). The Probable Error of a Mean. Biometrika. doi 10.2307/2331554. The t-test.
• Welch B (1947). The generalization of "Student's" problem when several different population variances are involved. Biometrika. doi 10.1093/biomet/34.1-2.28. The unequal-variance t-test.
• Mann H, Whitney D (1947). On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. Annals of Mathematical Statistics. doi 10.1214/aoms/1177730491. The Mann-Whitney U test.

What Each Test Does

The Independent Samples t-Test

The t-test compares the means of two independent groups. It calculates the difference between group means, divides by the standard error of that difference, and checks whether the resulting t-statistic is large enough to be unlikely under the null hypothesis of equal population means.

The t-test works best when the sampling distribution of the mean is close to normal. This is guaranteed when the data itself is normal. For non-normal data, the Central Limit Theorem says that, as the sample grows, the sampling distribution of the mean tends to become more normal. How fast this happens depends on the data shape, and strong skew and heavy tails can need much larger samples.

The Mann-Whitney U Test

The Mann-Whitney U (also called the Wilcoxon rank-sum test) works on ranks rather than raw values. It pools all observations from both groups, ranks them from smallest to largest, and tests whether one group tends to have higher ranks than the other.

The Mann-Whitney test uses ranks and does not assume normal data. It compares the groups' rank distributions. A location or median reading needs the groups to have the same shape. It also discards information about the magnitude of differences between values: the gap between ranks 1 and 2 is the same whether the actual values differ by 0.1 or 1000.

Side-by-Side Comparison

When the t-Test Is Robust Enough

The t-test can hold up better than many practitioners expect. How well it holds up depends on the data shape, the outliers, and the sample size.

The t-test works well even with non-normal data when:

• A larger sample. The Central Limit Theorem says that, as the sample grows, the sampling distribution of the mean tends to become more normal. The needed size depends on the data. Strong skew or heavy tails may need a larger sample, so check the data shape and outliers.
• Distributions are symmetric but non-normal. Symmetric distributions (uniform, bimodal symmetric, light-tailed) tend to pose fewer problems for the t-test, even with smaller samples.
• Sample sizes are equal. Equal group sizes make the t-test less sensitive to unequal variances. With unequal sizes plus non-normality, consider Welch's t-test.
• You care about mean differences. If the business question is about average performance (average revenue, mean response time), the t-test directly answers that question. Mann-Whitney answers a different question (stochastic dominance).

When Mann-Whitney Is Necessary

There are genuine scenarios where the Mann-Whitney is the correct choice:

• Ordinal data. Likert-scale responses (1-5 ratings), satisfaction rankings, or severity grades are ordinal. Ordinal values may not have equal steps between levels. A mean may not answer the question of interest, so consider whether a rank comparison fits the study question. Mann-Whitney compares rank distributions directly.
• Outlier-dominated data. If a few extreme values dominate the variance, the t-test loses power because the standard error inflates. Mann-Whitney ranks cap the influence of outliers. Revenue data with a few whale customers is a classic example.
• The research question is about ranks, not means. "Does treatment A tend to produce better outcomes than treatment B?" is a rank question. If you care about superiority rather than the size of the average difference, Mann-Whitney directly answers your question.
• Data is bounded or truncated. Floor and ceiling effects (e.g., a pain scale of 0-10 where many patients score 0 or 10) create non-normal distributions that violate t-test assumptions even with moderate samples.

Power and Outliers

Power is the chance a test finds a gap that is really there. It depends on the data, the sample size, and the size of the gap, and neither test is more powerful in every case. Because the Mann-Whitney test works on ranks, extreme values have limited influence on it. Outliers can inflate the variance the t-test uses. Revenue data with a few very large customers is one place where this matters.

Illustrative Numbers: A/B Test on Session Duration

These numbers are typed in by hand to show how the two tests can disagree. They are not from a real site. Two checkout flows are compared on time to complete checkout (seconds), 40 sessions per group.

The Data Problem

Session durations are right-skewed: most users finish in 30-90 seconds, but some abandon and return later (producing values of 300-900 seconds). The Shapiro-Wilk test rejects normality (p < 0.001).

t-Test Result

Group A mean: 78.3s (SD = 62.1)
Group B mean: 64.1s (SD = 55.8)
t(78) = 1.07, p = 0.287
Cohen's d = 0.24

Mann-Whitney Result

Group A median: 52s (IQR: 35-89)
Group B median: 41s (IQR: 28-72)
U = 604, p = 0.041
Rank-biserial r = 0.245

Here the Mann-Whitney detects a significant difference that the t-test misses. Why? The handful of extreme session durations (users who left and returned) inflate the standard deviations, making the t-test's denominator large. The Mann-Whitney, working on ranks, is unaffected by the magnitude of those outliers. For data like this, a rank test may fit the question better.

Decision Guide

Use the t-test when:

• Data is continuous (interval or ratio scale)
• The sample is large enough for the data shape (strong skew or heavy tails need more)
• The data are roughly symmetric
• You want to estimate the magnitude of the mean difference
• Equal (or similar) sample sizes
• No extreme outliers dominating variance

Use Mann-Whitney when:

• Data is ordinal (Likert scales, rankings)
• Small samples with severely skewed data
• Extreme outliers are present and meaningful (not errors)
• Data is bounded with floor/ceiling effects
• You care about rank ordering rather than mean differences
• Distribution shapes differ substantially between groups

Consider alternatives when:

• You have paired/matched data (use paired t-test or Wilcoxon signed-rank)
• You have 3+ groups (use ANOVA or Kruskal-Wallis)
• You want to control for covariates (use regression or ANCOVA)
• The outcome is binary (use chi-square or Fisher's exact test)