Two Proportion Z Test: A Practical Guide to Comparing Conversion Rates

You run an A/B test comparing two landing pages. Variant A converts at 3.2%, variant B at 3.8%. Your stakeholder asks: "Is B really better, or is this just noise?" You need an answer backed by proper methodology, not guesswork. That's what the two proportion z test gives you: a rigorous statistical test for comparing rates, conversion percentages, and proportions across two independent groups.

This article cuts through the theory. It gives a simulated worked example with a complete Python script, then covers the common ways proportion tests go wrong: linked observations, stopping early and too little data.

Worked Example: A Simulated Two-Group Test with Code

Question: In simulated data, did group B have a higher rate than group A?

Inputs: The script draws 1,200 results per group from made-up rates of 7% for A and 9% for B. The data are simulated, not from real users. The script built in a 2 point gap, so we know the true answer.

Method: A pooled, two-sided two-proportion z test at alpha 0.05. The interval uses each group's own rate. The script also computes an approximate power figure for a separate case.

Complete Python script:

import numpy as np
from scipy.stats import norm

SEED = 20261029
N = 1200
rng = np.random.default_rng(SEED)
# Simulated independent outcomes: A rate 7%, B rate 9%.
a = rng.binomial(1, 0.07, N)
b = rng.binomial(1, 0.09, N)
x_a, x_b = int(a.sum()), int(b.sum())
p_a, p_b = x_a / N, x_b / N
pooled = (x_a + x_b) / (2 * N)
se_null = np.sqrt(pooled * (1 - pooled) * (2 / N))
z = (p_b - p_a) / se_null
p_value = 2 * norm.sf(abs(z))
se_ci = np.sqrt(p_a * (1 - p_a) / N + p_b * (1 - p_b) / N)
ci_low = (p_b - p_a) - norm.ppf(0.975) * se_ci
ci_high = (p_b - p_a) + norm.ppf(0.975) * se_ci

# Approximate power for a two-sided pooled two-proportion z test.
n = 200
p1, p2, alpha = 0.05, 0.07, 0.05
p_null = (p1 + p2) / 2
se_null_power = np.sqrt(p_null * (1 - p_null) * (2 / n))
se_alt = np.sqrt(p1 * (1 - p1) / n + p2 * (1 - p2) / n)
critical = norm.ppf(1 - alpha / 2)
mean_z = (p2 - p1) / se_null_power
ratio = se_alt / se_null_power
power = (norm.cdf((-critical - mean_z) / ratio)
         + norm.sf((critical - mean_z) / ratio))

print(f"Simulated data, seed {SEED}")
print(f"Input rates: A = {0.07:.4%}; B = {0.09:.4%}; n = {N} per group")
print(f"A: {x_a}/{N} = {p_a:.4%}; B: {x_b}/{N} = {p_b:.4%}")
print(f"B minus A: {(p_b-p_a)*100:.3f} percentage points")
print(f"Pooled two-sided z test: z = {z:.3f}, p = {p_value:.4f}, alpha = 0.05")
print(f"Unpooled normal 95% CI for B minus A: "
      f"[{ci_low*100:.3f}, {ci_high*100:.3f}] percentage points")
print(f"Approximate power at n = {n} per group, rates 5% and 7%, "
      f"two-sided alpha = 0.05: {power:.1%}")

Real output from running the script with python3:

Simulated data, seed 20261029
Input rates: A = 7.0000%; B = 9.0000%; n = 1200 per group
A: 85/1200 = 7.0833%; B: 117/1200 = 9.7500%
B minus A: 2.667 percentage points
Pooled two-sided z test: z = 2.353, p = 0.0186, alpha = 0.05
Unpooled normal 95% CI for B minus A: [0.448, 4.886] percentage points
Approximate power at n = 200 per group, rates 5% and 7%, two-sided alpha = 0.05: 13.4%

Result: In this run, B was 2.667 percentage points above A (85/1200 = 7.083% against 117/1200 = 9.750%). The z statistic was 2.353 and the p-value was 0.0186, below 0.05. The 95% interval ran from 0.448 to 4.886 percentage points. The script built in a 2 point gap; the run's 2.667 is sampling noise around it, and the interval includes the true 2. In this simulated run the test gives evidence of a difference. It does not show that B will do better in a real campaign.

The last output line is a separate calculation, not part of the main test: it uses rates of 5% and 7% with 200 per group, and gives about 13.4% power. It is a normal approximation and depends on the test and its assumptions.

Limit: These are simulated results. A normal approximation may be unreliable for some data, and the example does not account for repeated users or other design problems.

When You Need a Two Proportion Z Test

Use the two proportion z test when you're comparing a binary outcome (yes/no, success/failure, converted/didn't convert) across two independent groups. The research question takes the form: "Is the rate in Group A different from the rate in Group B?"

Common applications:

• A/B testing: Does the new checkout flow increase purchase completion rate?
• Clinical studies: Does treatment reduce infection rate compared to placebo?
• Marketing experiments: Does email subject line A increase open rate vs subject line B?
• Quality control: Does Supplier A have a lower defect rate than Supplier B?
• Survey research: Do men and women respond "yes" to this question at different rates?

The key requirement: independent samples. Each observation must appear in exactly one group. If you're tracking the same users before and after a change, you don't have independent samples, you need McNemar's test for paired proportions instead.

The Hidden Implementation Details That Matter

Most proportion test tutorials give you the formula and call it done. But the difference between a valid test and garbage results lies in the setup. Here's what actually matters in practice.

Pooled vs Unpooled: When the Math Changes

The two proportion z test has a quirk: you calculate the standard error differently depending on whether you're testing a hypothesis or building a confidence interval.

For hypothesis testing (testing if p₁ = p₂), use the pooled proportion:

p_pooled = (x₁ + x₂) / (n₁ + n₂)
SE_pooled = sqrt(p_pooled * (1 - p_pooled) * (1/n₁ + 1/n₂))
z = (p₁ - p₂) / SE_pooled

Why pool? Under the null hypothesis, you're assuming the two proportions are equal. That they come from the same population. So you combine the data to estimate that common proportion.

For confidence intervals around the difference, use unpooled proportions:

SE_unpooled = sqrt(p₁(1-p₁)/n₁ + p₂(1-p₂)/n₂)
CI = (p₁ - p₂) ± z* × SE_unpooled

Why unpool? You're estimating the actual difference between two potentially different proportions, not testing if they're equal. Each group gets its own variance estimate.

Use the right approach for your question.

Sample Size and the Normal Approximation

The two proportion z test relies on a normal approximation to the binomial distribution. That approximation can be unreliable when counts are small.

Example: You have 100 visitors to variant A, 3 converted (3%). Can you test this?

• Successes: 3 (very few)
• Failures: 97

Be careful. With only 3 conversions, the normal approximation may be unreliable and the p-value may not be trustworthy. You need either more traffic or a different test (exact tests like Fisher's exact test work with small counts).

Planning sample size before running the test prevents this problem.

One-Tailed vs Two-Tailed: Choose Before You See Data

A two-tailed test asks: "Is there any difference between the groups?"
A one-tailed test asks: "Is Group A specifically higher than Group B?"

The one-tailed test has more statistical power for detecting effects in the predicted direction, but you must choose the direction before seeing the data. You can't look at results, notice B is higher than A, and then run a one-tailed test for "B > A", that's p-hacking.

Use one-tailed when: You only care about improvement in one direction (e.g., new treatment must be better than control, not just different)

Use two-tailed when: Any difference matters, or you're not sure which direction to expect

Default to two-tailed unless you have a strong theoretical reason for one-tailed. It's the more conservative choice.

Hand Calculation: Comparing Email Click-Through Rates

Let's work through a made-up scenario step by step, with the arithmetic shown. The counts are invented for illustration and are not the simulated data from the script above. You're testing two email subject lines to see which drives more click-throughs.

Setup:

• Group A (Control): "Your Weekly Newsletter" sent to 1,200 subscribers, 84 clicked through
• Group B (Variant): "5 Tools You Missed This Week" sent to 1,200 subscribers, 108 clicked through
• Research question: Does subject line B increase click-through rate compared to A?

Step 1: Check assumptions

• Independence: Each subscriber received only one email (randomized assignment). ✓
• Sample size: Group A has 84 successes and 1,116 failures. Group B has 108 successes and 1,092 failures. Neither group has tiny counts, so the normal approximation is reasonable here. ✓
• Randomization: Subscribers were randomly assigned to each variant. ✓

Step 2: Calculate observed proportions

p_A = 84 / 1200 = 0.070 (7.0%)
p_B = 108 / 1200 = 0.090 (9.0%)
Difference = 0.090 - 0.070 = 0.020 (2.0 percentage points)

Step 3: Calculate pooled proportion and standard error

p_pooled = (84 + 108) / (1200 + 1200) = 192 / 2400 = 0.080

SE_pooled = sqrt(0.080 × 0.920 × (1/1200 + 1/1200))
          = sqrt(0.0736 × 0.001667)
          = sqrt(0.0001227)
          = 0.01108

Step 4: Calculate z-statistic

z = (0.090 - 0.070) / 0.01108
  = 0.020 / 0.01108
  = 1.805

Step 5: Find p-value

For a two-tailed test with z = 1.805:
p-value = 0.071

Step 6: Interpret results

At α = 0.05, we fail to reject the null hypothesis (p = 0.071 > 0.05). While variant B had a 2 percentage point higher click-through rate, this difference is not statistically significant at the conventional threshold.

Practical interpretation: In this illustrative example, a 29% relative increase in click-through rate (from 7% to 9%) was observed, but with these sample sizes, random chance cannot be ruled out. You have three options:

1. Run the test longer to increase sample size and gain more power
2. Accept the suggestive evidence and implement variant B (acknowledging the uncertainty)
3. Consider the practical significance. A 2 point lift may be meaningful even without statistical certainty

Three Common Mistakes in Proportion Tests

Here are three common ways proportion tests go wrong, and how to avoid each.

Mistake 1: Non-Independent Observations

The error: Running a proportion test on data where the same subjects appear in both groups, or where observations within a group are correlated.

Hypothetical example: A product team tests a new feature by showing it to 500 users in Week 1 (20% adoption) and the same 500 users in Week 2 after improvements (35% adoption). They run a two proportion z test and declare the improvement significant.

Why it's wrong: The same users appear in both samples. Users who adopted in Week 1 are likely to adopt in Week 2 (they're correlated). The two proportion z test assumes independent samples. That each observation in Group A is unrelated to observations in Group B.

The fix: Use McNemar's test for paired binary data, or collect independent samples (different users in each group).

Mistake 2: Stopping Early Because You Hit Significance

The error: Planning to run an A/B test for two weeks, checking p-values daily, and stopping the test early when p < 0.05.

Why it's wrong: This is called "peeking" or "optional stopping," and it inflates your false positive rate. Each extra look gives you another chance to find a significant result by chance, so the real false positive rate ends up above the nominal 5%.

The fix: Pre-specify your sample size and run the test to completion, or use sequential testing methods designed for continuous monitoring (like alpha spending functions or group sequential designs).

Mistake 3: Running Underpowered Tests

The error: Running an A/B test with 200 observations per group, failing to find significance (p = 0.18), and concluding "there's no difference between variants."

Why it's wrong: A non-significant result doesn't prove the groups are equal. It might just mean you didn't have enough data to detect the difference. For a two-sided pooled z test at alpha 0.05, the script's normal approximation above gives about 13.4% power with 200 observations per group and rates of 5% and 7%. The estimate depends on the test and its assumptions. A real 2 point increase would usually be missed.

The fix: Calculate required sample size before running the test. For proportion tests:

n = (z_α/2 + z_β)² × [p₁(1-p₁) + p₂(1-p₂)] / (p₁ - p₂)²

Where z_α/2 = 1.96 for α = 0.05 two-tailed, and z_β = 0.84 for 80% power, a common choice.

Example calculation: To detect a change from 5% to 7% (2 point increase) with 80% power at α = 0.05:

n = (1.96 + 0.84)² × [0.05(0.95) + 0.07(0.93)] / (0.05 - 0.07)²
  = 7.84 × [0.0475 + 0.0651] / 0.0004
  = 7.84 × 0.1126 / 0.0004
  = 2,207 per group

This formula gives 2,207 observations per variant for an 80% chance of detecting this difference, assuming the formula's normal approximation holds. A test with only 200 per group is far below that.

Statistical Significance vs Practical Significance

Here's a scenario that confuses teams: an A/B test with a very large sample finds a tiny difference, say 0.2 percentage points, and the two proportion z test calls it statistically significant.

Your stakeholder asks: "Should we implement variant B?"

The statistical test says "these are different," but is a 0.2 percentage point increase worth the engineering effort to implement? That's a question statistics can't answer. It's about practical significance.

Statistical significance tells you whether the difference is likely real (not due to chance).
Practical significance tells you whether the difference matters for your business.

With large samples, tiny differences become statistically significant. A 0.1% lift in conversion rate might be "significant" with p < 0.001 but completely irrelevant in practice. Conversely, a 5% lift might be hugely meaningful even if p = 0.08 (not quite significant at α = 0.05).

Always report both:

• The statistical test result (p-value, confidence interval)
• The practical magnitude (absolute difference, relative lift, business impact)

Let stakeholders decide if the effect size is worth acting on, given the statistical uncertainty.

Power Analysis: Designing Tests That Can Actually Detect Effects

A common question after "is this significant?" is "how much data do I need?" That's a power analysis question. Statistical power is the probability of detecting a real effect when it exists: your test's ability to avoid false negatives.

A common choice is 80% power: if a real difference exists, you'd detect it 80% of the time. Going higher (90%, 95%) requires larger samples but reduces the risk of missing real effects.

Inputs Required for Sample Size Calculation

To calculate required sample size for a two proportion z test, you need four inputs:

1. Baseline rate (p₁): Current conversion rate, typically from historical data
2. Minimum detectable effect (MDE): Smallest difference worth detecting
3. Significance level (α): False positive rate you'll tolerate (a common choice is 0.05)
4. Power (1-β): Probability of detecting the effect if real (a common choice is 0.80)

The tricky one is MDE. This requires a business decision: what's the smallest lift that would change your decision? If a 1% absolute increase in conversion rate isn't worth the implementation cost, don't power your test to detect 1%. You're wasting sample size on differences you won't act on.

What to Do When You Can't Get Enough Data

Sometimes the required sample size exceeds what's realistically available. You calculate that you need 5,000 users per variant, but your traffic is limited. Options:

1. Run the test longer: keep collecting until you reach the planned sample size
2. Accept lower power: a lower power target needs smaller samples, but you're more likely to miss real effects
3. Target larger effects: If you power for a larger difference, the required sample size drops
4. Don't run the test: If you can't get adequate power, the test is unlikely to be conclusive: consider qualitative research or ship-and-monitor instead

Never run an underpowered test and interpret non-significance as "no effect." You're just wasting time and data.

Alternative Approaches: When Two Proportion Z Test Isn't Right

The two proportion z test is powerful but not universal. Here's when to use something else.

Fisher's Exact Test for Small Samples

When you have small counts, the normal approximation may be unreliable. Fisher's exact test calculates exact probabilities without relying on approximations.

Use when: Counts are small

Example: Testing a rare adverse event in a clinical study. Treatment group: 2/50 patients experienced the event. Control: 0/50. Fisher's exact test gives exact p-value without assuming normality.

McNemar's Test for Paired Data

When the same subjects provide data for both conditions (before/after, matched pairs), you have dependent samples. The two proportion z test assumes independence.

Use when: Same subjects measured twice, or matched pairs design

Example: 100 users try Feature A, then try Feature B. You measure which feature each user prefers. McNemar's test compares the proportion who switched preferences.

Chi-Square Test for More Than Two Groups

Comparing proportions across three or more groups? The chi-square test extends naturally to multiple groups.

Use when: Comparing conversion rates across 3+ groups simultaneously

Example: Testing four different email subject lines (A, B, C, D) in a single experiment. Chi-square tests if click-through rates differ across all four groups.

Note: For exactly two groups, chi-square and the two proportion z test are mathematically equivalent (χ² = z²).

Logistic Regression for Controlling Confounders

When you need to compare proportions while controlling for other variables (age, gender, baseline characteristics), logistic regression gives you adjusted estimates.

Use when: Groups differ on important characteristics you need to control for

Example: Comparing conversion rates between two marketing channels, but the channels attract different demographics. Logistic regression lets you estimate the channel effect while controlling for age, income, and location.

Reporting Results: What to Include in Your Analysis

A complete proportion test analysis reports more than just a p-value. Here's what to include for a defensible, reproducible result:

Essential Elements

1. Sample sizes and counts: n₁, n₂, x₁, x₂ for both groups
2. Observed proportions: p₁ and p₂ with percentages
3. Difference: Both absolute (p₁ - p₂) and relative ((p₁ - p₂)/p₂) differences
4. Test statistic and p-value: z-statistic and two-tailed p-value
5. Confidence interval: 95% CI for the difference
6. Effect size: Relative risk or odds ratio for interpretability
7. Assumptions check: Note that independence, sample size, and randomization assumptions were met

Example Report Template

Notice what this includes: the context, the data, the statistical results, the practical interpretation, and the recommended action. That's a complete analysis.

Your turn

Bring your own data and the question you actually need answered.

MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai

Frequently Asked Questions

What's the difference between a two proportion z test and a chi-square test?

For comparing two proportions, they're mathematically equivalent: the chi-square statistic equals the z-statistic squared. The z-test gives you a signed statistic (showing direction) and works naturally for one-tailed tests. Chi-square is always positive and better suited for multiple groups. Use the z-test when you have exactly two groups and care about direction.

How large should my sample size be for a two proportion z test?

The test uses a normal approximation, which can be unreliable when counts are small, so plan your sample size before you run the test. Smaller differences need larger samples. A power calculation takes your baseline rate, the smallest difference worth detecting, your alpha and your target power; the page's power section shows the formula and one worked case.

When should I use pooled vs unpooled variance for proportion tests?

For hypothesis testing (testing if p1 = p2), use the pooled proportion. You're assuming the null hypothesis is true, so you pool the data. For confidence intervals around the difference, use unpooled proportions since you're estimating the actual difference, not testing equality. Most statistical software handles this automatically.

Can I use a two proportion z test for repeated measures or matched pairs?

No. The two proportion z test assumes independent samples. If the same users appear in both groups (matched pairs, before-after on same subjects), use McNemar's test instead. Independence matters: when observations are linked, the p-value can be unreliable.

What p-value should I use to determine statistical significance?

A common choice is α=0.05, which means a 5% false positive rate when the null hypothesis is true. Choose your threshold before running the test based on the cost of being wrong. High-stakes decisions might warrant α=0.01. Running multiple tests? Apply Bonferroni correction by dividing α by the number of comparisons. Never adjust your threshold after seeing the results.

Running Your Own Proportion Test

You now have the methodology, the assumptions, the common pitfalls, and the interpretation framework. The final step: running the analysis on your own data.

The MCP Analytics two proportion z test tool handles the complete workflow: upload your data as a CSV (one column for group, one for outcome), and you get instant results with proper pooled variance, assumption checks, confidence intervals, and power analysis. The tool flags when assumptions are violated and recommends alternatives.

Every report includes embedded R code showing exactly how the analysis was performed, fully reproducible, with fixed random seeds and documented methodology. You can cite the analysis in reports with auto-generated APA/MLA/BibTeX references.

The Bottom Line on Proportion Testing

Comparing proportions is one of the most common statistical tasks in business and research. The two proportion z test gives you clear answers, when you use it correctly.

The three rules that matter: verify independence, check that counts are not tiny, and plan your analysis before seeing results. Skip these checks and your p-values may mislead you. Follow them and your conclusions are easier to defend.

Statistical significance tells you if an effect is likely real. Practical significance tells you if it matters. Always report both. Let stakeholders make decisions with full context: the statistical uncertainty and the business impact.

Before you run your next A/B test: calculate required sample size, pre-specify your stopping rule, and verify randomization. After you get results: check assumptions, report confidence intervals, and interpret in context. That's how you run proportion tests that produce answers you can trust.