Standard Ab Test
Executive Summary

The Verdict

treatment vs control on converted

Observations
100000
Variants
2
Control Group
control
Best Variant
treatment
Absolute Difference
-0.0009
P Value
0.667
Significant
no
THE VERDICT: treatment lowers converted by -0.1 pts (rel -0.8%), p = 0.67 — NOT statistically significant at the 95% confidence level. In practice: the data cannot distinguish this gap from random chance — which is not the same as proving the variants are equal. At the observed effect size, roughly 2,024,060 rows per variant (~4,048,120 total) would give an 80% chance of detecting a real difference this large.
What this means— written from the computed numbers · yours to edit

The new landing page (treatment) does not convert better than the old one. Treatment converts at 11.86%, while control converts at 11.95%—a difference of −0.09 percentage points (−0.8% relative), p = 0.667. This difference is not statistically significant; the data cannot rule out random chance as the explanation. To reliably detect a real difference this small would require roughly 4,048,120 total observations (2,024,060 per variant).

Overview

Analysis Overview

A/B test of 100,000 observations across 2 variants of group.

N Observations100000
N Variants2
Outcome Typebinary
Control Groupcontrol
What this means— written from the computed numbers · yours to edit

This analysis compares conversion rates across 2 variants using a two-proportion z-test on 100,000 observations. The treatment is compared against control as the reference group. Each variant's conversion rate is reported with a 95% confidence interval (Wilson method), which defines the range of true conversion rates consistent with the observed data. A 95% confidence interval that excludes zero indicates statistical significance at the 95% level—meaning the difference is unlikely to be due to random sampling variation.

Data Preparation

Data Quality

Row filtering, variant-group screening, and outcome type detection.

Initial Rows100000
Final Rows100000
Rows Removed0
Variants Kept2
Variants Dropped0
What this means— written from the computed numbers · yours to edit

All 100,000 rows were retained; no observations were dropped. Both variants (treatment and control) were usable, with no groups removed. The outcome variable, converted, contains only 0/1 values and was correctly detected as a binary conversion flag, with success coded as 1. Data quality checks passed with no exclusions needed.

Data Table

Variant Summary

Sample size and conversion rate per group with 95% confidence intervals.

VariantNRate Or MeanCI LowCI High
treatment500550.11860.11580.1215
control499450.11950.11670.1224
What this means— written from the computed numbers · yours to edit

Control achieves the higher conversion rate at 11.95% (95% CI: 11.67%–12.24%, n=49,945), while treatment converts at 11.86% (95% CI: 11.58%–12.15%, n=50,055). The groups are balanced in size, with treatment slightly larger. The confidence intervals overlap substantially, consistent with the non-significant test result. Control's rate is the decisive figure here—it is the benchmark against which treatment is measured.

Visualization

Outcome by Variant

converted per group, with 95% confidence intervals as error bars.

What this means— written from the computed numbers · yours to edit

Treatment and control conversion rates sit 0.09 percentage points apart (0.1186 vs 0.1195), with treatment trailing. The 95% confidence intervals overlap visibly, spanning roughly 11.58%–12.24% across both variants. This overlap reflects the statistical test result: the formal two-proportion z-test (p = 0.667) confirms the difference is consistent with random variation and not a reliable distinction between the variants.

Data Table

Statistical Test Results

Every comparison vs control with uplift, 95% CI, and p-value.

ComparisonAbsolute DiffRelative Uplift PCTCI LowCI HighP ValueSignificant
treatment vs control-0.0009-0.8-0.00490.00310.667no
What this means— written from the computed numbers · yours to edit

The treatment vs control comparison yields an absolute difference of −0.0009 (−0.09 percentage points), a relative uplift of −0.8%, and a 95% confidence interval of [−0.0049, 0.0031]. The p-value is 0.667, well above the 0.05 significance threshold. The confidence interval includes zero, confirming no statistically significant difference. This is the only comparison in the test; 0 of 1 comparisons are significant.

Rate this report Was this the answer you needed?
The exact source that produced this report — yours to keep, read, and re-run.
Download PDF
How this was computed method · R source · citation
The code that did it

A/B Test Analysis — Is the Difference Real?

Takes row-level experiment data (one row per user/session) with a variant column and an outcome column, detects the outcome type, runs the right significance test, and reports the uplift with a 95% confidence interval and a plain-English verdict.

Why This Method?

The correct test depends on the data: a 0/1 conversion outcome needs a two-proportion test (chi-square + pairwise vs control for 3+ variants); a numeric outcome needs a Welch t-test (ANOVA + Holm-adjusted pairwise Welch for 3+ variants). Automating that choice removes the most common A/B analysis mistakes.

What This Analysis Covers

  • Per-variant conversion rate (Wilson CI) or mean (t-based CI)
  • Absolute + relative uplift vs control with a 95% CI on the difference
  • Overall + pairwise significance tests, Cohen's d for numeric outcomes
  • An honest verdict, including a rough 80%-power sample-size estimate

when the result is not significant

Standard Library

Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {variant, outcome}. All narrative is derived from the user's own column names and computed values.

suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))

Core Analysis Pipeline

compute_shared <- function(df, params, col_map = list()) {
  # === SHARED EXPORTS ===
  #   initial_rows/final_rows/rows_removed   $ row accounting
  #   variant_h / outcome_h    $ humanized names of the two mapped columns
  #   outcome_type             $ "binary" | "numeric"
  #   success_label/success_how $ how the binary success level was chosen
  #   kept_levels/dropped_levels $ variant groups used / excluded (reported)
  #   n_missing/n_uncoercible/n_dropped_variant_rows $ drop accounting
  #   control / treatments / best_variant $ group roles
  #   gs                       $ per-group stats list (n, est, ci, sd, x)
  #   variant_summary_df       $ variant, n, rate_or_mean, ci_low, ci_high
  #   outcome_by_variant_df    $ variant, value, ci_low, ci_high
  #   test_results_df          $ comparison, absolute_diff, relative_uplift_pct,
  #                              ci_low, ci_high, p_value, significant
  #   primary                  $ headline comparison (best variant vs control)
  #   overall                  $ overall test for 3+ variants (label, p) or NULL
  #   sig / verdict / power_n  $ significance flag, verdict text, 80%-power n
  #   method_name / p_adjust_note
  #   metrics / json_output
  # === /SHARED EXPORTS ===

  variant_h <- humanize_semantic("variant", col_map)
  outcome_h <- humanize_semantic("outcome", col_map)

Step 1: Validate mapping + drop rows with missing values

initial_rows <- nrow(df)
  if (!"variant" %in% names(df)) {
    stop(sprintf("column_mapping must map &#x27;variant' (your %s column).", variant_h))
  }
  if (!"outcome" %in% names(df)) {
    stop(sprintf("column_mapping must map &#x27;outcome' (your %s column).", outcome_h))
  }
  if (initial_rows < 20) {
    stop(sprintf("Only %d rows — an A/B test needs at least 20 rows(and 5+ per %s).",
                 initial_rows, variant_h))
  }

  v_all <- trimws(as.character(df$variant))
  o_all <- df$outcome
  o_is_logical <- is.logical(o_all)
  o_chr_all <- trimws(as.character(o_all))
  ok <- !is.na(v_all) & v_all != "" & !is.na(o_all) & !is.na(o_chr_all) & o_chr_all != ""
  n_missing <- sum(!ok)
  v <- v_all[ok]
  o_chr <- o_chr_all[ok]
  o_log <- if (o_is_logical) o_all[ok] else NULL

Step 2: Variant groups — keep the top 4 by size, drop groups with n<5

tab <- sort(table(v), decreasing = TRUE)
  top_levels <- names(tab)[seq_len(min(4L, length(tab)))]
  beyond_top <- setdiff(names(tab), top_levels)
  small <- top_levels[as.integer(tab[top_levels]) < 5]
  kept_levels <- setdiff(top_levels, small)
  dropped_levels <- c(beyond_top, small)
  sel <- v %in% kept_levels
  n_dropped_variant_rows <- sum(!sel)
  v <- v[sel]
  o_chr <- o_chr[sel]
  if (!is.null(o_log)) o_log <- o_log[sel]
  if (length(kept_levels) < 2) {
    stop(sprintf("After cleaning, %s has %d usable group(s) — need at least 2 variants with 5+ rows each.",
                 variant_h, length(kept_levels)))
  }

Step 3: Outcome type detection (binary conversion vs numeric value)

outcome_type <- NULL
  success_label <- NA_character_
  success_how <- ""
  n_uncoercible <- 0L
  y <- NULL
  if (o_is_logical) {
    y <- as.numeric(o_log)
    outcome_type <- "binary"
    success_label <- "TRUE"
    success_how <- sprintf("%s is a logical(TRUE/FALSE) column; success = TRUE.", outcome_h)
  } else {
    conv <- suppressWarnings(as.numeric(o_chr))
    if (sum(!is.na(conv)) >= 0.95 * length(o_chr)) {
      bad <- is.na(conv)
      n_uncoercible <- sum(bad)
      if (n_uncoercible > 0) {
        v <- v[!bad]; conv <- conv[!bad]
      }
      uu <- unique(conv)
      if (all(uu %in% c(0, 1))) {
        y <- conv
        outcome_type <- "binary"
        success_label <- "1"
        success_how <- sprintf("%s contains only 0/1 values — treated as a conversion flag; success = 1.",
                               outcome_h)
      } else {
        y <- conv
        outcome_type <- "numeric"
        success_how <- sprintf("%s is numeric — comparing group means.", outcome_h)
      }
    } else {
      lv <- sort(unique(o_chr))
      if (length(lv) != 2) {
        stop(sprintf("%s has %d distinct non-numeric values — map a 0/1 conversion flag, a two-level yes/no column, or a numeric value.",
                     outcome_h, length(lv)))
      }
      kw <- c("yes", "true", "converted", "1", "success", "clicked", "purchased")
      hit <- lv[tolower(lv) %in% kw]
      if (length(hit) >= 1) {
        success_label <- hit[1]
        success_how <- sprintf("%s has two values(&#x27;%s' / '%s'); success = '%s' (matched a success keyword).",
                               outcome_h, lv[1], lv[2], success_label)
      } else {
        success_label <- lv[2]
        success_how <- sprintf("%s has two values(&#x27;%s' / '%s'); success = '%s' (the alphabetically-last level — no obvious success keyword found).",
                               outcome_h, lv[1], lv[2], success_label)
      }
      y <- as.numeric(o_chr == success_label)
      outcome_type <- "binary"
    }
  }

Re-check group sizes after any outcome-row drops

tab2 <- table(v)
  final_small <- names(tab2)[as.integer(tab2) < 5]
  if (length(final_small) > 0) {
    dropped_levels <- c(dropped_levels, final_small)
    sel2 <- !(v %in% final_small)
    n_dropped_variant_rows <- n_dropped_variant_rows + sum(!sel2)
    v <- v[sel2]; y <- y[sel2]
    kept_levels <- setdiff(kept_levels, final_small)
  }
  if (length(kept_levels) < 2) {
    stop(sprintf("After cleaning, %s has fewer than 2 variants with 5+ rows — cannot run a comparison.",
                 variant_h))
  }
  if (outcome_type == "binary") {
    if (all(y == 1) || all(y == 0)) {
      stop(sprintf("%s is constant(every usable row = %s) — there is nothing to compare.",
                   outcome_h, y[1]))
    }
  } else if (is.na(var(y)) || isTRUE(var(y) == 0)) {
    stop(sprintf("%s has zero variance — there is nothing to compare.", outcome_h))
  }

  final_rows <- length(y)
  rows_removed <- initial_rows - final_rows
  if (final_rows < 20) {
    stop(sprintf("Only %d usable rows after cleaning — need at least 20.", final_rows))
  }

Step 4: Per-variant estimates (Wilson CI for rates, t-based CI for means)

wilson_ci <- function(x, n, conf = 0.95) {
    z <- qnorm(1 - (1 - conf) / 2)
    p <- x / n
    denom <- 1 + z^2 / n
    centre <- (p + z^2 / (2 * n)) / denom
    half <- z * sqrt(p * (1 - p) / n + z^2 / (4 * n^2)) / denom
    c(max(0, centre - half), min(1, centre + half))
  }

Order groups by size (largest first)

grp_n <- as.integer(table(v)[kept_levels])
  names(grp_n) <- kept_levels
  kept_levels <- kept_levels[order(-grp_n)]
  grp_n <- grp_n[kept_levels]

  gs <- lapply(kept_levels, function(g) {
    yy <- y[v == g]
    n <- length(yy)
    if (outcome_type == "binary") {
      x <- sum(yy)
      ci <- wilson_ci(x, n)
      list(variant = g, n = n, est = x / n, ci_low = ci[1], ci_high = ci[2],
           x = x, sd = NA_real_)
    } else {
      m <- mean(yy)
      s <- sd(yy)
      half <- if (n > 1 && is.finite(s)) qt(0.975, n - 1) * s / sqrt(n) else NA_real_
      list(variant = g, n = n, est = m, ci_low = m - half, ci_high = m + half,
           x = NA_real_, sd = s)
    }
  })
  names(gs) <- kept_levels

Step 5: Control group — a level literally named control/a/baseline, else largest n

named_ctrl <- kept_levels[tolower(kept_levels) %in% c("control", "a", "baseline")]
  control <- if (length(named_ctrl) > 0) named_ctrl[1] else kept_levels[1]
  control_how <- if (length(named_ctrl) > 0) {
    sprintf("&#x27;%s' (named like a control group)", control)
  } else {
    sprintf("&#x27;%s' (the largest group)", control)
  }
  treatments <- setdiff(kept_levels, control)
  cg <- gs[[control]]

Step 6: Significance tests — pairwise vs control (+ overall for 3+ variants)

comp_rows <- list()
  for (g in treatments) {
    tg <- gs[[g]]
    d <- tg$est - cg$est
    eff <- NA_real_
    if (outcome_type == "binary") {
      pt <- tryCatch(suppressWarnings(prop.test(c(tg$x, cg$x), c(tg$n, cg$n))),
                     error = function(e) NULL)
      p <- if (!is.null(pt)) pt$p.value else NA_real_
      ci <- if (!is.null(pt)) as.numeric(pt$conf.int) else c(NA_real_, NA_real_)
    } else {
      tt <- tryCatch(t.test(y[v == g], y[v == control]), error = function(e) NULL)
      p <- if (!is.null(tt)) tt$p.value else NA_real_
      ci <- if (!is.null(tt)) as.numeric(tt$conf.int) else c(NA_real_, NA_real_)
      sp <- sqrt(((tg$n - 1) * tg$sd^2 + (cg$n - 1) * cg$sd^2) / (tg$n + cg$n - 2))
      eff <- if (is.finite(sp) && sp > 0) d / sp else NA_real_
    }
    rel <- if (is.finite(cg$est) && cg$est != 0) 100 * d / cg$est else NA_real_
    comp_rows[[g]] <- list(comparison = paste(g, "vs", control), variant = g,
                           d = d, rel = rel, ci_low = ci[1], ci_high = ci[2],
                           p = p, effect = eff)
  }

  p_adjust_note <- ""
  if (outcome_type == "numeric" && length(treatments) > 1) {
    ps <- sapply(comp_rows, function(r) r$p)
    adj <- p.adjust(ps, method = "holm")
    for (i in seq_along(comp_rows)) comp_rows[[i]]$p <- as.numeric(adj[i])
    p_adjust_note <- "Pairwise p-values are Holm-adjusted for multiple comparisons."
  }

  overall <- NULL
  if (length(kept_levels) > 2) {
    if (outcome_type == "binary") {
      ch <- tryCatch(suppressWarnings(chisq.test(table(v, y))), error = function(e) NULL)
      if (!is.null(ch)) overall <- list(label = "Overall(chi-square, all variants)",
                                        p = as.numeric(ch$p.value))
    } else {
      av <- tryCatch(summary(aov(y ~ factor(v))), error = function(e) NULL)
      p_over <- tryCatch(av[[1]][["Pr(>F)"]][1], error = function(e) NA_real_)
      if (!is.na(p_over)) overall <- list(label = "Overall(one-way ANOVA, all variants)",
                                          p = as.numeric(p_over))
    }
  }

  method_name <- if (outcome_type == "binary") {
    if (length(kept_levels) == 2) "two-proportion z-test(prop.test)"
    else "chi-square across all variants + pairwise two-proportion tests vs control"
  } else {
    if (length(kept_levels) == 2) "Welch two-sample t-test with Cohen&#x27;s d"
    else "one-way ANOVA + pairwise Welch t-tests vs control(Holm-adjusted)"
  }

Step 7: Headline comparison — the best-performing treatment vs control

ests <- sapply(treatments, function(g) gs[[g]]$est)
  ests <- ests[!is.na(ests)]          # never index which.max over possible NAs
  best_variant <- if (length(ests) > 0) names(ests)[which.max(ests)] else treatments[1]
  primary <- comp_rows[[best_variant]]
  sig <- isTRUE(!is.na(primary$p) && primary$p < 0.05)

80%-power sample-size estimate for the observed effect (only quoted when the result is not significant); tryCatch — skipped entirely on error.

power_n <- NULL
  if (!sig) {
    power_n <- tryCatch({
      n_est <- if (outcome_type == "binary") {
        ceiling(power.prop.test(p1 = cg$est, p2 = gs[[best_variant]]$est,
                                power = 0.80, sig.level = 0.05)$n)
      } else {
        tg <- gs[[best_variant]]
        sp <- sqrt(((tg$n - 1) * tg$sd^2 + (cg$n - 1) * cg$sd^2) / (tg$n + cg$n - 2))
        ceiling(power.t.test(delta = abs(primary$d), sd = sp,
                             power = 0.80, sig.level = 0.05)$n)
      }
      if (is.finite(n_est) && n_est > 0) n_est else NULL
    }, error = function(e) NULL)
  }

Step 8: The verdict (plain English, fully computed)

fmt_p <- function(p) {
    if (is.na(p)) "p unavailable"
    else if (p < 0.001) "p < 0.001"
    else paste0("p = ", signif(p, 2))
  }
  verb <- if (primary$d >= 0) "lifts" else "lowers"
  if (outcome_type == "binary") {
    gap_txt <- sprintf("%.1f pts", abs(100 * primary$d))
    rel_txt <- if (is.finite(primary$rel)) sprintf(" (rel %+.1f%%)", primary$rel) else ""
  } else {
    gap_txt <- sprintf("%.4g", abs(primary$d))
    rel_txt <- if (is.finite(primary$rel)) sprintf(" (rel %+.1f%%)", primary$rel) else ""
  }
  verdict <- sprintf("%s %s %s by %s%s%s, %s — %s at the 95%% confidence level.",
                     best_variant, verb, outcome_h,
                     if (primary$d >= 0) "+" else "-", gap_txt, rel_txt,
                     fmt_p(primary$p),
                     if (sig) "statistically significant" else "NOT statistically significant")
  if (!sig) {
    verdict <- paste0(verdict,
      " In practice: the data cannot distinguish this gap from random chance —",
      " which is not the same as proving the variants are equal.")
    if (!is.null(power_n)) {
      verdict <- paste0(verdict, sprintf(
        " At the observed effect size, roughly %s rows per variant(~%s total) would give an 80%% chance of detecting a real difference this large.",
        format(power_n, big.mark = ","), format(2 * power_n, big.mark = ",")))
    }
  }

Step 10: Metrics + machine answer

metrics <- list(
    `Observations`        = final_rows,
    `Variants`            = length(kept_levels),
    `Control Group`       = control,
    `Best Variant`        = best_variant,
    `Absolute Difference` = round(primary$d, 4),
    `P Value`             = signif(primary$p, 3),
    `Significant`         = if (sig) "yes" else "no"
  )

  json_output <- list(
    answer = paste0(
      "A/B test on ", format(final_rows, big.mark = ","), " rows across ",
      length(kept_levels), " variants of ", variant_h, ", measuring ", outcome_h,
      " (", if (outcome_type == "binary") "conversion rate" else "numeric mean", "). ",
      "Method: ", method_name, "; control = ", control_how, ". ", verdict,
      if (!is.null(overall)) paste0(" ", overall$label, ": ", fmt_p(overall$p), ".") else ""
    ),
    cards = lapply(
      c("tldr", "overview", "preprocessing", "variant_summary",
        "outcome_by_variant", "test_results"),
      function(cid) list(id = cid, metrics = metrics)
    )
  )

  list(
    initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
    variant_h = variant_h, outcome_h = outcome_h,
    outcome_type = outcome_type, success_label = success_label, success_how = success_how,
    kept_levels = kept_levels, dropped_levels = dropped_levels,
    n_missing = n_missing, n_uncoercible = n_uncoercible,
    n_dropped_variant_rows = n_dropped_variant_rows,
    control = control, control_how = control_how,
    treatments = treatments, best_variant = best_variant,
    gs = gs, comp_rows = comp_rows, primary = primary, overall = overall,
    sig = sig, verdict = verdict, power_n = power_n,
    method_name = method_name, p_adjust_note = p_adjust_note,
    variant_summary_df = variant_summary_df,
    outcome_by_variant_df = outcome_by_variant_df,
    test_results_df = test_results_df,
    metrics = metrics, json_output = json_output
  )
}
Your data has more stories to tell.Run any analysis on your own data — validated R modules, interactive reports, AI insights, and PDF export. 500 free credits on signup.
Try Free — No SignupSign Up Free
Download PDF · Print · Share ·Refresh this analysis ·Edit this report

Cite this analysis

Report an Issue

Tell us what's wrong. You'll get a free re-run of this analysis so you can try again with different parameters. If the re-run still doesn't meet your expectations, we'll refund your credits.

Want to run this analysis on your own data? Upload CSV — Free Analysis See Pricing