The Verdict
Which of the 2 variants of 'version' should you ship, and what does being wrong cost?
Gate 30 retains more players at 7 days, with 19.02% retention versus gate 40's 18.20%—a lift of 0.82 percentage points (95% credible interval 0.31 to 1.33 pp). Gate 30 is the best option with probability 0.999, meaning the posterior strongly favors it. The expected loss from shipping gate 30 is under 0.001 percentage points; staying on gate 40 costs 0.821 percentage points on average. The credible intervals do not overlap, and the entire interval for gate 30's advantage sits above zero, confirming the separation is real on this data.
Analysis Overview
Beta-Binomial posterior comparison of 2 variants of 'version' across 90,189 observations.
This analysis compares 2 gate placements across 90,189 player sessions using a Bayesian model that assigns each variant a full probability distribution over its true retention rate rather than a single point estimate. The 16,781 observed 7-day retentions are modeled as Binomial draws with Beta priors, producing exact posterior distributions: Beta(8503.00, 36199.00) for gate_30 and Beta(8280.00, 36199.00) for gate_40. Monte Carlo sampling from these posteriors (100,000 draws per variant, seed 42) generates decision-relevant probabilities: the chance each gate is best, the chance it beats its competitor, and the expected loss from shipping it. Every probability carries a Monte Carlo standard error of about 0.0001.
Data Quality
How the file was read, what was dropped, and how conversions were counted.
The file was read as one row per visitor, with 90,189 observations across 2 variants. Retention at 7 days was identified as TRUE in the 'retention 7' column and counted as a conversion. No rows were dropped for missing version or retention values, and no variants were excluded for small size. The pooled 7-day retention rate is 18.61% (16,781 conversions). The analysis assumes each row is an independent visitor; if the same player appears multiple times, those repeat visits are counted as separate rows, which would narrow confidence intervals below their true width.
Posterior Rates by Variant
Observed and posterior conversion rate per variant, each with a 95% credible interval.
| Variant | Trials | Conversions | Observed Rate PCT | Posterior Rate PCT | CI Low PCT | CI High PCT |
|---|---|---|---|---|---|---|
| gate_40 | 45489 | 8279 | 18.2 | 18.2 | 17.85 | 18.56 |
| gate_30 | 44700 | 8502 | 19.02 | 19.02 | 18.66 | 19.39 |
Gate 30 achieves a posterior 7-day retention rate of 19.022% (95% credible interval 18.659% to 19.387%) on 44,700 observations, while gate 40 posts 18.201% (95% CrI 17.848% to 18.557%) on 45,489 observations. Gate 30's interval is marginally wider (0.73 percentage points) than gate 40's (0.71 percentage points) despite similar sample sizes, a difference attributable to random sampling variation rather than structural imbalance. The posterior rates sit slightly closer together than raw rates because the uniform Beta(1.00, 1.00) prior pulls both estimates toward the center—a shrinkage that protects small arms but is negligible at these sample sizes.
Conversion Rate by Variant
Posterior conversion rate per variant with 95% credible intervals.
The short answer
Gate 30 stands clearly ahead of gate 40 in 7-day retention: 19.022% versus 18.201%, with no overlap between their credible intervals. Both are tightly pinned down because of large sample sizes.
The detail
The posterior conversion rates and 95% credible intervals are: gate 40 at 18.201% (17.848% to 18.557%) and gate 30 at 19.022% (18.659% to 19.387%). The interval widths are similar (0.71 pp and 0.73 pp respectively) because both gates have comparable sample sizes: 45,489 for gate 40 and 44,700 for gate 30. The absence of overlap means the two gates do not share any credible value for their true retention rate; the draw-by-draw comparison in the decision table confirms this separation translates to P(gate 30 beats gate 40) = 0.9993.
What this can't tell you
Visual separation of credible intervals is a rough guide; the precise comparison is the posterior probability that one rate exceeds the other, which appears in the decision table. The intervals assume each observation is independent; repeated visits by the same player would narrow them.
Posterior Lift vs Control
The full posterior distribution of the difference in conversion rate, in percentage points.
The short answer
Gate 30's lift over gate 40 centers at 0.82 percentage points, with 99.9% of the posterior above zero and 95% of the mass between 0.31 and 1.33 pp. The direction is settled; the magnitude is not.
The detail
The posterior distribution of the lift (gate 30 minus gate 40) in percentage points shows the bulk of probability centered around 0.82 pp, with 0.999 of the mass lying above zero. The 95% credible interval spans 0.31 to 1.33 pp, a range of 1.02 pp. This width means a decision that would flip somewhere inside that interval — for example, if your business value per percentage point of retention differed — cannot be made from this data alone. The fact that the entire interval sits above zero confirms gate 30 is ahead, but the exact size of the advantage carries residual uncertainty.
What this can't tell you
The posterior assumes the test ran to a fixed sample or that no stopping rule was applied. If the test was monitored and stopped at a threshold, the true lift distribution conditional on that stopping rule differs from what is shown here. A transaction-level or player-session export would help determine whether observations are truly independent.
Decision Table
Probability of beating the control, probability of being best, and the expected loss of shipping each variant.
| Variant | Prob Beats Control | Prob Best | Expected Loss Pp | Lift Pp | Lift CI Low Pp | Lift CI High Pp | Decision Note |
|---|---|---|---|---|---|---|---|
| gate_40 | — | 0.0007 | 0.8207 | 0 | — | — | The control. Shipping nothing costs an expected 0.821 percentage points of conversion rate against the best arm. |
| gate_30 | 0.9993 | 0.9993 | 0.0001 | 0.82 | 0.313 | 1.332 | Clears the conventional 0.95 bar for being best; the expected cost of shipping it is under 0.001 percentage points. |
The short answer
Gate 30 is the clear winner: it has a 0.9993 probability of being the best gate and an expected loss of under 0.001 percentage points if shipped. Staying on gate 40 carries an expected loss of 0.821 percentage points.
The detail
Gate 30: P(beats control gate 40) = 0.9993, P(best) = 0.9993, expected loss = 0.0001 pp, lift = 0.82 pp (95% CrI 0.31 to 1.33 pp). Gate 40 (control): P(best) = 0.0007, expected loss = 0.8207 pp. The expected loss column is the key decision metric: it quantifies the average conversion rate you forgo by shipping that variant when another is actually better. Gate 30's expected loss of under 0.001 pp is the smallest and clears the conventional 0.95 threshold for P(best).
What this can't tell you
The 0.95 threshold is a convention, not a law. Whether an expected loss of under 0.001 pp is acceptable depends on your business value per percentage point of retention — the analysis computes the risk, not the threshold. If the test was monitored and stopped when P(best) crossed 0.95, the reported 0.9993 overstates the accuracy of that decision rule.
Priors, Peeking, and What This Cannot Tell You
The prior's influence, a sensitivity check across three priors, and an honest account of the stopping-rule problem.
| Prior | Leader Under Prior | Prob Leader Is Best | Leader Rate PCT | Expected Loss Pp |
|---|---|---|---|---|
| Uniform Beta(1.00, 1.00) — the prior used above | gate_30 | 0.9993 | 19.02 | 0.0001 |
| Jeffreys Beta(0.50, 0.50) — the standard reference prior | gate_30 | 0.9993 | 19.02 | 0.0001 |
| Skeptical Beta centred on the pooled rate 18.61%, worth 50 pseudo-trials per variant | gate_30 | 0.9993 | 19.02 | 0.0001 |
The short answer
The prior contributes only 0.004% of gate 30's posterior weight, and the conclusion is robust across three different priors. Gate 30 remains the leader under all three, and P(best) stays at 0.9993. The posterior is valid regardless of monitoring, but stopping when a threshold is crossed introduces optimism that cannot be measured from this file.
The detail
The uniform prior Beta(1.00, 1.00) adds 2.00 pseudo-observations per variant against 90,189 total observations — negligible influence. Under a Jeffreys prior Beta(0.50, 0.50), gate 30 has P(best) = 0.9993 and posterior rate 19.021%. Under a skeptical prior Beta centered on the pooled rate (18.61%) worth 50 pseudo-trials per variant, gate 30 has P(best) = 0.9993 and posterior rate 19.02%. All three priors agree: gate 30 is best, with expected loss under 0.0001 pp. The posterior depends only on totals (16,781 conversions out of 90,189 observations), not on how many times you looked. That is a real advantage over fixed-horizon p-values. However, if you monitored the test and stopped the first time P(best) crossed a bar, the reported 0.999 overstates how often that decision rule would be right. Nothing in this data can measure the overstatement; the fix is to set the bar and minimum sample before looking.
What this can't tell you
The file does not record when the test was monitored or whether a stopping rule was applied. If a sequential stopping rule was used, the reported probabilities are valid as a Bayesian posterior but not as the long-run accuracy of that decision rule. Consider disclosing the stopping rule and minimum sample size in advance for future tests.
Methods & Disclosure
The model, the prior, the draws, the control, and the limits of what this analysis can decide.
| Item | Detail |
|---|---|
| Model | Beta-Binomial conjugate: each variant's conversion rate gets a Beta posterior with parameters (prior + conversions, prior + non-conversions). No model is fitted and no normal approximation is used, so the arms with few observations are handled correctly. |
| Prior | Beta(1.00, 1.00), uniform over the conversion rate. Posterior parameters run from Beta(8280.00, 36199.00) to Beta(8503.00, 37211.00) across the 2 variants. |
| Posterior draws | 100,000 draws per variant from each posterior, seed 42, so re-running this analysis on the same file returns the same probabilities. |
| Monte Carlo precision | The Monte Carlo standard error on gate_30's P(best) of 0.9993 is 0.0001, so the third decimal place is noise from the simulation rather than signal from the data. |
| Control group | 'gate_40' (the largest group, since no group is named like a control). Every lift, credible interval and P(beats control) in this report is measured against it. |
| Data shape | One row per visitor or session, aggregated to per-variant totals by this analysis. 'retention 7' is a TRUE/FALSE column; TRUE was counted as a conversion. |
| How conversions were counted | 16,781 conversions out of 90,189 observations overall, a pooled rate of 18.61%. |
| Does monitoring invalidate this? | The posterior depends only on the totals — 16,781 conversions out of 90,189 observations — and not on how many times you looked while the test ran. That is a real advantage over a fixed-horizon p-value, which is only valid for the sample size you committed to in advance. It is NOT a licence to stop whenever the number looks good. |
| What this cannot correct for | This analysis was run once, on a file that does not record when you looked. If you monitored the test and stopped the first time P(best) crossed a bar, the reported 0.999 for gate_30 overstates how often that decision rule would be right, and nothing in this data can measure by how much. The fix is to fix the rule — the bar, and the minimum sample — before you look, not to reinterpret the number afterwards. |
| Who sets the threshold | The expected loss of shipping gate_30 is under 0.001 percentage points of conversion rate. Whether that is small enough to act on is a business judgement about what a percentage point is worth to you — the analysis computes the risk, it cannot set the threshold. |
Each variant's posterior is Beta(prior + conversions, prior + non-conversions), computed exactly without normal approximation. Gate 30's posterior is Beta(8503.00, 36199.00); gate 40's is Beta(8280.00, 36199.00). The prior is Beta(1.00, 1.00), uniform over retention rate, contributing negligible weight (0.004% of gate 30's posterior). Decision probabilities come from 100,000 posterior draws per variant (seed 42), each carrying Monte Carlo standard error of about 0.0001. Gate 40 is the control (largest group). The analysis does not correct for stopping rules applied during monitoring—if you peeked and stopped at P(best) = 0.999, that reported probability overstates the true long-run success rate of that decision rule, and the data cannot measure the overstatement. The expected loss threshold is a business call, not a statistical one.
Bayesian A/B Test — Beta-Binomial Conjugate Analysis
Answers the question a frequentist significance test cannot: given what you have seen, what is the probability each variant is the best one, and what does it cost you to be wrong? Each variant's conversion rate gets a Beta posterior (conjugate to the Binomial likelihood); posterior draws then give P(variant beats control), P(variant is best), the expected loss (risk) of shipping each variant, and credible intervals on the lift.
Why This Method?
A p-value answers "how surprising would this data be if nothing were happening?" — a question nobody asked. The posterior answers the question people actually ask: how likely is B better than A, by how much, and what do I lose if I pick wrong. Because the Beta prior is conjugate to the Binomial likelihood, the posterior has a closed form and needs no fitting.
What This Analysis Covers
- Posterior conversion rate per variant with a 95% credible interval
- P(variant beats control) and P(variant is the best of all)
- Expected loss (risk) of shipping each variant
- The full posterior distribution of the lift, not just a point estimate
- An honest treatment of the "no peeking problem": what the posterior
really is and is not immune to, with the prior's influence quantified from the user's own sample sizes
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {variant, outcome, trials}. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
compute_shared <- function(df, params, col_map = list()) {
# === SHARED EXPORTS ===
# initial_rows/final_rows/rows_removed $ row accounting
# variant_h / outcome_h / trials_h $ humanized user column names
# shape / shape_label $ "row-per-unit" | "summary" + prose label
# success_how $ how the success level was determined (prose)
# variants $ character — kept variant labels (control first)
# trials / conv $ integer — per kept variant
# dropped_levels / drop_note $ variants excluded + why
# n_missing_rows $ rows dropped for a missing variant/outcome
# a0 / b0 $ the prior's Beta parameters
# post_a / post_b $ posterior Beta parameters per variant
# post_mean / ci_lo / ci_hi $ posterior rate + 95% credible interval
# control / control_how $ control label + how it was chosen
# p_beat / p_best / e_loss $ decision quantities per variant
# lift / lift_lo / lift_hi $ absolute lift vs control (proportion units)
# rel_lift / rel_lo / rel_hi $ relative lift vs control (fraction)
# leader / focus $ highest P(best) / best non-control variant
# decisive / separated $ leader clears 0.95 / clears the runner-up
# mc_se_leader $ Monte Carlo standard error on the leader's P(best)
# prior_weight_pct $ % of the leader's posterior weight from the prior
# sens_df $ P(best) for the leader under 3 priors
# posteriors_df / rates_df / lift_draws_df / decision_df / disclosure_df
# metrics / json_output
# === /SHARED EXPORTS ===
N_DRAWS <- 100000L # posterior draws per variant (seed fixed below)
MAX_VARIANTS <- 8L # keep at most this many arms
MIN_TRIALS_PER_VARIANT <- 5L
MIN_TOTAL_TRIALS <- 20L
SKEPTICAL_WEIGHT <- 50 # pseudo-trials per variant in the skeptical prior
variant_h <- humanize_semantic("variant", col_map)
outcome_h <- humanize_semantic("outcome", col_map)
trials_h <- humanize_semantic("trials", col_map)Step 1: Validate the mapping
initial_rows <- nrow(df)
if (!("variant" %in% names(df))) {
stop(sprintf("This analysis needs a variant column mapped — the column that says which version each row belongs to(mapped as '%s').",
variant_h))
}
if (!("outcome" %in% names(df))) {
stop(sprintf("This analysis needs an outcome column mapped — the conversion flag, or the conversion COUNT if your rows are already totalled by variant(mapped as '%s').",
outcome_h))
}Step 2: Which shape is this — one row per unit, or pre-aggregated totals?
The widget maps two columns (variant + outcome) and treats every row as one visitor. Mapping a third column of trial counts switches the module to summary mode, where each row is a whole variant.
shape <- if ("trials" %in% names(df)) "summary" else "row-per-unit"
v_all <- trimws(as.character(df$variant))
v_ok <- !is.na(v_all) & v_all != ""
success_how <- ""
if (shape == "summary") {Summary shape: outcome = conversions, trials = trials, one row per variant.
coerce_count <- function(v, label_h) {
ch <- trimws(as.character(v))
non_blank <- !is.na(ch) & ch != ""
conv <- suppressWarnings(as.numeric(ch))
if (sum(non_blank) == 0 ||
sum(!is.na(conv[non_blank])) < 0.95 * sum(non_blank)) {
stop(sprintf("The column '%s' does not look numeric — fewer than 95%% of its values parse as numbers. In summary mode it must hold whole counts.",
label_h))
}
conv
}
tr_all <- coerce_count(df$trials, trials_h)
cv_all <- coerce_count(df$outcome, outcome_h)
ok <- v_ok & !is.na(tr_all) & !is.na(cv_all)
n_missing_rows <- sum(!ok)
vv <- v_all[ok]; tt <- tr_all[ok]; cc <- cv_all[ok]
if (length(vv) == 0) {
stop(sprintf("No rows have a usable '%s', '%s' and '%s' together.",
variant_h, outcome_h, trials_h))
}
if (any(tt < 0) || any(cc < 0)) {
stop(sprintf("'%s' and '%s' must be counts of zero or more — negative values were found.",
outcome_h, trials_h))
}
bad <- which(cc > tt)
if (length(bad) > 0) {
stop(sprintf("'%s' exceeds '%s' for %d row(s) (for example %s: %s conversions out of %s). Conversions cannot outnumber the people who saw the variant.",
outcome_h, trials_h, length(bad), vv[bad[1]],
format(cc[bad[1]]), format(tt[bad[1]])))
}
agg_tr <- tapply(tt, vv, sum)
agg_cv <- tapply(cc, vv, sum)
success_how <- sprintf("Each row of your file is a whole variant: '%s' is the number of people who saw it and '%s' is the number who converted.",
trials_h, outcome_h)
} else {Row-per-unit shape: every row is one visitor; the outcome is a flag.
o_all <- df$outcome
o_is_logical <- is.logical(o_all)
o_chr_all <- trimws(as.character(o_all))
ok <- v_ok & !is.na(o_all) & !is.na(o_chr_all) & o_chr_all != ""
n_missing_rows <- sum(!ok)
vv <- v_all[ok]
o_chr <- o_chr_all[ok]
if (length(vv) == 0) {
stop(sprintf("No rows have both a '%s' and an '%s' value.", variant_h, outcome_h))
}
y <- NULL
if (o_is_logical) {
y <- as.numeric(o_all[ok])
success_how <- sprintf("'%s' is a TRUE/FALSE column; TRUE was counted as a conversion.", outcome_h)
} else {
num <- suppressWarnings(as.numeric(o_chr))
if (sum(!is.na(num)) >= 0.95 * length(o_chr)) {
keep_num <- !is.na(num)
vv <- vv[keep_num]; num <- num[keep_num]
n_missing_rows <- n_missing_rows + sum(!keep_num)
uu <- sort(unique(num))
if (!all(uu %in% c(0, 1))) {
stop(sprintf("'%s' holds %d distinct numeric values (for example %s and %s). A Bayesian A/B test models a CONVERSION RATE, so it needs a 0/1 flag per row — or map a trial-count column as well and give it conversion counts per variant.",
outcome_h, length(uu), f2(uu[1]), f2(uu[length(uu)])))
}
y <- num
success_how <- sprintf("'%s' holds only 0 and 1; 1 was counted as a conversion.", outcome_h)
} else {
lv <- sort(unique(o_chr))
if (length(lv) != 2) {
stop(sprintf("'%s' has %d distinct non-numeric values, so it cannot be read as a conversion flag. Map a 0/1 column, a two-level yes/no column, or add a trial-count column and supply conversion counts.",
outcome_h, length(lv)))
}
kw <- c("yes", "y", "true", "t", "converted", "conversion", "success",
"won", "win", "clicked", "purchased", "paid", "subscribed",
"signup", "signed up", "1")
hit <- lv[tolower(lv) %in% kw]
if (length(hit) >= 1) {
success_label <- hit[1]
success_how <- sprintf("'%s' has two values ('%s' and '%s'); '%s' was counted as a conversion because it matches a known success word.",
outcome_h, lv[1], lv[2], success_label)
} else {
success_label <- lv[2]
success_how <- sprintf("'%s' has two values ('%s' and '%s') and neither matches a known success word, so '%s' — the alphabetically later one — was counted as the conversion. Check that this is the direction you meant.",
outcome_h, lv[1], lv[2], success_label)
}
y <- as.numeric(o_chr == success_label)
}
}
agg_tr <- tapply(rep(1, length(y)), vv, sum)
agg_cv <- tapply(y, vv, sum)
}Step 3: Minimum-size guards, named in the user's own columns
all_levels <- names(agg_tr)
trials_all <- as.numeric(agg_tr[all_levels])
conv_all <- as.numeric(agg_cv[all_levels])
total_trials_all <- sum(trials_all)
if (total_trials_all < MIN_TOTAL_TRIALS) {
stop(sprintf("Only %s observation(s) of '%s' are usable — a Bayesian A/B test needs at least %d in total, and at least %d per group of '%s'.",
format(total_trials_all, big.mark = ","), variant_h,
MIN_TOTAL_TRIALS, MIN_TRIALS_PER_VARIANT, variant_h))
}Keep the largest arms; drop tiny ones rather than pooling them (pooling distinct variants into an "Other" bucket would invent a variant nobody ran).
ord <- order(-trials_all, all_levels)
ranked <- all_levels[ord]
beyond_cap <- if (length(ranked) > MAX_VARIANTS) ranked[(MAX_VARIANTS + 1):length(ranked)] else character(0)
candidates <- setdiff(ranked, beyond_cap)
too_small <- candidates[as.numeric(agg_tr[candidates]) < MIN_TRIALS_PER_VARIANT]
variants <- setdiff(candidates, too_small)
dropped_levels <- c(beyond_cap, too_small)
if (length(variants) < 2) {
stop(sprintf("After cleaning, '%s' has %d group(s) with at least %d observation(s) — a comparison needs at least two.",
variant_h, length(variants), MIN_TRIALS_PER_VARIANT))
}
trials <- as.numeric(agg_tr[variants])
conv <- as.numeric(agg_cv[variants])
if (sum(conv) == 0) {
stop(sprintf("Not one of the %s observations converted — every value of '%s' is a non-conversion, so there is no conversion rate to compare.",
format(sum(trials), big.mark = ","), outcome_h))
}
final_rows <- if (shape == "summary") length(variants) else sum(trials)
rows_removed <- initial_rows - final_rowsStep 4: Choose the control — a group named like one, else the largest
named_ctrl <- variants[tolower(variants) %in%
c("control", "a", "baseline", "original", "ctrl", "holdout")]
control <- if (length(named_ctrl) > 0) named_ctrl[1] else variants[safe_which_max(trials)]
control_how <- if (length(named_ctrl) > 0) {
sprintf("'%s' (its name identifies it as the control)", control)
} else {
sprintf("'%s' (the largest group, since no group is named like a control)", control)
}Put the control first so every table reads control-then-challengers.
variants <- c(control, setdiff(variants, control))
trials <- as.numeric(agg_tr[variants])
conv <- as.numeric(agg_cv[variants])
k <- length(variants)
ctrl_i <- 1LStep 5: Conjugate posteriors — Beta(a0 + conversions, b0 + failures)
a0 <- 1; b0 <- 1 # uniform prior over the conversion rate
post_a <- a0 + conv
post_b <- b0 + (trials - conv)
post_mean <- post_a / (post_a + post_b)
ci_lo <- stats::qbeta(0.025, post_a, post_b)
ci_hi <- stats::qbeta(0.975, post_a, post_b)
obs_rate <- conv / trialsStep 6: Posterior draws — every decision quantity comes from these
set.seed(42)
draw_matrix <- function(aa, bb) {
matrix(stats::rbeta(N_DRAWS * length(aa),
rep(aa, each = N_DRAWS),
rep(bb, each = N_DRAWS)),
nrow = N_DRAWS, ncol = length(aa))
}
D <- draw_matrix(post_a, post_b)
best_idx <- max.col(D, ties.method = "first")
row_max <- D[cbind(seq_len(N_DRAWS), best_idx)]
p_best <- as.numeric(tabulate(best_idx, nbins = k)) / N_DRAWS
e_loss <- colMeans(row_max - D)
ctrl_draws <- D[, ctrl_i]
p_beat <- colMeans(D > ctrl_draws)
p_beat[ctrl_i] <- NA_real_
lift <- post_mean - post_mean[ctrl_i] # exact posterior mean of the difference
lift_lo <- rep(NA_real_, k); lift_hi <- rep(NA_real_, k)
rel_lift <- rep(NA_real_, k); rel_lo <- rep(NA_real_, k); rel_hi <- rep(NA_real_, k)
for (i in seq_len(k)) {
if (i == ctrl_i) next
dl <- D[, i] - ctrl_draws
q <- stats::quantile(dl, c(0.025, 0.975), names = FALSE)
lift_lo[i] <- q[1]; lift_hi[i] <- q[2]
rl <- dl / ctrl_draws
qr <- stats::quantile(rl, c(0.025, 0.5, 0.975), names = FALSE)
rel_lo[i] <- qr[1]; rel_lift[i] <- qr[2]; rel_hi[i] <- qr[3]
}Step 7: Who leads, and is the lead real enough to act on?
lead_i <- safe_which_max(p_best)
leader <- variants[lead_i]
p_best_leader <- p_best[lead_i]
runner_i <- safe_which_max(replace(p_best, lead_i, NA_real_))
p_best_runner <- p_best[runner_i]
mc_se_leader <- sqrt(p_best_leader * (1 - p_best_leader) / N_DRAWS)
decisive <- p_best_leader >= 0.95
separated <- (p_best_leader - p_best_runner) > 0.05The reported lift picture always compares a challenger with the control.
non_ctrl <- setdiff(seq_len(k), ctrl_i)
focus_i <- non_ctrl[safe_which_max(p_best[non_ctrl])]
focus <- variants[focus_i]Step 8: How much of the answer is the prior, not the data?
prior_weight_pct <- 100 * (a0 + b0) / (a0 + b0 + trials[lead_i])
pooled_rate <- sum(conv) / sum(trials)
prior_defs <- list(
list(label = sprintf("Uniform Beta(%s, %s) — the prior used above", f2(a0), f2(b0)),
a = rep(a0, k), b = rep(b0, k)),
list(label = "Jeffreys Beta(0.50, 0.50) — the standard reference prior",
a = rep(0.5, k), b = rep(0.5, k)),
list(label = sprintf("Skeptical Beta centred on the pooled rate %s%%, worth %s pseudo-trials per variant",
pctf(pooled_rate), format(SKEPTICAL_WEIGHT)),
a = rep(SKEPTICAL_WEIGHT * pooled_rate, k),
b = rep(SKEPTICAL_WEIGHT * (1 - pooled_rate), k))
)
sens_rows <- list()
for (pd in prior_defs) {
set.seed(42)
Ds <- draw_matrix(pd$a + conv, pd$b + (trials - conv))
bi <- max.col(Ds, ties.method = "first")
pb <- as.numeric(tabulate(bi, nbins = k)) / N_DRAWS
rm_s <- Ds[cbind(seq_len(N_DRAWS), bi)]
el <- colMeans(rm_s - Ds)
sens_rows[[length(sens_rows) + 1]] <- data.frame(
prior = pd$label,
leader_under_prior = variants[safe_which_max(pb)],
prob_leader_is_best = round(max(pb), 4),
leader_rate_pct = round(100 * mean(Ds[, safe_which_max(pb)]), 3),
expected_loss_pp = round(100 * el[safe_which_max(pb)], 4),
stringsAsFactors = FALSE
)
}
sens_df <- do.call(rbind, sens_rows)
rownames(sens_df) <- NULL
sens_spread <- max(sens_df$prob_leader_is_best) - min(sens_df$prob_leader_is_best)
sens_agree <- length(unique(sens_df$leader_under_prior)) == 1Step 10: Disclosure — what the posterior is and is not immune to
peek_line <- paste0(
"The posterior depends only on the totals — ", format(sum(conv), big.mark = ","),
" conversions out of ", format(sum(trials), big.mark = ","),
" observations — and not on how many times you looked while the test ran. ",
"That is a real advantage over a fixed-horizon p-value, which is only valid ",
"for the sample size you committed to in advance. It is NOT a licence to stop ",
"whenever the number looks good."
)
stop_rule_line <- paste0(
"This analysis was run once, on a file that does not record when you looked. ",
"If you monitored the test and stopped the first time P(best) crossed a bar, ",
"the reported ", fmt_prob(p_best_leader), " for ", leader,
" overstates how often that decision rule would be right, and nothing in this ",
"data can measure by how much. The fix is to fix the rule — the bar, and the ",
"minimum sample — before you look, not to reinterpret the number afterwards."
)
prior_line <- paste0(
"The prior adds ", f2(a0 + b0), " pseudo-observations to each variant, which is ",
f3(prior_weight_pct), "% of ", leader, "'s ", format(trials[lead_i], big.mark = ","),
" observations. Under the three priors tested, P(best) for the leading variant spans ",
f3_min(sens_spread), " and the leading variant ",
if (sens_agree) "does not change" else "CHANGES, so the conclusion is prior-driven, not data-driven",
"."
)
threshold_line <- paste0(
"The expected loss of shipping ", leader, " is ", ppf_min(e_loss[lead_i], 3),
" percentage points of conversion rate. Whether that is small enough to act on ",
"is a business judgement about what a percentage point is worth to you — the ",
"analysis computes the risk, it cannot set the threshold."
)
disclosure_df <- data.frame(
item = c("Model", "Prior", "Posterior draws", "Monte Carlo precision",
"Control group", "Data shape", "How conversions were counted",
"Does monitoring invalidate this?", "What this cannot correct for",
"Who sets the threshold"),
detail = c(
sprintf("Beta-Binomial conjugate: each variant's conversion rate gets a Beta posterior with parameters (prior + conversions, prior + non-conversions). No model is fitted and no normal approximation is used, so the arms with few observations are handled correctly.")
,
sprintf("Beta(%s, %s), uniform over the conversion rate. Posterior parameters run from Beta(%s, %s) to Beta(%s, %s) across the %d variants.",
f2(a0), f2(b0), f2(min(post_a)), f2(min(post_b)),
f2(max(post_a)), f2(max(post_b)), k),
sprintf("%s draws per variant from each posterior, seed 42, so re-running this analysis on the same file returns the same probabilities.",
format(N_DRAWS, big.mark = ",")),
sprintf("The Monte Carlo standard error on %s's P(best) of %s is %s, so the third decimal place is noise from the simulation rather than signal from the data.",
leader, f4(p_best_leader), f4(mc_se_leader)),
sprintf("%s. Every lift, credible interval and P(beats control) in this report is measured against it.", control_how),
sprintf("%s %s", if (shape == "summary")
"Pre-aggregated totals: one row per variant." else
"One row per visitor or session, aggregated to per-variant totals by this analysis.",
success_how),
sprintf("%s conversions out of %s observations overall, a pooled rate of %s%%.",
format(sum(conv), big.mark = ","), format(sum(trials), big.mark = ","),
pctf(pooled_rate)),
peek_line,
stop_rule_line,
threshold_line
),
stringsAsFactors = FALSE
)
drop_note <- if (length(dropped_levels) > 0) {
sprintf("%d group(s) of '%s' were excluded (fewer than %d observations, or beyond the %d largest arms): %s. ",
length(dropped_levels), variant_h, MIN_TRIALS_PER_VARIANT, MAX_VARIANTS,
paste(dropped_levels, collapse = ", "))
} else {
sprintf("Every group of '%s' had enough observations to keep. ", variant_h)
}
shape_label <- if (shape == "summary") "pre-aggregated totals" else "one row per visitor"
metrics <- list(
`Variants Compared` = as.integer(k),
`Total Observations` = as.integer(round(sum(trials))),
`Control Group` = control,
`Leading Variant` = leader,
`Leader Conversion Rate` = paste0(pctf(post_mean[lead_i]), "%"),
`P(Leader Is Best)` = fmt_prob(p_best_leader),
`P(Leader Beats Control)` = if (lead_i == ctrl_i) "n/a(the control leads)" else fmt_prob(p_beat[lead_i]),
`Expected Loss(Leader)` = paste0(ppf_min(e_loss[lead_i], 3), " pp"),
`Lift vs Control` = if (lead_i == ctrl_i) "n/a(the control leads)" else
paste0(ppf(lift[lead_i]), " pp(95% CrI ", ppf(lift_lo[lead_i]), " to ",
ppf(lift_hi[lead_i]), ")"),
`Decision Status` = if (decisive) "leader clears 0.95" else "not yet decided at 0.95",
`Data Shape` = shape_label
)
verdict <- if (decisive) {
sprintf("%s is the best arm with probability %s, and shipping it carries an expected loss of %s percentage points.",
leader, fmt_prob(p_best_leader), ppf_min(e_loss[lead_i], 3))
} else if (!separated) {
sprintf("No arm has separated: %s leads with P(best) = %s against %s's %s, a gap this data cannot resolve.",
leader, fmt_prob(p_best_leader), variants[runner_i], fmt_prob(p_best_runner))
} else {
sprintf("%s leads with P(best) = %s, short of the conventional 0.95 bar, and shipping it carries an expected loss of %s percentage points.",
leader, fmt_prob(p_best_leader), ppf_min(e_loss[lead_i], 3))
}
json_output <- list(
answer = paste0(
"Bayesian A/B test of ", k, " variants of '", variant_h, "' across ",
format(sum(trials), big.mark = ","), " observations(",
format(sum(conv), big.mark = ","), " conversions), read from ", shape_label,
". Posterior conversion rates run from ", pctf(min(post_mean)), "% to ",
pctf(max(post_mean)), "%; the control is ", control, ". ", verdict, " ",
if (focus_i != ctrl_i) paste0(
focus, " sits ", ppf(lift[focus_i]), " percentage points ",
if (lift[focus_i] >= 0) "above" else "below", " the control(95% credible interval ",
ppf(lift_lo[focus_i]), " to ", ppf(lift_hi[focus_i]), " percentage points), and beats it with probability ",
fmt_prob(p_beat[focus_i]), ". ") else "",
"These posteriors do not depend on how often you checked the test, but they ",
"also cannot correct for a stopping rule they were never told about; the ",
"prior contributes ", f3(prior_weight_pct), "% of the leader's posterior weight."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "variant_posteriors",
"posterior_rates", "lift_distribution", "decision_table",
"priors_and_stopping", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
variant_h = variant_h, outcome_h = outcome_h, trials_h = trials_h,
shape = shape, shape_label = shape_label, success_how = success_how,
variants = variants, trials = trials, conv = conv, k = k,
dropped_levels = dropped_levels, drop_note = drop_note,
n_missing_rows = n_missing_rows,
a0 = a0, b0 = b0, post_a = post_a, post_b = post_b,
post_mean = post_mean, ci_lo = ci_lo, ci_hi = ci_hi, obs_rate = obs_rate,
control = control, control_how = control_how, ctrl_i = ctrl_i,
p_beat = p_beat, p_best = p_best, e_loss = e_loss,
lift = lift, lift_lo = lift_lo, lift_hi = lift_hi,
rel_lift = rel_lift, rel_lo = rel_lo, rel_hi = rel_hi,
leader = leader, lead_i = lead_i, p_best_leader = p_best_leader,
runner_i = runner_i, p_best_runner = p_best_runner,
focus = focus, focus_i = focus_i,
decisive = decisive, separated = separated, mc_se_leader = mc_se_leader,
n_draws = N_DRAWS, pooled_rate = pooled_rate,
prior_weight_pct = prior_weight_pct, sens_df = sens_df,
sens_spread = sens_spread, sens_agree = sens_agree,
peek_line = peek_line, stop_rule_line = stop_rule_line,
prior_line = prior_line, threshold_line = threshold_line,
posteriors_df = posteriors_df, rates_df = rates_df,
lift_draws_df = lift_draws_df, decision_df = decision_df,
disclosure_df = disclosure_df,
verdict = verdict, metrics = metrics, json_output = json_output
)
}