Can I Stop?
Stop - a winner has separated
You can stop for a winner: Variant B has separated from Control by 4.00 pp, and the approximately always-valid confidence sequence (1.79 pp to 6.52 pp) excludes zero. That conclusion is designed to hold, approximately (asymptotic; sample variances plugged into a normal mixture boundary), even though the experiment was re-tested after every one of its 6,000 observations. The boundary was first crossed at observation 1,650. An ordinary fixed-horizon test on the same data would have shown p below 0.05 at 941 point(s) before the approximately always-valid boundary was first crossed, across 7 separate excursion(s). The guarantee is not free: the approximately always-valid interval is 1.55 times wider than the fixed-horizon interval at this sample size, and it would take about 14,808 observations (2.47 times the current sample) for it to become as tight as the fixed-horizon interval already is. The guarantee is approximate (asymptotic; sample variances plugged into a normal mixture boundary) and covers this one comparison as specified; it does not protect a metric, a segment or an arm picked after seeing the results.
Analysis Overview
Approximately always-valid sequential monitoring of Variant B against Control across 6,000 observations.
A fixed-horizon test earns its 5% false-positive rate only if you look once, at a sample size decided before the data existed. This analysis instead re-tests after every observation and uses a boundary that is approximately valid at all of those looks at the same time (asymptotic; sample variances plugged into a normal mixture boundary): a mixture sequential probability ratio test with a normal mixture over the true effect. Rows are ordered by Signup Date, which was read as calendar dates. The effect is the mean of Converted for Variant B minus the mean for Control; the control arm was identified because its name identifies it as the control. The same cumulative data is also run through an ordinary Welch t-test at every point, so the two boundaries can be compared on your own numbers rather than on an assertion.
Data Quality
Row accounting, arm sizes, and how the ordering was resolved.
6,000 rows loaded; 6,000 analysed. No rows were incomplete. The outcome column Converted was already numeric and holds only 0 and 1, so the effect is a difference in rates. Control contributed 2,975 rows and Variant B contributed 3,025. 5,850 row(s) share an arrival value with an earlier row; tied rows keep their original file order, so the monitoring path steps through them one at a time. Nothing here can verify that assignment to Assigned Group was actually random, or that each row is a different person - both are assumptions the analysis inherits from how the experiment was run.
Effect and Boundaries Over the Experiment
The cumulative effect with the approximately always-valid and fixed-horizon boundaries drawn over it.
The middle series is the cumulative estimate of Variant B minus Control, recomputed as each observation arrives. The outer pair is the approximately always-valid boundary: crossing it is a valid stop however many times you looked, approximately (asymptotic; sample variances plugged into a normal mixture boundary). The inner pair is the ordinary fixed-horizon 0.05 boundary, drawn for contrast - it is the line a dashboard would show you, and it is narrower everywhere, which is precisely why watching it makes you stop too early. The cumulative effect first left the approximately always-valid corridor at observation 1,650. Both corridors narrow as data accumulates, but the approximately always-valid one narrows more slowly because it is paying for every look you might take.
The Two p-Values Side by Side
The fixed-horizon p-value and the approximately always-valid p-value at every analysis point.
The fixed-horizon p-value is recomputed from scratch at each point and is free to wander back above 0.05 after dipping below it - that is the behaviour that makes repeated checking dangerous. The approximately always-valid p-value is the running minimum of the inverse mixture likelihood ratio, so it only ever moves down, and it is a stopping criterion that holds approximately at any point in large samples (asymptotic; sample variances plugged into a normal mixture boundary). The fixed-horizon series dips under the 0.05 line at 941 point(s) before the approximately always-valid series first does: those are the moments a peeking team would have called a winner before the evidence allowed it. Both are computed from your own data at the same 5,680 points; the comparison is not an illustration.
The Decision, Line by Line
Everything the stopping decision rests on.
| Item | Detail |
|---|---|
| Decision | Stop - a winner has separated |
| Approximately always-valid boundary crossed | yes, first at observation 1,650 |
| Observations analysed | 6,000 |
| Effect (Variant B minus Control) | 4.00 pp |
| Approximately always-valid 95% confidence sequence (asymptotic; sample variances plugged into a normal mixture boundary) | 1.79 pp to 6.52 pp |
| Approximately always-valid p-value | < 0.001 |
| Fixed-horizon p-value at this sample size | < 0.001 |
| Smallest effect this test was set up to find | 2.96 pp |
| Control arm | Control (n = 2,975) |
| Treatment arm | Variant B (n = 3,025) |
The rule has three exits and one honest refusal. Stop for a winner when the confidence sequence excludes zero. Stop for futility when it lies entirely within 2.96 pp, the smallest effect this test was set up to find. Keep running when neither is true but both are still reachable. And when the interval is wider than that smallest effect, as it is at small sample sizes, no boundary is reachable at all and the analysis says so instead of reporting a null result it has not earned. Here the verdict is: stop - a winner has separated.
What Peeking Would Have Cost You
The fixed-horizon test and the approximately always-valid boundary, counted over the same analysis points.
| Check | Value |
|---|---|
| Analysis points examined (one after every observation) | 5,680 |
| Points where the fixed-horizon p-value was below 0.05 | 5,292 |
| Separate excursions below 0.05 by the fixed-horizon p-value | 7 |
| Points where the approximately always-valid boundary was crossed | 4,344 |
| Points before the first approximately always-valid crossing where the fixed-horizon test already said stop | 941 |
| First point the fixed-horizon test would have stopped | observation 660 |
| First point the approximately always-valid boundary was crossed | observation 1,650 |
Both tests were run at all 5,680 analysis points on your data. The fixed-horizon p-value fell below 0.05 at 5,292 of them, in 7 separate excursion(s); the approximately always-valid boundary was crossed at 4,344. The 941 point(s) where only the fixed-horizon test fired are the concrete cost of checking a dashboard: each one is a moment this experiment could have been stopped on evidence that had not actually arrived. Note that these counts assume the most aggressive schedule possible, a look after every observation; a team checking weekly would see fewer opportunities to be misled, but not zero.
The Price of Being Allowed to Peek
How much extra data the approximately always-valid guarantee costs on this experiment.
| Measure | Value |
|---|---|
| Approximately always-valid half-width at this sample size | 2.56 pp |
| Fixed-horizon half-width at this sample size | 1.65 pp |
| How much wider the approximately always-valid interval is | 1.55 times |
| Observations needed for the approximately always-valid interval to be as tight as the fixed-horizon one is now | 14,808 |
| That as a multiple of the current sample size | 2.47 times |
Approximately always-valid inference is not free and is not a strictly better test. It buys the right to look whenever you like, approximately (asymptotic; sample variances plugged into a normal mixture boundary), by widening the interval, which means it needs more data than a fixed-horizon test to reach the same conclusion. On this experiment the approximately always-valid interval is 2.56 pp on each side against the fixed-horizon 1.65 pp, a factor of 1.55. Reaching today's fixed-horizon precision under the approximately always-valid boundary would take about 14,808 observations, roughly 2.47 times what you have. If you genuinely can commit to a single analysis at a sample size fixed in advance, the fixed-horizon test is the more efficient choice and you should use it. This tool is for the common case where that commitment is not realistic.
Does the Verdict Survive a Different Alternative?
The same monitoring path re-run against three pre-specified alternatives.
| Alternative | Alternative Size | Half Width | Decision | Ever Crossed |
|---|---|---|---|---|
| 0.05 x the outcome's standard deviation | 1.48 | 2.64 | stop for a winner | yes |
| 0.10 x the outcome's standard deviation | 2.96 | 2.56 | stop for a winner | yes |
| 0.20 x the outcome's standard deviation | 5.92 | 2.68 | stop for a winner | yes |
The boundary has one tuning constant: tau, the effect size the test is aimed at. It is set here to 0.10 times the outcome's standard deviation (2.96 pp), read from how much Converted varies rather than from the gap between the arms, so it does not borrow the answer. A smaller tau makes the boundary tighter for small effects and looser for large ones; the futility rule moves with it directly, since tau also defines what counts as an effect worth acting on. The table re-runs the whole path at half and double the default. All three give the same verdict (stop for a winner), so the conclusion does not hinge on that choice. Strictly, tau should be fixed before the experiment starts; it is set from the outcome's spread here because a standard-library tool is not told what you designed for.
Methods and Disclosure
Every formula, constant and assumption behind the decision.
| Item | Detail |
|---|---|
| Test | Mixture sequential probability ratio test (mSPRT) on the difference in means between the two arms, implemented directly from the normal mixture - no group-sequential package is used. |
| Boundary | Robbins' normal-mixture boundary. The mixture likelihood ratio is Lambda = sqrt(V/(V+tau^2)) exp(delta^2 tau^2 / (2 V (V+tau^2))) with V the sampling variance of the effect; Ville's inequality bounds the chance that Lambda ever reaches 1/alpha at alpha when V is known. Here V is estimated from the sample variances observed so far and plugged in, so the bound is asymptotic (sample variances plugged into a normal mixture boundary): it holds approximately, and more closely as the sample grows. |
| Confidence sequence | Inverting the boundary gives effect +/- h with h = sqrt(2 V (V+tau^2)/tau^2 log(sqrt((V+tau^2)/V)/alpha)); this interval aims to cover the true effect at every sample size at once, not just at one pre-chosen sample size; with plug-in variances that coverage is approximate (asymptotic). |
| Pre-specified alternative (tau) | 2.96 pp - set at 0.10 times the outcome's standard deviation (29.60) over the first 300 observations, a calibration prefix the test does not use: the boundary is fixed before any observation it monitors, which the approximately always-valid guarantee requires. It is the effect size the boundary is tuned to find, and it is read from how much the outcome varies, not from the difference between the arms. The effect and the confidence sequence are computed on the 5,700 observations after the prefix. |
| Significance level | alpha = 0.05, applied to the whole monitoring path rather than to a single look. |
| Analysis schedule | The test is evaluated after every single observation from the point where both arms reach 10 rows - the most aggressive peeking possible - giving 5,680 analysis points. |
| Fixed-horizon comparison | At each of those points a Welch two-sample t-test is computed on the same cumulative data, purely so the two boundaries can be compared on this experiment's own numbers. |
| Effect definition | Mean of Variant B minus mean of Control, both being conversion rates, reported in percentage points. |
| Control arm chosen by | Control - its name identifies it as the control. |
| Arrival order read as | calendar dates in the column 'Signup Date'; 5,850 row(s) share an arrival value with an earlier row and keep their original file order. |
| What this cannot do | The guarantee is about the stopping rule, not about everything else. It holds for this one pre-specified comparison of these two arms on this one outcome. It does not cover choosing the metric, the segment or the arm after seeing the results, it cannot detect repeat visits by the same person, and it does not know whether the assignment was actually random. |
The whole analysis rests on one inequality: under the null, the chance that the mixture likelihood ratio ever reaches 1 divided by 0.05 is at most 0.05 - however many times you look, exactly only when the variance is known, so approximately here. The sample variances are plugged in, so the guarantee is approximately always-valid (asymptotic; sample variances plugged into a normal mixture boundary), not exact. Everything else is bookkeeping. The normal mixture is an approximation that improves with sample size, and the variance is estimated from the same data it is testing, so at very small samples the stated coverage is optimistic; that is one more reason the analysis refuses to declare a null result when no boundary is reachable. What it cannot repair is a question chosen after the fact: monitoring is made safe here, but selecting the outcome, the segment or the arm once the numbers are visible is a different problem, and this table is the place that says so plainly.
Sequential A/B Test - Peek With an Approximate Guarantee
Reads an ordered experiment log (who saw which variant, what happened, and when the observation arrived) and answers the question a running experiment actually raises: given everything so far, can I stop?
Why This Method?
A fixed-horizon p-value is only valid if you look once, at a sample size fixed before the data existed. Look after every visitor and the chance of seeing p < 0.05 somewhere along the way is far above 5%. A mixture sequential probability ratio test (mSPRT) replaces the fixed critical value with a boundary that is approximately always-valid (asymptotic; sample variances plugged into a normal mixture boundary) at every sample size simultaneously (asymptotic), so monitoring the experiment continuously does not inflate the false-positive rate beyond alpha in large samples (LAT-5113 ruling B: the sample variances are plug-ins, so the guarantee is asymptotic, not exact). The price is power, and this module states that price in the user's own numbers rather than selling peeking as free.
What This Analysis Covers
- The approximately always-valid confidence sequence for the treatment effect over time
- The current decision: stop for a winner, stop for futility, keep running,
or not enough data for any boundary to be reachable
- The naive fixed-horizon p-value computed alongside, and how many times it
would have crossed 0.05 while the approximately always-valid boundary did not
- The cost of the guarantee: how much wider the approximately always-valid interval is and
how many observations it needs to match today's fixed-horizon precision
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {variant, outcome, arrival}. The mSPRT, the confidence sequence and the boundary crossing are implemented directly from the normal mixture - no group-sequential package is used.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
Step 1: Confirm the mapping and humanize the user's column names
variant_h <- humanize_semantic("variant", col_map)
outcome_h <- humanize_semantic("outcome", col_map)
arrival_h <- humanize_semantic("arrival", col_map)
needed <- c("variant", "outcome", "arrival")
missing_keys <- needed[!(needed %in% names(df))]
if (length(missing_keys) > 0) {
labels <- c(variant = variant_h, outcome = outcome_h, arrival = arrival_h)
stop(sprintf(
paste0("Sequential testing needs three columns mapped: the variant each ",
"row saw, the outcome, and the order or date the observation ",
"arrived. Missing: %s."),
paste(labels[missing_keys], collapse = ", ")))
}Step 2: Read the arrival column as dates, or as a numeric order
arr_ch <- trimws(as.character(df$arrival))
arr_ch[arr_ch == "NA"] <- NA_character_
n_nonblank_arr <- sum(!is.na(arr_ch) & arr_ch != "")
parse_dates_vec <- function(ch) {
ok <- function(dd) sum(!is.na(dd)) >= 0.95 * sum(!is.na(ch) & trimws(ch) != "")
d <- suppressWarnings(as.Date(ch, format = "%Y-%m-%d"))
if (!ok(d)) d <- suppressWarnings(as.Date(lubridate::ymd(ch, quiet = TRUE)))
if (!ok(d)) d <- suppressWarnings(as.Date(lubridate::mdy(ch, quiet = TRUE)))
if (!ok(d)) d <- suppressWarnings(as.Date(lubridate::dmy(ch, quiet = TRUE)))
if (ok(d)) d else NULL
}
arrival_kind <- ""
ord_key <- NULL
if (n_nonblank_arr > 0) {
dts <- parse_dates_vec(arr_ch)
if (!is.null(dts)) {
ord_key <- as.numeric(dts)
arrival_kind <- "calendar dates"
} else {
nv <- suppressWarnings(as.numeric(arr_ch))
if (sum(!is.na(nv)) >= 0.95 * n_nonblank_arr) {
ord_key <- nv
arrival_kind <- "a numeric arrival order"
}
}
}
if (is.null(ord_key)) {
stop(sprintf(
paste0("The arrival column '%s' could not be read either as dates or as ",
"a numeric order, so the observations cannot be put in the order ",
"they arrived. Sequential testing is entirely about that order - ",
"map a date, a timestamp, or a running sequence number."),
arrival_h))
}Step 3: Read the outcome as a number (95% rule, then yes/no words)
y_raw <- df$outcome
outcome_read <- ""
if (is.numeric(y_raw)) {
y <- as.numeric(y_raw)
outcome_read <- "already numeric"
} else {
ch <- trimws(as.character(y_raw))
ch[ch == "NA"] <- NA_character_
nonblank <- !is.na(ch) & ch != ""
conv <- suppressWarnings(as.numeric(ch))
if (sum(nonblank) > 0 && sum(!is.na(conv)) >= 0.95 * sum(nonblank)) {
y <- conv
outcome_read <- "read as numbers"
} else {
lut <- c("yes" = 1, "no" = 0, "true" = 1, "false" = 0, "y" = 1, "n" = 0,
"t" = 1, "f" = 0, "converted" = 1, "not converted" = 0,
"1" = 1, "0" = 0, "success" = 1, "failure" = 0)
conv2 <- unname(lut[tolower(ch)])
if (sum(nonblank) > 0 && sum(!is.na(conv2)) >= 0.95 * sum(nonblank)) {
y <- conv2
outcome_read <- "read as yes/no labels turned into 1 and 0"
} else {
stop(sprintf(
paste0("The outcome column '%s' is not numeric and is not a yes/no ",
"flag - fewer than 95%% of its values could be turned into ",
"numbers. Map a numeric outcome(revenue, time, a 0/1 ",
"conversion flag) so the two arms can be compared."),
outcome_h))
}
}
}Step 4: Keep complete rows only
g_ch <- trimws(as.character(df$variant))
g_ch[g_ch == "" | g_ch == "NA"] <- NA_character_
keep <- !is.na(g_ch) & !is.na(y) & !is.na(ord_key)
n_dropped <- sum(!keep)
g_k <- g_ch[keep]; y_k <- y[keep]; o_k <- ord_key[keep]
final_rows <- length(y_k)
rows_removed <- initial_rows - final_rows
if (final_rows < 30) {
stop(sprintf(
paste0("Only %d complete row(s) remained after dropping rows missing ",
"%s, %s or %s. Sequential monitoring needs at least 30 ",
"observations before any boundary means anything."),
final_rows, variant_h, outcome_h, arrival_h))
}Step 5: Resolve the two arms - never chosen after seeing the outcome
tab <- sort(table(g_k), decreasing = TRUE)
tiny_arms <- names(tab)[tab < MIN_ARM_KEEP]
if (length(tiny_arms) > 0) {
keep2 <- !(g_k %in% tiny_arms)
g_k <- g_k[keep2]; y_k <- y_k[keep2]; o_k <- o_k[keep2]
final_rows <- length(y_k)
rows_removed <- initial_rows - final_rows
tab <- sort(table(g_k), decreasing = TRUE)
}
arms <- names(tab)
if (length(arms) < 2) {
stop(sprintf(
paste0("The variant column '%s' holds only %d group%s with at least %d ",
"observation%s(%s). A sequential A/B test compares exactly two ",
"arms."),
variant_h, length(arms), if (length(arms) == 1) "" else "s",
MIN_ARM_KEEP, "s", paste(arms, collapse = ", ")))
}
if (length(arms) > 2) {
stop(sprintf(
paste0("The variant column '%s' holds %d arms (%s). This analysis will ",
"not pick two of them for you: the approximately always-valid guarantee covers ",
"the comparison you specified before looking at the data, and ",
"choosing which arms to compare after seeing the results is ",
"exactly what it does not protect against. Filter the file to the ",
"two arms you want to compare and re-run."),
variant_h, length(arms), paste(arms, collapse = ", ")))
}Control is chosen by NAME, not by outcome - a fixed, stated rule.
ctrl_pat <- "^(control|baseline|a|original|holdout|current|old)$"
ctrl_hit <- arms[grepl(ctrl_pat, tolower(trimws(arms)))]
control_arm <- if (length(ctrl_hit) >= 1) ctrl_hit[1] else sort(arms)[1]
control_rule <- if (length(ctrl_hit) >= 1)
"its name identifies it as the control" else
"no arm was named as a control, so the alphabetically first label was used"
treat_arm <- setdiff(arms, control_arm)[1]Step 6: Order by arrival (ties keep their original file order)
ordv <- order(o_k, seq_along(o_k))
g_s <- g_k[ordv]; y_s <- y_k[ordv]; o_s <- o_k[ordv]
n_tied <- sum(duplicated(o_s))
n_total <- length(y_s)
if (!is.finite(stats::sd(y_s)) || stats::sd(y_s) <= 0) {
stop(sprintf(
paste0("Every retained row has the same value in the outcome column ",
"'%s', so there is no difference between the arms to test and no ",
"boundary to cross."),
outcome_h))
}
is_binary <- all(y_s %in% c(0, 1))
n_control_all <- sum(g_s == control_arm); n_treat_all <- n_total - n_control_allStep 6b: Calibration prefix (LAT-5113 review)
tau must be fixed before the observations it monitors, or the boundary is set from future data and the approximately always-valid guarantee no longer holds. So tau is read from the standard deviation of a calibration prefix (the first 5% of rows by arrival, at least 100, at most 20%, lengthened within that cap until the outcome varies), and the sequential test runs only on the observations AFTER it. Observation numbers shown are positions in the whole file.
burn_cap <- max(2L, as.integer(floor(0.2 * n_total)))
n_burn <- min(max(100L, as.integer(ceiling(0.05 * n_total))), burn_cap)
burn_sd <- function(b) stats::sd(y_s[seq_len(b)])
while (n_burn < burn_cap && !(is.finite(burn_sd(n_burn)) && burn_sd(n_burn) > 0)) n_burn <- n_burn + 1L
sd_all <- burn_sd(n_burn)
if (!is.finite(sd_all) || sd_all <= 0) {
stop(sprintf(
paste0("The outcome '%s' does not vary in the first %d observations (the ",
"calibration prefix, at most 20%% of the file), so the size of effect ",
"the boundary is tuned to cannot be set before the test starts."),
outcome_h, n_burn))
}
tau <- TAU_MULT * sd_all
t2 <- tau^2
g_s <- g_s[-seq_len(n_burn)]; y_s <- y_s[-seq_len(n_burn)]; o_s <- o_s[-seq_len(n_burn)]
n_seq <- length(y_s)
unit <- if (is_binary) "percentage points" else "units of the outcome"fmt_eff already carries "pp" for a binary outcome, so the spelled-out unit is appended only when it would not be a duplicate.
unit_suffix <- if (is_binary) "" else paste0(" ", unit)
to_disp <- function(x) if (is_binary) 100 * x else x
disp_digits <- if (is_binary) 2 else 4
fmt_eff <- function(x) trimws(paste0(.fmt_n(to_disp(x), disp_digits),
if (is_binary) " pp" else ""))Step 7: Cumulative statistics after every single observation
is_c <- as.integer(g_s == control_arm)
is_t <- 1L - is_c
nA <- cumsum(is_c); nB <- cumsum(is_t)
sA <- cumsum(y_s * is_c); sB <- cumsum(y_s * is_t)
qA <- cumsum(y_s^2 * is_c); qB <- cumsum(y_s^2 * is_t)
mA <- ifelse(nA > 0, sA / nA, NA_real_)
mB <- ifelse(nB > 0, sB / nB, NA_real_)
vA <- ifelse(nA > 1, (qA - nA * mA^2) / (nA - 1), NA_real_)
vB <- ifelse(nB > 1, (qB - nB * mB^2) / (nB - 1), NA_real_)
vA[!is.na(vA) & vA < 0] <- 0
vB[!is.na(vB) & vB < 0] <- 0
V <- vA / nA + vB / nB
delta <- mB - mA
valid <- nA >= MIN_PER_ARM & nB >= MIN_PER_ARM &
is.finite(V) & V > 0 & is.finite(delta)
idx <- which(valid)
if (length(idx) == 0) {
stop(sprintf(
paste0("Neither arm of '%s' reached %d observations with any variation ",
"in '%s', so the experiment cannot be monitored yet. %s has %d ",
"row(s) and %s has %d."),
variant_h, MIN_PER_ARM, outcome_h,
control_arm, sum(is_c), treat_arm, sum(is_t)))
}
h_path <- .av_halfwidth(V[idx], t2, ALPHA)
d_path <- delta[idx]
lam_path <- .av_lambda(d_path, V[idx], t2)
se_path <- sqrt(V[idx])
dfw <- (V[idx])^2 / ((vA[idx] / nA[idx])^2 / pmax(nA[idx] - 1, 1) +
(vB[idx] / nB[idx])^2 / pmax(nB[idx] - 1, 1))
dfw[!is.finite(dfw) | dfw < 1] <- 1
t_path <- d_path / se_path
p_fh_path <- 2 * stats::pt(-abs(t_path), df = dfw)
fh_hw_path <- stats::qt(1 - ALPHA / 2, df = dfw) * se_path
p_av_path <- cummin(pmin(1, 1 / lam_path))
crossed_av <- is.finite(h_path) & abs(d_path) > h_path
naive_sig <- is.finite(p_fh_path) & p_fh_path < ALPHA
n_points <- length(idx)
av_points <- sum(crossed_av)
naive_points <- sum(naive_sig)
naive_excursions <- sum(naive_sig & !c(FALSE, head(naive_sig, -1)))
cross_idx <- if (av_points > 0) which(crossed_av)[1] else NA_integer_LAT-5113: a fixed-horizon "stop" is a premature call only BEFORE the approximately always-valid boundary is first crossed. After it the effect has been established, so later fixed-horizon crossings are agreement, not false alarms (a planted 5% vs 8% test counted 952 true detections as "false alarms avoided").
before_cross <- if (is.na(cross_idx)) rep(TRUE, length(crossed_av)) else seq_along(crossed_av) < cross_idx
naive_only <- sum(naive_sig & !crossed_av & before_cross)
cross_n <- if (!is.na(cross_idx)) idx[cross_idx] + n_burn else NA_integer_
naive_first_idx <- if (naive_points > 0) which(naive_sig)[1] else NA_integer_
naive_first_n <- if (!is.na(naive_first_idx)) idx[naive_first_idx] + n_burn else NA_integer_Step 8: The current decision
LAT-5113: the approximately always-valid confidence sequence is the RUNNING INTERSECTION of every interval seen so far (each one approximately covers the truth at all times simultaneously, so their intersection does too), the same running logic that makes the approximately always-valid p-value a running minimum. The decision used to read only the last point's interval, which re-widens and can re-include zero after a valid crossing: a planted 3-point lift crossed at observation 1,196, had approximately always-valid p 0.008, and was told "no boundary is reachable". Now a crossing at any look is a stop for a winner, as the p-value says.
last <- length(idx)
effect <- d_path[last]
h_final <- h_path[last]
se_final <- se_path[last]
fh_hw_final <- fh_hw_path[last]
p_av <- p_av_path[last]
p_fh <- p_fh_path[last]The decision is tied to the CROSSING (any look whose interval excludes zero is a stop for a winner, the same event as the approximately always-valid p falling below alpha), and the reported interval always agrees with it. When the running intersection is EMPTY (two looks whose intervals cannot both cover the truth, e.g. a path that crossed up and later down; it review of #2289), the interval reported is the one at the FIRST crossing look, which excludes zero and names the direction decided, with a stated note that the intervals from different looks disagree (allowed in at most about alpha of tests).
run_cs <- function(hh) {
ok <- is.finite(hh)
crossed_t <- ok & abs(d_path) > hh
first <- if (any(crossed_t)) which(crossed_t)[1] else NA_integer_
lo <- cummax(ifelse(ok, d_path - hh, -Inf)); hi <- cummin(ifelse(ok, d_path + hh, Inf))
lo_l <- lo[last]; hi_l <- hi[last]
empty <- is.finite(lo_l) && is.finite(hi_l) && lo_l > hi_l
if (empty) { lo_l <- d_path[first] - hh[first]; hi_l <- d_path[first] + hh[first] }
list(lo = lo_l, hi = hi_l, crossed = !is.na(first), first = first, empty = empty)
}
av_cs <- run_cs(h_path)
av_low <- av_cs$lo
av_high <- av_cs$hi
cs_empty <- av_cs$empty
fh_low <- effect - fh_hw_final
fh_high <- effect + fh_hw_final
cs_note <- if (cs_empty) paste0(
"The intervals from different looks disagree: the effect first crossed the boundary at observation ",
format(idx[av_cs$first] + n_burn, big.mark = ","), " and later moved far enough the other way that no single ",
"range is inside every look's interval. This is allowed to happen in at most about ",
round(100 * ALPHA), "% of tests; the interval shown is the one at that first crossing, and a change ",
"in the effect over time(a novelty effect, a shift in who arrived) is worth checking before acting.") else NULL
decide <- function(cs) {
if (isTRUE(cs$crossed)) return("winner")
lo <- cs$lo; hi <- cs$hi
if (!is.finite(lo) || !is.finite(hi)) return("insufficient")
if ((hi - lo) / 2 > tau) return("insufficient")
if (max(abs(lo), abs(hi)) <= tau) return("futility")
"continue"
}
decision <- decide(av_cs)
decision_label <- switch(
decision,
winner = "Stop - a winner has separated",
futility = "Stop for futility - no effect worth acting on remains possible",
continue = "Keep running - no boundary has been crossed yet",
insufficient = "Not enough data yet - no boundary is reachable at this sample size")Step 9: The price of the guarantee, in this experiment's own numbers
frac_c <- sum(is_c) / n_seq
frac_c <- min(max(frac_c, 1e-6), 1 - 1e-6)
vA_f <- vA[idx][last]; vB_f <- vB[idx][last]
h_at_n <- function(n) {
Vn <- vA_f / (n * frac_c) + vB_f / (n * (1 - frac_c))
.av_halfwidth(Vn, t2, ALPHA)
}
grid <- unique(round(exp(seq(log(n_seq), log(n_seq * 5000), length.out = 6000))))
grid <- grid[is.finite(grid) & grid >= n_seq]
hg <- h_at_n(grid)
first_at_or_below <- function(target) {
okv <- which(is.finite(hg) & hg <= target)
if (length(okv) == 0) NA_real_ else grid[okv[1]] + n_burn
}
n_equiv <- first_at_or_below(fh_hw_final)
n_needed_futility <- if (decision == "insufficient")
first_at_or_below(tau) else NA_real_
width_ratio <- if (is.finite(fh_hw_final) && fh_hw_final > 0)
h_final / fh_hw_final else NA_real_Step 10: Sensitivity to the pre-specified alternative
tau_mults <- c(0.05, 0.10, 0.20)
tau_rows <- lapply(tau_mults, function(mm) {
tt <- mm * sd_all
hh <- .av_halfwidth(V[idx], tt^2, ALPHA)
dd <- decide(run_cs(hh))
data.frame(
alternative = paste0(.fmt_n(mm, 2), " x the outcome's standard deviation"),
alternative_size = .fmt_n(to_disp(tt), disp_digits),
half_width = .fmt_n(to_disp(hh[last]), disp_digits),
decision = switch(dd, winner = "stop for a winner",
futility = "stop for futility",
continue = "keep running",
insufficient = "not enough data"),
ever_crossed = if (sum(is.finite(hh) & abs(d_path) > hh) > 0) "yes" else "no",
stringsAsFactors = FALSE)
})
tau_df <- do.call(rbind, tau_rows)
tau_agreement <- length(unique(tau_df$decision)) == 1Step 12: One computed paragraph for the JSON answer
contrast_txt <- if (naive_only > 0) {
paste0("Along the way the fixed-horizon p-value fell below 0.05 at ",
format(naive_only, big.mark = ","), " analysis point(s) ",
if (!is.na(cross_n)) "before the approximately always-valid boundary was first crossed" else "where the approximately always-valid boundary was not crossed",
", in ", format(naive_excursions, big.mark = ","),
" separate excursion(s) of the fixed-horizon p-value",
if (!is.na(cross_n)) paste0(": calls a peeking team would have made before the evidence ",
"was strong enough to stop on, even though the effect later held up.")
else ": that gap is the false-positive risk peeking at a fixed-horizon test would have exposed you to here.")
} else if (naive_points > 0) {
paste0("The fixed-horizon p-value was below 0.05 at ",
format(naive_points, big.mark = ","),
" analysis point(s), all of them points where the approximately always-valid ",
"boundary had also been crossed, so on this data the two tests ",
"never disagreed.")
} else {
paste0("The fixed-horizon p-value never fell below 0.05 at any of the ",
format(n_points, big.mark = ","),
" analysis points, so on this data peeking would not have misled ",
"you - which is luck, not protection.")
}
json_output <- list(
answer = paste0(
decision_label, ". After ", format(n_total, big.mark = ","),
" observations the effect of ", treat_arm, " over ", control_arm,
" is ", fmt_eff(effect), " with an approximately always-valid(asymptotic; sample variances plugged into a normal mixture boundary) 95% confidence sequence of ",
fmt_eff(av_low), " to ", fmt_eff(av_high), unit_suffix,
" (approximately always-valid p-value ", .fmt_p(p_av), "). ", contrast_txt,
" The guarantee costs power: the approximately always-valid interval is ",
if (is.finite(width_ratio)) paste0(.fmt_n(width_ratio, 2), " times") else "wider than",
" the fixed-horizon interval at this sample size.",
if (!is.null(cs_note)) paste0(" ", cs_note) else ""
),
cards = lapply(
c("tldr", "overview", "preprocessing", "boundary_path", "pvalue_paths",
"decision", "peek_comparison", "power_cost", "tau_sensitivity", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows,
rows_removed = rows_removed, n_dropped = n_dropped, n_tied = n_tied,
variant_h = variant_h, outcome_h = outcome_h, arrival_h = arrival_h,
arrival_kind = arrival_kind, outcome_read = outcome_read,
tiny_arms = tiny_arms,
control_arm = control_arm, treat_arm = treat_arm, control_rule = control_rule,
n_control = n_control_all, n_treat = n_treat_all, n_total = n_total, n_burn = n_burn,
is_binary = is_binary, unit = unit, unit_suffix = unit_suffix,
to_disp = to_disp, fmt_eff = fmt_eff,
sd_all = sd_all, tau = tau, alpha = ALPHA, tau_mult = TAU_MULT,
min_per_arm = MIN_PER_ARM,
decision = decision, decision_label = decision_label, cs_empty = cs_empty, cs_note = cs_note,
effect = effect, av_low = av_low, av_high = av_high,
fh_low = fh_low, fh_high = fh_high,
h_final = h_final, fh_hw_final = fh_hw_final, se_final = se_final,
p_av = p_av, p_fh = p_fh,
cross_idx = cross_idx, cross_n = cross_n,
naive_first_idx = naive_first_idx, naive_first_n = naive_first_n,
n_points = n_points, naive_points = naive_points,
naive_excursions = naive_excursions, av_points = av_points,
naive_only = naive_only,
width_ratio = width_ratio, n_equiv = n_equiv,
n_needed_futility = n_needed_futility, grid_max = max(grid) + n_burn,
tau_df = tau_df, tau_agreement = tau_agreement,
boundary_path_df = boundary_path_df, pvalue_paths_df = pvalue_paths_df,
decision_df = decision_df, peek_df = peek_df, power_df = power_df,
methods_df = methods_df,
metrics = metrics, json_output = json_output
)
}Your turn
Bring your own data and the question you actually need answered.
MCP AnalyticsData science by email Send us your data and your question, and we will reply. ds@mcpanalytics.ai