Can I Stop?
Stop for futility — no effect worth acting on remains possible
The short answer
Stop the test for futility—no effect large enough to act on remains possible. The always-valid confidence interval (−0.79 pp to 0.61 pp) has narrowed entirely inside the 3.24 pp effect size this test was designed to detect, meaning every meaningful difference has been ruled out.
The detail
After 100,000 observations, the effect estimate is −0.09 pp with an always-valid p-value of 0.7449 (fixed-horizon p = 0.6597). The always-valid confidence sequence spans −0.79 to 0.61 pp—both bounds sit comfortably inside ±3.24 pp, the minimum detectable effect. The boundary was never crossed; no false alarms were avoided on this run because the fixed-horizon p-value never fell below 0.05 either. The guarantee costs precision: the always-valid interval is 1.73 times wider than the fixed-horizon interval at this sample size. Reaching the fixed-horizon precision would require approximately 331,443 observations, or 3.31 times the current sample.
What this can't tell you
This verdict applies only to the comparison as specified. It does not extend to other metrics, segments, or treatment arms selected post-hoc.
Analysis Overview
Always-valid sequential monitoring of treatment against control across 100,000 observations.
The short answer
This analysis replaces a one-time fixed-horizon test with a sequential boundary that remains valid at every observation point. You can now monitor your landing page test as data arrives without inflating false-positive risk, because the boundary adjusts for repeated looks rather than breaking the 5% guarantee that a single-look test provides.
The detail
A fixed-horizon p-value earns its 5% false-positive rate only if you decide your sample size before collecting data and look once. The moment you peek a second time, that 5% guarantee evaporates. This analysis uses a mixture sequential probability ratio test with a normal mixture over the true effect, re-testing after every observation with a boundary valid at all 99,976 analysis points simultaneously. Rows are ordered by timestamp. The effect is mean converted for treatment (n=50,055) minus mean converted for control (n=49,945). Control was identified by name, not by outcome direction. The same cumulative data also runs through an ordinary Welch t-test at every point for direct comparison.
What this can't tell you
The analysis covers this single comparison only; it does not protect a metric, segment, or arm chosen after observing results.
Data Quality
Row accounting, arm sizes, and how the ordering was resolved.
The short answer
All 100,000 rows loaded and analysed without loss. The outcome column converted is already numeric (0 and 1 only), so the effect is a difference in conversion rates. Arm sizes are balanced: control 49,945, treatment 50,055. Tied arrival timestamps were resolved by keeping original file order.
The detail
No rows removed; 100,000 final rows match 100,000 initial. The outcome converted holds only 0 and 1, making the effect a percentage-point difference in conversion rates. Control contributed 49,945 rows and treatment 50,055. Of the 100,000 rows, 99,977 share an arrival value with an earlier row; these tied rows retain their original file position, so the monitoring path steps through them one at a time rather than collapsing them.
What this can't tell you
Nothing here verifies that assignment to group was actually random, or that each row is a different person. Both are assumptions inherited from how the experiment was run. A transaction-level or user-level export would allow checking for repeat visits, but the current export is row-level.
Effect and Boundaries Over the Experiment
The cumulative effect with the always-valid and fixed-horizon boundaries drawn over it.
The short answer
The cumulative effect estimate stayed inside the always-valid corridor at all 99,976 analysis points, wandering between roughly −2 pp and +1 pp before settling near −0.09 pp. Both the always-valid boundary (outer) and the fixed-horizon boundary (inner) narrowed as data accumulated, but the always-valid boundary narrows more slowly because it pays for every look you might take.
The detail
The middle series is the cumulative effect (treatment minus control), recomputed as each observation arrives. The outer pair is the always-valid boundary—crossing it is a valid stop no matter how many times you looked. The inner pair is the fixed-horizon 0.05 boundary, drawn for contrast; it is narrower everywhere, which is precisely why watching it makes teams stop too early. The effect started at −6.67 pp at 25 observations, reached a low of about −1.83 pp around 2,670 observations, and gradually drifted toward zero, ending at −0.09 pp. No boundary crossing occurred.
What this can't tell you
The visualization assumes continuous monitoring after every observation. In practice, a weekly or daily check schedule would see fewer decision points and lower cumulative risk of false alarm, though not zero.
The Two p-Values Side by Side
The fixed-horizon p-value and the always-valid p-value at every analysis point.
The short answer
The fixed-horizon p-value wandered above 0.05 at all 99,976 analysis points, never triggering a false alarm on this run. The always-valid p-value, which only ever moves down, also never crossed 0.05. Both are computed from your own data at identical points; this is not an illustration.
The detail
The fixed-horizon p-value is recomputed from scratch at each point and is free to wander back above 0.05 after dipping below it—the behaviour that makes repeated checking dangerous. The always-valid p-value is the running minimum of the inverse mixture likelihood ratio, so it only ever decreases and is a legitimate stopping criterion at any point. Neither series dropped below 0.05 across all 99,976 analysis points. The fixed-horizon series reached a minimum of 0.1472 at 2,670 observations but climbed back above 0.5 and stayed there. The always-valid series remained high throughout, ending at 0.7449.
What this can't tell you
This run did not demonstrate the peeking risk because the fixed-horizon p-value never actually crossed 0.05. The danger is real but invisible here; it is a property of the procedure, not of this particular dataset.
The Decision, Line by Line
Everything the stopping decision rests on.
| Item | Detail |
|---|---|
| Decision | Stop for futility — no effect worth acting on remains possible |
| Always-valid boundary crossed | no |
| Observations analysed | 100,000 |
| Effect (treatment minus control) | -0.09 pp |
| Always-valid 95% confidence sequence | -0.79 pp to 0.61 pp |
| Always-valid p-value | 0.7449 |
| Fixed-horizon p-value at this sample size | 0.6597 |
| Smallest effect this test was set up to find | 3.24 pp |
| Control arm | control (n = 49,945) |
| Treatment arm | treatment (n = 50,055) |
The short answer
Stop for futility—the always-valid confidence interval (−0.79 pp to 0.61 pp) lies entirely within the smallest effect worth acting on (3.24 pp), so no remaining effect is large enough to justify continuing.
The detail
The decision rule has three exits and one refusal. Stop for a winner when the confidence sequence excludes zero. Stop for futility when it lies entirely within 3.24 pp (the smallest effect the test was set up to find). Keep running when neither is true but both are still reachable. Report "no boundary reachable" when the interval is wider than that smallest effect. Here: effect −0.09 pp, always-valid 95% confidence sequence −0.79 pp to 0.61 pp, always-valid p-value 0.7449, fixed-horizon p-value 0.6597. The interval sits squarely inside ±3.24 pp, ruling out every effect large enough to matter. Control: 49,945 rows. Treatment: 50,055 rows. Verdict: stop for futility.
What this can't tell you
Futility here means the data are consistent with no effect or a very small effect. It does not prove the treatment has zero impact in the population, only that the observed data do not support continuing to look for one.
What Peeking Would Have Cost You
The fixed-horizon test and the always-valid boundary, counted over the same analysis points.
| Check | Value |
|---|---|
| Analysis points examined (one after every observation) | 99,976 |
| Points where the fixed-horizon p-value was below 0.05 | 0 |
| Separate excursions below 0.05 by the fixed-horizon p-value | 0 |
| Points where the always-valid boundary was crossed | 0 |
| Points where the fixed-horizon test said stop but the always-valid boundary did not | 0 |
| First point the fixed-horizon test would have stopped | never |
| First point the always-valid boundary was crossed | never |
The short answer
Over 99,976 analysis points, the fixed-horizon p-value fell below 0.05 at 0 points in 0 separate excursions; the always-valid boundary was crossed 0 times. No point fired the fixed-horizon test alone, so this run does not demonstrate the peeking failure—but the risk is a property of the procedure, not of whether it happened to bite you.
The detail
Both tests were run at all 99,976 analysis points on your data. The fixed-horizon p-value never dropped below 0.05; the always-valid boundary was never crossed. The first point the fixed-horizon test would have stopped: never. The first point the always-valid boundary was crossed: never. No discrepancy between the two methods occurred on this particular run. Note that these counts assume the most aggressive schedule possible—a look after every observation; a team checking weekly would see fewer opportunities to be misled, but not zero.
What this can't tell you
The absence of a false alarm on this run does not mean the peeking risk is zero. The danger is baked into the fixed-horizon procedure itself. A longer or different dataset could easily trigger the mismatch, which is why the always-valid boundary exists.
The Price of Being Allowed to Peek
How much extra data the always-valid guarantee costs on this experiment.
| Measure | Value |
|---|---|
| Always-valid half-width at this sample size | 0.70 pp |
| Fixed-horizon half-width at this sample size | 0.40 pp |
| How much wider the always-valid interval is | 1.73 times |
| Observations needed for the always-valid interval to be as tight as the fixed-horizon one is now | 331,443 |
| That as a multiple of the current sample size | 3.31 times |
The short answer
The always-valid interval is 1.73 times wider than the fixed-horizon interval at this sample size. Reaching the fixed-horizon's current precision under the always-valid boundary would require about 331,443 observations—roughly 3.31 times what you have. If you can genuinely commit to a single analysis at a fixed sample size, use the fixed-horizon test; this tool is for the common case where that commitment is not realistic.
The detail
Always-valid inference buys the right to peek whenever you like by widening the interval, requiring more data than a fixed-horizon test to reach the same conclusion. On this experiment the always-valid half-width is 0.70 pp against the fixed-horizon 0.40 pp—a factor of 1.73. Reaching today's fixed-horizon precision (0.40 pp half-width) under the always-valid boundary would take about 331,443 observations, or 3.31 times the current 100,000. The trade-off is explicit: you gain the freedom to stop early or keep running; you pay with a wider interval at any fixed sample size.
What this can't tell you
This comparison assumes both methods are used on the same data. In practice, if you commit to a single pre-registered sample size and analysis, the fixed-horizon test is more efficient and should be used instead.
Does the Verdict Survive a Different Alternative?
The same monitoring path re-run against three pre-specified alternatives.
| Alternative | Alternative Size | Half Width | Decision | Ever Crossed |
|---|---|---|---|---|
| 0.05 x the outcome's standard deviation | 1.62 | 0.66 | stop for futility | no |
| 0.10 x the outcome's standard deviation | 3.24 | 0.70 | stop for futility | no |
| 0.20 x the outcome's standard deviation | 6.48 | 0.74 | stop for futility | no |
The short answer
The verdict holds across all three tuning choices: stop for futility at tau = 0.05, 0.10, and 0.20 times the outcome's standard deviation. The conclusion does not depend on the tuning constant.
The detail
The boundary's tuning parameter tau defines the effect size the test is aimed at. Here it is set to 0.10 times the outcome's standard deviation (3.24 pp), read from how converted varies rather than from the observed gap between arms. At tau = 0.05 (1.62 pp), the half-width is 0.66 pp and the decision is stop for futility. At tau = 0.10 (3.24 pp), the half-width is 0.70 pp and the decision is stop for futility. At tau = 0.20 (6.48 pp), the half-width is 0.74 pp and the decision is stop for futility. All three paths remain unviolated. Strictly, tau should be fixed before the experiment; it is set from the outcome's spread here because the tool is not configured with your design target.
What this can't tell you
This sensitivity check covers only the tuning constant. It does not address whether 3.24 pp (or any other tau) is the right business threshold for your test; that choice belongs to your experiment design, not to this analysis.
Methods and Disclosure
Every formula, constant and assumption behind the decision.
| Item | Detail |
|---|---|
| Test | Mixture sequential probability ratio test (mSPRT) on the difference in means between the two arms, implemented directly from the normal mixture — no group-sequential package is used. |
| Boundary | Robbins' normal-mixture boundary. The mixture likelihood ratio is Lambda = sqrt(V/(V+tau^2)) exp(delta^2 tau^2 / (2 V (V+tau^2))) with V the sampling variance of the effect; Ville's inequality bounds the chance that Lambda ever reaches 1/alpha at alpha, which is what makes continuous monitoring safe. |
| Confidence sequence | Inverting the boundary gives effect +/- h with h = sqrt(2 V (V+tau^2)/tau^2 log(sqrt((V+tau^2)/V)/alpha)); this interval covers the true effect at every sample size at once, not just at one pre-chosen sample size. |
| Pre-specified alternative (tau) | 3.24 pp — set at 0.10 times the outcome's standard deviation (32.39). It is the effect size the boundary is tuned to find, and it is read from how much the outcome varies, not from the difference between the arms. |
| Significance level | alpha = 0.05, applied to the whole monitoring path rather than to a single look. |
| Analysis schedule | The test is evaluated after every single observation from the point where both arms reach 10 rows — the most aggressive peeking possible — giving 99,976 analysis points. |
| Fixed-horizon comparison | At each of those points a Welch two-sample t-test is computed on the same cumulative data, purely so the two boundaries can be compared on this experiment's own numbers. |
| Effect definition | Mean of treatment minus mean of control, both being conversion rates, reported in percentage points. |
| Control arm chosen by | control — its name identifies it as the control. |
| Arrival order read as | calendar dates in the column 'timestamp'; 99,977 row(s) share an arrival value with an earlier row and keep their original file order. |
| What this cannot do | The guarantee is about the stopping rule, not about everything else. It holds for this one pre-specified comparison of these two arms on this one outcome. It does not cover choosing the metric, the segment or the arm after seeing the results, it cannot detect repeat visits by the same person, and it does not know whether the assignment was actually random. |
The short answer
The analysis uses a mixture sequential probability ratio test (mSPRT) with Robbins' normal-mixture boundary. Under the null, the chance the mixture likelihood ratio ever reaches 1/0.05 is capped at 0.05, no matter how many times you look. The confidence sequence inverts that boundary and covers the true effect at every sample size simultaneously.
The detail
The mixture likelihood ratio is Lambda = sqrt(V/(V+tau²)) exp(delta² tau² / (2V(V+tau²))), with V the sampling variance of the effect and tau = 3.24 pp (0.10 times the outcome's standard deviation, 32.39). Ville's inequality ensures that under the null, P(Lambda ≥ 1/alpha) ≤ alpha, making continuous monitoring safe. The confidence sequence is effect ± h, where h = sqrt(2V(V+tau²)/tau² log(sqrt((V+tau²)/V)/alpha)), covering the true effect at all sample sizes at once. A Welch two-sample t-test is computed at each of the 99,976 analysis points (evaluated after both arms reach 10 rows) for comparison. Effect is mean of treatment minus mean of control, both conversion rates, in percentage points. Control arm identified by name; arrival order read as calendar dates from the timestamp column.
What this can't tell you
The normal mixture is asymptotic, so at very small samples stated coverage is optimistic. Variance is estimated from the same data being tested, another reason the analysis refuses to declare a null when no boundary is reachable. Most critically, the guarantee covers the stopping rule and this one pre-specified comparison only—it does not protect choosing the outcome, segment, or arm after seeing results, nor does it detect repeat visits or verify random assignment.
Sequential A/B Test — Peek Safely
Reads an ordered experiment log (who saw which variant, what happened, and when the observation arrived) and answers the question a running experiment actually raises: given everything so far, can I stop?
Why This Method?
A fixed-horizon p-value is only valid if you look once, at a sample size fixed before the data existed. Look after every visitor and the chance of seeing p < 0.05 somewhere along the way is far above 5%. A mixture sequential probability ratio test (mSPRT) replaces the fixed critical value with a boundary that is valid at every sample size simultaneously, so monitoring the experiment continuously does not inflate the false-positive rate. The price is power, and this module states that price in the user's own numbers rather than selling peeking as free.
What This Analysis Covers
- The always-valid confidence sequence for the treatment effect over time
- The current decision: stop for a winner, stop for futility, keep running,
or not enough data for any boundary to be reachable
- The naive fixed-horizon p-value computed alongside, and how many times it
would have crossed 0.05 while the always-valid boundary did not
- The cost of the guarantee: how much wider the always-valid interval is and
how many observations it needs to match today's fixed-horizon precision
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {variant, outcome, arrival}. The mSPRT, the confidence sequence and the boundary crossing are implemented directly from the normal mixture — no group-sequential package is used.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
Step 1: Confirm the mapping and humanize the user's column names
variant_h <- humanize_semantic("variant", col_map)
outcome_h <- humanize_semantic("outcome", col_map)
arrival_h <- humanize_semantic("arrival", col_map)
needed <- c("variant", "outcome", "arrival")
missing_keys <- needed[!(needed %in% names(df))]
if (length(missing_keys) > 0) {
labels <- c(variant = variant_h, outcome = outcome_h, arrival = arrival_h)
stop(sprintf(
paste0("Sequential testing needs three columns mapped: the variant each ",
"row saw, the outcome, and the order or date the observation ",
"arrived. Missing: %s."),
paste(labels[missing_keys], collapse = ", ")))
}Step 2: Read the arrival column as dates, or as a numeric order
arr_ch <- trimws(as.character(df$arrival))
arr_ch[arr_ch == "NA"] <- NA_character_
n_nonblank_arr <- sum(!is.na(arr_ch) & arr_ch != "")
parse_dates_vec <- function(ch) {
ok <- function(dd) sum(!is.na(dd)) >= 0.95 * sum(!is.na(ch) & trimws(ch) != "")
d <- suppressWarnings(as.Date(ch, format = "%Y-%m-%d"))
if (!ok(d)) d <- suppressWarnings(as.Date(lubridate::ymd(ch, quiet = TRUE)))
if (!ok(d)) d <- suppressWarnings(as.Date(lubridate::mdy(ch, quiet = TRUE)))
if (!ok(d)) d <- suppressWarnings(as.Date(lubridate::dmy(ch, quiet = TRUE)))
if (ok(d)) d else NULL
}
arrival_kind <- ""
ord_key <- NULL
if (n_nonblank_arr > 0) {
dts <- parse_dates_vec(arr_ch)
if (!is.null(dts)) {
ord_key <- as.numeric(dts)
arrival_kind <- "calendar dates"
} else {
nv <- suppressWarnings(as.numeric(arr_ch))
if (sum(!is.na(nv)) >= 0.95 * n_nonblank_arr) {
ord_key <- nv
arrival_kind <- "a numeric arrival order"
}
}
}
if (is.null(ord_key)) {
stop(sprintf(
paste0("The arrival column '%s' could not be read either as dates or as ",
"a numeric order, so the observations cannot be put in the order ",
"they arrived. Sequential testing is entirely about that order — ",
"map a date, a timestamp, or a running sequence number."),
arrival_h))
}Step 3: Read the outcome as a number (95% rule, then yes/no words)
y_raw <- df$outcome
outcome_read <- ""
if (is.numeric(y_raw)) {
y <- as.numeric(y_raw)
outcome_read <- "already numeric"
} else {
ch <- trimws(as.character(y_raw))
ch[ch == "NA"] <- NA_character_
nonblank <- !is.na(ch) & ch != ""
conv <- suppressWarnings(as.numeric(ch))
if (sum(nonblank) > 0 && sum(!is.na(conv)) >= 0.95 * sum(nonblank)) {
y <- conv
outcome_read <- "read as numbers"
} else {
lut <- c("yes" = 1, "no" = 0, "true" = 1, "false" = 0, "y" = 1, "n" = 0,
"t" = 1, "f" = 0, "converted" = 1, "not converted" = 0,
"1" = 1, "0" = 0, "success" = 1, "failure" = 0)
conv2 <- unname(lut[tolower(ch)])
if (sum(nonblank) > 0 && sum(!is.na(conv2)) >= 0.95 * sum(nonblank)) {
y <- conv2
outcome_read <- "read as yes/no labels turned into 1 and 0"
} else {
stop(sprintf(
paste0("The outcome column '%s' is not numeric and is not a yes/no ",
"flag — fewer than 95%% of its values could be turned into ",
"numbers. Map a numeric outcome(revenue, time, a 0/1 ",
"conversion flag) so the two arms can be compared."),
outcome_h))
}
}
}Step 4: Keep complete rows only
g_ch <- trimws(as.character(df$variant))
g_ch[g_ch == "" | g_ch == "NA"] <- NA_character_
keep <- !is.na(g_ch) & !is.na(y) & !is.na(ord_key)
n_dropped <- sum(!keep)
g_k <- g_ch[keep]; y_k <- y[keep]; o_k <- ord_key[keep]
final_rows <- length(y_k)
rows_removed <- initial_rows - final_rows
if (final_rows < 30) {
stop(sprintf(
paste0("Only %d complete row(s) remained after dropping rows missing ",
"%s, %s or %s. Sequential monitoring needs at least 30 ",
"observations before any boundary means anything."),
final_rows, variant_h, outcome_h, arrival_h))
}Step 5: Resolve the two arms — never chosen after seeing the outcome
tab <- sort(table(g_k), decreasing = TRUE)
tiny_arms <- names(tab)[tab < MIN_ARM_KEEP]
if (length(tiny_arms) > 0) {
keep2 <- !(g_k %in% tiny_arms)
g_k <- g_k[keep2]; y_k <- y_k[keep2]; o_k <- o_k[keep2]
final_rows <- length(y_k)
rows_removed <- initial_rows - final_rows
tab <- sort(table(g_k), decreasing = TRUE)
}
arms <- names(tab)
if (length(arms) < 2) {
stop(sprintf(
paste0("The variant column '%s' holds only %d group%s with at least %d ",
"observation%s(%s). A sequential A/B test compares exactly two ",
"arms."),
variant_h, length(arms), if (length(arms) == 1) "" else "s",
MIN_ARM_KEEP, "s", paste(arms, collapse = ", ")))
}
if (length(arms) > 2) {
stop(sprintf(
paste0("The variant column '%s' holds %d arms (%s). This analysis will ",
"not pick two of them for you: the always-valid guarantee covers ",
"the comparison you specified before looking at the data, and ",
"choosing which arms to compare after seeing the results is ",
"exactly what it does not protect against. Filter the file to the ",
"two arms you want to compare and re-run."),
variant_h, length(arms), paste(arms, collapse = ", ")))
}Control is chosen by NAME, not by outcome — a fixed, stated rule.
ctrl_pat <- "^(control|baseline|a|original|holdout|current|old)$"
ctrl_hit <- arms[grepl(ctrl_pat, tolower(trimws(arms)))]
control_arm <- if (length(ctrl_hit) >= 1) ctrl_hit[1] else sort(arms)[1]
control_rule <- if (length(ctrl_hit) >= 1)
"its name identifies it as the control" else
"no arm was named as a control, so the alphabetically first label was used"
treat_arm <- setdiff(arms, control_arm)[1]Step 6: Order by arrival (ties keep their original file order)
ordv <- order(o_k, seq_along(o_k))
g_s <- g_k[ordv]; y_s <- y_k[ordv]; o_s <- o_k[ordv]
n_tied <- sum(duplicated(o_s))
n_total <- length(y_s)
sd_all <- stats::sd(y_s)
if (!is.finite(sd_all) || sd_all <= 0) {
stop(sprintf(
paste0("Every retained row has the same value in the outcome column ",
"'%s', so there is no difference between the arms to test and no ",
"boundary to cross."),
outcome_h))
}
tau <- TAU_MULT * sd_all
t2 <- tau^2
is_binary <- all(y_s %in% c(0, 1))
unit <- if (is_binary) "percentage points" else "units of the outcome"fmt_eff already carries "pp" for a binary outcome, so the spelled-out unit is appended only when it would not be a duplicate.
unit_suffix <- if (is_binary) "" else paste0(" ", unit)
to_disp <- function(x) if (is_binary) 100 * x else x
disp_digits <- if (is_binary) 2 else 4
fmt_eff <- function(x) trimws(paste0(.fmt_n(to_disp(x), disp_digits),
if (is_binary) " pp" else ""))Step 7: Cumulative statistics after every single observation
is_c <- as.integer(g_s == control_arm)
is_t <- 1L - is_c
nA <- cumsum(is_c); nB <- cumsum(is_t)
sA <- cumsum(y_s * is_c); sB <- cumsum(y_s * is_t)
qA <- cumsum(y_s^2 * is_c); qB <- cumsum(y_s^2 * is_t)
mA <- ifelse(nA > 0, sA / nA, NA_real_)
mB <- ifelse(nB > 0, sB / nB, NA_real_)
vA <- ifelse(nA > 1, (qA - nA * mA^2) / (nA - 1), NA_real_)
vB <- ifelse(nB > 1, (qB - nB * mB^2) / (nB - 1), NA_real_)
vA[!is.na(vA) & vA < 0] <- 0
vB[!is.na(vB) & vB < 0] <- 0
V <- vA / nA + vB / nB
delta <- mB - mA
valid <- nA >= MIN_PER_ARM & nB >= MIN_PER_ARM &
is.finite(V) & V > 0 & is.finite(delta)
idx <- which(valid)
if (length(idx) == 0) {
stop(sprintf(
paste0("Neither arm of '%s' reached %d observations with any variation ",
"in '%s', so the experiment cannot be monitored yet. %s has %d ",
"row(s) and %s has %d."),
variant_h, MIN_PER_ARM, outcome_h,
control_arm, sum(is_c), treat_arm, sum(is_t)))
}
h_path <- .av_halfwidth(V[idx], t2, ALPHA)
d_path <- delta[idx]
lam_path <- .av_lambda(d_path, V[idx], t2)
se_path <- sqrt(V[idx])
dfw <- (V[idx])^2 / ((vA[idx] / nA[idx])^2 / pmax(nA[idx] - 1, 1) +
(vB[idx] / nB[idx])^2 / pmax(nB[idx] - 1, 1))
dfw[!is.finite(dfw) | dfw < 1] <- 1
t_path <- d_path / se_path
p_fh_path <- 2 * stats::pt(-abs(t_path), df = dfw)
fh_hw_path <- stats::qt(1 - ALPHA / 2, df = dfw) * se_path
p_av_path <- cummin(pmin(1, 1 / lam_path))
crossed_av <- is.finite(h_path) & abs(d_path) > h_path
naive_sig <- is.finite(p_fh_path) & p_fh_path < ALPHA
n_points <- length(idx)
av_points <- sum(crossed_av)
naive_points <- sum(naive_sig)
naive_excursions <- sum(naive_sig & !c(FALSE, head(naive_sig, -1)))
naive_only <- sum(naive_sig & !crossed_av)
cross_idx <- if (av_points > 0) which(crossed_av)[1] else NA_integer_
cross_n <- if (!is.na(cross_idx)) idx[cross_idx] else NA_integer_
naive_first_idx <- if (naive_points > 0) which(naive_sig)[1] else NA_integer_
naive_first_n <- if (!is.na(naive_first_idx)) idx[naive_first_idx] else NA_integer_Step 8: The current decision
last <- length(idx)
effect <- d_path[last]
h_final <- h_path[last]
se_final <- se_path[last]
fh_hw_final <- fh_hw_path[last]
p_av <- p_av_path[last]
p_fh <- p_fh_path[last]
av_low <- effect - h_final
av_high <- effect + h_final
fh_low <- effect - fh_hw_final
fh_high <- effect + fh_hw_final
decide <- function(eff, hw) {
if (!is.finite(hw)) return("insufficient")
if (abs(eff) > hw) return("winner")
if (hw > tau) return("insufficient")
if (abs(eff) + hw <= tau) return("futility")
"continue"
}
decision <- decide(effect, h_final)
decision_label <- switch(
decision,
winner = "Stop — a winner has separated",
futility = "Stop for futility — no effect worth acting on remains possible",
continue = "Keep running — no boundary has been crossed yet",
insufficient = "Not enough data yet — no boundary is reachable at this sample size")Step 9: The price of the guarantee, in this experiment's own numbers
frac_c <- sum(is_c) / n_total
frac_c <- min(max(frac_c, 1e-6), 1 - 1e-6)
vA_f <- vA[idx][last]; vB_f <- vB[idx][last]
h_at_n <- function(n) {
Vn <- vA_f / (n * frac_c) + vB_f / (n * (1 - frac_c))
.av_halfwidth(Vn, t2, ALPHA)
}
grid <- unique(round(exp(seq(log(n_total), log(n_total * 5000), length.out = 6000))))
grid <- grid[is.finite(grid) & grid >= n_total]
hg <- h_at_n(grid)
first_at_or_below <- function(target) {
okv <- which(is.finite(hg) & hg <= target)
if (length(okv) == 0) NA_real_ else grid[okv[1]]
}
n_equiv <- first_at_or_below(fh_hw_final)
n_needed_futility <- if (decision == "insufficient")
first_at_or_below(tau) else NA_real_
width_ratio <- if (is.finite(fh_hw_final) && fh_hw_final > 0)
h_final / fh_hw_final else NA_real_Step 10: Sensitivity to the pre-specified alternative
tau_mults <- c(0.05, 0.10, 0.20)
tau_rows <- lapply(tau_mults, function(mm) {
tt <- mm * sd_all
hh <- .av_halfwidth(V[idx], tt^2, ALPHA)
dd <- decide(effect, hh[last])
data.frame(
alternative = paste0(.fmt_n(mm, 2), " x the outcome's standard deviation"),
alternative_size = .fmt_n(to_disp(tt), disp_digits),
half_width = .fmt_n(to_disp(hh[last]), disp_digits),
decision = switch(dd, winner = "stop for a winner",
futility = "stop for futility",
continue = "keep running",
insufficient = "not enough data"),
ever_crossed = if (sum(is.finite(hh) & abs(d_path) > hh) > 0) "yes" else "no",
stringsAsFactors = FALSE)
})
tau_df <- do.call(rbind, tau_rows)
tau_agreement <- length(unique(tau_df$decision)) == 1Step 12: One computed paragraph for the JSON answer
contrast_txt <- if (naive_only > 0) {
paste0("Along the way the fixed-horizon p-value fell below 0.05 at ",
format(naive_only, big.mark = ","), " analysis point(s) where the ",
"always-valid boundary was not crossed, in ",
format(naive_excursions, big.mark = ","),
" separate excursion(s) — that gap is the false-positive risk ",
"peeking at a fixed-horizon test would have exposed you to here.")
} else if (naive_points > 0) {
paste0("The fixed-horizon p-value was below 0.05 at ",
format(naive_points, big.mark = ","),
" analysis point(s), all of them points where the always-valid ",
"boundary had also been crossed, so on this data the two tests ",
"never disagreed.")
} else {
paste0("The fixed-horizon p-value never fell below 0.05 at any of the ",
format(n_points, big.mark = ","),
" analysis points, so on this data peeking would not have misled ",
"you — which is luck, not protection.")
}
json_output <- list(
answer = paste0(
decision_label, ". After ", format(n_total, big.mark = ","),
" observations the effect of ", treat_arm, " over ", control_arm,
" is ", fmt_eff(effect), " with an always-valid 95% confidence sequence of ",
fmt_eff(av_low), " to ", fmt_eff(av_high), unit_suffix,
" (always-valid p-value ", .fmt_p(p_av), "). ", contrast_txt,
" The guarantee costs power: the always-valid interval is ",
if (is.finite(width_ratio)) paste0(.fmt_n(width_ratio, 2), " times") else "wider than",
" the fixed-horizon interval at this sample size."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "boundary_path", "pvalue_paths",
"decision", "peek_comparison", "power_cost", "tau_sensitivity", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows,
rows_removed = rows_removed, n_dropped = n_dropped, n_tied = n_tied,
variant_h = variant_h, outcome_h = outcome_h, arrival_h = arrival_h,
arrival_kind = arrival_kind, outcome_read = outcome_read,
tiny_arms = tiny_arms,
control_arm = control_arm, treat_arm = treat_arm, control_rule = control_rule,
n_control = sum(is_c), n_treat = sum(is_t), n_total = n_total,
is_binary = is_binary, unit = unit, unit_suffix = unit_suffix,
to_disp = to_disp, fmt_eff = fmt_eff,
sd_all = sd_all, tau = tau, alpha = ALPHA, tau_mult = TAU_MULT,
min_per_arm = MIN_PER_ARM,
decision = decision, decision_label = decision_label,
effect = effect, av_low = av_low, av_high = av_high,
fh_low = fh_low, fh_high = fh_high,
h_final = h_final, fh_hw_final = fh_hw_final, se_final = se_final,
p_av = p_av, p_fh = p_fh,
cross_idx = cross_idx, cross_n = cross_n,
naive_first_idx = naive_first_idx, naive_first_n = naive_first_n,
n_points = n_points, naive_points = naive_points,
naive_excursions = naive_excursions, av_points = av_points,
naive_only = naive_only,
width_ratio = width_ratio, n_equiv = n_equiv,
n_needed_futility = n_needed_futility, grid_max = max(grid),
tau_df = tau_df, tau_agreement = tau_agreement,
boundary_path_df = boundary_path_df, pvalue_paths_df = pvalue_paths_df,
decision_df = decision_df, peek_df = peek_df, power_df = power_df,
methods_df = methods_df,
metrics = metrics, json_output = json_output
)
}