Standard Incrementality
Executive Summary

Executive Summary

How much of the movement in cigarette sales packs pc was incremental, against 45 holdout markets.

Markets
46
Treated Markets
1
Control Markets
45
Pre Periods
9
Post Periods
4
Incremental Lift
-11.18
Lift CI Low
-14.33
Lift CI High
-8.02
Relative Lift (%)
-13.1
Parallel Trends
VIOLATED
Read the caveat before the number. The parallel-trends assumption FAILS here: across the 9 pre-intervention periods, the treated markets were already pulling away from the controls by -0.67 in cigarette sales packs pc per period (p = 0.0162). Difference-in-differences credits the intervention with everything the two arms did differently after it started, so a pre-existing divergence is counted as lift. The treated markets ran -11.18 in cigarette sales packs pc per market-period against what the controls say they would have done anyway — a relative lift of -13.1% on a counterfactual level of 85.28 (95% interval -14.33 to -8.02, p < 0.001), but that number is NOT a clean incremental result — projected forward, the pre-trend alone accounts for -1.68 of it, or 15%. To get a trustworthy answer: collect more pre-intervention periods, pick control markets that actually tracked the treated ones before the change, or weight the controls into a synthetic control that matches the treated pre-period path. Standard errors are clustered on the 46 markets, so the interval reflects market-to-market variation rather than treating each market-period as independent. Difference-in-differences also assumes no other shock hit only the treated markets during the post window: a regional promotion, a competitor exit, a supply problem or weather that landed on one arm alone would be counted as lift.
What this means— written from the computed numbers · yours to edit

The short answer

California's 1989 Prop-99 tobacco program is associated with a reduction of about 11.18 packs per capita in cigarette sales relative to 45 control states, a 13.1% decline from the counterfactual level. However, this estimate is not a clean causal claim: the treated and control markets were already diverging before the intervention opened, pulling apart by 0.67 packs per capita per period (p = 0.0162). That pre-existing trend accounts for 15% of the estimated effect, so the true incremental lift is materially uncertain.

The detail

The difference-in-differences estimate is −11.18 packs per capita per market-period (95% CI −14.33 to −8.02, p < 0.001), representing a −13.1% relative lift against a counterfactual level of 85.28. The parallel-trends check, which is the integrity gate for the design, fails: across 9 pre-intervention periods, the treated market was already moving −0.67 packs per capita per period away from the controls (95% CI −1.21 to −0.13, p = 0.0162). All 8 pre-intervention event-study estimates exclude zero, confirming that divergence predates the policy. Projected forward, the pre-trend alone would account for −1.68 packs per capita, or 15% of the headline effect. Standard errors are clustered on 46 markets.

What this can't tell you

Because the arms were already on different trajectories, the −11.18 figure conflates the intervention's effect with pre-existing drift. The design would be strengthened by more pre-intervention periods to measure the trend precisely, control markets selected for their pre-intervention tracking of California rather than by convenience, or a synthetic control weighting the 45 available controls to match California's pre-period path.

Overview

Analysis Overview

Difference-in-differences on 46 markets in 'state code' — 1 treated against 45 held back.

N Units46
N Treated Units1
N Control Units45
N Pre Periods9
N Post Periods4
Standard Errorsunit-clustered (CR1)
What this means— written from the computed numbers · yours to edit

The short answer

The analysis compares cigarette sales in 1 treated market (California) to 45 control markets before and after 1989, using the controls as the counterfactual for what would have happened without the intervention. Two-way fixed effects absorb persistent differences in market size and shocks common to all states; unit-clustered standard errors account for market-to-market variation.

The detail

Cigarette sales packs per capita is regressed on the treated-by-after indicator in a two-way fixed-effects specification with fixed effects for all 46 markets and all 13 periods (9 pre, 4 post). The 45 control markets absorb whatever else moved cigarette sales in that window—inflation, national health campaigns, competitor behavior—providing a stronger counterfactual than a single series' own pre-period trend. Standard errors are clustered on the 46 markets (CR1), so the confidence interval reflects market-level variation rather than treating each market-period as independent. The method assumes no other shock hit only California during the post window; a regional promotion, supply disruption, or local weather event landing on one arm alone would be counted as lift.

What this can't tell you

The design cannot separate the intervention from pre-existing divergence between the arms. See the parallel-trends card for the severity and remedies.

Data Preparation

Data Preparation

How the raw rows became a treated-versus-control panel.

Initial Rows598
Final Rows598
Rows Dropped0
Rows Aggregated0
N Periods13
Balanced PanelTrue
What this means— written from the computed numbers · yours to edit

The short answer

All 598 rows loaded with no missing data. Each of the 46 markets appeared once per period, so no aggregation was needed. California (label '1') was coded as treated and the remaining 45 markets as controls; post-intervention periods (label '1' in 'post prop99') opened at 1989. The panel is balanced: 46 markets × 13 periods = 598 observations.

The detail

No rows were dropped or aggregated. The 'california' field identified the treated group (1 market, label '1') and controls (45 markets, label '0'). The 'post prop99' field identified the post-intervention window (label '1') starting at 1989, with 9 pre-periods and 4 post-periods. All 598 observations are present; the panel is balanced. The 'year' field contains 13 distinct period labels, ordered and placed on an evenly spaced synthetic timeline for charting; the date ticks are period order placeholders, not real calendar dates.

What this can't tell you

The 'year' field does not hold actual calendar dates, so the timing of the intervention relative to real-world events cannot be verified from this export. Consider a transaction-level or monthly export to pin the intervention window precisely if external events are suspected.

Visualization

Treated vs Control Over Time

Mean cigarette sales packs pc per period for each arm.

What this means— written from the computed numbers · yours to edit

The short answer

Before 1989, the treated market (California) and the 45 control markets both declined in cigarette sales, but California was already pulling away at −0.67 packs per capita per period. After the intervention, the gap widened sharply: California fell from 90.1 packs per capita in the last pre-period to 67.5 in the final period, while controls fell from 112.48 to 100.42 over the same span.

The detail

Mean cigarette sales packs per capita for the treated arm over 13 periods: 120.2 → 118.6 → 115.4 → 110.8 → 104.8 → 102.8 → 99.7 → 97.5 → 90.1 → 82.4 → 77.8 → 68.7 → 67.5. Control arm: 134.89 → 135.10 → 133.32 → 128.45 → 122.72 → 121.29 → 119.02 → 115.92 → 112.48 → 108.22 → 103.77 → 101.28 → 100.42. The pre-period gap was narrowing—the treated market was already moving faster downward—at −0.67 packs per capita per period. After 1989 (period 10 onward), the treated arm dropped sharply while the control trajectory flattened, widening the gap further.

What this can't tell you

The visual separation after 1989 cannot be attributed wholly to the intervention because the arms were already diverging before it. See the parallel-trends card for the magnitude of pre-trend bias.

Visualization

Effect by Period (Event Study)

Per-period treated-minus-control effect on cigarette sales packs pc, against the last pre-intervention period.

What this means— written from the computed numbers · yours to edit

The short answer

All 8 pre-intervention estimates exclude zero, with effects ranging from 3.06 to 7.69 packs per capita. This is direct evidence the arms were not on parallel paths before 1989. After the intervention, estimates step sharply negative: −10.19 and −10.53 in the final two periods, but the transition is gradual rather than immediate, consistent with a trend that began before the policy.

The detail

Pre-intervention estimates (periods 1–8, relative to period 9 baseline): 7.69 (CI 3.94–11.45), 5.89 (2.03–9.74), 4.47 (0.73–8.20), 4.74 (1.43–8.04), 4.46 (1.90–7.03), 3.89 (1.97–5.81), 3.06 (1.60–4.53), 3.96 (2.35–5.57). Period 9 is fixed at zero by construction. Post-intervention estimates: −3.44 (CI −4.43 to −2.45), −3.59 (−5.96 to −1.22), −10.19 (−13.03 to −7.36), −10.53 (−13.79 to −7.28). Standard errors are unit-clustered. All pre-estimates miss zero; post-estimates step away from zero and widen in magnitude, but the drift began before 1989.

What this can't tell you

The gradual pre-period divergence means the post-period effect cannot be cleanly separated from a continuation of the existing trend. More pre-intervention periods would sharpen this distinction.

Data Table

The Difference-in-Differences 2x2

Cell means before and after, for each arm, and their differences.

GroupBeforeAfterChange
Treated (1 market)106.774.1-32.56
Control (45 markets)124.8103.4-21.38
Difference (treated minus control)-18.14-29.32-11.18
What this means— written from the computed numbers · yours to edit

The short answer

California's treated markets fell 32.56 packs per capita from before to after, while control markets fell 21.38 over the same window. The difference—11.18 packs per capita—is what difference-in-differences attributes to the intervention. However, this raw number overstates the incremental lift because the two arms were not on parallel paths before the intervention started.

The detail

The four cell means are: treated before 106.66, treated after 74.10; control before 124.80, control after 103.42. The treated change is −32.56 packs per capita; the control change is −21.38. The difference between those changes is −11.18 packs per capita (95% CI: −14.33 to −8.02), which is the fixed-effects regression estimate on the balanced panel. This arithmetic is the foundation of the causal claim, but it assumes the control arm's change represents what would have happened to the treated arm absent the intervention. That assumption holds only if the arms were tracking each other before the policy switched on—a condition the parallel-trends check rejects.

What this can't tell you

The 2×2 table alone cannot detect whether the arms were already diverging before the intervention. That check belongs to the parallel-trends diagnostic, which fails here and undermines the causal reading of the 11.18 figure.

Data Table

Parallel Trends Check

Were the arms already diverging before the intervention?

CheckValueInterpretation
Differential pre-intervention trend per period (treated minus control), in cigarette sales packs pc-0.67Fitted on the 9 pre-intervention periods only: how fast the treated markets were pulling away from the controls BEFORE anything was switched on. Difference-in-differences assumes this is zero.
95% interval on that differential trend-1.21 to -0.13Standard errors are unit-clustered (CR1). An interval comfortably containing zero is what a clean design looks like.
p-value of the differential pre-trend0.0162Below 0.05 the arms were already diverging and the headline estimate cannot be read as causal.
Pre-intervention periods available9More pre-intervention periods make this check sharper; 9 are available here.
Pre-intervention event-study estimates whose 95% interval excludes zero8 of 8Each pre-intervention period is also estimated separately against the last pre-period; estimates that miss zero are direct evidence against parallel trends. 8 of them miss zero here.
Bias in the estimate if that pre-trend simply continued-1.68The pre-trend projected across the post window is -1.68, or 15% of the estimated effect — that is how much of the headline -11.18 could be pre-existing drift rather than the intervention.
VerdictVIOLATEDVIOLATED — the arms were already diverging by -0.67 per period before the intervention (p = 0.0162), so the difference-in-differences estimate is not a credible causal number
What this means— written from the computed numbers · yours to edit

The short answer

The parallel-trends assumption fails: over 9 pre-intervention periods, California was already moving −0.67 packs per capita per period away from the controls (p = 0.0162). This pre-existing divergence is counted as lift by difference-in-differences, inflating the causal estimate. Projected forward, the pre-trend alone accounts for −1.68 packs per capita, or 15% of the −11.18 headline effect.

The detail

Differential pre-intervention trend: −0.67 packs per capita per period (95% CI −1.21 to −0.13, p = 0.0162). All 8 pre-intervention event-study estimates exclude zero, confirming divergence. The 95% interval on the trend does not contain zero, failing the integrity check. If that trend had continued into the post-period (4 periods), it would account for −1.68 packs per capita, or 15% of the estimated −11.18. The design assumes the arms would have moved in parallel absent the intervention; this assumption is violated.

What this can't tell you

The headline −11.18 cannot be read as the true causal effect of Prop-99. To firm up the answer: collect more pre-intervention periods to measure the trend precisely rather than extrapolate it; select control markets that tracked California before the change; or build a synthetic control weighting the 45 available controls to match California's pre-period trajectory. A longer pre-period would be the most direct fix.

Rate this report Was this the answer you needed?
The exact source that produced this report — yours to keep, read, and re-run.
Download PDF
How this was computed method · R source · citation
The code that did it

Incrementality Test — How Much Lift Was Actually Incremental?

Geo-lift / holdout incrementality measurement by difference-in-differences on a panel of units (markets, stores, regions). Some units were treated (advertising switched on, a change rolled out); the rest were deliberately held back. The untreated units supply the counterfactual: what the treated units would have done anyway.

Why This Method?

A single series' own past can only tell you what it was trending toward. Untreated control units also absorb whatever else happened that period — a demand swing, a competitor, a holiday — because it hit both arms. That is why difference-in-differences with real holdouts is a far stronger causal claim than a before/after comparison of one series against its own pre-period trend.

What This Analysis Covers

  • The difference-in-differences estimate from a two-way fixed-effects

regression, with unit-clustered standard errors and a 95% interval

  • The parallel-trends check: were the arms already diverging before the

intervention? If so, the estimate is not credible and the report says so

  • An event study: per-period effect relative to the last pre-period
  • The 2x2 of cell means the whole method rests on

Standard Library

Platform standard-library module (LAT-1441): runs on ANY panel via the semantic mapping {unit, period, metric, treated, post}. All narrative is derived from the user's own column names and computed values.

suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))

Core Analysis Pipeline

Step 1: Every mapped column must be present

need <- c("unit", "period", "metric", "treated", "post")
  human <- c(unit = unit_name, period = period_name, metric = metric_name,
             treated = treated_name, post = post_name)
  missing_keys <- setdiff(need, names(df))
  if (length(missing_keys) > 0) {
    stop(sprintf(
      "Incrementality testing needs a market column(&#x27;%s'), a period column ('%s'), a numeric outcome ('%s'), a treatment-group flag ('%s') and a before/after flag ('%s'). %s could not be found in the data.",
      unit_name, period_name, metric_name, treated_name, post_name,
      paste0("&#x27;", paste(human[missing_keys], collapse = "', '"), "'")))
  }

Step 2: Coerce the outcome (95% rule) — refuse if it is not numeric

v <- df$metric
  if (!is.numeric(v)) {
    conv <- suppressWarnings(as.numeric(as.character(v)))
    n_orig <- sum(!is.na(v) & nzchar(trimws(as.character(v))))
    if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
      v <- conv
    } else {
      stop(sprintf(
        "&#x27;%s' is not numeric — measuring incremental lift needs a numeric outcome (sales, conversions, revenue).",
        metric_name))
    }
  }
  v <- as.numeric(v)

Step 3: Build the timeline. Dates are used when the period column

parses as dates; otherwise the distinct period labels are ordered (numerically when they are numbers) and laid out on an evenly spaced synthetic timeline purely so the charts have a time axis. Which of the two happened is stated in the report.

per_raw <- trimws(as.character(df$period))
  per_raw[is.na(df$period) | !nzchar(per_raw)] <- NA_character_
  pd <- parse_dates_robust(per_raw)
  n_nonblank <- sum(!is.na(per_raw))
  if (n_nonblank == 0) {
    stop(sprintf("&#x27;%s' is empty — there are no periods to compare before and after the intervention.",
                 period_name))
  }
  period_is_date <- sum(!is.na(pd)) >= 0.95 * n_nonblank

Step 4: Binarize the two flags, then keep complete rows only

unit_chr <- trimws(as.character(df$unit))
  unit_chr[is.na(df$unit) | !nzchar(unit_chr)] <- NA_character_

  tb_treated <- function(clean, levs) {
    list(level = levs[length(levs)],
         reason = sprintf(
           "neither label is a standard yes/no word, so the alphabetically later label &#x27;%s' was taken as the treated group",
           levs[length(levs)]))
  }
  tb_post <- function(clean, levs) {
    ord <- if (period_is_date) as.numeric(pd) else
      as.numeric(factor(per_raw, levels = sort(unique(per_raw))))
    mean_pos <- tapply(ord, clean, function(z) mean(z, na.rm = TRUE))
    mean_pos <- mean_pos[is.finite(mean_pos)]
    if (length(mean_pos) < 2) {
      return(list(level = levs[length(levs)],
                  reason = sprintf("the periods could not be ordered, so the alphabetically later label &#x27;%s' was taken as the after-intervention side",
                                   levs[length(levs)])))
    }
    lv <- names(mean_pos)[which.max(mean_pos)]
    list(level = lv,
         reason = sprintf("neither label is a standard before/after word, and &#x27;%s' falls later on the timeline on average", lv))
  }

  tr <- binarize_flag(
    df$treated, treated_name, "^(1|true|yes|y|t|treated|treatment|test|on|exposed)$",
    tb_treated,
    paste0("&#x27;", treated_name,
           "&#x27; holds only one value ('%s') — a holdout incrementality test needs BOTH treated markets and untreated control markets. Without controls there is no counterfactual, and the honest answer is that incremental lift cannot be measured from this data."))
  po <- binarize_flag(
    df$post, post_name, "^(1|true|yes|y|t|post|after|during|on)$",
    tb_post,
    paste0("&#x27;", post_name,
           "&#x27; holds only one value ('%s') — the analysis needs periods from BOTH before and after the intervention started."))

  treated01 <- ifelse(tr$clean == tr$on_level, 1L, 0L)
  post01    <- ifelse(po$clean == po$on_level, 1L, 0L)

  keep <- !is.na(unit_chr) & !is.na(per_raw) & !is.na(v) &
    !is.na(treated01) & !is.na(post01)
  n_dropped_rows <- initial_rows - sum(keep)
  w <- data.frame(
    unit = unit_chr[keep],
    period_label = per_raw[keep],
    period_date = if (period_is_date) pd[keep] else as.Date(NA),
    metric = v[keep],
    treated01 = treated01[keep],
    post01 = post01[keep],
    stringsAsFactors = FALSE)
  if (nrow(w) == 0) {
    stop(sprintf(
      "No rows survived cleaning — every row is missing at least one of &#x27;%s', '%s', '%s', '%s' or '%s'.",
      unit_name, period_name, metric_name, treated_name, post_name))
  }

Step 5: Order the periods and lay out the timeline

if (period_is_date) {
    ord_levels <- sort(unique(w$period_date))
    w$tidx <- match(w$period_date, ord_levels)
    axis_dates <- ord_levels
    period_axis_note <- sprintf(
      "&#x27;%s' parsed as dates, so the charts use the real calendar timeline",
      period_name)
  } else {
    labs <- unique(w$period_label)
    num <- suppressWarnings(as.numeric(labs))
    ord_levels <- if (all(!is.na(num))) labs[order(num)] else sort(labs)
    w$tidx <- match(w$period_label, ord_levels)
    axis_dates <- as.Date("2000-01-01") + (seq_along(ord_levels) - 1L)
    period_axis_note <- sprintf(
      "&#x27;%s' does not hold dates, so its %d distinct labels were ordered and placed on an evenly spaced synthetic timeline; the chart's date ticks are placeholders for period order, not real calendar dates",
      period_name, length(ord_levels))
  }
  axis_iso <- format(axis_dates, "%Y-%m-%d")
  period_label_of <- ord_levels
  n_periods <- length(ord_levels)

Step 6: Integrity of the design — the treatment flag must be a

property of the unit, and the before/after flag a property of the period

bad_units <- names(which(tapply(w$treated01, w$unit,
                                  function(z) length(unique(z))) > 1))
  if (length(bad_units) > 0) {
    stop(sprintf(
      "%d %s in &#x27;%s' (%s) %s marked as treated in some rows and untreated in others. Difference-in-differences needs '%s' to be constant within each %s — one arm per market for the whole window.",
      length(bad_units), pl(length(bad_units), "market"), unit_name,
      paste(head(bad_units, 3), collapse = ", "),
      pl(length(bad_units), "is", "are"), treated_name, unit_name))
  }
  bad_periods <- names(which(tapply(w$post01, w$tidx,
                                    function(z) length(unique(z))) > 1))
  if (length(bad_periods) > 0) {
    bad_lab <- period_label_of[as.integer(bad_periods)]
    stop(sprintf(
      "%d %s in &#x27;%s' (%s) %s marked both before and after the intervention. '%s' must be constant within each period — every period falls entirely on one side of the start date.",
      length(bad_lab), pl(length(bad_lab), "period"), period_name,
      paste(head(bad_lab, 3), collapse = ", "),
      pl(length(bad_lab), "is", "are"), post_name))
  }

Step 7: Collapse duplicate unit-period rows by SUM (rows are read as

amounts that add up within one market-period), then count the design

n_before_agg <- nrow(w)
  w <- aggregate(metric ~ unit + tidx + treated01 + post01, data = w, FUN = sum)
  n_dup_rows <- n_before_agg - nrow(w)
  w$period_label <- period_label_of[w$tidx]
  w$period_iso <- axis_iso[w$tidx]
  w <- w[order(w$unit, w$tidx), , drop = FALSE]
  rownames(w) <- NULL

  units_all <- unique(w$unit)
  n_units <- length(units_all)
  unit_arm <- tapply(w$treated01, w$unit, max)
  n_treated_units <- sum(unit_arm == 1)
  n_control_units <- sum(unit_arm == 0)
  pre_idx  <- sort(unique(w$tidx[w$post01 == 0]))
  post_idx <- sort(unique(w$tidx[w$post01 == 1]))
  n_pre <- length(pre_idx); n_post <- length(post_idx)

Step 8: Refuse rather than degrade

if (n_control_units < 1) {
    stop(sprintf(
      "Every market in &#x27;%s' is in the treatment group ('%s' = '%s') — a holdout incrementality test needs untreated control markets to build the counterfactual.",
      unit_name, treated_name, tr$on_level))
  }
  if (n_treated_units < 1) {
    stop(sprintf(
      "No market in &#x27;%s' is marked as treated ('%s' = '%s') — there is nothing to measure the lift of.",
      unit_name, treated_name, tr$on_level))
  }
  if (n_pre < 2) {
    stop(sprintf(
      "Only %d pre-intervention %s in &#x27;%s' — at least 2 are needed, because with a single pre-period there is no way to check whether the arms were already diverging.",
      n_pre, pl(n_pre, "period"), period_name))
  }
  if (n_post < 1) {
    stop(sprintf(
      "No period in &#x27;%s' is marked as after the intervention ('%s' = '%s') — there is no post window to measure.",
      period_name, post_name, po$on_level))
  }
  if (n_units < 4) {
    stop(sprintf(
      "Only %d %s in &#x27;%s' (%d treated, %d control) — at least 4 are needed for a difference-in-differences estimate to carry any usable uncertainty. With fewer, the honest answer is that the lift cannot be separated from market-to-market noise.",
      n_units, pl(n_units, "market"), unit_name, n_treated_units,
      n_control_units))
  }
  n_obs <- nrow(w)
  balanced <- n_obs == n_units * n_periods

Step 9: Two-way fixed-effects DiD. The interaction of treated and

post IS the difference-in-differences estimate; market and period fixed effects absorb the two main effects, so the interaction is fitted directly to keep the design matrix full rank.

w$did <- w$treated01 * w$post01
  fit_did <- stats::lm(metric ~ did + factor(unit) + factor(period_label),
                       data = w)
  if (is.na(coef(fit_did)[["did"]])) {
    stop(sprintf(
      "The treated-by-after term could not be estimated: it is perfectly explained by the market and period effects. Check that treated markets in &#x27;%s' are observed in BOTH the before and after windows of '%s'.",
      unit_name, post_name))
  }
  n_clusters <- n_units
  few_clusters <- n_clusters < 10
  V_did <- cluster_vcov(fit_did, w$unit)
  cluster_ok <- !is.null(V_did) && !few_clusters
  inf_did <- coef_inference(fit_did, "did", V_did, n_clusters, cluster_ok)
  se_basis <- inf_did$basis

Step 10: Relative lift is the absolute effect over the

counterfactual level (observed treated-post mean minus the effect).

mean_treated_post <- mean(w$metric[w$treated01 == 1 & w$post01 == 1])
  cf_level <- mean_treated_post - inf_did$est
  rel_ok <- is.finite(cf_level) && abs(cf_level) > 1e-9
  rel_scale <- if (rel_ok) 100 / abs(cf_level) else NA_real_
  rel_est <- if (rel_ok) inf_did$est * rel_scale else NA_real_
  rel_lo  <- if (rel_ok) inf_did$lo * rel_scale else NA_real_
  rel_hi  <- if (rel_ok) inf_did$hi * rel_scale else NA_real_

Step 11: THE integrity gate — parallel trends. Fit the pre-period

only and ask whether the treated arm was already moving differently.

pre_w <- w[w$post01 == 0, , drop = FALSE]
  pre_w$treat_t <- pre_w$treated01 * pre_w$tidx
  fit_pt <- tryCatch(
    stats::lm(metric ~ treat_t + factor(unit) + factor(period_label),
              data = pre_w),
    error = function(e) NULL)
  if (!is.null(fit_pt) && !is.na(coef(fit_pt)[["treat_t"]])) {
    V_pt <- cluster_vcov(fit_pt, pre_w$unit)
    inf_pt <- coef_inference(fit_pt, "treat_t", V_pt, n_clusters, cluster_ok)
  } else {
    inf_pt <- list(est = NA_real_, se = NA_real_, df = NA_real_, p = NA_real_,
                   lo = NA_real_, hi = NA_real_, basis = "not estimable")
  }
  pt_ok <- is.finite(inf_pt$est) && is.finite(inf_pt$p)

If the measured pre-trend simply carried on, this is the bias it would inject into the DiD estimate over the post window.

mean_post_gap <- mean(post_idx) - max(pre_idx)
  pt_bias <- if (pt_ok) inf_pt$est * mean_post_gap else NA_real_
  pt_bias_share <- if (pt_ok && abs(inf_did$est) > 1e-9)
    100 * abs(pt_bias) / abs(inf_did$est) else NA_real_
  pt_verdict <- if (!pt_ok) "unassessable" else if (inf_pt$p < 0.05) "violated" else
    if (inf_pt$p < 0.10) "borderline" else "supported"

Step 12: Event study — per-period effect against the last

pre-intervention period, which is normalised to zero by construction.

ref_idx <- max(pre_idx)
  es_terms <- setdiff(seq_len(n_periods), ref_idx)
  es_params <- n_units + 2 * n_periods
  es_method <- "regression"
  es_rows <- NULL
  if (es_params <= 500 && length(es_terms) > 0) {
    es_names <- paste0("es", es_terms)
    for (i in seq_along(es_terms)) {
      w[[es_names[i]]] <- as.integer(w$treated01 == 1 & w$tidx == es_terms[i])
    }
    fml <- stats::as.formula(paste0(
      "metric ~ ", paste(es_names, collapse = " + "),
      " + factor(unit) + factor(period_label)"))
    fit_es <- tryCatch(stats::lm(fml, data = w), error = function(e) NULL)
    if (!is.null(fit_es)) {
      V_es <- cluster_vcov(fit_es, w$unit)
      es_rows <- do.call(rbind, lapply(seq_along(es_terms), function(i) {
        ii <- coef_inference(fit_es, es_names[i], V_es, n_clusters, cluster_ok)
        data.frame(tidx = es_terms[i], effect = ii$est,
                   ci_low = ii$lo, ci_high = ii$hi,
                   stringsAsFactors = FALSE)
      }))
    }
  }
  if (is.null(es_rows)) {

How many PRE-period estimates are significantly non-zero — the visual half of the parallel-trends evidence.

pre_es <- es_rows[es_rows$tidx %in% setdiff(pre_idx, ref_idx), , drop = FALSE]
  pt_n_bad_pre <- if (nrow(pre_es) == 0) 0L else
    as.integer(sum(pre_es$ci_low > 0 | pre_es$ci_high < 0, na.rm = TRUE))
  pt_n_pre_est <- nrow(pre_es)

  event_study_df <- data.frame(
    period = axis_iso[es_rows$tidx],
    effect = round(es_rows$effect, 4),
    ci_high = round(es_rows$ci_high, 4),
    ci_low = round(es_rows$ci_low, 4),
    stringsAsFactors = FALSE)

Step 13: The picture people actually want — mean outcome per period

for each arm.

arm_means_df <- do.call(rbind, lapply(seq_len(n_periods), function(k) {
    do.call(rbind, lapply(c(1L, 0L), function(a) {
      sel <- w$tidx == k & w$treated01 == a
      if (!any(sel)) return(NULL)
      data.frame(period = axis_iso[k],
                 mean_metric = round(mean(w$metric[sel]), 4),
                 arm = if (a == 1L) "Treated" else "Control",
                 stringsAsFactors = FALSE)
    }))
  }))
  if (nrow(arm_means_df) > 2000) {
    keep_k <- sort(unique(round(seq(1, n_periods, length.out = 1000))))
    arm_means_df <- arm_means_df[arm_means_df$period %in% axis_iso[keep_k], ,
                                 drop = FALSE]
  }
  rownames(arm_means_df) <- NULL

Step 14: The 2x2 the whole method rests on

cell <- function(a, p) {
    sel <- w$treated01 == a & w$post01 == p
    if (!any(sel)) NA_real_ else mean(w$metric[sel])
  }
  t_pre <- cell(1, 0); t_post <- cell(1, 1)
  c_pre <- cell(0, 0); c_post <- cell(0, 1)
  raw_did <- (t_post - t_pre) - (c_post - c_pre)
  did_cells_df <- data.frame(
    group = c(sprintf("Treated(%d %s)", n_treated_units,
                      pl(n_treated_units, "market")),
              sprintf("Control(%d %s)", n_control_units,
                      pl(n_control_units, "market")),
              "Difference(treated minus control)"),
    before = round(c(t_pre, c_pre, t_pre - c_pre), 4),
    after = round(c(t_post, c_post, t_post - c_post), 4),
    change = round(c(t_post - t_pre, c_post - c_pre, raw_did), 4),
    stringsAsFactors = FALSE)

Step 15: Round ONCE — the same numbers feed tables and prose

did_r  <- round(inf_did$est, 2)
  did_lo <- round(inf_did$lo, 2); did_hi <- round(inf_did$hi, 2)
  did_p  <- inf_did$p
  rel_r  <- if (is.na(rel_est)) NA_real_ else round(rel_est, 1)
  rel_lo_r <- if (is.na(rel_lo)) NA_real_ else round(rel_lo, 1)
  rel_hi_r <- if (is.na(rel_hi)) NA_real_ else round(rel_hi, 1)
  raw_did_r <- round(raw_did, 2)
  pt_slope <- if (pt_ok) round(inf_pt$est, 3) else NA_real_
  pt_lo <- if (pt_ok) round(inf_pt$lo, 3) else NA_real_
  pt_hi <- if (pt_ok) round(inf_pt$hi, 3) else NA_real_
  pt_bias_r <- if (pt_ok) round(pt_bias, 2) else NA_real_
  pt_bias_share_r <- if (is.na(pt_bias_share)) NA_real_ else round(pt_bias_share, 1)
  cf_level_r <- round(cf_level, 2)
  did_significant <- is.finite(did_lo) && is.finite(did_hi) &&
    (did_lo > 0 || did_hi < 0)
  direction_word <- if (is.finite(did_r) && did_r >= 0) "above" else "below"

Step 16: The parallel-trends diagnostic table

pt_verdict_sentence <- switch(
    pt_verdict,
    violated = sprintf(
      "VIOLATED — the arms were already diverging by %s per period before the intervention(%s), so the difference-in-differences estimate is not a credible causal number",
      fmt_signed(pt_slope, 3), fmt_p_phrase(inf_pt$p)),
    borderline = sprintf(
      "BORDERLINE — the pre-period divergence of %s per period is not conclusive either way(%s); treat the estimate with caution",
      fmt_signed(pt_slope, 3), fmt_p_phrase(inf_pt$p)),
    supported = sprintf(
      "SUPPORTED — no detectable pre-intervention divergence(%s per period, %s). This is a failure to detect a violation, not proof that trends were parallel",
      fmt_signed(pt_slope, 3), fmt_p_phrase(inf_pt$p)),
    "UNASSESSABLE — the pre-period trend term could not be estimated from this panel")

  parallel_trends_df <- data.frame(
    check = c(
      sprintf("Differential pre-intervention trend per period(treated minus control), in %s", metric_name),
      "95% interval on that differential trend",
      "p-value of the differential pre-trend",
      "Pre-intervention periods available",
      "Pre-intervention event-study estimates whose 95% interval excludes zero",
      "Bias in the estimate if that pre-trend simply continued",
      "Verdict"),
    value = c(
      if (pt_ok) fmt_signed(pt_slope, 3) else "not estimable",
      if (pt_ok) paste0(fmt_n(pt_lo, 3), " to ", fmt_n(pt_hi, 3)) else "not estimable",
      if (pt_ok) fmt_p(inf_pt$p) else "not estimable",
      as.character(n_pre),
      sprintf("%d of %d", pt_n_bad_pre, pt_n_pre_est),
      if (pt_ok) fmt_signed(pt_bias_r) else "not estimable",
      toupper(pt_verdict)),
    interpretation = c(
      sprintf("Fitted on the %d pre-intervention %s only: how fast the treated markets were pulling away from the controls BEFORE anything was switched on. Difference-in-differences assumes this is zero.",
              n_pre, pl(n_pre, "period")),
      sprintf("Standard errors are %s. An interval comfortably containing zero is what a clean design looks like.",
              inf_pt$basis),
      "Below 0.05 the arms were already diverging and the headline estimate cannot be read as causal.",
      sprintf("More pre-intervention periods make this check sharper; %d %s available here.",
              n_pre, pl(n_pre, "is", "are")),
      sprintf("Each pre-intervention period is also estimated separately against the last pre-period; estimates that miss zero are direct evidence against parallel trends. %s",
              if (pt_n_bad_pre > 0)
                sprintf("%d of them %s zero here.", pt_n_bad_pre,
                        pl(pt_n_bad_pre, "misses", "miss"))
              else "None of them miss zero here."),
      if (pt_ok) sprintf(
        "The pre-trend projected across the post window is %s%s — that is how much of the headline %s could be pre-existing drift rather than the intervention.",
        fmt_signed(pt_bias_r),
        if (!is.na(pt_bias_share_r)) sprintf(", or %s%% of the estimated effect",
                                             fmt_n(pt_bias_share_r, 1)) else "",
        fmt_signed(did_r))
      else "The projection could not be computed because the pre-trend term is not estimable.",
      pt_verdict_sentence),
    stringsAsFactors = FALSE)

Step 17: Metrics + the one-paragraph computed answer

metrics <- list(
    `Markets`             = n_units,
    `Treated Markets`     = n_treated_units,
    `Control Markets`     = n_control_units,
    `Pre Periods`         = n_pre,
    `Post Periods`        = n_post,
    `Incremental Lift`    = did_r,
    `Lift CI Low`         = did_lo,
    `Lift CI High`        = did_hi,
    `Relative Lift(%)`   = rel_r,
    `Parallel Trends`     = toupper(pt_verdict)
  )

  lift_phrase <- paste0(
    fmt_signed(did_r), " in ", metric_name, " per market-period(95% interval ",
    fmt_n(did_lo), " to ", fmt_n(did_hi), ")",
    if (!is.na(rel_r)) paste0(", a relative lift of ", fmt_signed(rel_r, 1),
                              "% against a counterfactual level of ",
                              fmt_n(cf_level_r)) else "")
  answer <- if (pt_verdict == "violated") {
    paste0(
      "The parallel-trends assumption FAILS on this panel: treated and control markets in &#x27;",
      unit_name, "&#x27; were already diverging by ", fmt_signed(pt_slope, 3),
      " per period before the intervention(", fmt_p_phrase(inf_pt$p),
      "), so the difference-in-differences estimate of ", lift_phrase,
      " should NOT be read as incremental lift — an unknown part of it is pre-existing drift(projected bias ",
      fmt_signed(pt_bias_r),
      "). Fix the design before trusting a number: add more pre-intervention periods, choose control markets that tracked the treated ones before the change, or use a synthetic-control weighting of the controls.")
  } else {
    paste0(
      "Difference-in-differences across ", n_units, " markets(",
      n_treated_units, " treated, ", n_control_units, " held back) over ",
      n_pre, " pre and ", n_post, " post ", pl(n_post, "period"),
      ": the treated markets ran ", lift_phrase, " ", direction_word,
      " what the control markets say they would have done anyway. The interval ",
      if (did_significant) "excludes" else "includes",
      " zero, so the lift is ",
      if (did_significant) "distinguishable from" else "not distinguishable from",
      " noise. Parallel trends: ", toupper(pt_verdict), " (",
      fmt_p_phrase(inf_pt$p), " on the pre-period differential trend). ",
      "This assumes nothing else hit only the treated markets at the same time.")
  }

  json_output <- list(
    answer = answer,
    cards = lapply(
      c("tldr", "overview", "preprocessing", "arm_trends", "event_study",
        "did_table", "parallel_trends"),
      function(cid) list(id = cid, metrics = metrics)
    )
  )

  list(
    initial_rows = initial_rows, final_rows = n_obs,
    rows_removed = max(0, initial_rows - n_obs),
    n_dropped_rows = n_dropped_rows, n_dup_rows = n_dup_rows,
    unit_name = unit_name, period_name = period_name,
    metric_name = metric_name, treated_name = treated_name,
    post_name = post_name,
    treated_on = tr$on_level, treated_off = tr$off_level,
    treated_reason = tr$reason,
    post_on = po$on_level, post_off = po$off_level, post_reason = po$reason,
    period_is_date = period_is_date, period_axis_note = period_axis_note,
    n_units = n_units, n_treated_units = n_treated_units,
    n_control_units = n_control_units, n_periods = n_periods,
    n_pre = n_pre, n_post = n_post, n_obs = n_obs, balanced = balanced,
    first_post_label = period_label_of[min(post_idx)],
    ref_period_label = period_label_of[ref_idx],
    did_r = did_r, did_lo = did_lo, did_hi = did_hi, did_p = did_p,
    did_significant = did_significant, direction_word = direction_word,
    rel_r = rel_r, rel_lo = rel_lo_r, rel_hi = rel_hi_r, rel_ok = rel_ok,
    cf_level_r = cf_level_r, raw_did_r = raw_did_r,
    se_basis = se_basis, cluster_ok = cluster_ok, n_clusters = n_clusters,
    few_clusters = few_clusters,
    pt_slope = pt_slope, pt_lo = pt_lo, pt_hi = pt_hi, pt_p = inf_pt$p,
    pt_verdict = pt_verdict, pt_verdict_sentence = pt_verdict_sentence,
    pt_bias_r = pt_bias_r, pt_bias_share_r = pt_bias_share_r,
    pt_n_bad_pre = pt_n_bad_pre, pt_n_pre_est = pt_n_pre_est,
    es_method = es_method,
    arm_means_df = arm_means_df, event_study_df = event_study_df,
    did_cells_df = did_cells_df, parallel_trends_df = parallel_trends_df,
    metrics = metrics, json_output = json_output
  )
}
Your data has more stories to tell.Run any analysis on your own data — validated R modules, interactive reports, AI insights, and PDF export. 500 free credits on signup.
Try Free — No SignupSign Up Free
Download PDF · Print · Share ·Refresh this analysis ·Edit this report

Cite this analysis

Report an Issue

Tell us what's wrong. You'll get a free re-run of this analysis so you can try again with different parameters. If the re-run still doesn't meet your expectations, we'll refund your credits.

Want to run this analysis on your own data? Upload CSV — Free Analysis See Pricing