Executive Summary
Whether #Passengers is really trending or just noisy
The short answer
Airline passengers are rising significantly: 2.4617 per month over the 144-month window (Mann-Kendall p < 0.0001), a total increase of 345.586 passengers (351.1% of the starting level). However, the residuals are strongly serially correlated (lag-1 autocorrelation 0.728), which inflates the reported confidence—the true p-value is larger than < 0.0001.
The detail
Trend verdict: rising. Sen's slope is 2.4617 per month with a 95% confidence interval of 2.3219 to 2.6345. The Mann-Kendall p-value is < 0.0001 across 144 observations. The implied total change over the window (1949-01-01 to 1960-12-01) is 345.586 passengers. WARNING: detrended residuals show lag-1 autocorrelation of 0.73, above the 0.3 threshold. Positive autocorrelation makes consecutive points move together, shrinking the effective sample size and inflating trend-test confidence. Treat the significance verdict as optimistic.
What this can't tell you
The strong serial correlation means the effective degrees of freedom are lower than the nominal 144 points suggest. The pattern is real, but the precision of the p-value is overstated. A longer or independent time series would clarify whether this trend continues or stabilizes.
Analysis Overview
A Mann-Kendall trend test with Sen's slope on #Passengers over 144 time points.
The short answer
Airline passenger numbers show a clear upward trend over the 144-month window (January 1949 to December 1960), rising at a median pace of 2.4617 passengers per month. The Mann-Kendall test, which detects trends by comparing the ordering of values across time, confirms this movement is real and not random noise.
The detail
The Mann-Kendall test produced an S statistic of 8,327 (the count of later values higher minus lower across all time pairs), standardized to z = 14.382 with a two-sided p-value < 0.0001. Sen's slope—the median of all pairwise slopes between time points—estimates the trend at 2.4617 per month, with a 95% confidence interval of 2.3219 to 2.6345. Over the full 140.4-month window, this pace implies a total change of 345.586 passengers, or 351.1% of the trend line's starting level of 98.435. The test is rank-based and therefore immune to outliers and skewed distributions; 25 groups of tied values (51 repeated measurements) reduced the ordering information, which the variance correction accounts for.
What this can't tell you
The detrended residuals show lag-1 autocorrelation of 0.728, well above the 0.3 threshold. Positive serial correlation means consecutive months move together, shrinking the effective sample size and inflating the trend test's confidence—the reported p-value is optimistic, and the true significance is weaker than < 0.0001 suggests. To firm this up, consider a longer observation window that would dilute the effect of month-to-month carryover.
Data Quality
How the raw rows became an ordered time series.
The short answer
All 144 rows of passenger data loaded successfully with no missing values dropped or duplicates collapsed. The time axis was correctly parsed as calendar dates spanning January 1949 to December 1960, and the series was sorted chronologically before analysis.
The detail
Initial rows loaded: 144. Final rows after cleaning: 144. Rows dropped for missing values: 0. Duplicate time points collapsed: 0. The Month column was read as dates (1949-01-01 to 1960-12-01) and the series was sorted chronologically. With 10,296 possible point pairs, Sen's slope was estimated from a seeded random sample of 5,000 pairs (set.seed(42)) rather than computing all pairs, a practical approach that yields the same median slope estimate.
What this can't tell you
The export is a monthly aggregate; transaction-level or daily data would reveal whether the trend is driven by consistent growth across all travel periods or concentrated in specific seasons. The analysis cannot distinguish between genuine structural growth and temporary shifts in passenger reporting or airline capacity.
Series with Sen Trend Line
#passengers over Month, with the fitted Sen slope overlaid.
The short answer
The passenger series climbs steadily from 112 in January 1949 to higher levels by December 1960, rising faster than random fluctuation alone would produce. The Sen trend line (slope 2.4617 per month) cuts through the jagged monthly variation and shows the underlying upward direction clearly.
The detail
The series starts at 112 passengers in 1949-01-01 and ends at 432 in 1960-12-01, with the Sen trend line rising 2.4617 per month. The statistically significant verdict (p < 0.0001) reflects that the series works its way up faster than noise would explain. Monthly swings are visible—peaks in summer months (e.g., 199 in July and August 1951) and troughs in winter—but the trend line captures the median direction across all pairs of points.
What this can't tell you
The residuals are serially correlated (lag-1 autocorrelation 0.728), meaning smooth multi-point drifts can appear more trend-like than they statistically are. The visual slope matches the statistical verdict, but the confidence around that verdict is overstated. A deseasonalized or differenced view would show whether the trend is driven by a consistent month-over-month lift or by shifts in seasonal peaks and troughs.
Mann-Kendall Test Detail
The S statistic, its tie-corrected variance, z, p, and tie accounting.
| Statistic | Value | Interpretation |
|---|---|---|
| Mann-Kendall S | 8,327 | Concordant minus discordant pairs — positive means #Passengers tends to rise over Month |
| Variance of S (tie-corrected) | 335,164.3 | Null-hypothesis variance of S, reduced for the 25 group(s) of tied values |
| z statistic (continuity-corrected) | 14.382 | S standardized by its null variance, with the +/-1 continuity correction |
| Two-sided p-value | < 0.0001 | Below 0.05 — the upward trend is unlikely to be chance |
| Tied value groups | 25 (51 values) | Repeated values reduce the usable ordering information |
| Time points tested | 144 | Distinct time points of #Passengers after cleaning |
The short answer
The Mann-Kendall statistic S = 8,327 means that across all pairs of time points, later values are systematically higher than earlier ones far more often than chance would predict. The continuity-corrected z-score is 14.382, yielding p < 0.0001—the upward trend is real.
The detail
S = 8,327 (concordant minus discordant pairs). Tie-corrected null variance = 335,164.3, reduced from the original by 25 groups of tied values (51 repeated measurements). Continuity-corrected z = 14.382. Two-sided p < 0.0001. Time points tested: 144. The decisive cell is the p-value: below 0.05, confirming the upward tendency is unlikely to be noise. The tie correction matters because 51 of the 144 values are repeated, which carries no ordering information; the variance is adjusted accordingly.
What this can't tell you
The p-value assumes independence between observations, but the lag-1 autocorrelation of 0.728 violates that assumption. The true p-value is larger than < 0.0001. The test detects the direction and magnitude of trend, but not the precision of the significance level.
Sen's Slope and Implied Change
How fast #Passengers is moving, with its confidence interval.
| Statistic | Value | Interpretation |
|---|---|---|
| Sen slope (per month) | 2.4617 | Median of the 5,000 pairwise slopes, in units of #Passengers per month (a seeded random sample of the 10,296 possible pairs) |
| 95% confidence interval | 2.3219 to 2.6345 | Classical rank-based interval from the ordered pairwise slopes |
| Implied total change over the window | 345.586 | The Sen slope carried across the full window, in units of #Passengers |
| Implied change as % of starting level | 351.1% | Total change relative to the Sen trend line's starting level of 98.435 |
| Observed window | 1949-01-01 to 1960-12-01 (about 140.4 months) | The span of Month covered by the test |
The short answer
The median rate of change is 2.4617 passengers per month, with a 95% confidence interval of 2.3219 to 2.6345. The interval excludes zero, confirming the upward direction holds across the plausible range. Over the full 140.4-month window, this pace implies passengers grew by 345.586, or 351.1% of the trend line's starting level.
The detail
Sen's slope = 2.4617 per month (median of 5,000 seeded pairwise slopes from the 10,296 possible pairs). 95% confidence interval: 2.3219 to 2.6345, computed from the ordered pairwise slopes. Observed window: 1949-01-01 to 1960-12-01 (about 140.4 months). Implied total change: 345.586 passengers. Implied change as % of starting level: 351.1%, with the Sen trend line's starting level of 98.435. The slope is robust to outliers because it depends only on the median of pairwise comparisons, not on extreme values.
What this can't tell you
The confidence interval assumes independence; with lag-1 autocorrelation of 0.728, the true uncertainty around the slope is likely wider. The slope describes the linear trend over this specific 140.4-month window and does not forecast beyond it. A longer series would clarify whether this pace is sustainable or whether the trend is decelerating.
Methods and Checks
How the test was run, and whether its assumptions held.
| Check | Result | Notes |
|---|---|---|
| Trend test | Mann-Kendall, two-sided (z = 14.382, p < 0.0001) | Nonparametric: uses only the ordering of the values, so outliers and skew do not distort it |
| Tie correction | Applied (25 tie group(s)) | Tied values carry no ordering information; the variance of S is reduced accordingly |
| Continuity correction | Applied (+/-1 on S) | Improves the normal approximation for the discrete S statistic |
| Slope estimator | Sen's slope: 2.4617 per month | Median of pairwise slopes — robust to outliers, stated in the value's own units |
| Pairwise slopes used | 5,000 of 10,296 pairs (seeded sample) | With this many points the full pair set is large; a seeded sample of 5,000 pairs estimates the same median |
| Serial-correlation check | Lag-1 autocorrelation of residuals = 0.728 (threshold 0.3) — CAUTION FIRED | See the caution: serial correlation inflates the trend test's confidence |
The short answer
The Mann-Kendall test is rank-based and requires no normality assumption; it compares every pair of time points to ask whether later values are systematically higher. Sen's slope is the median of all pairwise slopes and is robust to outliers. However, the detrended residuals show lag-1 autocorrelation of 0.728, well above the 0.3 threshold—a critical violation that inflates the trend test's confidence.
The detail
Trend test: Mann-Kendall, two-sided (z = 14.382, p < 0.0001), nonparametric and based only on ordering. Tie correction: applied (25 tie groups). Continuity correction: applied (+/-1 on S). Slope estimator: Sen's slope 2.4617 per month, the median of pairwise slopes, robust to outliers. Pairwise slopes: 5,000 of 10,296 pairs (seeded sample, set.seed(42)). Serial-correlation check: lag-1 autocorrelation of residuals = 0.728 (threshold 0.3)—CAUTION FIRED. Positive autocorrelation makes consecutive points move together, shrinking the effective sample size and inflating the trend test's confidence. The true p-value is larger than < 0.0001.
What this can't tell you
The analysis assumes independence, which is violated. The reported significance is optimistic. To validate the trend strength, consider a longer observation window or a method that explicitly accounts for autocorrelation (e.g., prewhitening or a generalized least-squares trend test).
Trend Test — Is It Really Rising?
Tests whether an ordered series shows a real monotonic trend or just noise: the classical Mann-Kendall trend test (tie-corrected variance, continuity-corrected normal approximation) paired with Sen's slope — the median of all pairwise slopes — with its classical rank-based confidence interval, plus an honest serial-correlation caution.
Why This Method?
Mann-Kendall + Sen's slope is the standard nonparametric trend pair: it needs no normality assumption, shrugs off outliers, and separates "the line drifts up" from "the noise happened to end high". Sen's slope then states the trend in the value's own units per period.
What This Analysis Covers
- Mann-Kendall S, tie-corrected variance, z, and two-sided p
- Sen's slope with its classical confidence interval and implied change
- The series plotted with the Sen trend line overlaid
- A lag-1 autocorrelation check that warns when the test's confidence
is inflated by serial correlation
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {date, value}. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
Step 1: Read the time axis — real dates first, else a sortable index
raw_seq <- df$date
non_blank <- !(is.na(raw_seq) | !nzchar(trimws(as.character(raw_seq))))
n_nonblank <- sum(non_blank)
d <- parse_dates_robust(raw_seq)
frac_dates <- if (n_nonblank > 0) sum(!is.na(d)) / n_nonblank else 0
numseq <- suppressWarnings(as.numeric(as.character(raw_seq)))
frac_num <- if (n_nonblank > 0) sum(!is.na(numseq)) / n_nonblank else 0
if (frac_dates >= 0.8) {
t_raw <- as.numeric(d)
seq_kind <- "date"
} else if (frac_num >= 0.95 && n_nonblank > 0) {
t_raw <- numseq
seq_kind <- "numeric"
} else {
stop(sprintf("The '%s' column could not be read as dates or as a sortable numeric index — a trend test needs an orderable time axis.",
date_name))
}Step 2: Coerce the value (95% rule) and drop unusable rows
mv <- df$value
if (!is.numeric(mv)) {
conv <- suppressWarnings(as.numeric(as.character(mv)))
n_orig <- sum(!is.na(mv) & nzchar(as.character(mv)))
if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
mv <- conv
} else {
stop(sprintf("The '%s' column is not numeric — a trend test needs a numeric value to track.",
value_name))
}
}
mv <- as.numeric(mv)
keep <- !is.na(mv) & !is.na(t_raw)
n_dropped <- sum(!keep)
t_kept <- t_raw[keep]
x_kept <- mv[keep]Step 3: Sort by time and collapse duplicate time points (mean)
ord <- order(t_kept)
t_kept <- t_kept[ord]
x_kept <- x_kept[ord]
n_dup_collapsed <- 0L
if (anyDuplicated(t_kept)) {
n_before <- length(t_kept)
agg <- tapply(x_kept, t_kept, mean)
t_num <- as.numeric(names(agg))
x <- as.numeric(agg)
o2 <- order(t_num)
t_num <- t_num[o2]; x <- x[o2]
n_dup_collapsed <- n_before - length(x)
} else {
t_num <- t_kept
x <- x_kept
}
n_points <- length(x)
final_rows <- n_points
rows_removed <- initial_rows - final_rows
if (n_points < 8) {
stop(sprintf("Only %d usable %s of '%s' over '%s' — a trend test needs at least 8 time points.",
n_points, pts(n_points), value_name, date_name))
}
if (isTRUE(max(x) == min(x))) {
stop(sprintf("'%s' has no variation — every one of the %d values equals %s, so there is no trend to test.",
value_name, n_points, fmt_num(x[1], 4)))
}Step 4: Time-range prose + inferred reporting period
gaps <- diff(t_num)
step <- stats::median(gaps)
if (seq_kind == "date") {
seq_range_text <- sprintf("%s to %s",
format(as.Date(min(t_num), origin = "1970-01-01"), "%Y-%m-%d"),
format(as.Date(max(t_num), origin = "1970-01-01"), "%Y-%m-%d"))
period_word <- if (step <= 1.5) "day"
else if (step >= 6 && step <= 8) "week"
else if (step >= 13 && step <= 15) "fortnight"
else if (step >= 28 && step <= 31.5) "month"
else if (step >= 84 && step <= 95) "quarter"
else if (step >= 350 && step <= 380) "year"
else sprintf("%s-day period", fmt_num(step, 1))
} else {
seq_range_text <- sprintf("%s to %s", fmt_num(min(t_num)), fmt_num(max(t_num)))
period_word <- "step"
}Step 5: Mann-Kendall S with tie-corrected variance (hand-rolled)
S = sum over all i<j of sign(x_j - x_i); the inner loop is vectorized so large series stay fast. Ties in the VALUES reduce Var(S) through the classical correction term.
S <- 0
for (i in seq_len(n_points - 1)) {
S <- S + sum(sign(x[(i + 1):n_points] - x[i]))
}
tie_tab <- table(x)
tie_sizes <- as.numeric(tie_tab[tie_tab > 1])
n_tie_groups <- length(tie_sizes)
n_tied_values <- if (n_tie_groups > 0) sum(tie_sizes) else 0L
varS <- (n_points * (n_points - 1) * (2 * n_points + 5) -
sum(tie_sizes * (tie_sizes - 1) * (2 * tie_sizes + 5))) / 18
if (!is.finite(varS) || varS <= 0) {
stop(sprintf("'%s' has effectively no variation (all %d values are tied) — there is no trend to test.",
value_name, n_points))
}Continuity-corrected normal approximation, two-sided p
z_stat <- if (S > 0) (S - 1) / sqrt(varS) else if (S < 0) (S + 1) / sqrt(varS) else 0
p_value <- 2 * stats::pnorm(-abs(z_stat))Step 6: Sen's slope — median of pairwise slopes (per t-unit)
All N = n(n-1)/2 pairs when N <= 5000; otherwise a seeded random sample of 5000 pairs (set.seed(42)) — and the prose says so.
n_pairs <- n_points * (n_points - 1) / 2
slopes_sampled <- n_pairs > 5000
if (!slopes_sampled) {
ii <- rep(seq_len(n_points - 1), times = (n_points - 1):1)
jj <- unlist(lapply(seq_len(n_points - 1), function(i) (i + 1):n_points))
} else {
set.seed(42)
ii <- integer(0); jj <- integer(0)
while (length(ii) < 5000) {
a <- sample.int(n_points, 8000, replace = TRUE)
b <- sample.int(n_points, 8000, replace = TRUE)
ok <- a < b
ii <- c(ii, a[ok]); jj <- c(jj, b[ok])
if (length(ii) >= 5000) { ii <- ii[1:5000]; jj <- jj[1:5000] }
}
}
slopes <- (x[jj] - x[ii]) / (t_num[jj] - t_num[ii])
slopes <- slopes[is.finite(slopes)]
slopes <- sort(slopes)
n_pairs_used <- length(slopes)
sen_slope_t <- stats::median(slopes)Classical rank-based CI for Sen's slope: C = z_{0.975} * sqrt(Var(S)); the CI bounds sit at ranks (N - C)/2 and (N + C)/2 + 1 of the ordered pairwise slopes. When the pairs were sampled, the same ranks are taken as quantiles of the sampled slope distribution.
C_alpha <- stats::qnorm(0.975) * sqrt(varS)
rank_lo <- (n_pairs - C_alpha) / 2
rank_hi <- (n_pairs + C_alpha) / 2 + 1
p_lo <- min(max(rank_lo / n_pairs, 1 / n_pairs_used), 1)
p_hi <- min(max(rank_hi / n_pairs, 1 / n_pairs_used), 1)
ci_t <- as.numeric(stats::quantile(slopes, probs = c(p_lo, p_hi),
type = 1, names = FALSE))
ci_lo_t <- ci_t[1]
ci_hi_t <- ci_t[2]Express the slope per reporting period (per t-unit x median spacing)
sen_slope_period <- sen_slope_t * step
ci_lo_period <- ci_lo_t * step
ci_hi_period <- ci_hi_t * stepStep 7: Implied change over the observed window
Sen intercept (Conover): median of (x_i - slope t_i); the trend line is intercept + slope t.
intercept <- stats::median(x - sen_slope_t * t_num)
span_t <- max(t_num) - min(t_num)
total_change <- sen_slope_t * span_t
start_level <- intercept + sen_slope_t * min(t_num)
pct_change <- if (is.finite(start_level) && abs(start_level) > 1e-12) {
100 * total_change / abs(start_level)
} else NA_real_Step 8: Serial-correlation caution — lag-1 autocorrelation of the
residuals from a simple linear fit. Positive autocorrelation makes the Mann-Kendall variance too small, so the p-value overstates confidence.
fit <- stats::lm(x ~ t_num)
e <- stats::residuals(fit)
em <- mean(e)
r1 <- sum((e[-n_points] - em) * (e[-1] - em)) / sum((e - em)^2)
autocorr_flag <- is.finite(r1) && abs(r1) > 0.3
autocorr_caution <- if (!autocorr_flag) {
sprintf("Lag-1 autocorrelation of the detrended residuals is %s — low enough that the Mann-Kendall p-value can be read at face value.",
fmt_num(r1, 2))
} else if (r1 > 0) {
sprintf("WARNING: the detrended residuals are strongly serially correlated(lag-1 autocorrelation %s, above the 0.3 threshold). Positive autocorrelation makes consecutive %s move together, which shrinks the effective sample size and INFLATES the trend test's confidence — the true p-value is larger than the %s reported here. Treat the significance verdict as optimistic.",
fmt_num(r1, 2), pts(2), fmt_p(p_value))
} else {
sprintf("WARNING: the detrended residuals show strong negative serial correlation(lag-1 autocorrelation %s, beyond the 0.3 threshold). The normal approximation behind the Mann-Kendall p-value assumes independent observations, so the reported p-value should be treated with caution.",
fmt_num(r1, 2))
}Step 9: Verdict
significant <- is.finite(p_value) && p_value < 0.05
trend_up <- S > 0
verdict <- if (!significant) "No significant trend"
else if (trend_up) "Rising" else "Falling"
direction_word <- if (trend_up) "upward" else if (S < 0) "downward" else "flat"Step 10: Chart dataset — actual series + the Sen trend line
(<= 1000 points per series; always keep the first and last point)
if (n_points <= 1000) {
keep_idx <- seq_len(n_points)
} else {
set.seed(42)
keep_idx <- sort(unique(c(1L, n_points,
sample(2:(n_points - 1), 998))))
}
period_chr <- if (seq_kind == "date") {
format(as.Date(t_num[keep_idx], origin = "1970-01-01"), "%Y-%m-%d")
} else {
as.character(t_num[keep_idx])
}
fitted_line <- intercept + sen_slope_t * t_num[keep_idx]
trend_series_df <- rbind(
data.frame(period = period_chr,
value = round(x[keep_idx], 4),
series = "Actual",
stringsAsFactors = FALSE),
data.frame(period = period_chr,
value = round(fitted_line, 4),
series = "Sen trend",
stringsAsFactors = FALSE)
)
rownames(trend_series_df) <- NULLStep 11: Test-detail table
mk_test_df <- data.frame(
statistic = c("Mann-Kendall S", "Variance of S(tie-corrected)",
"z statistic(continuity-corrected)", "Two-sided p-value",
"Tied value groups", "Time points tested"),
value = c(fmt_num(S, 0), fmt_num(varS, 1), fmt_num(z_stat, 3),
fmt_p(p_value), sprintf("%d(%d values)", n_tie_groups, n_tied_values),
fmt_num(n_points, 0)),
interpretation = c(
sprintf("Concordant minus discordant pairs — positive means %s tends to rise over %s", value_name, date_name),
sprintf("Null-hypothesis variance of S, reduced for the %d group(s) of tied values", n_tie_groups),
"S standardized by its null variance, with the +/-1 continuity correction",
if (significant) sprintf("Below 0.05 — the %s trend is unlikely to be chance", direction_word)
else "At or above 0.05 — the ordering is consistent with noise",
if (n_tie_groups > 0) "Repeated values reduce the usable ordering information"
else "No repeated values — the full ordering information was used",
sprintf("Distinct time points of %s after cleaning", value_name)
),
stringsAsFactors = FALSE
)
rownames(mk_test_df) <- NULLStep 12: Slope table
window_desc <- sprintf("%s(%s)", seq_range_text,
if (seq_kind == "date") sprintf("about %s %ss", fmt_num(span_t / step, 1), period_word)
else sprintf("%s %ss", fmt_num(span_t / max(step, 1e-12), 1), period_word))
sen_slope_df <- data.frame(
statistic = c(sprintf("Sen slope(per %s)", period_word),
"95% confidence interval",
"Implied total change over the window",
"Implied change as % of starting level",
"Observed window"),
value = c(fmt_num(sen_slope_period, 4),
sprintf("%s to %s", fmt_num(ci_lo_period, 4), fmt_num(ci_hi_period, 4)),
fmt_num(total_change, 3),
if (is.na(pct_change)) "n/a(starting level is ~0)"
else sprintf("%s%%", fmt_num(pct_change, 1)),
window_desc),
interpretation = c(
sprintf("Median of the %s pairwise slopes, in units of %s per %s%s",
format(n_pairs_used, big.mark = ","), value_name, period_word,
if (slopes_sampled) sprintf(" (a seeded random sample of the %s possible pairs)",
format(n_pairs, big.mark = ",")) else ""),
"Classical rank-based interval from the ordered pairwise slopes",
sprintf("The Sen slope carried across the full window, in units of %s", value_name),
if (is.na(pct_change)) "The trend-line starting level is approximately zero, so a percentage is not meaningful"
else sprintf("Total change relative to the Sen trend line's starting level of %s", fmt_num(start_level, 3)),
sprintf("The span of %s covered by the test", date_name)
),
stringsAsFactors = FALSE
)
rownames(sen_slope_df) <- NULLStep 13: Methods / checks table
methods_df <- data.frame(
check = c("Trend test", "Tie correction", "Continuity correction",
"Slope estimator", "Pairwise slopes used",
"Serial-correlation check"),
result = c(sprintf("Mann-Kendall, two-sided(z = %s, p %s)", fmt_num(z_stat, 3),
if (p_value < 0.0001) fmt_p(p_value) else paste0("= ", fmt_p(p_value))),
if (n_tie_groups > 0) sprintf("Applied(%d tie group(s))", n_tie_groups) else "Not needed(no ties)",
"Applied(+/-1 on S)",
sprintf("Sen's slope: %s per %s", fmt_num(sen_slope_period, 4), period_word),
if (slopes_sampled) sprintf("%s of %s pairs(seeded sample)",
format(n_pairs_used, big.mark = ","),
format(n_pairs, big.mark = ","))
else sprintf("All %s pairs", format(n_pairs_used, big.mark = ",")),
sprintf("Lag-1 autocorrelation of residuals = %s(threshold 0.3) — %s",
fmt_num(r1, 3), if (autocorr_flag) "CAUTION FIRED" else "clear")),
notes = c("Nonparametric: uses only the ordering of the values, so outliers and skew do not distort it",
"Tied values carry no ordering information; the variance of S is reduced accordingly",
"Improves the normal approximation for the discrete S statistic",
"Median of pairwise slopes — robust to outliers, stated in the value's own units",
if (slopes_sampled) "With this many points the full pair set is large; a seeded sample of 5,000 pairs estimates the same median"
else "The complete pairwise slope set was computed — no sampling",
if (autocorr_flag) "See the caution: serial correlation inflates the trend test's confidence"
else "Residuals look serially independent enough for the standard test"),
stringsAsFactors = FALSE
)
rownames(methods_df) <- NULL
metrics <- list(
`Observations` = n_points,
`Trend Verdict` = verdict,
`Sen Slope` = round(sen_slope_period, 4),
`Slope Units` = sprintf("per %s", period_word),
`MK p-value` = fmt_p(p_value),
`Total Change` = round(total_change, 3),
`Lag-1 Autocorrelation` = round(r1, 3),
`Autocorrelation Caution` = if (autocorr_flag) "yes — read the warning" else "none"
)
json_output <- list(
answer = paste0(
"Mann-Kendall trend test on ", n_points, " time ", pts(n_points), " of ",
value_name, " over ", date_name, " (", seq_range_text, "): ",
if (significant) {
paste0("a statistically significant ", direction_word, " trend(S = ",
fmt_num(S, 0), ", z = ", fmt_num(z_stat, 2), ", p ",
if (p_value < 0.0001) fmt_p(p_value) else paste0("= ", fmt_p(p_value)), "). ")
} else {
paste0("no statistically significant monotonic trend(S = ", fmt_num(S, 0),
", z = ", fmt_num(z_stat, 2), ", p = ", fmt_p(p_value),
") — the movement is consistent with noise. ")
},
"Sen's slope estimates ", fmt_num(sen_slope_period, 4), " per ", period_word,
" (95% CI ", fmt_num(ci_lo_period, 4), " to ", fmt_num(ci_hi_period, 4),
"), implying a total change of ", fmt_num(total_change, 3),
if (!is.na(pct_change)) paste0(" (", fmt_num(pct_change, 1), "% of the starting level)") else "",
" across the window.",
if (autocorr_flag) paste0(" CAUTION: lag-1 autocorrelation of the residuals is ",
fmt_num(r1, 2),
" — serial correlation inflates the test's confidence, so the p-value is optimistic.") else ""
),
cards = lapply(
c("tldr", "overview", "preprocessing", "trend_chart",
"mk_test", "sen_slope", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
date_name = date_name, value_name = value_name,
seq_kind = seq_kind, seq_range_text = seq_range_text,
n_points = n_points, n_dropped = n_dropped, n_dup_collapsed = n_dup_collapsed,
period_word = period_word, step = step,
S = S, varS = varS, z_stat = z_stat, p_value = p_value,
n_tie_groups = n_tie_groups, n_tied_values = n_tied_values,
sen_slope_t = sen_slope_t, sen_slope_period = sen_slope_period,
ci_lo_period = ci_lo_period, ci_hi_period = ci_hi_period,
n_pairs = n_pairs, n_pairs_used = n_pairs_used, slopes_sampled = slopes_sampled,
intercept = intercept, start_level = start_level,
total_change = total_change, pct_change = pct_change,
r1 = r1, autocorr_flag = autocorr_flag, autocorr_caution = autocorr_caution,
trend_up = trend_up, significant = significant,
verdict = verdict, direction_word = direction_word,
trend_series_df = trend_series_df, mk_test_df = mk_test_df,
sen_slope_df = sen_slope_df, methods_df = methods_df,
metrics = metrics, json_output = json_output
)
}