Executive Summary
Best-fitting shape for charges
Medical insurance charges follow a lognormal distribution best among the 6 candidates fitted. The lognormal fit (meanlog = 9.0987, sdlog = 0.9192) achieves an AIC of 27,923.58, leading gamma—the runner-up—by 72.59 AIC points, a gap large enough to be a real decision. The goodness-of-fit test does not reject the lognormal (p = 0.056), meaning the data is consistent with that shape. At 1,338 observations, however, goodness-of-fit tests are highly sensitive and reject almost any candidate because no real dataset follows a textbook distribution exactly. The lognormal is the best fit among the candidates offered, not a fact about how charges are generated.
Analysis Overview
Maximum-likelihood distribution fitting on charges across 1,338 values.
Maximum-likelihood fitting tests 6 candidate distributions against 1,338 medical insurance charges to find the shape that best describes the data. The method estimates the parameters of each candidate that make the observed values most probable, then ranks them by the Akaike Information Criterion, which balances fit quality against model complexity. Two count-based candidates (Poisson and negative binomial) were skipped because charges contain fractional values. This analysis answers which named distribution fits best—a different question from whether the data is normal enough for a t-test, which a separate normality screening tool addresses.
Data Quality
Readable values, ties, and which candidates were applicable.
All 1,338 rows carried readable charge values; no imputation was required. One value repeated an already-present figure due to rounding or discrete recording. Charges range from 1,122 to 63,770, all strictly positive, so every positive-support distribution applies. Poisson and negative binomial were inapplicable because they require whole numbers and charges contains fractional values. The presence of one tie makes the Kolmogorov-Smirnov p-value approximate rather than exact, a minor limitation on that test's precision.
Candidate Ranking
Every fitted candidate with its parameters, log-likelihood, AIC and BIC.
| Distribution | Family | Parameters | Log Likelihood | Aic | Delta Aic | Bic |
|---|---|---|---|---|---|---|
| lognormal | continuous | meanlog = 9.0987, sdlog = 0.9192 | -1.396e+04 | 2.792e+04 | 0 | 2.793e+04 |
| gamma | continuous | shape = 1.4104, rate = 0.0001 | -1.4e+04 | 2.8e+04 | 72.59 | 2.801e+04 |
| weibull | continuous | shape = 1.1756, scale = 14,100 | -1.401e+04 | 2.803e+04 | 103.7 | 2.804e+04 |
| exponential | continuous | rate = 0.0001 | -1.404e+04 | 2.808e+04 | 158.5 | 2.809e+04 |
| normal | continuous | mean = 13,270, sd = 12,105 | -1.448e+04 | 2.896e+04 | 1036 | 2.897e+04 |
| uniform | continuous | min = 1,122, max = 63,770 | -1.478e+04 | 2.956e+04 | 1638 | 2.957e+04 |
The lognormal distribution ranks first with AIC 27,923.58. Gamma places second at AIC 27,996.18, trailing by 72.59 points—a decisive gap that indicates the lognormal is genuinely the better choice. Weibull (AIC 28,027.32), exponential (AIC 28,082.05), normal (AIC 28,959.26), and uniform (AIC 29,561.21) follow in order of declining fit. The Bayesian Information Criterion, which penalizes extra parameters more heavily, agrees with AIC: lognormal remains the winner. The 72.59-point AIC gap to gamma is large enough that the two shapes are practically distinguishable on this data; gaps under about 2 would indicate the candidates are indistinguishable.
Observed vs Fitted
The histogram of charges with the winning fit drawn over it, bin by bin.
The fitted lognormal tracks the observed histogram reasonably through the middle range of charges but systematically overpredicts the lower tail. The largest single gap is 46.72 observations in the bin from 1,122 to 3,210, where the fit expects 161.28 but observes 208 values. The fit recovers through the middle bins (3,210 to 22,005) where predicted and observed counts align closely, then diverges again at the upper tail, underpredicting very high charges. This pattern—tracking the middle well while missing the tails—is common and matters most when the tail drives the decision (a service-level target, a risk limit, or a capacity headroom).
Q-Q Plot
Observed quantiles against the quantiles the winning fit predicts.
Observed and theoretical quantiles correlate at 0.9084, indicating a moderately tight relationship overall. The Q-Q plot shows the fitted lognormal pulls away from the diagonal at both ends. At the lower tail, observed charges lie above the diagonal, meaning the fit underpredicts the smallest values. At the upper tail, observed charges curve below the diagonal, indicating the fit overstates how far the largest charges extend. The middle of the distribution hugs the diagonal closely. Because a fit can match the center while being badly wrong in the tail—and the tail is usually what a model is fitted for—the curvature at the top end is the critical finding: the lognormal understates the magnitude of the largest charges.
Goodness of Fit
Formal tests of each fitted candidate against the observed data.
| Distribution | Test | Statistic | P Value | Anderson Darling | Ad Reading |
|---|---|---|---|---|---|
| lognormal | Kolmogorov-Smirnov | 0.0366 | 0.056 | 3.942 | strong misfit (above 3.88) |
| gamma | Kolmogorov-Smirnov | 0.0774 | <0.001 | 12.28 | strong misfit (above 3.88) |
| weibull | Kolmogorov-Smirnov | 0.0844 | <0.001 | 13.58 | strong misfit (above 3.88) |
| exponential | Kolmogorov-Smirnov | 0.0961 | <0.001 | 19.6 | strong misfit (above 3.88) |
| normal | Kolmogorov-Smirnov | 0.1885 | <0.001 | 85.14 | strong misfit (above 3.88) |
| uniform | Kolmogorov-Smirnov | 0.5148 | <0.001 | 857.1 | strong misfit (above 3.88) |
The lognormal is the only candidate not rejected at the 5 percent significance level, with a Kolmogorov-Smirnov p-value of 0.056. All other candidates—gamma (p < 0.001), Weibull (p < 0.001), exponential (p < 0.001), normal (p < 0.001), and uniform (p < 0.001)—are rejected. The Anderson-Darling statistic, which weights tails more heavily, shows the lognormal at 3.942, just above the 3.88 boundary for strong misfit, while all others fall well above it. Passing does not confirm the lognormal is the true generator; several different shapes can pass the same data, and at 1,338 values the tests are so sensitive that they reject almost any candidate because no real dataset follows a textbook distribution exactly.
Methods & Disclosure
How the fits were estimated, ranked, and tested — and what the result does not say.
| Item | Detail |
|---|---|
| Estimation | Maximum likelihood. Closed-form estimators for normal, lognormal, exponential, uniform and poisson; base optim (Nelder-Mead, two passes) for gamma, weibull and the negative binomial. A search that fails to converge is reported as such, never as a fit. |
| Ranking | Akaike Information Criterion, with the Bayesian Information Criterion alongside. The gap to the runner-up here is 72.59 AIC points; a gap under about 2 means the two candidates are practically indistinguishable on this data. |
| Comparability across families | Continuous candidates are scored on a probability density and discrete ones on a probability mass, so their likelihoods are on different scales. AIC is compared only WITHIN a family, never across the dashed line between them. |
| Candidates skipped | poisson (requires whole numbers; charges contains fractional values); negative binomial (requires whole numbers; charges contains fractional values) |
| Candidates that did not converge | none — every applicable candidate converged |
| Goodness of fit | Kolmogorov-Smirnov (base stats) and an Anderson-Darling statistic computed directly from its definition for continuous candidates; a Pearson chi-square on merged integer cells for discrete ones. Because the parameters were estimated from the same data the tests score, both p-values are optimistic — the true rejection rate is higher than the nominal one. |
| Effect of sample size | At 1,338 values the goodness-of-fit tests are highly sensitive: at this sample size they reject almost any candidate, because no real dataset follows a textbook distribution exactly. A small p-value here flags a detectable departure, not necessarily one large enough to matter for your use. |
| Uniform fit caveat | The uniform fit places its boundaries exactly at the smallest and largest observed values, which flatters its likelihood. Treat a uniform win with suspicion unless the quantity really is bounded. |
| What a fit is | A fitted distribution is a description that survived comparison with the alternatives offered here, not a fact about the process that generated charges. A shape outside this panel may fit better still. |
| Which tool answers which question | This analysis asks which named shape best describes charges as a whole. The separate normality and outlier screening tool asks a different question — whether a column is close enough to normal, and free enough of outliers, for a t-test or a regression to be trustworthy. Use that one before a parametric test; use this one before a simulation, a queueing model or a pricing model. |
Maximum-likelihood estimation was applied to 6 candidates using closed-form estimators for normal, lognormal, exponential, uniform, and Poisson, and numerical optimization (Nelder-Mead) for gamma, Weibull, and negative binomial. All applicable candidates converged. Ranking used the Akaike Information Criterion within the continuous family; the 72.59-point gap to gamma indicates practical distinguishability. Poisson and negative binomial were skipped because charges contain fractional values. Goodness-of-fit used Kolmogorov-Smirnov and Anderson-Darling tests; both p-values are optimistic because parameters were estimated from the same data the tests score. The lognormal is the best fit among candidates offered on this sample, not a claim about the process generating charges. A separate normality and outlier screening tool answers whether the data is normal enough for a t-test.
Distribution Fitting — What Shape Does This Column Follow?
Fits a panel of candidate probability distributions to one numeric column by maximum likelihood, ranks them by AIC and BIC, tests each fit for goodness of fit, and shows the winner drawn over a histogram of the actual data plus a quantile-quantile plot.
Why This Method?
Pricing models, queueing models, reliability targets, risk capital and simulation all start from an assumed distribution. Guessing that shape from a histogram is unreliable. Fitting every plausible candidate by maximum likelihood and ranking them on the same information criterion makes the choice explicit, comparable, and reproducible — and makes visible how close the runners-up were.
What This Analysis Covers
- Maximum-likelihood fits of normal, lognormal, gamma, weibull, exponential
and uniform, plus poisson and negative binomial when the data is counts
- A ranking by AIC and BIC inside each family, with the runner-up gap
- Kolmogorov-Smirnov and Anderson-Darling goodness of fit for continuous
candidates, a Pearson chi-square goodness of fit for discrete ones
- The winning fit drawn over the observed histogram, and a Q-Q plot
- An explicit statement of what the sample size does to the tests' power
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {value}. All narrative is derived from the user's own column name and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Core Analysis Pipeline
Step 1: Find the mapped column and humanize its name
initial_rows <- nrow(df)
if (!("value" %in% names(df))) {
stop("column_mapping must map one numeric column to 'value'.")
}
var_name <- unname(humanize_semantic("value", col_map))
if (length(var_name) == 0 || !nzchar(var_name)) var_name <- "the mapped column"Step 2: Coerce to numeric under the 95 percent rule
v_raw <- df[["value"]]
if (!is.numeric(v_raw)) {
conv <- suppressWarnings(as.numeric(as.character(v_raw)))
n_orig <- sum(!is.na(v_raw) & trimws(as.character(v_raw)) != "")
if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
v_raw <- conv
} else {
stop(sprintf(paste0("The column mapped for fitting, %s, is not numeric — ",
"fewer than 95 percent of its non-blank entries could ",
"be read as numbers. Map a numeric column instead."),
var_name))
}
}Missing values are set aside, never imputed: an imputed value would change the very shape this analysis is trying to describe.
x_all <- suppressWarnings(as.numeric(v_raw))
n_missing <- as.integer(sum(!is.finite(x_all)))
x <- x_all[is.finite(x_all)]
n <- length(x)
final_rows <- n
rows_removed <- initial_rows - final_rows
if (n < 20) {
stop(sprintf(paste0("Distribution fitting needs at least 20 usable values in %s; ",
"only %d were found. With fewer than 20 values no candidate ",
"shape can be told apart from any other."), var_name, n))
}
sd_x <- sd(x)
if (!is.finite(sd_x) || sd_x == 0) {
stop(sprintf(paste0("%s is constant — every usable row holds the value %s. ",
"A constant column has no distribution to fit."),
var_name, fmt_num(x[1])))
}
n_ties <- as.integer(n - length(unique(x)))Step 3: Screen candidates against their domains
A candidate whose support does not contain the data is SKIPPED with a stated reason. It is never fitted to nonsense and never silently dropped.
x_min <- min(x); x_max <- max(x)
strictly_positive <- x_min > 0
count_shaped <- (x_min >= 0) && all(abs(x - round(x)) < 1e-8)
pos_reason <- sprintf(
"requires strictly positive values; the smallest value in %s is %s",
var_name, fmt_num(x_min))
int_reason <- if (x_min < 0) {
sprintf("requires non-negative whole numbers; %s contains negative values(smallest is %s)",
var_name, fmt_num(x_min))
} else {
sprintf("requires whole numbers; %s contains fractional values", var_name)
}
fits <- list()
for (dist in .CANDIDATES) {
applicable <- TRUE; reason <- ""
if (dist %in% c("lognormal", "gamma", "weibull", "exponential") && !strictly_positive) {
applicable <- FALSE; reason <- pos_reason
}
if (dist %in% .DISCRETE && !count_shaped) {
applicable <- FALSE; reason <- int_reason
}
if (!applicable) {
fits[[dist]] <- list(dist = dist,
family = if (dist %in% .DISCRETE) "discrete" else "continuous",
status = "skipped", reason = reason,
params = numeric(0), k = NA_integer_,
loglik = NA_real_, aic = NA_real_, bic = NA_real_)
} else {
fits[[dist]] <- .fit_candidate(dist, x)
}
}
status_vec <- vapply(fits, function(f) f$status, character(1))
fitted_names <- names(fits)[status_vec == "fitted"]
skipped <- names(fits)[status_vec == "skipped"]
failed <- names(fits)[status_vec == "did not converge"]
if (length(fitted_names) == 0) {
stop(sprintf(paste0("No candidate distribution could be fitted to %s: every ",
"candidate was either outside its valid domain or failed ",
"to converge."), var_name))
}Step 4: Rank within family
AIC compares models fitted to the SAME data on the SAME likelihood scale. Continuous candidates use a density, discrete candidates a probability, so their AIC values are NOT comparable to one another. Ranking therefore happens inside each family and the report says so.
fam_vec <- vapply(fits[fitted_names], function(f) f$family, character(1))
aic_vec <- vapply(fits[fitted_names], function(f) f$aic, numeric(1))
bic_vec <- vapply(fits[fitted_names], function(f) f$bic, numeric(1))
ll_vec <- vapply(fits[fitted_names], function(f) f$loglik, numeric(1))
k_vec <- vapply(fits[fitted_names], function(f) as.integer(f$k), integer(1))
delta_vec <- rep(NA_real_, length(fitted_names))
names(delta_vec) <- fitted_names
for (fam in unique(fam_vec)) {
idx <- which(fam_vec == fam)
ok_idx <- idx[is.finite(aic_vec[idx])]
if (length(ok_idx) > 0) {
delta_vec[ok_idx] <- aic_vec[ok_idx] - min(aic_vec[ok_idx])
}
}
primary_family <- if (count_shaped) "discrete" else "continuous"
prim_idx <- which(fam_vec == primary_family & is.finite(aic_vec))
if (length(prim_idx) == 0) {
prim_idx <- which(is.finite(aic_vec))
primary_family <- fam_vec[prim_idx[which.min(aic_vec[prim_idx])]]
prim_idx <- which(fam_vec == primary_family & is.finite(aic_vec))
}NA-filtered before which.min (LAT-1445 crash class).
win_i <- prim_idx[which.min(aic_vec[prim_idx])]
winner <- fits[[fitted_names[win_i]]]
other_prim <- setdiff(prim_idx, win_i)
runner_up_gap <- if (length(other_prim) > 0) {
min(aic_vec[other_prim]) - aic_vec[win_i]
} else NA_real_
runner_up_name <- if (length(other_prim) > 0) {
fitted_names[other_prim[which.min(aic_vec[other_prim])]]
} else "none"
ord <- order(fam_vec != primary_family, aic_vec)
fit_ranking_df <- data.frame(
distribution = fitted_names[ord],
family = fam_vec[ord],
parameters = vapply(fitted_names[ord],
function(nm) .param_string(fits[[nm]]), character(1)),
log_likelihood = round(ll_vec[ord], 2),
aic = round(aic_vec[ord], 2),
delta_aic = round(delta_vec[ord], 2),
bic = round(bic_vec[ord], 2),
stringsAsFactors = FALSE
)
rownames(fit_ranking_df) <- NULLStep 5: Goodness of fit
Continuous candidates: Kolmogorov-Smirnov from base stats plus an Anderson-Darling statistic computed directly from its definition. Discrete candidates: a Pearson chi-square on the binned counts, because the Kolmogorov-Smirnov test assumes a continuous distribution.
gof_rows <- list()
ks_p_map <- setNames(rep(NA_real_, length(fitted_names)), fitted_names)
for (nm in fitted_names) {
f <- fits[[nm]]
pf <- .dist_pfun(nm, f$params)
if (f$family == "continuous") {
kt <- tryCatch(suppressWarnings(ks.test(x, pf)), error = function(e) NULL)
kstat <- if (!is.null(kt)) as.numeric(kt$statistic) else NA_real_
kp <- if (!is.null(kt)) as.numeric(kt$p.value) else NA_real_
a2 <- .anderson_darling(x, pf)
gof_rows[[length(gof_rows) + 1]] <- data.frame(
distribution = nm, test = "Kolmogorov-Smirnov",
statistic = if (is.finite(kstat)) round(kstat, 4) else NA_real_,
p_value = fmt_p(kp),
anderson_darling = if (is.finite(a2)) round(a2, 3) else NA_real_,
ad_reading = .ad_verdict(a2),
stringsAsFactors = FALSE)
ks_p_map[[nm]] <- kp
} else {
cg <- .chisq_gof(round(x), .dist_pmf(nm, f$params), f$k)
gof_rows[[length(gof_rows) + 1]] <- data.frame(
distribution = nm,
test = if (is.na(cg$df)) "chi-square(too few cells)"
else sprintf("chi-square on %d cells, %d df", cg$cells, cg$df),
statistic = if (is.finite(cg$stat)) round(cg$stat, 4) else NA_real_,
p_value = fmt_p(cg$p),
anderson_darling = NA_real_,
ad_reading = "not applicable to a discrete fit",
stringsAsFactors = FALSE)
ks_p_map[[nm]] <- cg$p
}
}
gof_df <- do.call(rbind, gof_rows)
gof_df <- gof_df[order(match(gof_df$distribution, fit_ranking_df$distribution)), , drop = FALSE]
rownames(gof_df) <- NULL
winner_gof_p <- ks_p_map[[winner$dist]]
n_not_rejected <- sum(is.finite(ks_p_map) & ks_p_map >= 0.05)Step 6: Observed versus fitted, bin by bin (the overlay)
win_pf <- .dist_pfun(winner$dist, winner$params)
if (winner$family == "discrete") {
xi <- round(x)
top <- min(max(xi), 25)
vals <- 0:top
obs <- as.integer(table(factor(xi, levels = vals)))
pmf <- .dist_pmf(winner$dist, winner$params)
pr <- suppressWarnings(pmf(vals)); pr[!is.finite(pr)] <- 0
labs <- as.character(vals)
if (max(xi) > top) {
obs <- c(obs, as.integer(sum(xi > top)))
pr <- c(pr, max(0, 1 - sum(pr)))
labs <- c(labs, paste0(top + 1, " or more"))
}
expected <- n * pr
} else {
nb <- max(8L, min(30L, as.integer(ceiling(sqrt(n)))))
brks <- seq(x_min, x_max, length.out = nb + 1)
obs <- as.integer(table(cut(x, breaks = brks, include.lowest = TRUE)))
lo <- suppressWarnings(win_pf(brks[-(nb + 1)]))
hi <- suppressWarnings(win_pf(brks[-1]))
pr <- hi - lo
pr[!is.finite(pr)] <- 0
expected <- n * pr
labs <- sprintf("%s to %s", fmt_num(brks[-(nb + 1)]), fmt_num(brks[-1]))
}
overlay_df <- data.frame(
bin = rep(labs, each = 2),
count = as.numeric(rbind(obs, round(expected, 2))),
series = rep(c("Observed", paste0("Fitted ", winner$dist)), times = length(labs)),
stringsAsFactors = FALSE
)
rownames(overlay_df) <- NULL
bin_gap <- max(abs(obs - expected))
worst_bin <- labs[which.max(abs(obs - expected))]Step 7: Q-Q points (thinned deterministically to at most 1,000)
win_qf <- .dist_qfun(winner$dist, winner$params)
xs <- sort(x)
pp <- (seq_len(n) - 0.5) / n
theo <- suppressWarnings(win_qf(pp))
idx <- if (n > 1000) unique(round(seq(1, n, length.out = 1000))) else seq_len(n)
qq_df <- data.frame(theoretical = theo[idx], sample = xs[idx],
stringsAsFactors = FALSE)
qq_df <- qq_df[is.finite(qq_df$theoretical) & is.finite(qq_df$sample), , drop = FALSE]
qq_df$theoretical <- round(qq_df$theoretical, 4)
qq_df$sample <- round(qq_df$sample, 4)
rownames(qq_df) <- NULL
qq_cor <- if (nrow(qq_df) > 2) {
suppressWarnings(cor(qq_df$theoretical, qq_df$sample))
} else NA_real_Step 8: The power statement — computed, and it names the actual n
power_text <- if (n < 50) {
paste0("With only ", n, " value", .s(n), " the goodness-of-fit tests have ",
"very little power: at this sample size they fail to reject almost ",
"any candidate shape, so a large p-value here means \"too little ",
"data to tell them apart\", not \"the data is this distribution\".")
} else if (n <= 1000) {
paste0("At ", format(n, big.mark = ","), " values the goodness-of-fit tests ",
"have moderate power: they will catch a badly wrong shape but not a ",
"subtly wrong one, so a large p-value means the candidate is not ",
"contradicted rather than confirmed.")
} else {
paste0("At ", format(n, big.mark = ","), " values the goodness-of-fit tests ",
"are highly sensitive: at this sample size they reject almost any ",
"candidate, because no real dataset follows a textbook distribution ",
"exactly. A small p-value here flags a detectable departure, not ",
"necessarily one large enough to matter for your use.")
}Step 9: Disclosure table
skip_detail <- if (length(skipped) > 0) {
paste(vapply(skipped, function(nm)
paste0(nm, " (", fits[[nm]]$reason, ")"), character(1)), collapse = "; ")
} else "none — every candidate was inside its valid domain"
fail_detail <- if (length(failed) > 0) {
paste(vapply(failed, function(nm)
paste0(nm, " (", fits[[nm]]$reason, ")"), character(1)), collapse = "; ")
} else "none — every applicable candidate converged"
methods_df <- data.frame(
item = c(
"Estimation",
"Ranking",
"Comparability across families",
"Candidates skipped",
"Candidates that did not converge",
"Goodness of fit",
"Effect of sample size",
"Uniform fit caveat",
"What a fit is",
"Which tool answers which question"),
detail = c(
paste0("Maximum likelihood. Closed-form estimators for normal, lognormal, ",
"exponential, uniform and poisson; base optim(Nelder-Mead, two ",
"passes) for gamma, weibull and the negative binomial. A search ",
"that fails to converge is reported as such, never as a fit."),
paste0("Akaike Information Criterion, with the Bayesian Information ",
"Criterion alongside. The gap to the runner-up here is ",
if (is.finite(runner_up_gap)) fmt_num(runner_up_gap, 2) else "not applicable",
" AIC points; a gap under about 2 means the two candidates are ",
"practically indistinguishable on this data."),
paste0("Continuous candidates are scored on a probability density and ",
"discrete ones on a probability mass, so their likelihoods are on ",
"different scales. AIC is compared only WITHIN a family, never ",
"across the dashed line between them."),
skip_detail,
fail_detail,
paste0("Kolmogorov-Smirnov(base stats) and an Anderson-Darling statistic ",
"computed directly from its definition for continuous candidates; a ",
"Pearson chi-square on merged integer cells for discrete ones. ",
"Because the parameters were estimated from the same data the tests ",
"score, both p-values are optimistic — the true rejection rate is ",
"higher than the nominal one."),
power_text,
paste0("The uniform fit places its boundaries exactly at the smallest and ",
"largest observed values, which flatters its likelihood. Treat a ",
"uniform win with suspicion unless the quantity really is bounded."),
paste0("A fitted distribution is a description that survived comparison ",
"with the alternatives offered here, not a fact about the process ",
"that generated ", var_name, ". A shape outside this panel may fit ",
"better still."),
paste0("This analysis asks which named shape best describes ", var_name,
" as a whole. The separate normality and outlier screening tool ",
"asks a different question — whether a column is close enough to ",
"normal, and free enough of outliers, for a t-test or a regression ",
"to be trustworthy. Use that one before a parametric test; use this ",
"one before a simulation, a queueing model or a pricing model.")),
stringsAsFactors = FALSE
)
metrics <- list(
`Observations` = n,
`Best Fitting Shape` = winner$dist,
`Best Fit AIC` = round(winner$aic, 2),
`Runner-Up AIC Gap` = if (is.finite(runner_up_gap)) round(runner_up_gap, 2) else NA_real_,
`Candidates Fitted` = as.integer(length(fitted_names)),
`Candidates Skipped` = as.integer(length(skipped))
)
gap_phrase <- if (!is.finite(runner_up_gap)) {
paste0("It was the only candidate in the ", primary_family, " family that could be fitted.")
} else if (runner_up_gap < 2) {
paste0("The gap to ", runner_up_name, " is only ", fmt_num(runner_up_gap, 2),
" AIC points, so the two are practically indistinguishable on this data ",
"and the ranking should not be read as a decision between them.")
} else {
paste0("It leads ", runner_up_name, " by ", fmt_num(runner_up_gap, 2),
" AIC points.")
}
json_output <- list(
answer = paste0(
"Fitted ", length(fitted_names), " candidate distribution",
.s(length(fitted_names)), " to ", var_name, " across ",
format(n, big.mark = ","), " value", .s(n),
" by maximum likelihood. The best fit inside the ", primary_family,
" family is the ", winner$dist, " (", .param_string(winner),
"), at AIC ", fmt_num(winner$aic, 2), ". ", gap_phrase, " ",
if (length(skipped) > 0) paste0(
length(skipped), " candidate", .s(length(skipped)), " ",
if (length(skipped) == 1) "was" else "were",
" skipped as inapplicable to this column: ",
paste(skipped, collapse = ", "), ". ") else "",
power_text,
" A fitted distribution is a description that beat the alternatives, not a ",
"fact about the process that generated ", var_name, "."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "fit_comparison",
"distribution_fit", "qq_plot", "goodness_of_fit", "methods"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
var_name = var_name, x = x, n = n, n_missing = n_missing, n_ties = n_ties,
x_min = x_min, x_max = x_max,
strictly_positive = strictly_positive, count_shaped = count_shaped,
fits = fits, fitted_names = fitted_names, skipped = skipped, failed = failed,
primary_family = primary_family, winner = winner,
runner_up_gap = runner_up_gap, runner_up_name = runner_up_name,
fit_ranking_df = fit_ranking_df, gof_df = gof_df,
overlay_df = overlay_df, qq_df = qq_df, methods_df = methods_df,
bin_gap = bin_gap, worst_bin = worst_bin, qq_cor = qq_cor,
winner_gof_p = winner_gof_p, n_not_rejected = n_not_rejected,
power_text = power_text, gap_phrase = gap_phrase,
metrics = metrics, json_output = json_output
)
}