Executive Summary
Whether predicted probability means what it says about diabetes
The short answer
The model is mildly miscalibrated with an expected calibration error of 0.0237, but the verdict is fragile: it changes across different binning choices, ranging from "well calibrated" to "clearly miscalibrated."
The detail
Across 768 rows, predicted probability equals observed outcome at the aggregate level (both 34.9%), so the model is neither systematically over- nor under-confident overall. A logistic recalibration returns a slope of 1.000 with a 95% confidence interval that includes 1, meaning the spread of predictions is consistent with being appropriate. The worst bin holding at least 10 rows covers predictions from 0.50 to 0.60: 38 rows predicted an average of 54.5% but 63.2% actually were '1' (95% interval 47.3% to 76.6%). The reliability term of 0.0012 is only 0.8% of the Brier score, so recalibration would fix almost nothing. Across four binning schemes, the expected calibration error ranged over 0.0396 (from 0.0137 to 0.0533), and the verdict changed between well calibrated and clearly miscalibrated. The direction of error is binning-independent but its magnitude is not.
What this can't tell you
The size of the miscalibration cannot be pinned down from binning alone. The verdict depends critically on how the probability range is divided. Consider examining the per-bin table with confidence intervals to understand where disagreement actually occurs, rather than relying on a single headline metric.
Analysis Overview
Calibration of predicted probability against diabetes across 768 observations.
The diabetes risk model outputs probabilities that are mildly miscalibrated—when it says 70%, the event happens somewhat less often than 70% across the full range. An expected calibration error of 0.0237 quantifies the typical gap between predicted and observed rates. The Brier score of 0.1527 decomposes into three parts: reliability (0.0012, the only piece recalibration could fix), resolution (0.0741, the model's discriminative power), and uncertainty (0.2272, inherent outcome difficulty). Across 768 observations, the model predicted 34.9% on average, matching the observed 34.9%, so there is no systematic over- or under-confidence overall—the miscalibration is distributed across probability ranges.
Data Quality
Positive-class definition, the probability scale, dropped rows, and class balance.
All 768 rows carried both outcome and predicted probability; zero rows were dropped. The outcome split 268 positive (34.9%) and 500 negative cases. Predicted probabilities ranged from 0.002 to 0.993 and were read as probabilities as supplied, requiring no rescaling. No prediction sat at exactly 0 or 1, so logit transformation proceeded without nudging. Rows grouped into 10 equal-width bins across 0 to 1, and all 10 bins held at least 10 rows, so no thin bin dominates the picture and every bin provides credible evidence.
Reliability Curve
Observed outcome rate against predicted probability, per bin of predicted probability.
The short answer
The model's predictions sit on both sides of the perfect-calibration line with no consistent tilt. Six of the 10 bins fall below the diagonal (outcome happened less often than predicted) and four above, indicating scattered local mismatches rather than systematic over- or under-confidence across the probability range.
The detail
The reliability curve plots mean predicted probability against observed outcome rate for each bin. The worst-performing bin that clears the row threshold covers predictions from 0.50 to 0.60: 38 rows predicted an average of 54.5% but the observed rate was 63.2% (95% interval 47.3% to 76.6%), placing this bin well above the diagonal. At the low end, the bin at mean predicted 0.0574 observed only 0.0455. The pattern is a loose scatter rather than a consistent upward or downward tilt. All 10 bins hold at least 10 rows, so no single thin bin drives the picture.
What this can't tell you
Individual bin placement can be affected by sampling variability, particularly in bins holding fewer rows. The confidence intervals in the per-bin table show which deviations are large enough to be evidence of genuine miscalibration versus noise. The pattern here does not indicate a simple rescaling problem that a single recalibration slope could fix.
Bin-by-Bin Calibration
Predicted versus observed per bin, with a binomial confidence interval on every observed rate.
| Bin | N | Mean Predicted | Observed Rate | CI Low | CI High | Gap | Reading |
|---|---|---|---|---|---|---|---|
| 0.00 to 0.10 | 154 | 0.0574 | 0.0455 | 0.0222 | 0.0908 | 0.012 | the prediction falls inside the outcome's confidence interval |
| 0.10 to 0.20 | 153 | 0.1441 | 0.1373 | 0.0916 | 0.2007 | 0.0068 | the prediction falls inside the outcome's confidence interval |
| 0.20 to 0.30 | 105 | 0.2488 | 0.2476 | 0.1749 | 0.3381 | 0.0012 | the prediction falls inside the outcome's confidence interval |
| 0.30 to 0.40 | 86 | 0.3445 | 0.3953 | 0.2986 | 0.501 | -0.0508 | the prediction falls inside the outcome's confidence interval |
| 0.40 to 0.50 | 59 | 0.4457 | 0.4068 | 0.2909 | 0.5341 | 0.0389 | the prediction falls inside the outcome's confidence interval |
| 0.50 to 0.60 | 38 | 0.5447 | 0.6316 | 0.4728 | 0.7662 | -0.0869 | the prediction falls inside the outcome's confidence interval |
| 0.60 to 0.70 | 51 | 0.6558 | 0.6275 | 0.4903 | 0.7468 | 0.0283 | the prediction falls inside the outcome's confidence interval |
| 0.70 to 0.80 | 49 | 0.7453 | 0.7551 | 0.6191 | 0.854 | -0.0098 | the prediction falls inside the outcome's confidence interval |
| 0.80 to 0.90 | 46 | 0.849 | 0.8696 | 0.7433 | 0.9388 | -0.0205 | the prediction falls inside the outcome's confidence interval |
| 0.90 to 1.00 | 27 | 0.9385 | 0.8519 | 0.6752 | 0.9408 | 0.0867 | the prediction falls inside the outcome's confidence interval |
All 10 bins show predicted probability falling inside the 95% Wilson confidence interval around the observed rate—no bin provides evidence of miscalibration when accounting for binomial uncertainty. The largest gap is in the 0.50–0.60 bin: 38 rows, mean prediction 0.5447, observed rate 0.6316 (95% CI 0.4728–0.7662), gap of −0.0869. The smallest gap is in the 0.20–0.30 bin: 105 rows, mean prediction 0.2488, observed rate 0.2476, gap of 0.0012. The discipline here is that a prediction only signals miscalibration when it falls outside the confidence interval; none do. Every bin holds at least 10 rows, eliminating the concern that thin bins are posting large gaps on noise alone.
Calibration Fit & Error
Calibration intercept and slope from a logistic recalibration, plus expected and maximum calibration error.
| Term | Estimate | CI Low | CI High | Interpretation |
|---|---|---|---|---|
| Calibration intercept (with slope free) | 0 | -0.1957 | 0.1957 | Intercept of the logistic fit of the outcome on the predicted logit. 0 is ideal. |
| Calibration slope | 1 | 0.8487 | 1.151 | Slope of that fit. 1 is ideal; below 1 means the predictions are too extreme, above 1 means they are too timid. |
| Calibration-in-the-large | 0 | -0.1805 | 0.1805 | Intercept when the slope is held at 1 — whether the predictions are too high or too low overall. 0 is ideal. |
| Mean predicted probability | 0.349 | — | — | The average probability the model put on '1'. |
| Observed base rate | 0.349 | — | — | The share of rows that actually were '1'. |
| Expected calibration error (ECE) | 0.0237 | — | — | Row-weighted average gap between predicted and observed across the 10 bins used. 0 is perfect. |
| Maximum calibration error (bins with enough rows) | 0.0869 | — | — | The largest gap in any bin holding at least 10 rows — the worst error that is not just small-sample noise. |
| Maximum calibration error (all bins) | 0.0869 | — | — | The largest gap in any non-empty bin, including thin ones. Read it against the row counts in the bin table before acting on it. |
The logistic recalibration yields a slope of 1.000 (95% CI 0.8487–1.1513) and intercept of 0.000, both ideal. The slope's confidence interval includes 1, so the spread of the predictions is consistent with being about right—the model is not systematically too extreme or too timid. The intercept of 0.000 confirms no overall over- or under-forecasting: mean predicted probability was 0.349 against observed base rate 0.349. Expected calibration error of 0.0237 is the row-weighted average gap across the 10 bins. Maximum calibration error is 0.0869 in bins holding at least 10 rows, the same as the maximum across all non-empty bins, confirming that thin bins are not driving a false worst-case reading.
Brier Score Decomposition
The Brier score split into reliability, resolution and uncertainty.
| Component | Value | Interpretation |
|---|---|---|
| Brier score | 0.1527 | Mean squared distance between the predicted probability and the 0/1 outcome across 768 rows. Lower is better; 0 is perfect. |
| Reliability (lower is better) | 0.0012 | How far the predictions sit from what actually happened inside each bin. This is the part of the score that recalibration can remove. |
| Resolution (higher is better) | 0.074 | How far the bins' outcome rates move away from the overall base rate of 34.9%. This is the part that comes from the model actually knowing something. |
| Uncertainty (fixed by the data) | 0.2272 | The variance of the outcome itself, base rate times one minus base rate. No model can reduce this — it is the difficulty of the problem. |
| Within-bin residual | -0.0016 | Brier minus (reliability - resolution + uncertainty). The decomposition is computed over 10 bins, so this leftover is the spread of the predictions inside each bin. |
The Brier score of 0.1527 splits as: reliability 0.0012, resolution 0.0741, uncertainty 0.2272, with a within-bin residual of −0.0016. Reliability (0.8% of the total) is the only component recalibration can remove; it measures how far predictions sit from outcomes inside each bin. Resolution (0.0741) is the credit for pushing bins away from the base rate of 34.9%; it represents what the model actually knows and would be lost if the model simply predicted 34.9% for every row. Uncertainty (0.2272) is fixed by the outcome itself—the variance of a 34.9% base rate—and is beyond any model's reach. The small reliability term indicates that the model's main problem is not dishonesty but limited discrimination.
Distribution of Predicted Probabilities
How predicted probability is spread across the 0 to 1 range.
The busiest bucket is 0.10–0.15, holding 11.5% of the 768 rows. The distribution is bimodal: 23.6% of predictions sit at the extremes (below 0.10 or above 0.90), while 50.5% cluster in the middle (0.20–0.80). The model makes strong claims at the extremes; those are the regions most worth checking against the bin table. Because calibration numbers are weighted averages over this distribution, regions the model rarely visits contribute little to the headline error no matter how wrong the model is there. The concentration in the low-probability tail (0.00–0.20, 20.5% of rows) and the spread across mid-range probabilities mean that calibration errors in the 0.50–0.60 range carry meaningful weight in the overall expected calibration error.
Does the Verdict Survive Re-binning?
Expected and maximum calibration error recomputed under several binnings.
| Binning | N Bins | Ece | Mce | Verdict |
|---|---|---|---|---|
| 5 equal-width bins | 5 | 0.0137 | 0.0222 | well calibrated |
| 10 equal-width bins | 10 | 0.0237 | 0.0869 | mildly miscalibrated |
| 20 equal-width bins | 20 | 0.0533 | 0.1951 | clearly miscalibrated |
| 10 equal-count bins | 10 | 0.0368 | 0.0852 | mildly miscalibrated |
The verdict does not survive re-binning. Across four binnings—5 equal-width, 10 equal-width, 20 equal-width, and 10 equal-count—the expected calibration error ranged from 0.0137 to 0.0533, and the reading shifted: well calibrated (5 bins), mildly miscalibrated (10 equal-width and 10 equal-count), and clearly miscalibrated (20 bins). Coarser bins (5) average opposite errors away and read kinder; finer bins (20) split into noise and read harsher. The direction of error—average prediction (34.9%) minus observed base rate (34.9%)—is binning-independent, but the size is not. Treat the magnitude of miscalibration as uncertain; the per-bin confidence intervals in the calibration_bins table provide a more stable picture than any single headline number.
What Correction Would Buy
Platt scaling and isotonic regression applied to the same predictions, reported in-sample.
| Method | Brier | Ece | Note |
|---|---|---|---|
| As delivered (no correction) | 0.1527 | 0.0237 | The predicted probabilities exactly as supplied. |
| Platt scaling (logistic recalibration) | 0.1527 | 0.0237 | The fitted intercept and slope applied back to the predicted logit. It can shift and stretch the probabilities but cannot change their ORDER, so it leaves discrimination untouched. Fitted on these same rows. |
| Isotonic regression (pool-adjacent-violators) | 0.1472 | 0 | A non-decreasing step function fitted by pooling adjacent violators, here 17 distinct steps. It is far more flexible than Platt scaling and is therefore the most optimistic in-sample. |
Platt scaling (logistic recalibration) applied to the predicted logits leaves the Brier score at 0.1527 and expected calibration error at 0.0237, no improvement. Isotonic regression (17 distinct steps) lowers the Brier to 0.1472, a 3.6% in-sample improvement, and achieves an expected calibration error of 0. However, both corrections were fitted on the same rows they are scored on, making these numbers in-sample and optimistic—an upper bound, not a realistic expectation. This optimism is most severe for isotonic regression, which has 17 free parameters and can bend to this data. Neither method can improve discrimination (AUC remains 0.839) because monotone corrections never reorder rows. Before trusting any correction, fit it on one data split and measure it on a held-out split.
Calibration Is Not Discrimination
The ranking question alongside the honesty question, and why they are not the same.
| Metric | Value | Interpretation |
|---|---|---|
| AUC (discrimination) | 0.839 | Probability that a random '1' row is scored above a random '0' row — good ranking. 0.500 means no ranking ability at all. |
| Observed base rate | 34.9% | The share of rows that were '1'. |
| Brier score | 0.15273 | The delivered probabilities scored against the outcomes. |
| Brier score of a base-rate-only forecast | 0.22719 | What you would score by predicting 34.9% for every single row. That forecast is perfectly calibrated and ranks nothing. |
| Brier skill score | 0.3278 | How much the delivered probabilities beat that base-rate-only forecast; 0 means no better, 1 would be perfect. |
Calibration and discrimination answer different questions. Calibration asks whether the probability is honest; discrimination asks whether the score ranks rows correctly. The AUC of 0.839 indicates good ranking—a random positive case is scored above a random negative case 83.9% of the time. A forecast that simply predicted the base rate (34.9%) for every row would be perfectly calibrated (all bins would sit on the diagonal) but rank nothing (AUC 0.500). The delivered probabilities score a Brier of 0.15273 versus 0.22719 for that base-rate-only forecast, a Brier skill score of 0.3278. That skill comes entirely from the resolution term (0.0741), the model's ability to separate cases; calibration alone cannot supply it. Fixing calibration moves numbers onto the diagonal but does not increase what the model knows.
Classifier Calibration — Are the Predicted Probabilities Honest?
Takes predicted probabilities plus the true binary outcome and answers the question "when this model says 70%, does it happen 70% of the time?".
Delivers the reliability curve (predicted vs observed rate per bin, against the perfect-calibration diagonal, with binomial confidence intervals per bin), the Brier score decomposed into reliability / resolution / uncertainty, the calibration intercept and slope from a logistic recalibration, expected and maximum calibration error, the histogram of predicted probabilities, a binning-sensitivity check, and the effect of two remedies — Platt scaling and isotonic regression.
Why This Method?
A single number cannot say whether probabilities are honest. The Brier score mixes three different things together; splitting it into reliability (how far the probabilities sit from what happened), resolution (how much they separate outcomes) and uncertainty (how hard the problem was) is what turns one number into a diagnosis. The reliability curve then shows exactly WHERE the model is wrong, and the calibration slope says whether the predictions are systematically too extreme or too timid.
What This Analysis Covers
- The reliability curve with binomial (Wilson) confidence intervals per bin
- The Brier score decomposed into reliability, resolution and uncertainty
- Calibration intercept and slope from a logistic recalibration
- Expected (ECE) and maximum (MCE) calibration error
- The distribution of predicted probabilities
- Whether the verdict survives a different binning
- Platt scaling and isotonic regression as remedies, reported in-sample
- Discrimination (AUC) alongside calibration, and why they are not the same
Standard Library
Platform standard-library module (LAT-1441): runs on ANY dataset via the semantic mapping {actual, score} — the same input shape as the ROC analysis. All narrative is derived from the user's own column names and computed values.
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Step 1: Row accounting + semantic column discovery
initial_rows <- nrow(df)
if (!"actual" %in% names(df)) {
stop("column_mapping must map an 'actual' column (the true yes/no outcome).")
}
if (!"score" %in% names(df)) {
stop("column_mapping must map a 'score' column (the predicted probability).")
}
actual_name <- humanize_semantic("actual", col_map)
score_name <- humanize_semantic("score", col_map)Step 2: Choose the positive class and binarize (no row dropping yet)
v_raw <- df$actual
if (is.logical(v_raw)) {
y_all <- as.integer(v_raw)
positive_label <- "TRUE"
negative_label <- "FALSE"
positive_rule <- "the boolean TRUE value was taken as the positive class"
} else {
vc <- trimws(as.character(v_raw))
vc[is.na(v_raw) | vc == "" | toupper(vc) %in% c("NA", "N/A", "NULL")] <- NA_character_
lv <- sort(unique(vc[!is.na(vc)]))
if (length(lv) < 2) {
stop(sprintf(
paste0("The outcome column('%s') holds only one distinct value (%s) — a ",
"calibration analysis needs rows where the event happened and rows ",
"where it did not."),
actual_name,
if (length(lv) == 1) paste0("'", lv[1], "'") else "none"))
}
pat <- paste0("^(1|yes|y|true|t|positive|pos|disease|churn|churned|fraud|",
"default|defaulted|converted|won|win|success|event|flagged)$")
hits <- lv[grepl(pat, tolower(lv))]
if (length(hits) >= 1) {
positive_label <- hits[length(hits)]
positive_rule <- paste0("'", positive_label,
"' matched a conventional positive label")
} else {
positive_label <- lv[length(lv)]
positive_rule <- paste0("no conventional positive label was found, so '",
positive_label,
"' (alphabetically last) was used as the positive class")
}
neg_levels <- setdiff(lv, positive_label)
negative_label <- if (length(neg_levels) == 1) neg_levels[1]
else paste0("not ", positive_label)
if (length(lv) > 2) {
positive_rule <- paste0(positive_rule,
"; the outcome had more than two distinct values, so this is a one-against-the-rest evaluation")
}
y_all <- ifelse(is.na(vc), NA_integer_, as.integer(vc == positive_label))
}Step 3: Coerce the score to numeric under the 95% rule
sc_raw <- df$score
if (is.numeric(sc_raw)) {
sc_all <- as.numeric(sc_raw)
} else {
chr <- trimws(as.character(sc_raw))
chr[chr == ""] <- NA_character_
n_nonblank <- sum(!is.na(chr))
sc_all <- suppressWarnings(as.numeric(chr))
n_ok <- sum(!is.na(sc_all))
if (n_nonblank == 0 || n_ok < 0.95 * n_nonblank) {
stop(sprintf(
paste0("The predicted-probability column('%s') is not numeric — only %s of %s ",
"non-blank values could be read as numbers, and at least 95%% is required. ",
"Calibration needs a predicted probability per row, not a text label."),
score_name, format(n_ok, big.mark = ","), format(n_nonblank, big.mark = ",")))
}
}Step 4: Keep only complete (outcome, probability) pairs
keep <- !is.na(sc_all) & !is.na(y_all)
p <- as.numeric(sc_all[keep])
y <- as.integer(y_all[keep])
final_rows <- length(y)
rows_removed <- initial_rows - final_rows
if (final_rows < 30) {
stop(sprintf(
paste0("Only %s rows have both a usable outcome('%s') and a numeric predicted ",
"probability('%s') — a calibration analysis needs at least 30."),
format(final_rows, big.mark = ","), actual_name, score_name))
}Step 5: Decide what scale the predictions are on
rng_lo <- min(p); rng_hi <- max(p)
if (rng_lo >= 0 && rng_hi <= 1) {
scale_rule <- paste0("the values ran from ", fnum(rng_lo, 3), " to ",
fnum(rng_hi, 3), ", so they were read as probabilities as given")
} else if (rng_lo >= 0 && rng_hi <= 100) {
scale_rule <- paste0("the values ran from ", fnum(rng_lo, 2), " to ",
fnum(rng_hi, 2), ", so they were read as percentages and divided by 100")
p <- p / 100
} else {
stop(sprintf(
paste0("The column mapped as the predicted probability('%s') ranges from %s to %s, ",
"which is neither a probability(0 to 1) nor a percentage(0 to 100). ",
"Calibration asks whether predicted PROBABILITIES are honest, so a raw ",
"score or a logit has no calibration to check. To measure how well such a ",
"score ranks the two classes of '%s', use the ROC analysis instead."),
score_name, fnum(rng_lo, 3), fnum(rng_hi, 3), actual_name))
}Step 6: Class-balance guard, named in the user's own columns
n1 <- sum(y == 1L); n0 <- sum(y == 0L)
if (n1 < 5 || n0 < 5) {
stop(sprintf(
paste0("A calibration analysis needs at least 5 rows of each outcome, but among the ",
"usable rows '%s' has %s '%s' and %s '%s'."),
actual_name, format(n1, big.mark = ","), positive_label,
format(n0, big.mark = ","), negative_label))
}
base_rate <- mean(y)
mean_pred <- mean(p)
cal_large <- mean_pred - base_rateStep 7: Brier score and its Murphy decomposition over the main bins
brier <- mean((p - y)^2)
unc <- base_rate * (1 - base_rate)
bss <- if (unc > 0) 1 - brier / unc else NA_real_Bin count scales with the data: never fewer than 4, never more than 10, and never more than one bin per 10 rows.
k_main <- max(4L, min(10L, as.integer(floor(final_rows / 10))))
bins_main <- make_bins(p, y, seq(0, 1, length.out = k_main + 1L))
reliable_min <- max(10L, as.integer(ceiling(0.01 * final_rows)))
em <- bin_errors(bins_main, final_rows, reliable_min)
ece <- em$ece; mce_all <- em$mce_all; mce_reliable <- em$mce_reliable
n_bins_used <- em$n_bins; thin_bins <- em$thin_bins
worst_row <- em$worst_row
rel <- sum(bins_main$n * (bins_main$mean_predicted - bins_main$observed_rate)^2,
na.rm = TRUE) / final_rows
res <- sum(bins_main$n * (bins_main$observed_rate - base_rate)^2,
na.rm = TRUE) / final_rows
resid <- brier - (rel - res + unc)Step 8: Logistic recalibration — calibration intercept and slope
The predictions are clamped away from 0 and 1 so the logit is finite.
eps <- min(0.5 / final_rows, 1e-6)
pc <- pmin(pmax(p, eps), 1 - eps)
n_clamped <- sum(p <= eps | p >= 1 - eps)
lp <- log(pc / (1 - pc))
lp_var <- stats::var(lp)
slope_estimable <- is.finite(lp_var) && lp_var > 0
cal_int <- NA_real_; cal_int_lo <- NA_real_; cal_int_hi <- NA_real_
cal_slope <- NA_real_; cal_slope_lo <- NA_real_; cal_slope_hi <- NA_real_
citl <- NA_real_; citl_lo <- NA_real_; citl_hi <- NA_real_
slope_note <- ""
p_platt <- rep(base_rate, final_rows)
citl_fit <- suppressWarnings(try(
stats::glm(y ~ 1, family = stats::binomial(), offset = lp), silent = TRUE))
if (!inherits(citl_fit, "try-error")) {
cf <- summary(citl_fit)$coefficients
citl <- cf[1, 1]
citl_lo <- cf[1, 1] - 1.959964 * cf[1, 2]
citl_hi <- cf[1, 1] + 1.959964 * cf[1, 2]
}
if (slope_estimable) {
fit <- suppressWarnings(try(
stats::glm(y ~ lp, family = stats::binomial()), silent = TRUE))
if (!inherits(fit, "try-error") && nrow(summary(fit)$coefficients) >= 2) {
cf <- summary(fit)$coefficients
cal_int <- cf[1, 1]
cal_int_lo <- cf[1, 1] - 1.959964 * cf[1, 2]
cal_int_hi <- cf[1, 1] + 1.959964 * cf[1, 2]
cal_slope <- cf[2, 1]
cal_slope_lo <- cf[2, 1] - 1.959964 * cf[2, 2]
cal_slope_hi <- cf[2, 1] + 1.959964 * cf[2, 2]
p_platt <- as.numeric(stats::predict(fit, type = "response"))
} else {
slope_note <- paste0("the logistic recalibration did not converge, so the ",
"calibration slope is not reported")
}
} else {
slope_note <- paste0("every row carries the same predicted probability, so the ",
"calibration slope is not estimable — there is no spread to ",
"stretch or shrink")
}
slope_reading <- if (is.na(cal_slope)) {
slope_note
} else if (cal_slope_hi < 1) {
paste0("the slope of ", fnum(cal_slope, 3), " sits entirely below 1, so the ",
"predictions are too extreme(over-confident): high predictions run above ",
"what actually happens and low predictions run below it")
} else if (cal_slope_lo > 1) {
paste0("the slope of ", fnum(cal_slope, 3), " sits entirely above 1, so the ",
"predictions are too timid(under-confident): they are squeezed toward the ",
"middle relative to what actually happens")
} else {
paste0("the slope of ", fnum(cal_slope, 3), " has a 95% interval that includes 1, ",
"so the spread of the predictions is consistent with being about right")
}Step 9: Isotonic regression by pool-adjacent-violators, implemented here
ux <- sort(unique(p))
gi <- match(p, ux)
g_n <- integer(length(ux)); g_s <- numeric(length(ux))
for (i in seq_along(gi)) {
g <- gi[i]
g_n[g] <- g_n[g] + 1L
g_s[g] <- g_s[g] + y[i]
}
iso_group <- pava_fit(g_s / g_n, as.numeric(g_n))
p_iso <- iso_group[gi]
iso_steps <- length(unique(round(iso_group, 12)))
platt_brier <- mean((p_platt - y)^2)
iso_brier <- mean((p_iso - y)^2)
platt_bins <- make_bins(p_platt, y, seq(0, 1, length.out = k_main + 1L))
iso_bins <- make_bins(p_iso, y, seq(0, 1, length.out = k_main + 1L))
platt_ece <- bin_errors(platt_bins, final_rows, reliable_min)$ece
iso_ece <- bin_errors(iso_bins, final_rows, reliable_min)$eceStep 10: Discrimination — rank AUC (ties get average ranks)
rk <- rank(p)
auc <- (sum(rk[y == 1L]) - as.numeric(n1) * (n1 + 1) / 2) /
(as.numeric(n1) * as.numeric(n0))
auc_band_txt <- auc_band(auc)Step 11: Binning sensitivity — does the verdict survive re-binning?
k_half <- max(3L, as.integer(k_main %/% 2L))
k_dbl <- min(20L, as.integer(k_main * 2L))
ew_ks <- sort(unique(c(k_half, k_main, k_dbl)))
binning_rows <- list()
for (kk in ew_ks) {
bb <- make_bins(p, y, seq(0, 1, length.out = kk + 1L))
ee <- bin_errors(bb, final_rows, reliable_min)
binning_rows[[length(binning_rows) + 1L]] <- data.frame(
binning = paste0(kk, " equal-width bins"),
n_bins = ee$n_bins,
ece = round(ee$ece, 4),
mce = round(ee$mce_reliable, 4),
verdict = ece_band(ee$ece),
stringsAsFactors = FALSE
)
}
q_breaks <- unique(as.numeric(stats::quantile(
p, probs = seq(0, 1, length.out = k_main + 1L), type = 7, na.rm = TRUE)))
q_breaks[1] <- min(q_breaks[1], 0); q_breaks[length(q_breaks)] <- max(q_breaks[length(q_breaks)], 1)
qb <- make_bins(p, y, q_breaks)
qe <- bin_errors(qb, final_rows, reliable_min)
binning_rows[[length(binning_rows) + 1L]] <- data.frame(
binning = paste0(max(1L, length(q_breaks) - 1L), " equal-count bins"),
n_bins = qe$n_bins,
ece = round(qe$ece, 4),
mce = round(qe$mce_reliable, 4),
verdict = ece_band(qe$ece),
stringsAsFactors = FALSE
)
binning_df <- do.call(rbind, binning_rows)
rownames(binning_df) <- NULL
verdicts <- unique(binning_df$verdict)
binning_agrees <- length(verdicts) == 1L
ece_spread <- max(binning_df$ece, na.rm = TRUE) - min(binning_df$ece, na.rm = TRUE)Distribution of the predicted probabilities — where the model actually lives.
hist_k <- min(20L, max(5L, as.integer(floor(final_rows / 5))))
hb <- make_bins(p, y, seq(0, 1, length.out = hist_k + 1L))
hist_df <- data.frame(
bucket = paste0(fnum(hb$bin_lo, 2), " to ", fnum(hb$bin_hi, 2)),
count = hb$n,
share = round(hb$n / final_rows, 4),
stringsAsFactors = FALSE
)
h_ok <- hb$n
busiest_i <- which.max(h_ok)
busiest_bucket <- hist_df$bucket[busiest_i]
busiest_share <- hist_df$share[busiest_i]
mid_share <- sum(hb$n[hb$bin_lo >= 0.2 & hb$bin_hi <= 0.8]) / final_rows
edge_share <- sum(hb$n[hb$bin_hi <= 0.1 | hb$bin_lo >= 0.9]) / final_rows
remedy_df <- data.frame(
method = c("As delivered(no correction)",
"Platt scaling(logistic recalibration)",
"Isotonic regression(pool-adjacent-violators)"),
brier = round(c(brier, platt_brier, iso_brier), 5),
ece = round(c(ece, platt_ece, iso_ece), 4),
note = c(
"The predicted probabilities exactly as supplied.",
paste0("The fitted intercept and slope applied back to the predicted logit. It can ",
"shift and stretch the probabilities but cannot change their ORDER, so it ",
"leaves discrimination untouched. Fitted on these same rows."),
paste0("A non-decreasing step function fitted by pooling adjacent violators, here ",
format(iso_steps, big.mark = ","), " distinct steps. It is far more flexible ",
"than Platt scaling and is therefore the most optimistic in-sample.")),
stringsAsFactors = FALSE
)
discrim_df <- data.frame(
metric = c("AUC(discrimination)", "Observed base rate", "Brier score",
"Brier score of a base-rate-only forecast", "Brier skill score"),
value = c(fnum(auc, 3), fpct(base_rate), fnum(brier, 5), fnum(unc, 5),
fnum(bss, 4)),
interpretation = c(
paste0("Probability that a random '", positive_label, "' row is scored above a random '",
negative_label, "' row — ", auc_band_txt,
" ranking. 0.500 means no ranking ability at all."),
paste0("The share of rows that were '", positive_label, "'."),
"The delivered probabilities scored against the outcomes.",
paste0("What you would score by predicting ", fpct(base_rate),
" for every single row. That forecast is perfectly calibrated and ranks nothing."),
paste0("How much the delivered probabilities beat that base-rate-only forecast; ",
"0 means no better, 1 would be perfect.")),
stringsAsFactors = FALSE
)Step 14: KPI metrics + machine channels
metrics <- list(
`Observations` = final_rows,
`Calibration` = cal_verdict,
`ECE` = round(ece, 4),
`Calibration Slope` = if (is.na(cal_slope)) "not estimable" else round(cal_slope, 3),
`Brier Score` = round(brier, 4),
`AUC(Discrimination)`= round(auc, 3)
)
calibration_summary <- list(
n = final_rows, base_rate = base_rate, mean_pred = mean_pred,
cal_large = cal_large, brier = brier, reliability = rel, resolution = res,
uncertainty = unc, residual = resid, bss = bss,
ece = ece, mce_all = mce_all, mce_reliable = mce_reliable,
n_bins = n_bins_used, k_main = k_main, reliable_min = reliable_min,
thin_bins = thin_bins,
cal_int = cal_int, cal_int_lo = cal_int_lo, cal_int_hi = cal_int_hi,
cal_slope = cal_slope, cal_slope_lo = cal_slope_lo, cal_slope_hi = cal_slope_hi,
citl = citl, slope_estimable = slope_estimable,
auc = auc, auc_band = auc_band_txt,
platt_brier = platt_brier, iso_brier = iso_brier,
platt_ece = platt_ece, iso_ece = iso_ece, iso_steps = iso_steps,
n_clamped = n_clamped, binning_agrees = binning_agrees,
ece_spread = ece_spread, verdict = cal_verdict,
positive_label = positive_label, negative_label = negative_label
)
json_output <- list(
answer = paste0(
"Calibration of ", score_name, " against ", actual_name, " (positive = '",
positive_label, "') across ", format(final_rows, big.mark = ","),
" rows: the predictions are ", cal_verdict, ", with an expected calibration error of ",
fnum(ece, 4), " over ", n_bins_used, " bins. ", direction, ". ",
"The Brier score of ", fnum(brier, 4), " splits into reliability ", fnum(rel, 4),
", resolution ", fnum(res, 4), " and uncertainty ", fnum(unc, 4),
" — the reliability part is what a recalibration could remove. ",
if (is.na(cal_slope))
paste0("The calibration slope is not estimable because ", slope_note, ". ")
else
paste0("The logistic recalibration gives an intercept of ", fnum(cal_int, 3),
" and a slope of ", fnum(cal_slope, 3), " (1.000 is ideal): ",
slope_reading, ". "),
binning_sentence, " ", discrim_sentence
),
cards = lapply(
c("tldr", "overview", "preprocessing", "reliability_curve", "calibration_bins",
"calibration_fit", "brier_decomposition", "score_histogram",
"binning_sensitivity", "recalibration", "discrimination"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows, final_rows = final_rows, rows_removed = rows_removed,
actual_name = actual_name, score_name = score_name,
positive_label = positive_label, negative_label = negative_label,
positive_rule = positive_rule, scale_rule = scale_rule,
n1 = n1, n0 = n0, base_rate = base_rate, mean_pred = mean_pred, cal_large = cal_large,
brier = brier, rel = rel, res = res, unc = unc, resid = resid, bss = bss,
ece = ece, mce_all = mce_all, mce_reliable = mce_reliable,
n_bins_used = n_bins_used, k_main = k_main, reliable_min = reliable_min,
thin_bins = thin_bins,
cal_int = cal_int, cal_slope = cal_slope, citl = citl,
cal_slope_lo = cal_slope_lo, cal_slope_hi = cal_slope_hi,
slope_estimable = slope_estimable, slope_note = slope_note,
slope_reading = slope_reading,
auc = auc, auc_band_txt = auc_band_txt,
platt_brier = platt_brier, iso_brier = iso_brier,
platt_ece = platt_ece, iso_ece = iso_ece, iso_steps = iso_steps,
n_clamped = n_clamped, binning_agrees = binning_agrees, ece_spread = ece_spread,
cal_verdict = cal_verdict, direction = direction,
binning_sentence = binning_sentence, thin_sentence = thin_sentence,
worst_sentence = worst_sentence, discrim_sentence = discrim_sentence,
insample_sentence = insample_sentence, clamp_sentence = clamp_sentence,
busiest_bucket = busiest_bucket, busiest_share = busiest_share,
mid_share = mid_share, edge_share = edge_share,
bins_main = bins_main,
reliability_df = reliability_df, bin_table_df = bin_table_df,
brier_df = brier_df, fit_df = fit_df, hist_df = hist_df,
binning_df = binning_df, remedy_df = remedy_df, discrim_df = discrim_df,
calibration_summary = calibration_summary, metrics = metrics,
json_output = json_output
)
}