Executive Summary
Model fit and top drivers of SalePrice across 2,930 observations
The short answer
Overall Qual is the strongest driver of house prices. The model explains 79.4% of SalePrice variation across 2,930 homes, with typical errors of 36,232 dollars. All 8 drivers are statistically significant.
The detail
Linear regression on 2,930 observations explains 79.4% of the variation in SalePrice (adjusted R² = 0.794). Overall Qual is the strongest driver (p = 1.07e-163), followed by Gr Liv Area. Typical prediction error (RMSE) is 36,232.11 in the same units as SalePrice. All 8 driver terms are statistically significant at p < 0.05.
What this can't tell you
Strong overall fit does not guarantee accurate predictions for individual homes. Residual diagnostics show heteroscedasticity—prediction uncertainty widens at higher prices. The model's strength in the central price range (100k–200k, 62.2% of data) does not extend equally to the extremes.
Analysis Overview
Ordinary least squares regression of SalePrice on 8 driver column(s) across 2,930 observations.
The short answer
This analysis uses linear regression to explain house sale prices from 8 structural and property features. The model accounts for 79.4% of the variation in SalePrice, with typical prediction errors of about 36,232 dollars. Each coefficient tells you the expected price change per one-unit increase in that feature, holding all others constant.
The detail
Ordinary least squares regression fitted 2,930 observations of SalePrice on 8 drivers: Gr Liv Area, Overall Qual, Year Built, Total Bsmt SF, Garage Area, Lot Area, Full Bath, and TotRms AbvGrd. The model achieves R² = 0.794 (adjusted 0.794) and RMSE = 36,232.11. All 2,930 rows retained—no missing SalePrice values. Numeric drivers imputed at the median; all mapped columns were usable. SalePrice ranges from 12,789 to 755,000 (median 160,000).
What this can't tell you
Residual spread grows at higher predicted values (heteroscedasticity), meaning the linear model is less precise at the top of the price range. A transformed outcome or additional drivers could tighten predictions for higher-priced homes. Consider whether order-level or transaction-context data (market conditions, time on market) might explain some of the remaining scatter.
Data Quality
Missing value handling, driver typing, and exclusions.
The short answer
All 2,930 rows had usable sale prices and were retained. Missing numeric values were filled with column medians; text fields were grouped into categories. The dataset is complete and ready for modeling.
The detail
Initial load: 2,930 rows. Final rows: 2,930 (rows removed: 0). Every observation had a valid SalePrice. SalePrice range: 12,789 to 755,000 (median 160,000). Numeric drivers received median imputation for missing values; categorical drivers were encoded as factor levels with rare levels grouped into 'Other'. All 8 mapped driver columns were usable and included in the model.
What this can't tell you
Median imputation assumes missingness is random within each column. If missing values were systematically related to unobserved factors (e.g., incomplete appraisals for certain property types), the imputed values could bias coefficient estimates. A detailed missing-data report by driver would clarify whether imputation was appropriate.
Distribution of the Outcome
How SalePrice is distributed across the dataset.
The short answer
House prices cluster in the (100000,200000] range, which contains 1,821 observations (62.2% of the dataset). The full span runs from [0,100000] to (700000,800000], with only 2 houses above 700,000. This concentration anchors the regression: coefficients describe movement within the observed range.
The detail
The 2,930 SalePrice values distribute as follows: [0,100000] (252 obs), (100000,200000] (1,821 obs), (200000,300000] (627 obs), (300000,400000] (166 obs), (400000,500000] (47 obs), (500000,600000] (11 obs), (600000,700000] (4 obs), (700000,800000] (2 obs). The modal range (100000,200000] holds 62.2% of all observations, making it the domain where the model is most densely informed.
What this can't tell you
The heavily right-skewed distribution with 62.2% of data in one bin suggests the model's coefficients are most reliable for the typical price range. Predictions outside (100000,200000] rest on extrapolation or sparse data. Consider whether this distribution matches your target market.
Coefficient Estimates
Effect of each driver on SalePrice with standard errors, 95% confidence intervals, and significance.
| Term | Estimate | Std Error | T Value | P Value | CI Low | CI High | Significance |
|---|---|---|---|---|---|---|---|
| Intercept | -8.194e+05 | 5.953e+04 | -13.77 | 8.08e-42 | -9.362e+05 | -7.027e+05 | *** |
| Gr Liv Area | 56.44 | 2.837 | 19.89 | 1.1e-82 | 50.88 | 62 | *** |
| Overall Qual | 2.189e+04 | 752.1 | 29.1 | 1.07e-163 | 2.041e+04 | 2.336e+04 | *** |
| Year Built | 377.7 | 31.25 | 12.09 | 7.33e-33 | 316.5 | 439 | *** |
| Total Bsmt SF | 26.97 | 1.953 | 13.81 | 4.4e-42 | 23.14 | 30.8 | *** |
| Garage Area | 44.25 | 4.131 | 10.71 | 2.71e-26 | 36.15 | 52.35 | *** |
| Lot Area | 0.7346 | 0.0915 | 8.028 | 1.42e-15 | 0.5552 | 0.9141 | *** |
| Full Bath | -5375 | 1743 | -3.084 | 0.0021 | -8793 | -1957 | ** |
| TotRms AbvGrd | -1673 | 742.2 | -2.254 | 0.0243 | -3128 | -217.6 | * |
The short answer
Overall Qual has the largest effect: each point increase in quality rating is associated with a 21,885 dollar increase in price. Gr Liv Area adds 56.44 dollars per square foot. All effects are statistically significant; confidence intervals exclude zero.
The detail
Coefficient estimates with 95% confidence intervals (all p < 0.05): Overall Qual: 21,885.12 [20,410.46 to 23,359.78], p = 1.07e-163; Gr Liv Area: 56.44 [50.88 to 62.00], p = 1.1e-82; Year Built: 377.74 [316.47 to 439.01], p = 7.33e-33; Total Bsmt SF: 26.97 [23.14 to 30.80], p = 4.4e-42; Garage Area: 44.25 [36.15 to 52.35], p = 2.71e-26; Lot Area: 0.73 [0.56 to 0.91], p = 1.42e-15; Full Bath: −5,375.31 [−8,793.18 to −1,957.44], p = 0.0021; TotRms AbvGrd: −1,672.97 [−3,128.35 to −217.59], p = 0.0243. All intervals exclude zero, confirming reliable effects.
What this can't tell you
Negative coefficients for Full Bath and TotRms AbvGrd are counterintuitive and suggest multicollinearity or omitted-variable bias. These features may proxy for other property characteristics not in the model. Consider whether these drivers interact with quality or size measures in ways the linear model does not capture.
Driver Ranking
Relative statistical influence of each driver, scaled 0-100.
The short answer
Overall Qual dominates, with an importance score of 100. Gr Liv Area (68.4) is the second-strongest driver. The remaining six drivers trail significantly, with Lot Area (27.6) in sixth place.
The detail
Driver importance ranked by t-statistic scaled 0–100: Overall Qual (100, positive, p = 1.07e-163); Gr Liv Area (68.4, positive, p = 1.1e-82); Total Bsmt SF (47.5, positive, p = 4.4e-42); Year Built (41.5, positive, p = 7.33e-33); Garage Area (36.8, positive, p = 2.71e-26); Lot Area (27.6, positive, p = 1.42e-15); Full Bath (10.6, negative, p = 0.0021); TotRms AbvGrd (7.7, negative, p = 0.0243). The top two drivers (Overall Qual and Gr Liv Area) account for the bulk of statistical influence; the bottom two are much weaker.
What this can't tell you
Importance rankings reflect statistical signal strength, not practical or market importance. A weak statistical signal does not mean a feature is irrelevant to pricing—it may reflect measurement error, omitted interactions, or confounding by unmeasured factors. Transaction-level context (e.g., days on market, seller motivation) could reveal why some structural features show weaker associations.
Top Driver Effect
Observed relationship between the strongest numeric driver and the outcome.
The short answer
SalePrice rises steeply with Overall Qual—a strong positive relationship (r = 0.78) across the plotted observations. Overall Qual ranges from 2 to 10 in the data. At quality 2, prices cluster around 55,000–60,000; at quality 3 and above, they spread upward, reflecting the regression's 21,885.1184 dollar-per-point coefficient.
The detail
The correlation is r = 0.78 across 1,000 plotted observations. Overall Qual spans 2 to 10. Sample points show: at quality 2, prices are 55,000, 60,000, 57,625; at quality 3, prices range from 51,689 to 130,000; at quality 4, prices range from 34,900 to 173,000. The regression coefficient (21,885.1184) describes the best-fit slope; residual scatter around this line reflects unmodelled factors and measurement noise.
What this can't tell you
The scatter plot shows association, not causation. High correlation does not imply that improving a house's quality rating will raise its price by the coefficient amount—other unmeasured factors (location, market timing, condition) drive both quality perception and price. The relationship is observational.
Predicted vs Actual
Model predictions against observed SalePrice.
The short answer
Model predictions track actual prices closely overall—most points lie near the diagonal. With R² = 0.794, the model captures the bulk of price variation. However, scatter increases at higher predicted values, indicating larger errors for expensive homes. Typical error is 36,232.11 dollars.
The detail
Predicted vs actual SalePrice shows the regression line's fit. Points on the diagonal represent perfect predictions. Examples: predicted 152,347.27, actual 165,000 (error +12,652.73); predicted 228,062.31, actual 200,000 (error −28,062.31); predicted 293,027.40, actual 383,970 (error +90,942.60). RMSE = 36,232.11. The pattern is tight at lower prices and loosens at higher prices, consistent with heteroscedasticity observed in residual diagnostics.
What this can't tell you
Heteroscedasticity means the model's confidence intervals are not uniformly reliable across the price range. Predictions for homes above 300,000 dollars carry larger typical errors than those in the modal range (100,000–200,000). The linear model may underestimate uncertainty for high-value properties.
Residual Diagnostics
Prediction errors across the fitted range — a check of the linear model's assumptions.
The short answer
Residuals (actual minus predicted price) center near −1,944.79 with spread 38,557.56. Error magnitude grows at higher predicted values—the model is less precise for expensive homes. This heteroscedasticity violates the linear model's constant-variance assumption, suggesting a transformed outcome or additional drivers could improve stability.
The detail
Residuals show 46.3% of predictions undershoot the actual value. Spread (standard deviation of residuals) is 38,557.56. At fitted values around 45,562, residuals range from −12,243 to +18,938; at fitted values around 323,726, residuals span −21,726 to +90,943. The widening spread at higher predictions indicates heteroscedasticity. Mean residual ≈ −1,944.79 suggests slight systematic underprediction, though this is small relative to RMSE.
What this can't tell you
Heteroscedasticity indicates the linear model's assumption of equal error variance is violated. Standard errors and confidence intervals computed under homoscedasticity may understate true uncertainty at high price points. A log-transformed outcome or weighted least squares could address this. The presence of outliers (e.g., residual +90,942.60) warrants investigation of high-value properties.
Linear Regression — Outcome Drivers
Fits an ordinary least squares regression to explain a numeric outcome from one or more driver columns chosen by the user. Reports coefficient estimates with confidence intervals and significance, ranks drivers by standardized effect, and surfaces model fit and residual diagnostics.
Why This Method?
Linear regression is the workhorse of explanatory analytics: coefficients are directly interpretable ("one unit of X changes Y by b"), significance tests separate signal from noise, and diagnostics reveal when the linear story is incomplete. It is the right first model for almost any "what drives this number?" question.
What This Analysis Covers
- Outcome distribution
- Coefficient table with 95% confidence intervals and p-values
- Driver ranking by standardized effect size
- Effect of the strongest numeric driver
- Predicted vs actual and residual diagnostics
Standard Library
This is a platform standard-library module (LAT-1441): it runs on ANY dataset via the semantic mapping {outcome, driver_1..driver_N}. All narrative text is derived from the user's own column names and computed values — nothing dataset-specific is hardcoded.
suppressPackageStartupMessages(library(htmltools))
suppressPackageStartupMessages(library(jsonlite))
suppressPackageStartupMessages(library(plotly))
suppressPackageStartupMessages(library(DT))
suppressPackageStartupMessages(library(htmlwidgets))
suppressPackageStartupMessages(library(arrow))
suppressPackageStartupMessages(library(knitr))
suppressPackageStartupMessages(library(rmarkdown))
suppressPackageStartupMessages(library(dplyr))
suppressPackageStartupMessages(library(tidyr))
suppressPackageStartupMessages(library(ggplot2))
suppressPackageStartupMessages(library(stringr))
suppressPackageStartupMessages(library(lubridate))
suppressPackageStartupMessages(library(broom))
suppressPackageStartupMessages(library(Matrix))
suppressPackageStartupMessages(library(cluster))
suppressPackageStartupMessages(library(data.table))Step 1: Row accounting + semantic column discovery
initial_rows <- nrow(df)
if (!"outcome" %in% names(df)) {
stop("column_mapping must map an 'outcome' column (the numeric value to explain)")
}
driver_cols <- grep("^driver_[0-9]+$", names(df), value = TRUE)
driver_cols <- driver_cols[order(as.integer(sub("^driver_", "", driver_cols)))]
if (length(driver_cols) == 0) {
stop("column_mapping must map at least one driver column(driver_1)")
}
outcome_name <- humanize_semantic("outcome", col_map)
driver_names <- setNames(humanize_semantic(driver_cols, col_map), driver_cols)Step 2: Coerce outcome to numeric; drop rows with missing outcome
df$outcome <- suppressWarnings(as.numeric(df$outcome))
df <- df[!is.na(df$outcome), , drop = FALSE]
if (nrow(df) < 10) {
stop(sprintf(
"Only %d rows have a usable numeric value in the outcome column('%s'). At least 10 are required.",
nrow(df), outcome_name))
}Step 3: Type each driver — numeric if >=95% of values convert, else factor
dropped_drivers <- character(0)
for (dc in driver_cols) {
v <- df[[dc]]
if (!is.numeric(v)) {
conv <- suppressWarnings(as.numeric(as.character(v)))
n_orig <- sum(!is.na(v) & as.character(v) != "")
if (n_orig > 0 && sum(!is.na(conv)) >= 0.95 * n_orig) {
df[[dc]] <- conv
}
}
v <- df[[dc]]
if (is.numeric(v)) {Numeric: impute NA with median
med <- median(v, na.rm = TRUE)
if (is.na(med)) { dropped_drivers <- c(dropped_drivers, dc); next }
v[is.na(v)] <- med
df[[dc]] <- v
} else {Categorical: blank/NA -> "Missing"; lump beyond 12 levels into "Other"
v <- as.character(v)
v[is.na(v) | trimws(v) == ""] <- "Missing"
tab <- sort(table(v), decreasing = TRUE)
if (length(tab) > 12) {
keep <- names(tab)[1:12]
v[!(v %in% keep)] <- "Other"
}
if (length(unique(v)) > nrow(df) / 2) {Near-unique text column (an ID, not a driver) — exclude
dropped_drivers <- c(dropped_drivers, dc)
next
}
df[[dc]] <- factor(v)
}
}Step 4: Drop zero-variance drivers
for (dc in setdiff(driver_cols, dropped_drivers)) {
v <- df[[dc]]
zero_var <- if (is.numeric(v)) {
isTRUE(var(v, na.rm = TRUE) == 0) || is.na(var(v, na.rm = TRUE))
} else {
length(unique(v)) <= 1
}
if (zero_var) dropped_drivers <- c(dropped_drivers, dc)
}
model_drivers <- setdiff(driver_cols, dropped_drivers)
if (length(model_drivers) == 0) {
stop("No usable driver columns remained after cleaning(all were constant, empty, or identifier-like).")
}
df_clean <- df[, c(model_drivers, "outcome"), drop = FALSE]
final_rows <- nrow(df_clean)
rows_removed <- initial_rows - final_rowsStep 5: Guard — need clearly more rows than model terms
n_terms_est <- sum(sapply(model_drivers, function(dc) {
if (is.factor(df_clean[[dc]])) nlevels(df_clean[[dc]]) - 1 else 1
}))
while (n_terms_est >= final_rows - 5 && length(model_drivers) > 1) {
drop_dc <- model_drivers[length(model_drivers)]
dropped_drivers <- c(dropped_drivers, drop_dc)
model_drivers <- model_drivers[-length(model_drivers)]
df_clean <- df_clean[, c(model_drivers, "outcome"), drop = FALSE]
n_terms_est <- sum(sapply(model_drivers, function(dc) {
if (is.factor(df_clean[[dc]])) nlevels(df_clean[[dc]]) - 1 else 1
}))
}Step 6: Fit OLS
model <- lm(outcome ~ ., data = df_clean)
glance_df <- as.data.frame(broom::glance(model))Step 7: Coefficient table with 95% CIs and significance stars
tidy_df <- broom::tidy(model, conf.int = TRUE, conf.level = 0.95)
coefficients_df <- data.frame(
term = humanize_term(tidy_df$term, model_drivers, col_map),
estimate = round(tidy_df$estimate, 4),
std_error = round(tidy_df$std.error, 4),
t_value = round(tidy_df$statistic, 3),
p_value = signif(tidy_df$p.value, 3),
ci_low = round(tidy_df$conf.low, 4),
ci_high = round(tidy_df$conf.high, 4),
significance = ifelse(is.na(tidy_df$p.value), "redundant",
ifelse(tidy_df$p.value < 0.001, "***",
ifelse(tidy_df$p.value < 0.01, "**",
ifelse(tidy_df$p.value < 0.05, "*",
ifelse(tidy_df$p.value < 0.1, ".", ""))))),
stringsAsFactors = FALSE
)Step 8: Driver importance — max |t| across a driver's terms, 0-100 scale
Direction from the sign of the strongest term (numeric drivers only).
term_raw <- tidy_df$term[tidy_df$term != "(Intercept)"]
term_t <- tidy_df$statistic[tidy_df$term != "(Intercept)"]
term_est <- tidy_df$estimate[tidy_df$term != "(Intercept)"]
term_p <- tidy_df$p.value[tidy_df$term != "(Intercept)"]
imp_rows <- lapply(model_drivers, function(dc) {
idx <- which(startsWith(term_raw, dc))
# Aliased (perfectly collinear) drivers have NA statistics — no
# estimable independent effect; exclude from the ranking instead of
# crashing on which.max(all-NA) -> integer(0).
idx <- idx[!is.na(term_t[idx])]
if (length(idx) == 0) return(NULL)
best <- idx[which.max(abs(term_t[idx]))]
data.frame(
semantic = dc,
driver = driver_names[[dc]],
abs_t = abs(term_t[best]),
direction = if (is.numeric(df_clean[[dc]])) {
if (term_est[best] > 0) "positive" else "negative"
} else "categorical",
p_value = signif(term_p[best], 3),
stringsAsFactors = FALSE
)
})
importance_df <- do.call(rbind, Filter(Negate(is.null), imp_rows))
if (!is.null(importance_df) && nrow(importance_df) > 0) {
importance_df <- importance_df[order(-importance_df$abs_t), , drop = FALSE]
max_t <- max(importance_df$abs_t, na.rm = TRUE)
importance_df$importance_score <- if (is.finite(max_t) && max_t > 0) {
round(100 * importance_df$abs_t / max_t, 1)
} else 0
rownames(importance_df) <- NULL
} else {
importance_df <- data.frame(semantic = character(0), driver = character(0),
abs_t = numeric(0), direction = character(0),
p_value = numeric(0), importance_score = numeric(0),
stringsAsFactors = FALSE)
}
top_driver_name <- if (nrow(importance_df) > 0) importance_df$driver[1] else "Unknown"
top_driver_sem <- if (nrow(importance_df) > 0) importance_df$semantic[1] else NA_character_Step 9: Outcome distribution — 7 pretty bins
brks <- pretty(range(df_clean$outcome, na.rm = TRUE), n = 7)
bins <- cut(df_clean$outcome, breaks = brks, include.lowest = TRUE, dig.lab = 10)
dist_tbl <- table(bins)
outcome_distribution_df <- data.frame(
outcome_range = as.character(names(dist_tbl)),
count = as.integer(dist_tbl),
stringsAsFactors = FALSE
)Step 10: Top numeric driver effect scatter — <=1000 rows
numeric_drivers_ranked <- importance_df$semantic[
sapply(importance_df$semantic, function(s) is.numeric(df_clean[[s]]))]
top_numeric_sem <- if (length(numeric_drivers_ranked) > 0) numeric_drivers_ranked[1] else NA_character_
top_numeric_driver <- if (!is.na(top_numeric_sem)) driver_names[[top_numeric_sem]] else NA_character_
if (!is.na(top_numeric_sem)) {
set.seed(42)
idx <- if (nrow(df_clean) > 1000) sample(nrow(df_clean), 1000) else seq_len(nrow(df_clean))
top_driver_effect_df <- data.frame(
driver_value = df_clean[[top_numeric_sem]][idx],
outcome = df_clean$outcome[idx],
stringsAsFactors = FALSE
)
top_driver_effect_df <- top_driver_effect_df[order(top_driver_effect_df$driver_value), ]
rownames(top_driver_effect_df) <- NULL
} else {
top_driver_effect_df <- data.frame(driver_value = numeric(0), outcome = numeric(0),
stringsAsFactors = FALSE)
}Step 11: Predicted vs actual and residuals — <=1000 samples
fitted_vals <- fitted(model)
resid_vals <- residuals(model)
set.seed(42)
ridx <- if (length(fitted_vals) > 1000) sample(length(fitted_vals), 1000) else seq_along(fitted_vals)
predicted_actual_df <- data.frame(
predicted = round(fitted_vals[ridx], 4),
actual = round(df_clean$outcome[ridx], 4),
stringsAsFactors = FALSE
)
rownames(predicted_actual_df) <- NULL
residuals_df <- data.frame(
fitted = round(fitted_vals[ridx], 4),
residual = round(resid_vals[ridx], 4),
stringsAsFactors = FALSE
)
rownames(residuals_df) <- NULLStep 12: KPI metrics (user-facing keys)
r2 <- glance_df$r.squared[1]
ar2 <- glance_df$adj.r.squared[1]
rmse <- sqrt(mean(resid_vals^2, na.rm = TRUE))
n_sig <- sum(coefficients_df$p_value < 0.05 & coefficients_df$term != "Intercept", na.rm = TRUE)
metrics <- list(
`Observations` = final_rows,
`R Squared` = round(r2, 3),
`Adjusted R Squared` = round(ar2, 3),
`RMSE` = round(rmse, 3),
`Significant Drivers` = n_sig,
`Top Driver` = top_driver_name
)Step 13: json_output machine channel
json_output <- list(
answer = paste0(
"Linear regression of ", outcome_name, " on ",
length(model_drivers), " driver(s) across ",
format(final_rows, big.mark = ","), " rows: R² = ", round(r2, 3),
" (adjusted ", round(ar2, 3), "), RMSE = ", round(rmse, 3), ". ",
top_driver_name, " is the strongest driver(", n_sig,
" of ", nrow(coefficients_df) - 1, " terms significant at p<0.05)."
),
cards = lapply(
c("tldr", "overview", "preprocessing", "outcome_distribution",
"coefficients_table", "driver_importance", "top_driver_effect",
"predicted_vs_actual", "residual_diagnostics"),
function(cid) list(id = cid, metrics = metrics)
)
)
list(
initial_rows = initial_rows,
final_rows = final_rows,
rows_removed = rows_removed,
outcome_name = outcome_name,
driver_names = driver_names,
model_drivers = model_drivers,
dropped_drivers = dropped_drivers,
df_clean = df_clean,
model = model,
glance_df = glance_df,
coefficients_df = coefficients_df,
importance_df = importance_df,
top_driver_name = top_driver_name,
top_driver_sem = top_driver_sem,
outcome_distribution_df = outcome_distribution_df,
top_driver_effect_df = top_driver_effect_df,
top_numeric_driver = top_numeric_driver,
predicted_actual_df = predicted_actual_df,
residuals_df = residuals_df,
rmse = rmse,
metrics = metrics,
json_output = json_output
)
}Compute shared resources
shared <- compute_shared(df, params, col_map)