Regression Methods: Complete Guide to Choosing the Right Model

Regression is the workhorse of predictive analytics. Whether you are forecasting revenue, estimating customer lifetime value, or identifying which features drive conversion, some form of regression is a useful starting point for many prediction questions, depending on the outcome and the shape of the relationship. The challenge is not whether to use regression -- it is which regression to use. Pick ordinary least squares when your features are not strongly correlated and you want simple, readable coefficients. When correlated features make the coefficients unstable, Ridge shrinks them toward each other. Pick Lasso when you want the model to drop weak features, and remember it may zero out variables you care about.

This guide maps regression methods to the problems they suit. Each section says when a method fits and links to its own guide. The example below compares OLS, Ridge, and Lasso on made-up data. If you already know your problem type, jump to the comparison table. If you are starting from scratch, follow the decision flowchart at the bottom.

Find the method for your question

Start with the question you are asking. Each guide says when its method does not fit.

Worked Example: OLS, Ridge and Lasso When Two Inputs Move Together

This example shows OLS, Ridge and Lasso only, and only on made-up data. It says nothing about logistic, polynomial, quantile or negative binomial regression, and it does not rank the methods on real data. Each method has its own guide: linear regression (OLS), Ridge and Lasso. Ridge and Lasso were introduced by Hoerl and Kennard (1970) and Tibshirani (1996); see Sources below.

Question: When two inputs track each other, how do OLS, Ridge and Lasso compare on test error and on the weights they give each input? Test RMSE is the usual size of a guess error on rows the model did not see. Lower is better.

Inputs: Simulated data, seed 42, repeated 500 times. Each repeat makes 200 rows with two inputs, x1 and x2, where x2 closely tracks x1. The outcome is y = 3 * x1 + 3 * x2 plus random noise with SD 0.5. Each repeat fits on 160 rows and scores on 40 rows.

Method: Fit OLS, Ridge with alpha 1 and Lasso with alpha 0.1 on the same rows. Record the test RMSE, the x1 weight, and the sum of the two weights.

import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso

SEED = 42
runs = 500
n = 200
rng = np.random.default_rng(SEED)
models = {
    "OLS": lambda: LinearRegression(),
    "Ridge (alpha=1)": lambda: Ridge(alpha=1.0),
    "Lasso (alpha=0.1)": lambda: Lasso(alpha=0.1, max_iter=10000),
}
rmse = {k: [] for k in models}
coef1 = {k: [] for k in models}
coef_sum = {k: [] for k in models}
for _ in range(runs):
    x1 = rng.normal(size=n)
    x2 = 0.95 * x1 + rng.normal(scale=0.15, size=n)
    y = 3.0 * x1 + 3.0 * x2 + rng.normal(scale=0.5, size=n)
    X = np.column_stack([x1, x2])
    for name, make in models.items():
        m = make().fit(X[:160], y[:160])
        err = y[160:] - m.predict(X[160:])
        rmse[name].append(np.sqrt(np.mean(err ** 2)))
        coef1[name].append(m.coef_[0])
        coef_sum[name].append(m.coef_.sum())

print(f"Simulated data, seed {SEED}; {runs} repeats of: 200 rows, "
      "x2 closely tracks x1, y = 3*x1 + 3*x2 + noise (SD 0.5)")
print("Each repeat fits on 160 rows and scores on 40 rows.")
print("Model | mean test RMSE | SD of x1 weight across repeats | "
      "mean of (x1 + x2) weights")
for name in models:
    print(f"{name} | {np.mean(rmse[name]):.3f} | "
          f"{np.std(coef1[name]):.3f} | {np.mean(coef_sum[name]):.3f}")
print("True weights: x1 = 3, x2 = 3, sum = 6")

Simulated data, seed 42; 500 repeats of: 200 rows, x2 closely tracks x1, y = 3*x1 + 3*x2 + noise (SD 0.5)
Each repeat fits on 160 rows and scores on 40 rows.
Model | mean test RMSE | SD of x1 weight across repeats | mean of (x1 + x2) weights
OLS | 0.501 | 0.256 | 6.001
Ridge (alpha=1) | 0.501 | 0.166 | 5.981
Lasso (alpha=0.1) | 0.514 | 0.263 | 5.887
True weights: x1 = 3, x2 = 3, sum = 6

Result: All three models predict about equally well: mean test RMSE 0.501 for OLS, 0.501 for Ridge and 0.514 for Lasso, against a built-in noise SD of 0.5. OLS and Ridge are tied; Lasso is a little higher. The sum of the two weights stays near the built-in 6 (6.001, 5.981, 5.887). What changes is the split between the two weights: the x1 weight swings by an SD of 0.26 across repeats for OLS, 0.17 for Ridge and 0.26 for Lasso. Lasso's sum sits below 6 because its penalty shrinks the weights.

Limit: The inputs were built to track each other and both true weights were set at 3, so this pattern is by construction. It shows what correlated inputs do to the weights. It does not show which method is better on real data.

Quick Comparison

When to Use Each Method

Decision Flowchart

Related Topics

Regression does not exist in isolation. These techniques complement regression modeling at different stages of the pipeline.

Regression in Business Applications

Regression is the mathematical engine behind marketing ROI measurement, budget allocation, and media mix modeling. See how these methods apply to real business problems: