Can I trust these predictions?

Before building a churn model, what score does it have to beat?

The data. standard_classification_v2.csv, the published sample file for Classification: 600 rows; columns customer_id, region, plan, tenure_months, support_tickets, monthly_charge, churned. Sample data, not a customer's.

The calculation. Python, standard library only. This exact code ran against the file when this page was built, and printed the output below it.

import csv
rows = list(csv.DictReader(open("standard_classification_v2.csv")))
y = [int(r["churned"]) for r in rows]
p = sum(y) / len(y)
print(len(y), "customers, churn rate", f"{p:.1%}")
print("accuracy of always guessing the larger class:",
      f"{max(p, 1 - p):.1%}")
print("Brier score of always predicting the rate:",
      round(sum((p - v) ** 2 for v in y) / len(y), 3))
plan = {}
for r in rows:
    plan.setdefault(r["plan"], []).append(int(r["churned"]))
for k, v in sorted(plan.items()):
    print(k, len(v), "customers, churn", f"{sum(v) / len(v):.1%}")

600 customers, churn rate 44.8%
accuracy of always guessing the larger class: 55.2%
Brier score of always predicting the rate: 0.247
basic 292 customers, churn 57.2%
enterprise 102 customers, churn 26.5%
pro 206 customers, churn 36.4%

What it says. A model that never looks at the data is right 55.2% of the time by always guessing the larger class, and scores a Brier score of 0.247 by predicting the overall rate for everyone. A churn model is only useful if it beats both on data it was not trained on. Plan alone already separates customers: churn runs from 26.5% to 57.2% across plans.

What it does not say. These are in-sample rates on one file. A fair comparison scores the model on rows it was not trained on; the library analysis holds out part of the data for that, and reports calibration as well as accuracy.

Run Classification on this file or your own, or open its stored sample report.