Groups inside groups

Do the four raters agree on quality scores, or does it depend on who scored the sample?

The data. standard_icc_v2.csv, the published sample file for Intraclass correlation: 120 rows; columns Sample ID, Rater Name, Quality Score, Session. Sample data, not a customer's.

The calculation. Python, standard library only. This exact code ran against the file when this page was built, and printed the output below it.

import csv
rows = list(csv.DictReader(open("standard_icc_v2.csv")))
raters = sorted({r["Rater Name"] for r in rows})
by = {}
for r in rows:
    if r["Quality Score"]:  # first score per rater and sample
        by.setdefault(r["Sample ID"], {}).setdefault(
            r["Rater Name"], float(r["Quality Score"]))
full = [list(v.values()) for v in by.values() if len(v) == len(raters)]
n, k = len(full), len(raters)
grand = sum(map(sum, full)) / (n * k)
msb = k * sum((sum(s) / k - grand) ** 2 for s in full) / (n - 1)
msw = sum((x - sum(s) / k) ** 2
          for s in full for x in s) / (n * (k - 1))
icc = (msb - msw) / (msb + (k - 1) * msw)
print(n, "of", len(by), "samples scored by all", k, "raters")
print("one-way ICC(1):", round(icc, 3))

28 of 30 samples scored by all 4 raters
one-way ICC(1): 0.878

What it says. An intraclass correlation of 0.88 means about 88% of the spread in scores comes from real differences between samples, and the rest from raters disagreeing on the same sample. Samples sit in groups of ratings here, which is why the variance is split by level rather than treating every score as independent.

What it does not say. Only the 28 samples that all 4 raters scored are used; 2 were incomplete. A one-way ICC does not separate a rater who is consistently harsh from random disagreement; the library analysis reports the two-way forms as well.

Run Intraclass correlation on this file or your own, or open its stored sample report.