Measurement System Analysis

Fleiss κ (≥ 2 raters)

fleiss-kappa — overall agreement of two or more raters for categorical data.

Statusvalidated

Version2.0.0

Minitab equivalentStat > Quality Tools > Attribute Agreement Analysis (multiple appraisers, kappa summary)

Description

Fleiss κ extends Cohen's idea to more than two raters and any number of categories. It averages the per-subject pairwise agreement P_i and normalises it against the chance agreement P_e derived from the class marginals. The classical formula (`fleiss-1971`) requires an equal number of ratings per subject; when rater counts differ across subjects Qprovement switches to the kappa variant per Randolph (free-marginal), for which no closed-form SE exists — SE₀, z and p are then NaN. The closed-form standard error of Fleiss, Nee & Landis (1979) holds under H₀ κ = 0 only; it therefore feeds the one-sided z-test and no confidence interval (Minitab reports none either).

Formulas

P_i

P_{i} = \frac{\sum _{j} n _{ij}^{2} - n _{i \cdot}}{n _{i \cdot} ( n _{i \cdot} - 1 )}

Observed pairwise agreement at subject i (n_ij = number of ratings in class j)

P̄

\overset{ˉ}{P} = \frac{1}{N} i = 1 \sum N P_{i}

Mean pairwise agreement across all subjects

P_e

P_{e} = j \sum \overset{p}{ˉ}_{j}^{2}, \overset{p}{ˉ}_{j} = \frac{\sum _{i} n _{ij}}{N \cdot n ˉ}

Chance agreement derived from the class marginals

κ = \frac{P ˉ - P _{e}}{1 - P _{e}}

Fleiss kappa coefficient

Randolph (unbalanced)

κ_{R} = \frac{K P ˉ - 1}{K - 1}

Free-marginal variant for unequal n_i·; K = number of classes

SE₀(κ)

SE_{0}^{2} = \frac{2}{N n ( n - 1 )} \cdot \frac{( \sum _{j} p ˉ _{j} q ˉ _{j} ) ^{2} - \sum _{j} p ˉ _{j} q ˉ _{j} ( q ˉ _{j} - p ˉ _{j} )}{( \sum _{j} p ˉ _{j} q ˉ _{j} ) ^{2}}, \overset{q}{ˉ}_{j} = 1 - \overset{p}{ˉ}_{j}

Standard error under H₀ κ = 0 per Fleiss, Nee & Landis (1979), n = ratings per subject; reduces to 2/(N n (n−1)) for binary data; used for the test only

z, p

z = \frac{κ}{SE _{0}}, p = 1 - Φ (z)

One-sided test of H₀ κ = 0 against κ > 0 (as in Minitab)

Assumptions

At least two independent raters classify the same subjects
Categorical ratings (binary or nominal); pairwise weighted-kappa is more informative for ordinal scales
For the standard error under H₀: equal rater count per subject

Limitations

For unequal n_i· (Randolph fallback) no test is defined — only the point estimate κ is available
Fleiss κ is unweighted by definition — for ordinal scales it masks small misclassifications

References

Fleiss, J. L. (1971) — Measuring nominal scale agreement among many raters, Psychological Bulletin 76(5)
Randolph, J. J. (2005) — Free-marginal multirater kappa, Joensuu Learning and Instruction Symposium
Fleiss, J. L., Nee, J. C. M., & Landis, J. R. (1979) — Large sample variance of kappa in the case of different sets of raters, Psychological Bulletin 86(5), 974–977
Fleiss, J. L., Levin, B. & Paik, M. C. (2003) — Statistical Methods for Rates and Proportions, 3rd Edition, Wiley (§18)