Fleiss κ

Kappa statistic for more than two raters on nominal or binary characteristics — overall agreement.

Also known as: Fleiss' kappa, κ (Fleiss), multirater kappa

Fleiss κ generalises Cohen κ to $\geq 3$ raters and yields one overall agreement value. For each part $i$ and class $j$ let $n_{ij}$ be the number of ratings of class $j$ .

P_{i} = \frac{\sum _{j} n _{ij}^{2} - n _{i \cdot}}{n _{i \cdot} ( n _{i \cdot} - 1 )}, κ = \frac{P ˉ - P _{e}}{1 - P _{e}}, P_{e} = j \sum \overset{p}{ˉ}_{j}^{2}

$P_{i}$ is the pairwise agreement within a part, $\overset{ˉ}{P}$ its mean. $\overset{p}{ˉ}_{j}$ are the class marginals. Under unbalanced rater counts the engine switches to the Randolph variant. $H_{0} : κ = 0$ is tested with the standard error under $H_{0}$ after Fleiss/Nee/Landis 1979 (Fleiss/Levin/Paik 2003, as in Minitab): $z = κ / SE_{0}$ , one-sided $p$ . Qprovement reports no confidence interval for Fleiss κ, and neither does Minitab.

Used in

MSA Type 5 (attribute)

In the Algorithm Lab

msa/fleiss-kappa

Sources

Fleiss, J.L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382.
Fleiss, J.L., Nee, J.C.M., & Landis, J.R. (1979). Large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86(5), 974–977.
Randolph, J.J. (2005). Free-marginal multirater kappa: An alternative to Fleiss' fixed-marginal multirater kappa. Joensuu Learning and Instruction Symposium.
Fleiss, J.L., Levin, B., & Paik, M.C. (2003). Statistical Methods for Rates and Proportions, 3rd ed. Wiley — Kap. 18.

Fleiss κ

See also

Used in

In the Algorithm Lab

Sources