Measurement System Analysis

Cohen κ (two raters)

cohen-kappa — agreement between two raters for categorical data, corrected for chance.

Statusvalidated

Version1.1.0

Minitab equivalentStat > Quality Tools > Attribute Agreement Analysis (paired appraisers)

Description

Cohen κ compares the observed agreement p_o with the chance agreement p_e derived from the marginal distributions. κ = 1 → perfect agreement, κ = 0 → chance, κ < 0 → worse than chance. Cornerstone of all kappa KPIs in Qprovement; MSA Type 5 computes it pairwise for every rater pair and it is the base for fleiss-kappa and weighted-kappa. The Fleiss/Cohen/Everitt (1969) SE approximation feeds a symmetric Wald confidence interval; the SE under H₀ gives the one-sided z-test for κ > 0.

Formulas

p_o

p_{o} = \frac{1}{N} i \sum n_{ii}

Observed agreement: proportion of cases on the cross-tabulation diagonal

p_e

p_{e} = i \sum \frac{n _{i \cdot} n _{\cdot i}}{N ^{2}}

Chance agreement derived from the observed marginal distributions

κ = \frac{p _{o} - p _{e}}{1 - p _{e}}

Cohen kappa coefficient

SE(κ)

SE (κ) = \frac{p _{o} ( 1 - p _{o} )}{N ( 1 - p _{e} ) ^{2}}

Standard error per Fleiss/Cohen/Everitt (1969)

95 %-KI

κ \pm z_{1 - α /2} \cdot SE (κ)

Symmetric Wald confidence interval

SE₀(κ)

SE_{0}^{2} = \frac{\sum _{i, j} p _{i \cdot} p _{\cdot j} [ w _{ij} - ( w ˉ _{i \cdot} + w ˉ _{\cdot j} ) ] ^{2} - p _{e}^{2}}{N ( 1 - p _{e} ) ^{2}}

Standard error under H₀ κ = 0 per Fleiss/Cohen/Everitt (1969); w̄_i· = Σ_j p_·j w_ij, w̄_·j = Σ_i p_i· w_ij, unweighted w = identity. Used for the test only, not for the CI

z, p

z = \frac{κ}{SE _{0}}, p = 1 - Φ (z)

One-sided test of H₀ κ = 0 against κ > 0 (as in Minitab)

Assumptions

Two raters classify the same set of subjects on the same scale
Categorical ratings (binary or nominal); for ordinal scales weighted-kappa is more appropriate
Independent ratings (raters do not influence each other)

Limitations

κ is low when one class is rare, even under near-perfect agreement (kappa paradox)
The Wald CI relies on a large-sample approximation — CI edges are imprecise for very small N

References

Cohen, J. (1960) — A coefficient of agreement for nominal scales, Educational and Psychological Measurement 20(1)
Fleiss, J. L., Cohen, J. & Everitt, B. S. (1969) — Large sample standard errors of kappa and weighted kappa, Psychological Bulletin 72(5)
AIAG — Measurement Systems Analysis Reference Manual, 4th Edition (MSA-4), Kapitel III-C: Attribute Measurement Systems Study