Measurement System Analysis

Weighted κ (ordinal, linear/quadratic)

weighted-kappa — agreement with distance-weighted disagreements, for ordinal scales (e.g. grades 1–5).

Statusvalidated

Version1.1.0

Minitab equivalentStat > Quality Tools > Attribute Agreement Analysis (ordinal, weighted kappa)

Description

Weighted κ generalises Cohen κ by scoring disagreements not binary (0/1) but by their distance on the ordinal scale. Small discrepancies (grade 3 vs. 4) count less than large ones (grade 1 vs. 5). Two standard variants: linear (w_ij = 1 − |i−j|/(k−1)) and quadratic (w_ij = 1 − ((i−j)/(k−1))²). The quadratic form per Cicchetti & Allison (1971) is the de-facto standard and matches Minitab. Without weighting weighted κ collapses to cohen-kappa.

Formulas

W (linear)

w_{ij} = 1 - \frac{∣ i - j ∣}{k - 1}

Linear weight matrix; k = number of ordinal levels

W (quadratisch)

w_{ij} = 1 - (\frac{i - j}{k - 1})^{2}

Quadratic weight matrix (Cicchetti-Allison standard)

p_o (weighted)

p_{o} = i, j \sum w_{ij} \cdot p_{ij}

Weighted observed agreement (p_ij = relative cell frequency)

p_e (weighted)

p_{e} = i, j \sum w_{ij} \cdot p_{i \cdot} p_{\cdot j}

Weighted chance agreement

κ_w

κ_{w} = \frac{p _{o} - p _{e}}{1 - p _{e}}

Weighted kappa coefficient

SE₀(κ)

SE_{0}^{2} = \frac{\sum _{i, j} p _{i \cdot} p _{\cdot j} [ w _{ij} - ( w ˉ _{i \cdot} + w ˉ _{\cdot j} ) ] ^{2} - p _{e}^{2}}{N ( 1 - p _{e} ) ^{2}}

Standard error under H₀ κ = 0 per Fleiss/Cohen/Everitt (1969); w̄_i· = Σ_j p_·j w_ij, w̄_·j = Σ_i p_i· w_ij, unweighted w = identity. Used for the test only, not for the CI

z, p

z = \frac{κ}{SE _{0}}, p = 1 - Φ (z)

One-sided test of H₀ κ = 0 against κ > 0 (as in Minitab)

Assumptions

Ordinal scale with a defined ordering of the levels
Two raters classify the same subjects
Weighting scheme is a deliberate choice (linear = mild, quadratic = strict penalty for large disagreements)

Limitations

Not meaningful for purely nominal scales — use cohen-kappa or fleiss-kappa instead
The choice of linear vs. quadratic changes κ noticeably; use the same variant for comparability

References

Cohen, J. (1968) — Weighted kappa: Nominal scale agreement with provision for scaled disagreement, Psychological Bulletin 70(4)
Cicchetti, D. V. & Allison, T. (1971) — A new procedure for assessing reliability of scoring EEG sleep recordings, American Journal of EEG Technology 11(3)
Fleiss, J. L., Cohen, J. & Everitt, B. S. (1969) — Large sample standard errors of kappa and weighted kappa, Psychological Bulletin 72(5)