MSA Type 5 (attribute)
Attribute Measurement System Analysis — multiple appraisers, kappa-based agreement (Cohen, Fleiss, weighted)
Purpose of the Study
MSA Type 5 (Attribute Measurement System Analysis) evaluates inspection processes that do not produce continuous readings but a categorical decision — binary (OK/NOK), nominal (defect classes A/B/C without order) or ordinal (grades 1–5, ordered). The module uses kappa-based statistics to assess how reliably and reproducibly multiple appraisers assign the same category to the same parts.
Agreement: Fraction of ratings where two or more appraisers assign the same category. Raw agreement overstates reliability because it counts chance-level agreement — hence the kappa statistics correct for the chance component.
Repeatability: Ability of a single appraiser to assign the same part to the same category on repeated ratings. Computed as the fraction of (part × appraiser) combinations in which all replicates match exactly.
Effectiveness (with reference): Fraction of an appraiser’s individual ratings that agree with the reference. Requires traceable reference calls — without a reference column the consensus fallback kicks in and only the between-appraiser metrics remain fully informative.
Boundary: MSA Type 1/2/4 handle variable measurements (mm, N, °C) and yield Cg/Cgk, Gage R&R or linearity metrics. Type 5 in contrast evaluates attribute inspection processes — visual checks, classification, sorting — and delivers κ, effectiveness and — for binary characteristics with a reference — signal-detection metrics.
Data Requirements
- Long layout (default) with three required columns: Part ID, Appraiser ID, Rating. Each row is one individual rating.
- Optional reference column with the true category for each part. If missing, the consensus fallback kicks in.
- Optional replicate column. If missing, the engine auto-numbers replicates per (part × appraiser) starting at 1.
- At least 2 distinct parts, 2 distinct appraisers and 2 distinct rating classes present in the data.
- For repeatability: at least 2 replicates per (part × appraiser) — otherwise the engine emits W_LOW_REP_COUNT.
- AIAG recommendation for release studies: 50 parts · 3 appraisers · 3 replicates, with roughly 50 % borderline parts in the sample.
Long or wide data layout: Long layout: one rating per row (part, appraiser, rating, optionally replicate). Wide layout as in Minitab's "multiple columns": choose "Wide" under Data layout, tick the part column and all rating columns and enter the trials per appraiser. Columns count in worksheet order — all trials of appraiser 1 first, then appraiser 2 —, the appraiser name is the common start of the column names (A-1, A-2 → A). Example: "binary, wide data layout".
Characteristic type: Binary (2 classes, positive first) → Cohen κ + miss/false-alarm rate + signal detection. Nominal (≥ 3 classes, unordered) → Cohen κ + Fleiss κ. Ordinal (ordered grades) → additionally weighted κ (linear or quadratic) so that small misclassifications weigh less than large ones. For binary data the most frequent rating counts as OK (positive); if that is wrong, use "Swap positive" — otherwise miss and false-alarm rate are swapped.
Reference vs. consensus: Traceable reference calls (expert set, master sample) give the most reliable effectiveness. Without a reference the consensus fallback derives a majority vote per part; ties push the part into **meta.ambiguousParts and exclude it from all vs-reference metrics. κ between appraisers is not affected.
Formulas
The core of Type 5 are the kappa statistics. They measure the share of agreement that exceeds chance — from 1 (perfect) through 0 (chance level) to negative (systematic disagreement).
Cohen κ (two appraisers)
Observed agreement: / N — diagonal sum of the cross tab divided by the total number of rating pairs.
Chance agreement: ) / N² — expected agreement from the marginal distributions of both appraisers.
Cohen κ: . SE(κ) after Fleiss/Cohen/Everitt (1969); 95 % CI . The test against chance uses , computed under : κ = 0 (Fleiss/Cohen/Everitt 1969).
Fleiss κ (≥ 2 appraisers)
Fleiss κ: per part , = mean. = class marginal. with . Under unbalanced rater counts the engine switches to the Randolph variant and records this in the method field.
z and p per κ: , one-sided p value for : κ = 0. A small p only says the agreement is better than chance — not that it is good enough. Fleiss κ has no confidence interval: the closed-form variance holds only under and serves the test, not an interval (Fleiss, Levin & Paik 2003). Minitab reports none either. p below 0.001 shows as "< 0.001".
κ within appraiser: Fleiss κ per appraiser across his or her own trials: the trials act as raters. Measures how well an appraiser agrees with themselves — chance-corrected, unlike repeatability. Only parts with at least two trials count; unequal trial counts use the Randolph variant.
Weighted κ (ordinal)
Weighted κ: Weight matrix : linear or quadratic . Near misclassifications (grade 3 ⟷ 4) are penalised more mildly than distant ones (1 ⟷ 5). .
Effectiveness + Wilson score CI
Effectiveness: eff = #{(part, rep) with rating = reference} / #{(part, rep) : part ∉ ambiguous}. Fraction of individual ratings that agree with the reference — per appraiser.
Wilson score CI: For a proportion p̂ = x/n: (p̂ + z²/(2n) ± z·√(p̂(1−p̂)/n + z²/(4n²))) / (1 + z²/n). Better coverage than the Wald interval for small n and proportions near 0 or 1.
The significance level α affects the Cohen κ confidence intervals and the Wilson intervals. The p values of the κ tests are reported independently of α.
Miss rate, false-alarm rate, bias (binary with reference only)
Miss rate: miss = P(rating = positive | reference = negative). Escape — a truly NOK part was rated OK.
False-alarm rate: fa = P(rating = negative | reference = positive). False alarm — a truly OK part was rated NOK.
Bias rate: bias = miss − fa. Greater than 0: the appraiser is too lenient; less than 0: too strict.
Signal detection (binary with reference only)
d′ (sensitivity): d′ = z(hit) − z(fa) with hit = P(rating=pos | reference=pos) and fa as above. Discrimination between OK and NOK; higher = better.
Criterion c (decision threshold): c = −½ · (z(hit) + z(fa)). c > 0: appraiser is conservative (tends to "NOK"), c < 0: liberal (tends to "OK"). c ≈ 0: neutral threshold.
Hautus (1995) edge correction: at hit ∈ {0, 1} or fa ∈ {0, 1} the values are adjusted by ± 0.5/N (log-linear correction) so that z(·) does not diverge.
Error Details
The key figures tell you how good the gage is. The two detail tables tell you what is causing it — and only that lets you improve anything. Both exclude parts without a known reference from the error counts while still listing them.
Ratings per part: One row per part with the reference, each appraiser's ratings, the number of deviations from the reference and the number of self-inconsistent appraisers. Individual ratings are counted, not appraisers: a consistently wrong appraiser therefore weighs more than a one-off slip. Rows are sorted by severity; the filter hides the undisputed parts. When deviations cluster on a few parts, what is usually missing there is a boundary sample — not training.
Misclassifications per appraiser: One row per appraiser and kind of misclassification ("NOK → OK" means: reference NOK, rated OK), plus the parts affected. The mixed row counts the (part, appraiser) pairs whose repeats did not all come out the same — that is disagreement with oneself and needs no reference. One-sided misclassifications indicate a systematic bias (too strict or too lax), two-sided ones rather an unclear inspection instruction.
All appraisers agree: Share of parts where all appraisers rated the same across all repeats. Needs no reference and is stricter than any single figure: one dissenting appraiser drops the part.
All agree and correct: The same condition, plus the unanimous rating must match the reference. The gap to the tile next to it is the most interesting number in the whole analysis: it counts the parts where the team was unanimous and still wrong. No agreement measure finds those — κ included, because κ measures agreement, not correctness.
Verdict & Traffic Lights
The verdict follows the selected verdict rule: AIAG MSA 4th Ed. ch. III-C (attribute study) by default, Bosch Heft 10 as an option. The verdict criteria table shows every rated criterion with value, appraiser, limits and status. The decisive row is highlighted and is also named next to the verdict.
AIAG MSA (4th ed.), ch. III-C
- Fleiss κ: good ≥ 0.75 · marginal ≥ 0.40 · below unacceptable
- Effectiveness (lowest): good ≥ 90 % · marginal ≥ 80 % · below unacceptable
- Miss rate (highest): good ≤ 2 % · marginal ≤ 5 % · above unacceptable
- False-alarm rate (highest): good ≤ 5 % · marginal ≤ 10 % · above unacceptable
The worst criterion decides. Miss rate and false-alarm rate count only for binary data with a reference column. When the reference is the appraisers' consensus, an error shared by the majority stays invisible. Both rates are then shown in the appraiser table for information only, and a note below the verdict criteria says so. An appraiser without NOK reference parts has no miss rate, one without OK parts no false-alarm rate; that appraiser is skipped for the criterion.
Bosch Heft 10
As in Heft 10 (2019, Verfahren 7, p. 29), the smallest of four Fleiss κ is rated: between appraisers, within each appraiser, each appraiser vs. reference and all appraisers together vs. reference. Against the reference, the reference counts as one more rater beside the trials. Limits: ≥ 0.9 capable, ≥ 0.7 conditionally capable, below that not capable. Under Bosch, the "κ vs. reference" table and the "Avg. κ vs. reference" tile show this Fleiss κ per appraiser, under AIAG Cohen's κ. Fleiss κ comes with a z test only, not a confidence interval. The κ dots in the result tables follow the limits of the selected rule.
Ambiguous parts without reference: When the consensus fallback finds no majority vote for a part (tie for binary/nominal, no unique median for ordinal), the part lands in meta.ambiguousParts and is excluded only from effectiveness, miss/false-alarm and SDT. Kappa between appraisers is retained.
Interpretation text: Next to the verdict stand Fleiss κ with z and p and the decisive criterion; the interpretation paragraph names, where applicable, the warnings W_UNBALANCED_REPS, W_AMBIGUOUS_CONSENSUS and W_LOW_REP_COUNT.
References
- AIAG (2010)Measurement Systems Analysis (MSA) Reference Manual4th ed., Automotive Industry Action Group, Southfield, MI, ISBN 978-1-60534-211-5Ch. III-C: default verdict rule from Fleiss κ, effectiveness, miss rate and false-alarm rate.
- Fleiss, Joseph L.; Nee, John C. M.; Landis, J. Richard (1979)Large sample variance of kappa in the case of different sets of ratersPsychological Bulletin, Vol. 86, No. 5, p. 974–977doi:10.1037/0033-2909.86.5.974Standard error of Fleiss κ under H0 for z and p.
- Melzer, Almut (2019)Six Sigma – kompakt und praxisnah: Prozessverbesserung effizient und erfolgreich implementieren2nd ed., Springer Gabler, Wiesbaden, ISBN 978-3-658-23754-7doi:10.1007/978-3-658-23755-4Cross-check of the attribute study: miss and false-alarm rate in the verdict, Bosch rule set, z and p per κ, within-appraiser κ.
- Robert Bosch GmbH (2019)Qualitätsmanagement in der Bosch-Gruppe, Technische Statistik, Heft 10: Fähigkeit von Mess- und PrüfprozessenRobert Bosch GmbH, StuttgartOptional verdict rule: smallest κ, capable from 0.9, conditionally capable from 0.7.
Examples
This module ships with the following example datasets — load any of them in the app with a single click.
Available in the following cycles
- DMAIC: Measure
- DMADV: Measure
- 8D: D2 — Describe the Problem