Why not just report percent agreement?
Two raters can agree often just by chance, especially when most items share one label. That is why Cohen proposed kappa in 1960; as McHugh's review puts it, "Jacob Cohen critiqued use of percent agreement due to its inability to account for chance agreement." Kappa "can range from -1 to +1," where 0 means agreement no better than chance. McHugh also warns that Cohen's suggested interpretation "may be too lenient for health related studies because it implies that a score as low as 0.41 might be acceptable."
Sources: Cohen 1960; McHugh 2012.
Which agreement metric fits which label type?
Pick the metric from the shape of the label, not from habit. Categories call for a kappa-type statistic, continuous values for an ICC, regions for an overlap score, and events in a time series for an event-matching score. The Metrics Reloaded recommendations make the same point for image analysis: match the metric to the problem type.
| Metric | Label type | Raters | What it measures | Watch out for |
|---|---|---|---|---|
| Percent agreement | Any | 2+ | Share of items where raters gave the same label | Ignores agreement expected by chance |
| Cohen's kappa | Nominal (e.g., present/absent) | 2 | Agreement corrected for chance | Can be low despite high agreement when one class is rare (see below) |
| Weighted kappa | Ordinal (e.g., severity grades) | 2 | Chance-corrected agreement with partial credit for near-misses | Result depends on the weighting scheme; report it |
| Fleiss' kappa | Nominal | More than 2 | Chance-corrected agreement across many raters | Same prevalence sensitivity as Cohen's kappa |
| Krippendorff's alpha | Nominal to ratio | 2+ | A general reliability coefficient for coded data | Report the data level (nominal, ordinal, interval) you used |
| Intraclass correlation (ICC) | Continuous (e.g., measurements, scores) | 2+ | Reliability of continuous ratings | There are several ICC forms; report model, type and definition |
| Dice / IoU (Jaccard) | Segmentation masks | 2 (pairwise) | Overlap between two raters' regions | Small structures are penalized more by a few pixels of disagreement (our view) |
| Event-level F1 | Events in time (e.g., EEG seizures) | 2 (pairwise) | Matching of events, combining sensitivity and precision | Define the matching rule (any-overlap or time-aligned) |
Sources: Cohen 1960; Cohen 1968; Fleiss 1971; Landis & Koch 1977; Hayes & Krippendorff 2007; Shrout & Fleiss 1979; Koo & Li 2016; Taha & Hanbury 2015; Maier-Hein et al. 2024 (Metrics Reloaded); Dan et al. 2025 (SzCORE). The "Watch out for" column is a short summary; the Dice row's note is our view.
How do I choose and report an ICC?
Koo and Li note that "there are 10 forms of ICCs," so the form has to be stated. Their guideline says reporting should include the "software information, 'model,' 'type,' and 'definition' selections." For interpretation, they suggest reading the 95% confidence interval: "values less than 0.5, between 0.5 and 0.75, between 0.75 and 0.9, and greater than 0.90 are indicative of poor, moderate, good, and excellent reliability, respectively." The original ICC forms come from Shrout and Fleiss (1979).
Sources: Koo & Li 2016; Shrout & Fleiss 1979.
What is the kappa paradox?
Feinstein and Cicchetti showed that "a high value of p0 can be drastically lowered by a substantial imbalance in the table's marginal totals." In plain terms, when one label is rare (seizures in hours of EEG, a rare finding on imaging), raters can agree on almost every item and still get a modest kappa. Their companion paper gives the fix: "the omnibus value of kappa should always be accompanied by separate individual values of ppos and pneg," that is, positive agreement and negative agreement reported separately.
Sources: Feinstein & Cicchetti 1990; Cicchetti & Feinstein 1990.
Worked example (illustrative)
These numbers are made up to show the arithmetic. Two raters label 100 EEG segments as seizure or no seizure. Both say seizure on 4, only rater A says seizure on 3, only rater B says seizure on 3, and both say no seizure on 90.
| Statistic | Calculation | Value |
|---|---|---|
| Observed agreement (p0) | (4 + 90) / 100 | 0.94 |
| Chance agreement (pe) | 0.07 × 0.07 + 0.93 × 0.93 | 0.87 |
| Cohen's kappa | (0.94 − 0.87) / (1 − 0.87) | 0.54 |
| Positive agreement (ppos) | 2 × 4 / (2 × 4 + 3 + 3) | 0.57 |
| Negative agreement (pneg) | 2 × 90 / (2 × 90 + 3 + 3) | 0.97 |
The raters agree on 94% of segments, but kappa is 0.54 and positive agreement is 0.57: they agree on "no seizure" almost always and on "seizure" only about half the time. Reporting all three numbers tells a reviewer far more than any one of them. For seizure detection specifically, the SzCORE framework scores "at the granularity level of events," and uses F1 "as a combined measure containing information on both sensitivity and precision."
Sources: Cicchetti & Feinstein 1990; Dan et al. 2025 (SzCORE). Example values are illustrative arithmetic, not study data.
How should disagreements be adjudicated?
In our view, a workable pattern is: two or more raters label each item independently, without seeing each other's labels; agreement is computed per label class before anyone resolves anything; then disagreements go to a senior reader or a consensus meeting, with the reason recorded. Keep the pre-adjudication agreement numbers. They describe how hard the task is and how uncertain the reference standard is, and they are what a reader of your validation will ask for. Rising disagreement on one class is usually a sign the labeling guideline needs a clearer definition there.
This section is our practice recommendation, not a quoted standard.
Related resources
Frequently asked questions
What is a good kappa for medical labels?
There is no single cutoff. McHugh's review warns that Cohen's suggested interpretation may be too lenient for health-related studies because it implies a score as low as 0.41 might be acceptable. Report kappa with positive and negative agreement and judge it against how the labels will be used.
Should I use Cohen's kappa or Fleiss' kappa?
Cohen's kappa is for two raters on nominal labels; Fleiss' kappa extends kappa-type agreement to more than two raters. For ordinal labels with two raters, weighted kappa gives partial credit for near-misses.
Why is my kappa low when my raters almost always agree?
That is the kappa paradox: when one label is rare, kappa can be drastically lowered by the imbalance in marginal totals even when observed agreement is high. Report positive and negative agreement alongside kappa.
Which agreement metric should I use for segmentation or EEG events?
For segmentation masks, overlap scores such as Dice or IoU (Jaccard). For events in time, such as EEG seizures, event-level scoring with an F1 score that combines sensitivity and precision, with the event-matching rule stated.
Scope it in 30 minutes
Book a scoping call and we'll name the smallest engagement that moves you forward. Most teams start with the fixed-scope Initial Audit.
Book a 30-min scoping callPrefer to write? Send a short note.