Cohen’s Kappa Calculator (Inter-Rater Agreement)
Measure how well two raters agree beyond chance. Paste their agreement matrix (rows = rater A’s categories, columns = rater B’s, in the same order) and get Cohen’s kappa with a 95% confidence interval, plus observed and chance-expected agreement. Works for any number of categories (2×2, 3×3, …). Nothing is uploaded.
How to read it
What kappa tells you
When two raters classify the same items, some agreement happens by chance alone. Cohen’s kappa corrects for that:
\[\kappa = \frac{p_o - p_e}{1 - p_e}\]
where pₒ is the observed agreement and pₑ the agreement expected if both raters labelled at random. κ = 1 is perfect agreement, κ = 0 is exactly chance, and negative values mean worse than chance.
A widely used (if rough) guide from Landis & Koch (1977):
| κ | Interpretation |
|---|---|
| < 0.00 | Less than chance |
| 0.01–0.20 | Slight |
| 0.21–0.40 | Fair |
| 0.41–0.60 | Moderate |
| 0.61–0.80 | Substantial |
| 0.81–1.00 | Almost perfect |
Kappa is sensitive to how common the categories are (the “prevalence problem”), so report it alongside the raw observed agreement.
Do it in R
m <- matrix(c(40, 10, 15, 35), nrow = 2, byrow = TRUE)
psych::cohen.kappa(m) # kappa with confidence interval
# or: vcd::Kappa(m)For agreement on a continuous measurement (not categories), you’d use an intraclass correlation (ICC) instead — get in touch if that’s your case.
FAQ
Frequently asked questions
Kappa or percent agreement?
Report both. Percent agreement is intuitive but ignores chance; kappa corrects for it. When categories are very unbalanced, kappa can look low even with high raw agreement — showing both makes the picture honest.
More than two raters?
Cohen’s kappa is for exactly two raters. For three or more, use Fleiss’ kappa; for ordered categories, weighted kappa; for continuous ratings, the ICC.
Setting up a reliability study or coding scheme? We can design and analyse it.