When two raters classify the same items, how much do they really agree?
Your browser does not support embedded video. Download the video (MP4).
:::
Everything in the video, ready to copy:
# Inter-rater agreement: Cohen's kappa (base R) # Two raters classify 50 items; their agreement table: tab <- matrix(c(20, 5, 3, 22), nrow = 2, byrow = TRUE, dimnames = list(Rater1 = c("+", "-"), Rater2 = c("+", "-"))) tab # Observed vs chance-expected agreement po <- sum(diag(tab)) / sum(tab) pe <- sum(rowSums(tab) * colSums(tab)) / sum(tab)^2 c(observed = po, expected = pe) # Cohen's kappa - agreement beyond chance kappa <- (po - pe) / (1 - pe) kappa
Download this R script · ▶ Practice in the playground
When to use it. Use kappa to quantify agreement corrected for chance: 0 is chance-level, 1 is perfect; above ~0.6 is usually considered good agreement.
Try the Cohen’s kappa calculator.
Next: Cronbach’s Alpha: Reliability →