Created 24.i.22<\/p>\n\n\n\n
I did a peer review of a paper recently and met an old chestnut: that the inter-rater agreement reported was good because the simple agreement rates were “good”. This is nonsense and that has been written about for probably a century and alternative ways summarising agreement rates have been around for a long time. Jacob Cohen invented his “chance corrected” “coefficient of agreement for nominal scales”, kappa in 1960 (Cohen, 1960). That made me think it might be useful to have a blog post here, perhaps actually several, linking with demonstrations of the issues in my “R SAFAQ” (Self-Answered Frequently (self) Asked Questions” (a.k.a. Rblog). <\/p>\n\n\n\n
The issue is very simple: if the thing that is rated is not around 50:50 in the ratings, then agreement even by chance is going to be high. Let’s say two raters are asked to rate a series of photos of facial expressions for the presence of “quizzically raised eyebrows” and the rate of photos that look even remotely quizzical they are given is only 10% and let’s suppose they are told that the rate is about 10% and use that information. <\/p>\n\n\n\n
Now if they have absolutely no agreement, i.e. only chance agreement about what constitutes a “quizzically raised eyebrow” they may well still each rate about 90%. In that case by chance alone rater B will rate as quizzical 10% of the photos that rater A rated as quizzical: rate of agreement 10% * 10% = one in a hundred, 1% agreement. However, rater B will rate as not quizzical 90% of the 90% of photos that rater A rated as not quizzical: rate of agreement 90% * 90% = 81%. So their raw agreement rate is 82% which sounds pretty good until we realise that it arose by pure chance. Here’s an aesthetically horrible table of that for n = 100 and the perfect chance level of agreement. (In real life, sampling vagaries mean it wouldn’t be quite as neat as this but it wouldn’t be far off this.)<\/p>\n\n\n\n
| \n n <\/th>\n | \n Rated quizzical by rater B <\/th>\n | \n Rated NOT quizzical by rater B <\/th>\n | \n Row totals <\/th>\n <\/tr>\n |
|---|---|---|---|
| \n Rated quizzical by rater A <\/td>\n | \n 1 <\/td>\n | \n 9 <\/td>\n | \n 10 <\/td>\n <\/tr>\n |
| \n Rated NOT quizzical by rater A <\/td>\n | \n 9 <\/td>\n | \n 81 <\/td>\n | \n 90 <\/td>\n <\/tr>\n |
| \n Column totals: <\/td>\n | \n 10 <\/td>\n | \n 90 <\/td>\n | \n 100 <\/td>\n <\/tr>\n <\/table>\n<\/div> |