{"id":2761,"date":"2022-01-26T11:44:29","date_gmt":"2022-01-26T11:44:29","guid":{"rendered":"https:\/\/www.psyctc.org\/psyctc\/?post_type=docs&p=2761"},"modified":"2022-01-26T11:44:29","modified_gmt":"2022-01-26T11:44:29","password":"","slug":"inter-rater-agreement-reliability","status":"publish","type":"docs","link":"https:\/\/www.psyctc.org\/psyctc\/glossary2\/inter-rater-agreement-reliability\/","title":{"rendered":"Inter-rater agreement\/reliability"},"content":{"rendered":"\n

One way to assess how good, how usable, a rating system is is to get more than one person to use the system to do ratings of the same things where what is rated has a broad coverage of the possible ratings in the system. Ratings may be undimensional and continuous, e.g. “how much warmth in this clip do you think this carer is expressing about …”, or categorical “into which of the categories would you put this AAI interview? The index of inter-rater agreement you are most likely to see for categorical ratings is Cohen’s kappa<\/a>, a correlation<\/a> coefficient is better for continuous ratings.<\/p>\n\n\n\n

Details<\/h4>\n\n\n\n

Inter-rater reliability\/agreement is an extremely general way of assessing a rating system. Cohen’s kappa, the first really widely recognised and used description of the issues and how to use joint ratings in psychological situations, applied to just two raters rating a set of things but there are extensions and alternatives that can be used for more than two raters.<\/p>\n\n\n\n

Try also<\/h4>\n\n\n\n

Reliability
Bias<\/a>
Cohen’s kappa<\/a>
Weighted kappa
Interview measures
Rating scales
Classification
Nominal\/category scaling<\/a>
Ordinal scaling
<\/a>Stevens’ levels of measurement<\/a>
AAI<\/p>\n\n\n\n

Chapters<\/h4>\n\n\n\n

Chapter 3.<\/p>\n\n\n\n

History<\/h4>\n\n\n\n

Created 26\/1\/22.<\/p>\n","protected":false},"excerpt":{"rendered":"

One way to assess how good, how usable, a rating system is is to get more than one person to use the system to do ratings of the same things where what is rated has a broad coverage of the possible ratings in the system. Ratings may be undimensional and continuous, e.g. “how much warmth … Continue reading Inter-rater agreement\/reliability<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":""},"doc_category":[18],"glossaries":[],"doc_tag":[],"knowledge_base":[],"class_list":["post-2761","docs","type-docs","status-publish","hentry","doc_category-om-book"],"year_month":"2026-04","word_count":183,"total_views":"1247","reactions":{"happy":"0","normal":"0","sad":"0"},"author_info":{"name":"chris","author_nicename":"chris","author_url":"https:\/\/www.psyctc.org\/psyctc\/author\/chris\/"},"doc_category_info":[{"term_name":"All OM book glossary entries","term_url":"https:\/\/www.psyctc.org\/psyctc\/glossary\/non-knowledgebase\/om-book\/"}],"doc_tag_info":[],"knowledge_base_info":[],"knowledge_base_slug":[],"_links":{"self":[{"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/docs\/2761","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/docs"}],"about":[{"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/types\/docs"}],"author":[{"embeddable":true,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/comments?post=2761"}],"version-history":[{"count":1,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/docs\/2761\/revisions"}],"predecessor-version":[{"id":2762,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/docs\/2761\/revisions\/2762"}],"wp:attachment":[{"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/media?parent=2761"}],"wp:term":[{"taxonomy":"doc_category","embeddable":true,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/doc_category?post=2761"},{"taxonomy":"glossaries","embeddable":true,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/glossaries?post=2761"},{"taxonomy":"doc_tag","embeddable":true,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/doc_tag?post=2761"},{"taxonomy":"knowledge_base","embeddable":true,"href":"https:\/\/www.psyctc.org\/psyctc\/wp-json\/wp\/v2\/knowledge_base?post=2761"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}