Comparison studies on agreement coefficients with emphasis on missing data

PhD student Alexandra de Raadt studied methods to deal with missing data when using agreement coefficients.

In various fields of science the categorization of people into categories is required. An example is the assignment of people with mental health problems to classes of mental disorders by a psychologists. A diagnosis may provide a person more insight into his or her problems, which is often a prerequisite for finding the right treatment.

A nominal rating instrument has high reliability if persons obtain the same classification under similar conditions. In other words, a classification is considered reliable if the raters agree on their rating. A coefficient that is commonly used for measuring the degree of agreement between two raters is Cohen’s kappa. Cohen’s kappa is a standard tool for assessing greement between nominal classifications in social and medical sciences.

Missing data (or missing values) are a common problem in many fields of science. In agreement studies, missing data may occur due to missed appointments or dropout of persons. However, missing data may also be the result of rater performance. If a particular category is missing, or if a category is not fully understood, a rater may choose not to rate the unit. How missing data may affect the quantification of inter-rater agreement has not been studied comprehensively.

In this dissertation we mainly focused on the impact of missing data on kappa coefficients. The results show that a coefficient that uses missing data for a more precise estimation of the expected agreement, multiple imputation methods and listwise deletion are able to handle missing agreement data sufficiently.

Proefschrift

Comparison studies on agreement coefficients with emphasis on missing data

Medewerkers

Alexandra de Raadt (PhD student)
Matthijs Warrens
Roel Bosker