A researcher who wants to establish score reliability in an observational study typically determines
A) how well observer-assigned scores correlate with an external criterion.
B) the extent of agreement among observers.
C) how well the observation predicts other events.

Question

Quizplus · Accepted Answer

The most common method for establishing score reliability in observational studies is to determine the agreement among multiple observers. This is typically done by having the observers independently assign scores to the same set of observations, and then calculating the extent to which their scores are in agreement. A common measure of agreement is Cohen's kappa coefficient. The other options given, correlating observer-assigned scores with an external criterion and examining the predictive validity of the observation, are both related to validity rather than reliability.

A Researcher Who Wants to Establish Score Reliability in an Observational