Comparison between inter-rater reliability and inter-rater agreement in performance assessment

<p><b>INTRODUCTION</b>Over the years, performance assessment (PA) has been widely employed in medical education, Objective Structured Clinical Examination (OSCE) being an excellent example. Typically, performance assessment involves multiple raters, and therefore, consistency among the scores provided by the auditors is a precondition to ensure the accuracy of the assessment. Inter-rater agreement and inter-rater reliability are two indices that are used to ensure such scoring consistency. This research primarily examined the relationship between inter-rater agreement and inter-rater reliability.</p><p><b>MATERIALS AND METHODS</b>This study used 3 sets of simulated data that was based on raters' evaluation of student performance to examine the relationship between inter-rater agreement and inter-rater reliability.</p><p><b>RESULTS</b>Data set 1 had high inter-rater agreement but low inter-rater reliability, data set 2 had high inter-rater reliability but low inter-rater agreement, and data set 3 had high inter-rater agreement and high inter-rater reliability.</p><p><b>CONCLUSION</b>Inter-rater agreement and inter-rater reliability can but do not necessarily coexist. The presence of one does not guarantee that of the other. Inter-rater agreement and inter-rater reliability are both important for PA. The former shows stability of scores a student receives from different raters, while the latter shows the consistence of scores across different students from different raters.</p>

Clinical skills in final-year medical students: the relationship between self-reported confidence and direct observation by faculty or residents

<p><b>INTRODUCTION</b>Many students, while performing clinical skills such as medical interviewing/ communication, physical examination, and procedural tasks, have never been observed by faculty members or residents. This study aimed to explore the relationships between final-year medical students' self-reported confidence and the frequency of direct observation by faculty member or resident while conducting these clinical skills.</p><p><b>MATERIALS AND METHODS</b>Medical students at China Medical University in Taiwan participated in the survey. Before graduating, they were asked to answer a questionnaire about (1) their confidence in performing 17 clinical skills including medical interviewing/communication, physical examination, and procedural tasks, and (2) the number of times they had been directly observed by faculty members or residents during student-patient encounters.</p><p><b>RESULTS</b>Many students reported never having been observed by a faculty member while they performed history taking/communication (46% to 84%), physical examination (36% to 42%), or procedural tasks (41% to 81%). It was found that residents had observed the students more frequently than the faculty members. The correlations between self-reported confidence and the corresponded direct observation were small to medium but significant. However, no difference was found between observation by a faculty member and by a resident.</p><p><b>CONCLUSIONS</b>This study confirmed that many medical students have not been directly observed in clinical training; and that those who were observed more often, expressed more self-reported confidence. Some assessment measures, which focus on direct observation and feedback during student-patient encounters, may improve the students' confidence.</p>