Contribution of examiner subjectivity and patient heterogeneity on long case examination.

OBJECTIVE: To evaluate the effect of examiner subjectivity and heterogeneity among the cases on scores from the Royal College of Physicians (RCPT) long case examination. MATERIAL AND METHOD: Data from internal medicine candidates who performed clinical part of RCPT board certifying examination in academic year 2008 were collected. For each candidate, scores from pair of examiners for each of the long case was stratified based on disease category according to the course syllabus into 3 groups; very common, common and uncommon diseases. The scores also categorized according to difficulty level subjectively and rated by the examiners into 3 levels; easy, moderate and difficult. Mean scores in each group of encounters were compared using ANOVA. RESULTS: There were 21 examination centers involved with 1,840 number of encounters by 232 candidates. Among 437 patients that have been used for the long case, common scenarios (27.6% of the total) were cirrhosis, hyperthyroidism, cerebral thrombosis, bronchogenic carcinoma, rheumatic heart disease and thalassemia. Mean and SD of scores from the very common, common and uncommon diseases were 75.5 +/- 11.6, 75.6 +/- 10.6 and 74.7 +/- 11.3 respectively, with no statistical significant difference between the groups. Mean and SD of scores from the easy, moderate and difficult cases were 76.1 +/- 10.5, 74.8 +/- 11.0, 75.5 +/- 10.9 respectively. The moderate group has the lowest score with a statistical significant difference from other groups (p = 0.042). CONCLUSION: In current RCPT long case examination, difficulty of the case appears to contribute to variation in scores derived from the examiners. Measures for score adjustment and examiner calibration should be implemented in the future.

Reliability and validity of long case and short case in internal medicine board certification examination.

OBJECTIVE: To be certified for the Thai Board of Internal Medicine, each candidate must pass both written and clinical examinations performed in different academic years. The present study aimed to assess the reliability and validity of the long case and short case which contribute major fractions in the clinical part of board certifying examination. MATERIAL AND METHOD: Data from 585 internal medicine residents entering a first-attempt clinical part in board certifying examination during the academic year 2005-2007 were collected. Inter-rater reliability and construct validity of the long case and short case were then examined. RESULTS: Good to excellent intraclass correlation (ICC) of scores from different examiners was demonstrated (ICC between 0.71 and 0.97) and the variation ranged from 15.3 to 27.3%. For different occasions of examination, class normalized gain was between -0.7 and -9.0% and negative individual normalized gain was observed in 45.6% to 48.2% of the candidates. CONCLUSION: Acceptable inter-rater reliability was demonstrated in long case and short case in clinical examination for the Thai Board of Internal Medicine. But construct validity for this type of clinical assessment was not established.