Reliability of OSTE in the health professions education exit examination of college of physicians and surgeons, Pakistan: a psychometric analysis

ABSTRACT

To determine reliability and validity of objectively structured teaching examination [OSTE] and the difference in the examination scores of two consecutive Health Professions Education batches. Cross-sectional analytical study. The College of Physicians and Surgeons Pakistan, Karachi, from November 2011 to April 2012. The OSTEs were designed on the basis of examination blueprints. Scores of participants of two consecutive years [group-I n = 14, group-II n = 15] were analyzed by using SPSS version 16. Mean +/- SD was computed and overall means were compared by t test. Reliability was determined by Cronbach's coefficient. Standard error of measurement [SEM] was computed for precision of scores. OSTE stations were not uni-dimensional, factor analysis including varimax rotation was performed. For convergent validity [Pearson correlations] was determined between scores of OSTE, lectures delivered and workshops conducted. The overall mean score was 7.64 +/- 0.65 in group-I and 7.33 +/- 1.03 in group-II [p = 0.429], [group-I SEM = 0.35 and group-II, SEM = 0.48]. Factor analysis yielded four factors in group-I and five factors in group-II with Eigenvalues > 1. The Pearson's correlation of average scores of OSTE and lecture was significant for group-I [r = 0.61, p 0.019], and insignificant for group-II [r = 0.46, p = 0.08]. Correlation between OSTE and workshops conducted was insignificant in group-I [r = 0.51, p = 0.057] and significant in group-II [r = 0.53, p = 0.04]. OSTE is a reliable, valid assessment tool for discerning competency among learners of HPE program