ABSTRACT
Determine the quality of eleven knowledge based multiple-choice examinations [A-J] and compare pass-fail rates before and after standard setting as a criterion of credibility; intra-class correlation coefficients [ICC] for inter-rater reliability and correlations between mean test scores and standards set for all individual items as a measure of validity. Determine the effect of standard setting an exam 121 days after it had been previously standard set and two options removed using item analysis. Design: Cohort study. In-course examinations delivered to 85-111 medical and dental students in one medical school. Intervention: Modified method of Angoff standard setting multiple-choice examinations. KR-20 coefficients of reliability, mean test scores, point biserial coefficients, standards set and item score correlation, intra-class correlations and pass-fail rates. All the examinations were of high quality according to the criteria used. The pass-fail rates of exams A and B not standard set were 56.4% and 51.8% which then fell to a range 6% - 29% [C-J] after standard setting. The ICC was 0.69 + 0.12, indicating good inter-rater reliability while the MTS - item Diff correlations were low. The theoretical exam confirmed an acceptable standard set in terms of a statistically insignificant change in pass-fail rates. These results support the use of standard setting as being credible and responsive, but only a moderate level of validity was achieved