Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37.

The process of setting a standard when pass/fail decisions have to be made inevitably involves judgment about the point on the test score scale where performance is deemed to be adequate for the purpose for which the examination is set. As with any process which involves human judgment, setting this standard is likely to include a certain degree of error, which may result in some false positive and false negative decisions. The customary practice of maintaining a constant point on the test score scale at which pass/fail separations are made cannot be justified, as examinations vary in difficulty. The aim of standard setting procedures is to minimize such errors while accounting for the varying difficulty of examinations. A standard may be norm-referenced, where it is dependent on the performance of the particular group of examinees, or criterion-referenced, where it is based on predetermined criteria, irrespective of examinee performance. Where certification of competence is the primary purpose of an examination, the latter is preferred as the decision to be made is whether an individual is competent to practise rather than competent compared to peers. Several methods of standard setting have been used, some of which are based solely on predetermined criteria, while others compromise between norm- and criterion-referenced standards. This guide examines the more commonly used methods of standard setting, illustrates the procedure used in each with the help of an example, and discusses the advantages and disadvantages associated with the use of each. The common errors made by judges in the standard setting process are pointed out and the manner in which judges should be selected, trained and instructed emphasized. A method used for equating similar tests set at different times with the intention of maintaining standards from one examination to the next is illustrated with an example. Finally, the guide proposes a practical method for arriving at a pre-determined standard by the proportionate selection of test-items of known relative difficulties in relation to minimally competent examinees.

Structured problem-related anatomy demonstrations: making order of random teaching events.

BACKGROUND: A standardized learning activity called "Structured Problem-Related Anatomy Demonstrations" was recently implemented. This activity is intended to be a substitute to the need-based instruction method in which students schedule appointments in small groups or individually with faculty to discuss issues and difficulties related to anatomy the students have encountered during their tutorial discussions and self-study learning. DESCRIPTION: Students in a given year were divided into 2 groups. Each group was further divided into 4 small groups (n = 10) who rotated through 4 "stations." A faculty member who demonstrated a specific aspect of anatomy related to the weekly health problem staffed each of the stations. EVALUATION: The effect of the new method of instruction on students' performance in end-of-unit examinations was not statistically significant. A 5-point scale questionnaire seeking the perceptions of 3rd- and 4th-year medical students (n = 176), who had experience with both methods, was used. The means +/- standard deviation of students' responses to items related to organization was 3.61 +/- 0.55, to knowledge was 4.29 +/- 0.73, to integration was 3.91 +/- 0.73, and to skills was 3.84 +/- 0.83. The vast majority of students expressed their overall support for continuing the new system (4.54 +/- 0.58). CONCLUSIONS: The new method increased students' satisfaction and confidence while maintaining their level of performance in final assessments. Faculty have more control over their schedules and can devote more quality time to teaching and research.