The contribution of prescription chart design and familiarity to prescribing error: a prospective, randomised, cross-over study.

PURPOSE OF STUDY: Initiatives to standardise hospital paper-based prescription charts are underway in various countries in an effort to reduce prescribing errors. The aim of this study was to investigate the extent to which prescribing error rates are influenced by prescription chart design and familiarity. STUDY DESIGN: In this prospective, randomised, cross-over study, Foundation Year 1 doctors working in five Scottish National Health Service (NHS) Boards participated in study sessions during which they were asked to prescribe lists of medications for five fictional patients using a different design of paper prescription chart for each patient. Each doctor was timed completing each set of prescriptions, and each chart was subsequently assessed against a predefined list of possible errors. A mixed modelling approach using three levels of variables (design of and familiarity with a chart, prescribing speed and individual prescriber) was employed. RESULTS: A total of 72 Foundation Year 1 doctors participated in 10 data-collection sessions. Differences in prescription chart design were associated with significant variations in the rates of prescribing error. The charts from NHS Highland and NHS Grampian produced significantly higher error rates than the other three charts. Participants who took longer to complete their prescriptions made significantly fewer errors, but familiarity with a chart did not predict error rate. CONCLUSIONS: This study has important implications for prescription chart design and prescribing education. The inverse relationship between the time taken to complete a prescribing task and the rate of error emphasises the importance of attention to detail and workload as factors in error causation. Further work is required to identify the characteristics of prescription charts that are protective against errors.

The standard error of measurement is a more appropriate measure of quality for postgraduate medical assessments than is reliability: an analysis of MRCP(UK) examinations.

BACKGROUND: Cronbach's alpha is widely used as the preferred index of reliability for medical postgraduate examinations. A value of 0.8-0.9 is seen by providers and regulators alike as an adequate demonstration of acceptable reliability for any assessment. Of the other statistical parameters, Standard Error of Measurement (SEM) is mainly seen as useful only in determining the accuracy of a pass mark. However the alpha coefficient depends both on SEM and on the ability range (standard deviation, SD) of candidates taking an exam. This study investigated the extent to which the necessarily narrower ability range in candidates taking the second of the three part MRCP(UK) diploma examinations, biases assessment of reliability and SEM. METHODS: a) The interrelationships of standard deviation (SD), SEM and reliability were investigated in a Monte Carlo simulation of 10,000 candidates taking a postgraduate examination. b) Reliability and SEM were studied in the MRCP(UK) Part 1 and Part 2 Written Examinations from 2002 to 2008. c) Reliability and SEM were studied in eight Specialty Certificate Examinations introduced in 2008-9. RESULTS: The Monte Carlo simulation showed, as expected, that restricting the range of an assessment only to those who had already passed it, dramatically reduced the reliability but did not affect the SEM of a simulated assessment. The analysis of the MRCP(UK) Part 1 and Part 2 written examinations showed that the MRCP(UK) Part 2 written examination had a lower reliability than the Part 1 examination, but, despite that lower reliability, the Part 2 examination also had a smaller SEM (indicating a more accurate assessment). The Specialty Certificate Examinations had small Ns, and as a result, wide variability in their reliabilities, but SEMs were comparable with MRCP(UK) Part 2. CONCLUSIONS: An emphasis upon assessing the quality of assessments primarily in terms of reliability alone can produce a paradoxical and distorted picture, particularly in the situation where a narrower range of candidate ability is an inevitable consequence of being able to take a second part examination only after passing the first part examination. Reliability also shows problems when numbers of candidates in examinations are low and sampling error affects the range of candidate ability. SEM is not subject to such problems; it is therefore a better measure of the quality of an assessment and is recommended for routine use.

Performance in the MRCP(UK) Examination 2003-4: analysis of pass rates of UK graduates in relation to self-declared ethnicity and gender.

BACKGROUND: Male students and students from ethnic minorities have been reported to under-perform in undergraduate medical examinations. We examined the effects of ethnicity and gender on pass rates in UK medical graduates sitting the Membership of the Royal Colleges of Physicians in the United Kingdom [MRCP(UK)] Examination in 2003-4. METHODS: Pass rates for each part of the examination were analysed for differences between graduate groupings based on self-declared ethnicity and gender. RESULTS: All candidates declared their gender, and 84-90% declared their ethnicity. In all three parts of the examination, white candidates performed better than other ethnic groups (P < 0.001). In the MRCP(UK) Part 1 and Part 2 Written Examinations, there was no significant difference in pass rate between male and female graduates, nor was there any interaction between gender and ethnicity. In the Part 2 Clinical Examination (Practical Assessment of Clinical Examination Skills, PACES), women performed better than did men (P < 0.001). Non-white men performed more poorly than expected, relative to white men or non-white women. Analysis of individual station marks showed significant interaction between candidate and examiner ethnicity for performance on communication skills (P = 0.011), but not on clinical skills (P = 0.176). Analysis of overall average marks showed no interaction between candidate gender and the number of assessments made by female examiners (P = 0.151). CONCLUSION: The cause of these differences is most likely to be multifactorial, but cannot be readily explained in terms of previous educational experience or differential performance on particular parts of the examination. Potential examiner prejudice, significant only in the cases where there were two non-white examiners and the candidate was non-white, might indicate different cultural interpretations of the judgements being made.