Simulation studies for the evaluation of health information technologies: experiences and results.

It is essential for new health information technologies (IT) to undergo rigorous evaluations to ensure they are effective and safe for use in real-world situations. However, evaluation of new health IT is challenging, as field studies are often not feasible when the technology being evaluated is not sufficiently mature. Laboratory-based evaluations have also been shown to have insufficient external validity. Simulation studies seem to be a way to bridge this gap. The aim of this study was to evaluate, using a simulation methodology, the impact of a new prototype of an electronic medication management system on the appropriateness of prescriptions and drug-related activities, including laboratory test ordering or medication changes. This article presents the results of a controlled simulation study with 50 simulation runs, including ten doctors and five simulation patients, and discusses experiences and lessons learnt while conducting the study. Although the new electronic medication management system showed tendencies to improve medication safety when compared with the standard system, this tendency was not significant. Altogether, five distinct situations were identified where the new medication management system did help to improve medication safety. This simulation study provided a good compromise between internal validity and external validity. However, several challenges need to be addressed when undertaking simulation evaluations including: preparation of adequate test cases; training of participants before using unfamiliar applications; consideration of time, effort and costs of conducting the simulation; technical maturity of the evaluated system; and allowing adequate preparation of simulation scenarios and simulation setting. Simulation studies are an interesting but time-consuming approach, which can be used to evaluate newly developed health IT systems, particularly those systems that are not yet sufficiently mature to undergo field evaluation studies.

3,520 medication errors evaluated to assess the potential for IT-based decision support.

We have previously studied system failures involved in medication errors using a limited number of root cause analyses as source. The aim of this study was to describe a larger number of medication errors with respect to harm, involved medicines and involved system problems - thus providing information for the development of IT-based decision support. We evaluated 3,520 medication error reports derived from 12 months of consecutive reporting from 13 hospitals in the Capital Region of Denmark. We found 0.65% errors with serious harm and 16% with moderate harm. A small number of medicines were involved in the majority of the errors. The problems in the medication error process were heterogeneous. Some were related to specific medicines and others were related to the computerized order entry system. Accordingly decision support targeted at specific medicines and improved IT systems are part of the continuing work to reduce the frequency of medication errors.

Lessons learnt from conducting a high fidelity simulation test in health IT.

Testing IT-systems by use of simulation requires a thorough planning and preparation in order to create a realistic clinical environment. For a successful test through simulation a dedicated test team to control the environment is needed, as well as people to play the role of patients and staff. Relevant artifacts and elaborate scenarios ensure the narrative. This paper explores the preliminary work and execution of an extensive test of a Computerized Order Entry System prototype. Central to the setup of the test is a script which outlines the method by guiding the preparation and execution.