ABSTRACT
This article describes the key features of realist (realistic) evaluation and illustrates their application using, as an example, a simulation-based course for final year medical students. The use of simulation-based education (SBE) is increasing and so too is the evidence supporting its value as a powerful technique which can lead to substantial educational benefits. Accompanying these changes is a call for research into its use to be more theory-driven and to investigate both 'Did it work?' and as importantly 'Why did it work (or not)?' An evaluation methodology that is capable of answering both questions is realist evaluation. Realist evaluation is an emerging methodology that is suited to evaluating complex interventions such as SBE. The realist philosophy positions itself between positivist and constructivist paradigms and seeks to answer the question 'What works for whom, in what circumstances and why?' In seeking to answer this question, realist evaluation sets out to identify three fundamental components of an intervention, namely context, mechanism and outcome. Educational programmes work (successful outcomes) when theory-driven interventions (mechanisms) are applied to groups under appropriate conditions (context). Realist research uses a mixed methods (qualitative and quantitative) approach to gathering data in order to test the proposed context-mechanism-outcome (CMO) configurations of the intervention under investigation. Realist evaluation offers a valuable methodology for researchers investigating interventions utilising simulation-based education. By investigating and understanding the context, mechanisms and outcomes of SBE interventions, realist evaluation can provide the deeper level of understanding being called for.
ABSTRACT
Medical emergencies in general practice are uncommon and their management requires good teamwork, communication and effective use of the available resources by the whole primary care team. To address this need the Montagu Clinical Simulation Centre developed and delivered a half-day simulation-based medical emergencies course for primary care teams (GPs, practice nurses and administrative staff). Each half-day course comprises two simulated medical emergencies, which are video-taped and then debriefed. The course was evaluated using a multi-level approach by seeking the staff's reactions to the course, their learning, the behaviour changes produced and the results for the organisation. We gained this information through self-reporting using end-of-course and follow-up questionnaires. The immediate feedback was very positive, showing they had learnt the objectives set. We then surveyed all those who had attended training between 2003 and 2007. A follow-up questionnaire was developed and sent to the practice managers who then co-ordinated their completion and return using a pre-paid return envelope. The survey was carried out in two cohorts. The first set of follow-up questionnaires was sent out in September 2004 to all those that had attended up to the end of May 2004 and the second set sent out in May 2008 to those who had attended between June 2004 and May 2007. Of the 338 available to complete the follow-up questionnaire, 208 responded, giving a response rate of 62%. Eighty percent of practices had made changes to equipment, emergency protocols or training and 20% of the participants had been involved in managing a medical emergency since attending the course. Of those, 86% indicated that their management and confidence had improved, that they were better able to take a lead role, give instruction and delegate tasks as required. Based on the self-reported perceptions of learning, we believe that the lessons learnt have been translated into positive changes at a personal and practice level for all members of the primary care team. The participants valued the chance to train as a complete unit. We plan to evaluate simulation-based training in the workplace to see if this has a greater impact.
