Impact of Simulation-Based Training on Radiation Therapists' Workload, Situation Awareness, and Performance.

PURPOSE: This study aimed to assess the impact of simulation-based training intervention on radiation therapy therapist (RTT) mental workload, situation awareness, and performance during routine quality assurance (QA) and treatment delivery tasks. METHODS AND MATERIALS: As part of a prospective institutional review board-approved study, 32 RTTs completed routine QA and treatment delivery tasks on clinical scenarios in a simulation laboratory. Participants, randomized to receive (n = 16) versus not receive (n = 16) simulation-based training had pre- and postintervention assessments of mental workload, situation awareness, and performance. We used linear regression models to compare the postassessment scores between the study groups while controlling for baseline scores. Mental workload was quantified subjectively using the NASA Task Load Index. Situation awareness was quantified subjectively using the situation awareness rating technique and objectively using the situation awareness global assessment technique. Performance was quantified based on procedural compliance (adherence to preset/standard QA timeout tasks) and error detection (detection and correction of embedded treatment planning errors). RESULTS: Simulation-based training intervention was associated with significant improvements in overall performance (P < .01), but had no significant impact on mental workload or subjective/objective quantifications of situation awareness. CONCLUSIONS: Simulation-based training might be an effective tool to improve RTT performance of QA-related tasks.

Validity of Simulation-Based Assessment for Accreditation Council for Graduate Medical Education Milestone Achievement.

INTRODUCTION: The Accreditation Council for Graduate Medical Education requires biannual evaluation of anesthesiology residents on 25 subcompetency milestones. Some milestone domains are particularly challenging to repeatedly and reliably observe during clinical care. Simulation-Based Milestones Assessment (SBMA) may help overcome these challenges. However, few studies have examined the external validation of simulation assessment scores (ie, the relationships between simulation-based assessment scores and other standard measures of ability) for milestones. This study analyzed whether SBMA scores (1) discriminate by postgraduate year, (2) improve over time, and (3) correlate with traditional measures of performance. METHODS: This is a retrospective analysis of 55 residents' SBMA data from 30 scenarios for two academic years. Each scenario was evaluated for time-in-training discrimination. Scenarios were then analyzed for SBMA scoring trends over time, and SBMA scores were compared with residents' clinical evaluations. RESULTS: Twenty-four SBMA scenarios discriminated by postgraduate year. Repeated measure analysis of variance showed statistically significant between-session score improvements (F (3, 54) = 17.79, P < 0.001). Pearson correlation coefficients demonstrated moderate to strong correlation between SBMA and clinical evaluations: January 2015 r = 0.67, P < 0.01 (n = 27); May 2015 r = 0.43, P = 0.09 (n = 17); November 2015 r = 0.70, P < 0.01 (n = 24); and April 2016 r = 70, P < 0.01 (n = 27). CONCLUSIONS: The associations between SBMA scores and experience level, time-in-training, and clinical performance evaluations provide evidence that SBMA may be used as metrics of residents' Accreditation Council for Graduate Medical Education milestone competencies.

A Descriptive Survey of Anesthesiology Residency Simulation Programs: How Are Programs Preparing Residents for the New American Board of Anesthesiology APPLIED Certification Examination?

BACKGROUND: Anesthesiology residency programs may need new simulation-based programs to prepare residents for the new Objective Structured Clinical Examination (OSCE) component of the American Board of Anesthesiology (ABA) Primary Certification process. The design of such programs may require significant resources, including faculty time, expertise, and funding, as are currently needed for structured oral examination (SOE) preparation. This survey analyzed the current state of US-based anesthesiology residency programs regarding simulation-based educational programming for SOE and OSCE preparation. METHODS: An online survey was distributed to every anesthesiology residency program director in the United States. The survey included 15 to 46 questions, depending on each respondent's answers. The survey queried current practices and future plans regarding resident preparation specifically for the ABA APPLIED examination, with emphasis on the OSCE. Descriptive statistics were summarized. χ and Fisher exact tests were used to test the differences in proportions across groups. Spearman rank correlation was used to examine the association between ordinal variables. RESULTS: The responding 66 programs (49%) were a representative sample of all anesthesiology residencies (N = 136) in terms of geographical location (χ P = .58). There was a low response rate from small programs that have 12 or fewer clinical anesthesia residents. Ninety-one percent (95% confidence interval [CI], 84%-95%) of responders agreed that it is the responsibility of the program to specifically prepare residents for primary certification, and most agreed that it is important to practice SOEs (94%; 95% CI, 88%-97%) and OSCEs (89%; 95% CI, 83%-94%). While 100% of respondents reported providing mock SOEs, only 31% (95% CI, 24%-40%) of respondents provided mock OSCE experiences. Of those without an OSCE program, 75% (95% CI, 64%-83%) reported plans to start one. The most common reasons for not having an OSCE program already in place, and the perceived challenges for implementing an OSCE program, were the same: lack of time (faculty and residents), expertise in OSCE development and assessment, and funding. CONCLUSIONS: The results provide data from residency programs for benchmarking their simulation curriculum and ABA APPLIED Examination preparation offerings. Despite agreement that residency programs should prepare residents for the ABA APPLIED Examination, many programs have yet to implement an OSCE preparation program, in part due to lack of financial resources, faculty expertise, and time. Additionally, in contrast to the SOE, the OSCE is a new format for ABA primary certification. As a result, the lack of consensus concerning preparation needs could be related to the amount information that is available regarding the examination content and assessment process.

Improving radiation oncology providers' workload and performance: Can simulation-based training help?

PURPOSE: To help with ongoing safety challenges in radiation therapy (RT), the objective of this research was to develop and assess the impact of a simulation-based training intervention on radiation oncology providers' workload and performance during treatment planning and quality assurance (QA) tasks. METHODS AND MATERIALS: Eighteen radiation oncology professionals completed routine treatment planning and QA tasks on 2 clinical scenarios in a simulation laboratory as part of a prospective institutional review board-approved study. Workload was measured at the end of each assessment/scenario using the NASA Task-Load Index. Performance was quantified based on procedural compliance (adherence to preset/standard QA tasks), time-to-scenario completion, and clinically relevant performance. Participants were then randomized to receive (vs not receive) simulation-based training intervention (eg, standardized feedback on workload and performance) and underwent repeat measurements of workload and performance. Pre- and postintervention changes in workload and performance from participants who received (vs did not receive) were compared using 2-way analysis of variance. RESULTS: Simulation-based training was associated with significant improvements in procedural compliance (P = .01) and increases in time-to-scenario completion (P < .01) but had no significant impact on subjective workload or clinically relevant performance. CONCLUSION: Simulation-based training may be a tool to improve procedural compliance of RT professionals and to acquire new skills and knowledge to proactively maintain RT professionals' preoccupation with patient safety.

Emergency Manuals Improved Novice Physician Performance During Simulated ICU Emergencies.

Background: Emergency manuals, which are safety essentials in non-medical high-reliability organizations (e.g., aviation), have recently gained acceptance in critical medical environments. Of the existing emergency manuals in anesthesiology, most are geared towards intraoperative settings. Additionally, most evidence supporting their efficacy focuses on the study of physicians with at least some meaningful experience as a physician. Our aim was to evaluate whether an emergency manual would improve the performance of novice physicians (post-graduate year [PGY] 1 or first year resident) in managing a critical event in the intensive care unit (ICU). Methods: PGY1 interns (n=41) were assessed on the management of a simulated critical event (unstable bradycardia) in the ICU. Participants underwent a group allocation process to either a control group (n=18) or an intervention group (emergency manual provided, n=23). The number of successfully executed treatment and diagnostic interventions completed was evaluated over a ten minute (600 seconds) simulation for each participant. Results: The participants using the emergency manual averaged 9.9/12 (83%) interventions, compared to an average of 7.1/12 (59%) interventions (p < 0.01) in the control group. Conclusions: The use of an emergency manual was associated with a significant improvement in critical event management by individual novice physicians in a simulated ICU patient (23% average increase).

A Smartphone-based Decision Support Tool Improves Test Performance Concerning Application of the Guidelines for Managing Regional Anesthesia in the Patient Receiving Antithrombotic or Thrombolytic Therapy.

BACKGROUND: The American Society of Regional Anesthesia and Pain Medicine (ASRA) consensus statement on regional anesthesia in the patient receiving antithrombotic or thrombolytic therapy is the standard for evaluation and management of these patients. The authors hypothesized that an electronic decision support tool (eDST) would improve test performance compared with native physician behavior concerning the application of this guideline. METHODS: Anesthesiology trainees and faculty at 8 institutions participated in a prospective, randomized trial in which they completed a 20-question test involving clinical scenarios related to the ASRA guidelines. The eDST group completed the test using an iOS app programmed to contain decision logic and content of the ASRA guidelines. The control group completed the test by using any resource in addition to the app. A generalized linear mixed-effects model was used to examine the effect of the intervention. RESULTS: After obtaining institutional review board's approval and informed consent, 259 participants were enrolled and randomized (eDST = 122; control = 137). The mean score was 92.4 ± 6.6% in the eDST group and 68.0 ± 15.8% in the control group (P < 0.001). eDST use increased the odds of selecting correct answers (7.8; 95% CI, 5.7 to 10.7). Most control group participants (63%) used some cognitive aid during the test, and they scored higher than those who tested from memory alone (76 ± 15% vs. 57 ± 18%, P < 0.001). There was no difference in time to completion of the test (P = 0.15) and no effect of training level (P = 0.56). CONCLUSIONS: eDST use improved application of the ASRA guidelines compared with the native clinician behavior in a testing environment.

Effect of a cognitive aid on adherence to perioperative assessment and management guidelines for the cardiac evaluation of noncardiac surgical patients.

BACKGROUND: The 2007 American College of Cardiologists/American Heart Association Guidelines on Perioperative Cardiac Evaluation and Care for Noncardiac Surgery is the standard for perioperative cardiac evaluation. Recent work has shown that residents and anesthesiologists do not apply these guidelines when tested. This research hypothesized that a decision support tool would improve adherence to this consensus guideline. METHODS: Anesthesiology residents at four training programs participated in an unblinded, prospective, randomized, cross-over trial in which they completed two tests covering clinical scenarios. One quiz was completed from memory and one with the aid of an electronic decision support tool. Performance was evaluated by overall score (% correct), number of incorrect answers with possibly increased cost or risk of care, and the amount of time required to complete the quizzes both with and without the cognitive aid. The primary outcome was the proportion of correct responses attributable to the use of the decision support tool. RESULTS: All anesthesiology residents at four institutions were recruited and 111 residents participated. Use of the decision support tool resulted in a 25% improvement in adherence to guidelines compared with memory alone (P < 0.0001), and participants made 77% fewer incorrect responses that would have resulted in increased costs. Use of the tool was associated with a 3.4-min increase in time to complete the test (P < 0.001). CONCLUSIONS: Use of an electronic decision support tool significantly improved adherence to the guidelines as compared with memory alone. The decision support tool also prevented inappropriate management steps possibly associated with increased healthcare costs.

Threat and error management for anesthesiologists: a predictive risk taxonomy.

PURPOSE OF REVIEW: Patient care in the operating room is a dynamic interaction that requires cooperation among team members and reliance upon sophisticated technology. Most human factors research in medicine has been focused on analyzing errors and implementing system-wide changes to prevent them from recurring. We describe a set of techniques that has been used successfully by the aviation industry to analyze errors and adverse events and explain how these techniques can be applied to patient care. RECENT FINDINGS: Threat and error management (TEM) describes adverse events in terms of risks or challenges that are present in an operational environment (threats) and the actions of specific personnel that potentiate or exacerbate those threats (errors). TEM is a technique widely used in aviation, and can be adapted for the use in a medical setting to predict high-risk situations and prevent errors in the perioperative period. A threat taxonomy is a novel way of classifying and predicting the hazards that can occur in the operating room. TEM can be used to identify error-producing situations, analyze adverse events, and design training scenarios. SUMMARY: TEM offers a multifaceted strategy for identifying hazards, reducing errors, and training physicians. A threat taxonomy may improve analysis of critical events with subsequent development of specific interventions, and may also serve as a framework for training programs in risk mitigation.

Decision-making and safety in anesthesiology.

PURPOSE OF REVIEW: Anesthesiologists work in a complex environment that is intolerant of errors. Cognitive errors, or errors in thought processes, are mistakes that a clinician makes despite 'knowing better'. Several new studies provide a better understanding of how to manage risk while making better decisions. RECENT FINDINGS: Heuristics, or mental shortcuts, allow physicians to make decisions quickly and efficiently but may be responsible for errors in diagnosis and treatment. Using simple 'decision-making checklists' can help healthcare providers to make the correct decisions by monitoring their own thought processes. Anesthesiologists can adopt risk assessment tools that were originally developed for use by pilots to determine the hazards associated with a particular clinical management strategy. SUMMARY: Effective decision-making and risk management reduce the risk of adverse events in the operating room. This article proposes several new decision-making and risk assessment tools for use in the operating room.