Development and Validation of a Trauma-Informed Care Communication Skills Assessment Tool.

BACKGROUND: Trauma-informed care (TIC) is growing in medical education as healthcare systems recognize trauma's impact on health outcomes. TIC acknowledges and responds to the effects of trauma on physical, psychological, and emotional health. As TIC trainings are developed and delivered to healthcare professionals across the learner continuum, curricula need evaluation beyond learner satisfaction and knowledge to better assess changes in skills. We developed the Gap Kalamazoo Communication Skills Assessment Form for Trauma-Informed Care (GKCSAF-TIC) to evaluate pediatric trainees' communication skills in TIC. OBJECTIVE: Describe the development and validity evidence of the GKCSAF-TIC in assessing pediatric residents' TIC skills during standardized patient encounters. METHODS: We developed and implemented the TIC communication skills assessment tool in a one-year prospective cohort study involving pediatric residents. We conducted simulated patient encounters conducted before and after TIC training, with two pediatric faculty attendings assessing each encounter. We gathered validity evidence using Messick's framework, focusing on content, response process, internal structure, and relationship with other variables. RESULTS: We analyzed 57 standardized patient encounters with 33 pediatric interns, including 23 pre-post matched pairs. The development process and rater training supported content and response process validity. Internal consistency, measured by Cronbach's alpha, ranged from 0.93 to 0.96, while inter-rater reliability, measured by intraclass correlations, ranged from 0.80 to 0.83. There was a significant improvement in scores from pre-training to post-training (3.7/5 to 4.05/5; p < 0.05). CONCLUSION: The GKCSAF-TIC demonstrated strong preliminary validity and offers educators a valuable means to assess and provide formative feedback to pediatric trainees about TIC. WHAT'S NEW?: Pediatric trainees are learning about trauma-informed care and need formative feedback on their practice of TIC in the clinical setting. The GKCSAF-TIC can be used to evaluate and provide formative feedback to trainees about their communication skills in providing trauma-informed care.

Developing a generalizable pediatric ECMO emergency checklist for clinical specialist: Progress and challenges.

INTRODUCTION: Extracorporeal membrane oxygenation (ECMO) emergencies require skilled clinical specialist (CS) who manage ECMO circuits. While tools for assessing CS skills have been published, there is significant variation in protocols and circuit design. This study aims to further develop these checklists to produce a generalizable ECMO skill assessment with adequate validity evidence to support its use as a summative evaluation tool. METHODS: An initial survey determined variation in ECMO circuit components and configurations, and the original checklists and simulations were altered through a modified Delphi process. The finalized checklist and simulation were then assessed for validity and reliability. Three trained raters assessed ten simulations from five subjects at two different institutions using two circuit designs. Data analysis was conducted using a fully crossed subject x rater x circuit generalizability (G) and decision (D) study. RESULTS: The G-study coefficient was 0 with 0% variance across subject and circuit. The greatest variance was among raters (28.7%). Significant variance was also associated with the subject and pump type relationship (27%). CONCLUSION: Despite the rigorous process used to modify the assessment, generalizability was poor. Lack of familiarity with center-specific circuit design played a key role. Future endeavors in ECMO skill assessment should focus either on developing and validating site-specific tools or standardizing circuit designs.

Society for Simulation in Healthcare Guidelines for Simulation Training.

BACKGROUND: Simulation has become a staple in the training of healthcare professionals with accumulating evidence on its effectiveness. However, guidelines for optimal methods of simulation training do not currently exist. METHODS: Systematic reviews of the literature on 16 identified key questions were conducted and expert panel consensus recommendations determined using the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) methodology. OBJECTIVE: These evidence-based guidelines from the Society for Simulation in Healthcare intend to support healthcare professionals in decisions on the most effective methods for simulation training in healthcare. RESULTS: Twenty recommendations on 16 questions were determined using GRADE. Four expert recommendations were also provided. CONCLUSIONS: The first evidence-based guidelines for simulation training are provided to guide instructors and learners on the most effective use of simulation in healthcare.

Educational and Patient Care Impacts of In Situ Simulation in Healthcare: A Systematic Review.

ABSTRACT: This systematic review was performed to assess the effectiveness of in situ simulation education. We searched databases including MEDLINE and Embase for studies comparing in situ simulation with other educational approaches. Two reviewers screened articles and extracted information. Sixty-two articles met inclusion criteria, of which 24 were synthesized quantitatively using random effects meta-analysis. When compared with current educational practices alone, the addition of in situ simulation to these practices was associated with small improvements in clinical outcomes, including mortality [odds ratio, 0.66; 95% confidence interval (CI), 0.55 to 0.78], care metrics (standardized mean difference, -0.34; 95% CI, -0.45 to -0.21), and nontechnical skills (standardized mean difference, -0.52; 95% CI, -0.99 to -0.05). Comparisons between in situ and traditional simulation showed mixed learner preference and knowledge improvement between groups, while technical skills showed improvement attributable to in situ simulation. In summary, available evidence suggests that adding in situ simulation to current educational practices may improve patient mortality and morbidity.

Barriers and Facilitators to Implementing Simulation-Based Translational Research: A Qualitative Study.

INTRODUCTION: Translational research has been identified as a research priority for the National Institutes of Health (NIH) and the Society for Simulation in Healthcare (SSH). Despite a larger focus on translational research in recent years, the overall amount of simulation-based translational research remains low. Greater understanding of how to approach translational simulation is required to inform novice simulation and translational researchers. This study sought to answer the following research questions: How do simulation experts describe the barriers and facilitators to implementing translational simulation programs? How do simulation experts describe their various approaches to implementing translational simulation programs? What recommendations do simulation experts describe for overcoming barriers to implementing translational simulation programs? METHODS: A qualitative instrumental case study was used to elicit multiple instances of translational simulation research to gather an in-depth description from study participants. Three data sources were used: documents, semistructured interviews, and a focus group. RESULTS: Data analyses revealed 5 major themes: clarifying goals and definitions, special considerations, social networking, research, and factors external to the simulation program. CONCLUSIONS: Key findings include a lack of a standardized definitions for translational simulation and simulation-based translational research, the challenge of demonstrating the value of translational simulation, and the need for translational simulation programs to be integrated into departmental quality, patient safety, and risk management work. The findings and advice from the experts in this research can assist new researchers or those encountering challenges in implementing translational simulations.

Setting an Agenda: Results of a Consensus Process on Research Directions in Distance Simulation.

BACKGROUND: The COVID-19 pandemic forced rapid implementation and refinement of distance simulation methodologies in which participants and/or facilitators are not physically colocated. A review of the distance simulation literature showed that heterogeneity in many areas (including nomenclature, methodology, and outcomes) limited the ability to identify best practice. In April 2020, the Healthcare Distance Simulation Collaboration was formed with the goal of addressing these issues. The aim of this study was to identify future research priorities in the field of distance simulation using data derived from this summit. METHODS: This study analyzed textual data gathered during the consensus process conducted at the inaugural Healthcare Distance Simulation Summit to explore participant perceptions of the most pressing research questions regarding distance simulation. Participants discussed education and patient safety standards, simulation facilitators and barriers, and research priorities. Data were qualitatively analyzed using an explicitly constructivist thematic analysis approach, resulting in the creation of a theoretical framework. RESULTS: Our sample included 302 participants who represented 29 countries. We identified 42 codes clustered within 4 themes concerning key areas in which further research into distance simulation is needed: (1) safety and acceptability, (2) educational/foundational considerations, (3) impact, and (4) areas of ongoing exploration. Within each theme, pertinent research questions were identified and categorized. CONCLUSIONS: Distance simulation presents several challenges and opportunities. Research around best practices, including educational foundation and psychological safety, are especially important as is the need to determine outcomes and long-term effects of this emerging field.

The Concise Assessment of Leader Management Tool: Evaluation of Healthcare Provider Leadership During Real-Life Pediatric Emergencies.

INTRODUCTION: Resuscitation events in pediatric critical and emergency care are high risk, and strong leadership is an important component of an effective response. The Concise Assessment of Leadership Management (CALM) tool, designed to assess the strength of leadership skills during pediatric crises, has shown promising validity and reliability in simulated settings. The objective of this study was to generate further validity and reliability evidence for the CALM by applying it to real-life emergency events. METHODS: A prospective, video-based study was conducted in an academic pediatric emergency department. Three reviewers independently applied the CALM tool to the assessment of pediatric emergency department physicians as they led both a cardiac arrest and a sepsis event. Time to critical event (epinephrine, fluid, and antibiotic administration) was collected via video review. Based on Kane's framework, we conducted fully crossed, person × event × rater generalizability (G) and decision (D) studies. Interrater reliability was calculated using Gwet AC 2 and intraclass correlation coefficients. Time to critical events was correlated with CALM scores using Spearman coefficient. RESULTS: Nine team leaders were assessed in their leadership of 2 resuscitations each. The G coefficient was 0.68, with 26% subject variance, 20% rater variance, and no case variance. Thirty-three percent of the variance (33%) was attributed to third-order interactions and unknown factors. Gwet AC 2 was 0.3 and intraclass correlation was 0.58. The CALM score and time to epinephrine correlated at -0.79 ( P = 0.01). The CALM score and time to fluid administration correlated at -0.181 ( P = 0.64). CONCLUSIONS: This study provides additional validity evidence for the CALM tool's use in this context if used with multiple raters, aligning with data from the previous simulation-based CALM validity study. Further development may improve reliability. It also serves as an exemplar of the rigors of conducting validity work within medical simulation.

Development and Validation of a Point-of-Care-Ultrasound Image Quality Assessment Tool: The POCUS IQ Scale.

OBJECTIVES: We aimed to develop a standardized scoring tool to measure point-of-care ultrasound (POCUS) image quality and to determine validity evidence for its use to assess lung ultrasound image quality. METHODS: The POCUS Image Quality (POCUS IQ) scale was developed by POCUS-trained physicians to assess sonographers' image acquisition skills by evaluating image quality for any POCUS application. The scale was piloted using lung images of healthy standardized patients acquired by three expert sonographers compared to three novices before and after training. All images (experts, novices pre-training, novices post-training) were scored on the POCUS IQ scale by three blinded POCUS-trained physicians. Reliability was assessed with fully-crossed generalizability and decision studies. Validity was assessed using Messick's framework. RESULTS: Content validity was supported by the tool's development process of literature review, expert consensus, and pilot testing. Response process was supported by reviewer training and the blinded scoring process. Relation to other variables was supported by scores relating to sonographer experience: median expert score = 10.5/14 (IQR: 4), median novice pre-training score = 6/14 (IQR: 2.25), and novices' improvement after training (median post-training score = 12/14, IQR: 3.25). Internal structure was supported by internal consistency data (coefficient alpha = 0.84, omega coefficient = 0.91) and the generalizability study showing the main contributor to score variability was the sonographer (51%). The G-coefficient was 0.89, suggesting very good internal structure, however, Gwet's AC2  was 0.5, indicating moderate interrater reliability. The D study projected a minimum of 1 reviewer and 2 patients are needed for good psychometric reliability. CONCLUSIONS: The POCUS scale has good preliminary validity evidence as an assessment tool for lung POCUS image acquisition skills. Further studies are needed to demonstrate its utility for other POCUS applications and as a feedback tool for POCUS learners.

Characterizing preferred terms for geographically distant simulations: distance, remote and telesimulation.

Background: Simulationists lack standard terms to describe new practices accommodating pandemic restrictions. A standard language around these new simulation practices allows ease of communication among simulationists in various settings. Methods: We explored consensus terminology for simulation accommodating geographic separation of participants, facilitators or equipment. We used an iterative process with participants of two simulation conferences, with small groups and survey ranking. Results: Small groups (n = 121) and survey ranking (n = 54) were used with distance, remote, and telesimulation as leading terms. Each was favored by a third of the participants without consensus. Conclusion: This research has deepened our understanding of how simulationists interpret this terminology, including the derived themes: (1) physical distance/separation, (2) overarching nature of the term and (3) implications from existing terms. We further deepen the conceptual discussion on healthcare simulation aligned with the search of the terminologies. We propose there are nuances that prevent an early consensus recommendation. A taxonomy of descriptors specifying the conduct of distance, remote and telesimulation is preferred.

Preparing and Presenting Validation Studies: A Guide for the Perplexed.

SUMMARY STATEMENT: Simulated environments are frequently used for learner assessment, and a wide array of assessment instruments have been created to assist with this process. It is important, therefore, that clear, compelling evidence for the validity of these assessments be established. Contemporary theory recognizes instrument validity as a unified construct that links a construct to be assessed with a population, an environment of assessment, and a decision to be made using the scores. In this article, we present a primer on 2 current frameworks (Messick and Kane), define the elements of each, present a rubric that can be used by potential authors to structure their work, and offer examples of published studies showing how each framework has been successfully used to make a validity argument. We offer this with the goal of improving the quality of validity-related publications, thereby advancing the quality of assessment in healthcare simulation.

The Different Challenges in Being an Adult Versus a Pediatric Intensivist.

There is little current research comparing stress, burnout, and resilience in pediatric and adult intensive care practitioners. This article analyzes data derived from a 2018 qualitative study of burnout and resilience among ICU providers to explore differences that may exist between the pediatric and adult domains of practice. DESIGN: This study represents a thematic subanalysis of textual data derived from a larger qualitative study of ICU provider burnout and resilience. SETTING: Six international critical care units (Australia, Israel, United States). SUBJECTS: Physicians working at the above sites who had been practicing as intensivists for a minimum period of 4 years. INTERVENTIONS: None. MEASUREMENTS AND MAIN RESULTS: Data were collected using a semistructured interview process, and resulting transcripts were analyzed using postpositivist framework analysis. A secondary analysis was then performed separately on pediatric and adult datasets using the initial coding framework as a template. Three themes related to perceived differences were noted: differences in the patient characteristics within both cohorts, differences in the relationships between staff and family, and personal biases of individual intensivists. Pediatric and adult practitioners differed in their perceptions of the patient's perceived responsibility for their illness. Emotional responses to the stressor of child abuse (particularly as they related to clinician-family relationships) also differed. The stress of dealing with family expectations of patient survival even in dire circumstances was unique to the pediatric environment. Both pediatric and adult practitioners commented on the perceived difficulty of assuming the opposite role. Differences in life expectancy and mortality rate were significant factors in this. CONCLUSIONS: Although similar stressors exist within each group, meaningful differences in how these are perceived and personally processed by individual clinicians exist. Better understanding of these differences will assist attempts to enhance the resilience and provide career guidance to aspiring intensive care clinicians.

Current Research Priorities in Healthcare Simulation: Results of a Delphi Survey.

INTRODUCTION: In 2011 and 2017, the Society for Simulation in Healthcare Research Committee convened summits to develop a forward-thinking agenda for simulation research. After the second summit, the Society for Simulation in Healthcare Research Committee sought expert opinion on the most important research questions in healthcare simulation. This study used systematic methodology to develop a prioritized research agenda for healthcare simulation. METHODS: A modified Delphi approach was conducted in 3 survey rounds. During round 1, individuals with expertise in healthcare simulation research were recruited to submit important research questions. Submitted questions were reviewed and duplicates were removed. Remaining questions were synthesized into a concise, high-level list for further rating. During round 2, these reformulated questions were distributed to the same experts who ranked their importance on a 5-point Likert scale. Average question importance ratings were calculated and shared during round 3, and a final vote was taken to identify the highest priority items. RESULTS: Seventeen experts submitted 74 questions, which were reduced to 21 reformulated items. Variability in expert responses decreased significantly across survey rounds, indicating that consensus had been achieved. The top 3 research question identified by the experts were related to (1) the impact of system level simulation interventions on system efficiency, patient safety, and patient outcomes; (2) the return on investment of simulation for healthcare systems, and (3) whether a dose-response relationship exists between simulation training and performance/patient outcomes. CONCLUSIONS: The agenda developed in this study may help guide and focus researcher efforts and funding agency decisions, ultimately helping advance the field.

From Concept to Publication: Effectiveness of the International Network for Simulation-Based Pediatric Innovation, Research, and Education Project Development Process at Generating Simulation Scholarship.

BACKGROUND: As simulation matures, it is critical to develop pathways for researchers. A recent analysis, however, demonstrates a low conversion rate between abstract and peer-reviewed journal publication in our field. The International Network for Simulation-based Pediatric Innovation, Research, and Education has used the ALERT Presentation process for the past decade as a means of accelerating research. In this study, we analyze the scholarly products attributable to ALERT Presentations. METHODS: Surveys were distributed to all International Network for Simulation-based Pediatric Innovation, Research, and Education Advanced Look Exploratory Research Template (ALERT) Presentation first authors from January 2011 through January 2020. Presenters were asked to provide information on abstracts, grants, journal publications, and book chapters related to their ALERT Presentation, as well as basic demographic information. A structured literature search was conducted for those ALERT Presentations whose authors did not return a survey. The resulting database was descriptively analyzed, and statistical correlations between demographic variables and scholarship were examined. RESULTS: One hundred sixty-five new ALERT presentations were presented over 10 years. We identified 361 associated scholarly works (170 conference abstracts, 125 peer-reviewed journal publications, 65 grants, and 1 book chapter). Sixty-one percent (101 of 165) of ALERT Presentations produced at least 1 item of scholarship, and 59% (34 of 58) of ALERT Presentations that resulted in at least 1 abstract also led to at least 1 peer-reviewed journal article. Presenter gender was associated with likelihood of journal publication. CONCLUSIONS: The ALERT Presentation process is an effective approach for facilitating the development of projects that result in disseminated scholarship. Wider adoption may benefit other simulation and education research networks.

A Decade Later-Progress and Next Steps for Pediatric Simulation Research.

SUMMARY STATEMENT: A decade ago, at the time of formation of the International Network for Pediatric Simulation-based Innovation, Research, and Education, the group embarked on a consensus building exercise. The goal was to forecast the facilitators and barriers to growth and maturity of science in the field of pediatric simulation-based research. This exercise produced 6 domains critical to progress in the field: (1) prioritization, (2) research methodology and outcomes, (3) academic collaboration, (4) integration/implementation/sustainability, (5) technology, and (6) resources/support/advocacy. This article reflects on and summarizes a decade of progress in the field of pediatric simulation research and suggests next steps in each domain as we look forward, including lessons learned by our collaborative grass roots network that can be used to accelerate research efforts in other domains within healthcare simulation science.

Simulathon 2020: Integrating Simulation Period Prevalence Methodology Into the COVID-19 Disaster Management Cycle in India.

SUMMARY STATEMENT: The disaster management cycle is an accepted model that encompasses preparation for and recovery from large-scale disasters. Over the past decade, India's Pediatric Simulation Training and Research Society has developed a national-scale simulation delivery platform, termed the Simulathon , with a period prevalence methodology that integrates with core aspects of this model. As an exemplar of the effectiveness of this approach, we describe the development, implementation, and outcomes of the 2020 Simulathon, conducted from April 20 to May 20 in response to the nascent COVID-19 pandemic disaster. We conclude by discussing how aspects of the COVID-19 Simulathon enabled us to address key aspects of the disaster management cycle, as well as challenges that we encountered. We present a roadmap by which other simulation programs in low- and middle-income countries could enact a similar process.

Attention-Deficit/Hyperactivity Disorder Practice Patterns: A Survey of Kentucky Pediatric Providers.

OBJECTIVE: Kentucky ranks among the highest in the nation for attention-deficit/hyperactivity disorder (ADHD) prevalence in children aged 4 to 17 years. In 2011, the American Academy of Pediatrics (AAP) released a clinical practice guideline based on the DSM-IV. A guideline revision based on the Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM-5) was released in October 2019. In this study, we assess and describe pediatric providers' ADHD practices using the 2011 guideline and DSM-5 diagnostic criteria. METHODS: This was a cross-sectional, survey-based descriptive study. Kentucky Chapter of the AAP (KY AAP) members were anonymously surveyed. The results were examined for trends in routine practice. RESULTS: Fifty-eight general pediatricians and pediatric residents responded to the survey, yielding a 38% (58/154) response rate. Among respondents performing routine diagnosis of ADHD (N = 51), 73% (37/51) used DSM-5 criteria. Most providers usually or always initially assessed for coexisting behavioral conditions (96%; 49/51), developmental conditions (78%; 39/51), and adverse childhood experiences (73%; 37/51). Among respondents performing routine management of ADHD (N = 55), only 11% (6/55) of respondents indicated that they titrated stimulant medications every 3 to 7 days. After initiation of medication, 78% of providers scheduled a follow-up visit within 2 to 4 weeks. During subsequent visits, only half indicated discussing behavioral interventions, screening for coexisting conditions, and reviewing follow-up teacher-rated ADHD scales. CONCLUSION: Pediatricians in the KY AAP adhere to the DSM-5 criteria for diagnosing ADHD. Pediatric providers' practices would benefit from education in improvements in pharmacotherapy titration, surveillance of coexisting conditions associated with ADHD, discussion of psychosocial interventions, and school support strategies.

Standardized Vaccine-Hesitant Patients in the Assessment of the Effectiveness of Vaccine Communication Training.

OBJECTIVES: To determine if training residents in a structured communication method elicits specific behaviors in a laboratory model of interaction with vaccine-hesitant parents. STUDY DESIGN: Standardized patients portraying vaccine-hesitant parents were used to assess the effectiveness of training in the Announce, Inquire, Mirror, Secure (AIMS) Method for Healthy Conversations. Blinded pediatric residents were pseudorandomized to receive AIMS or control training and underwent pre- and post-training encounters with blinded standardized patients. Encounters were assessed by blinded raters using a novel tool. Participant confidence and standardized patient evaluations of the participants' general communication skills were assessed. RESULTS: Ratings were available for 27 AIMS and 26 control participants. Statistically significant increases in post-training scores (maximum = 30) were detected in AIMS, but not in control, participants (median, 21.3 [IQR, 19.8-24.8] vs 18.8 [IQR, 16.9-20.9]; P < .001). Elements (maximum score = 6) with significant increases were Inquire (0.67 [IQR, 0-1.76] vs -0.33 [IQR, -0.67 to 0.33]; P < .001); Mirror (1.33 [IQR, 0 to 2] vs -0.33 [IQR, -0.92 to 0]; P < .001) and Secure (0.33 [IQR, 0 to 1.67] vs -0.17 [IQR, -0.67 to 0.33]; P = .017). Self-confidence increased equally in both groups. Standardized patients did not detect a difference in communication skills after training and between groups. Internal consistency and inter-rater reliability of the assessment tool were modest. CONCLUSIONS: Standardized patients proved useful in studying the effectiveness of structured communication training, but may have been limited in their ability to perceive a difference between groups owing to the predetermined encounter outcome of vaccine refusal. AIMS training should be studied in real-world scenarios to determine if it impacts vaccine acceptance.

Recommendations and Guidelines for the Use of Simulation to Address Structural Racism and Implicit Bias.

SUMMARY STATEMENT: Simulation-based education is a particularly germane strategy for addressing the difficult topic of racism and implicit bias due to its immersive nature and the paradigm of structured debriefing. Researchers have proposed actionable frameworks for implicit bias education, particularly outlining the need to shift from recognition to transformation, with the goal of changing discriminatory behaviors and policies. As simulation educators tasked with training health care professionals, we have an opportunity to meet this need for transformation. Simulation can shift behaviors, but missteps in design and implementation when used to address implicit bias can also lead to negative outcomes. The focus of this article is to provide recommendations to consider when designing simulation-based education to specifically address racism and implicit bias.