Reliability of prehospital patient classification in helicopter emergency medical service missions.

BACKGROUND: Several scores and codes are used in prehospital clinical quality registries but little is known of their reliability. The aim of this study is to evaluate the inter-rater reliability of the American Society of Anesthesiologists physical status (ASA-PS) classification system, HEMS benefit score (HBS), International Classification of Primary Care, second edition (ICPC-2) and Eastern Cooperative Oncology Group (ECOG) performance status in a helicopter emergency medical service (HEMS) clinical quality registry (CQR). METHODS: All physicians and paramedics working in HEMS in Finland and responsible for patient registration were asked to participate in this study. The participants entered data of six written fictional missions in the national CQR. The inter-rater reliability of the ASA-PS, HBS, ICPC-2 and ECOG were evaluated using an overall agreement and free-marginal multi-rater kappa (Κfree). RESULTS: All 59 Finnish HEMS physicians and paramedics were invited to participate in this study, of which 43 responded and 16 did not answer. One participant was excluded due to unfinished data entering. ASA-PS had an overall agreement of 40.2% and Κfree of 0.28 in this study. HBS had an overall agreement of 44.7% and Κfree of 0.39. ICPC-2 coding had an overall agreement of 51.5% and Κfree of 0.47. ECOG had an overall agreement of 49.6% and Κfree of 0.40. CONCLUSION: This study suggests a marked inter-rater unreliability in prehospital patient scoring and coding even in a relatively uniform group of practitioners working in a highly focused environment. This indicates that the scores and codes should be specifically designed or adapted for prehospital use, and the users should be provided with clear and thorough instructions on how to use them.

The reliability and accuracy of operational system data in a nationwide helicopter emergency medical services mission database.

AIM: The aim of this study was to evaluate the reliability and accuracy of documentation in FinnHEMS database, which is a nationwide helicopter emergency service (HEMS) clinical quality registry. METHODS: This is a nationwide study based on written fictional clinical scenarios. Study subjects were HEMS physicians and paramedics, who filled in the clinical quality registry based on the clinical scenarios. The inter-rater -reliability of the collected data was analyzed with percent agreement and free-marginal multi-rater kappa. RESULTS: Dispatch coding had a percent agreement of 91% and free-marginal multi-rater kappa value of 0.83. Coding for transportation or mission cancellation resulted in an agreement of 84% and free-marginal kappa value of 0.68. An agreement of 82% and a kappa value of 0.73 for dispatcher coding was found. Mission end, arrival at hospital and HEMS unit dispatch -times had agreements from 80 to 85% and kappa values from 0.61 to 0.73. The emergency call to dispatch centre time had an agreement of 71% and kappa value of 0.56. The documentation of pain had an agreement of 73% on both the first and second measurements. All other vital parameters had less than 70% agreement and 0.40 kappa value in the first measurement. The documentation of secondary vital parameter measurements resulted in agreements from 72 to 91% and kappa values from 0.43 to 0.64. CONCLUSION: Data from HEMS operations can be gathered reliably in a national clinical quality registry. This study revealed some inaccuracies in data registration and data quality, which are important to detect to improve the overall reliability and validity of the HEMS clinical quality register.

Development of an instrument for the evaluation of advanced life support performance.

BACKGROUND: Assessing advanced life support (ALS) competence requires validated instruments. Existing instruments include aspects of technical skills (TS), non-technical skills (NTS) or both, but one instrument for detailed assessment that suits all resuscitation situations is lacking. This study aimed to develop an instrument for the evaluation of the overall ALS performance of the whole team. METHODS: This instrument development study had four phases. First, we reviewed literature and resuscitation guidelines to explore items to include in the instrument. Thereafter, we interviewed resuscitation team professionals (n = 66), using the critical incident technique, to determine possible additional aspects associated with the performance of ALS. Second, we developed an instrument based on the findings. Third, we used an expert panel (n = 20) to assess the validity of the developed instrument. Finally, we revised the instrument based on the experts' comments and tested it with six experts who evaluated 22 video recorded resuscitations. RESULTS: The final version of the developed instrument had 69 items divided into adherence to guidelines (28 items), clinical decision-making (5 items), workload management (12 items), team behaviour (8 items), information management (6 items), patient integrity and consideration of laymen (4 items) and work routines (6 items). The Cronbach's α values were good, and strong correlations between the overall performance and the instrument were observed. CONCLUSION: The instrument may be useful for detailed assessment of the team's overall performance, but the numerous items make the use demanding. The instrument is still under development, and more research is needed to determine its psychometric properties.

Evaluating helicopter emergency medical missions: a reliability study of the HEMS benefit and NACA scores.

BACKGROUND: The benefits of the Helicopter Emergency Medical Service (HEMS) and dispatch accuracy are continuously debated, and a widely accepted score to measure the benefits of the mission is lacking. The HEMS Benefit Score (HBS) has been used in Finnish helicopter emergency medical services, but studies are lacking. The National Advisory Committee for Aeronautics (NACA) score is widely used to measure the severity of illness or injury in the pre-hospital setting, but it has many critics due to its subjectivity. We investigated the inter-rater and rater-against-reference reliability of these scores. METHODS: Twenty-five fictional HEMS missions were created by an expert panel. A total of 22 pre-hospital physicians were recruited to participate in the study from two different HEMS bases. The participants received written instructions on the use of the scores. Intraclass correlation coefficients (ICCs) and mean differences between rater-against-reference values were calculated. RESULTS: A total of 17 physicians participated in the study. The ICC was 0.70 (95% CI 0.57-0.83) for the HBS and 0.65 (95% CI 0.51-0.79) for the NACA score. Mean differences between references and raters were -0.09 (SD 0.72) for the HBS and 0.28 (SD 0.61) for the NACA score, indicating that raters scored some lower NACA values than reference values formed by an expert panel. CONCLUSION: The HBS and NACA score had substantial inter-rater reliability. In addition, the rater-against-reference values were acceptable, though large differences were observed between individual raters and references in some clinical cases.