Development and Evaluation of Patient Safety Interventions: Perspectives of Operational Safety Leaders and Patient Safety Organizations.

OBJECTIVES: The purpose of this study is to understand how patient safety professionals from healthcare facilities and patient safety organizations develop patient safety interventions and the resources used to support intervention development. METHODS: Semistructured interviews were conducted with patient safety professionals at nine healthcare facilities and nine patient safety organizations. Interview data were qualitatively analyzed, and findings were organized by the following: patient safety solutions and interventions, use of external databases, and evaluation of patient safety solutions. RESULTS: Development of patient safety interventions across healthcare facilities and patient safety organizations was similar and included literature searches, internal brainstorming, and interviews. Nearly all patient safety professionals at healthcare facilities reported contacting colleagues at other healthcare facilities to learn about similar safety issues and potential interventions. Additionally, less than half of patient safety professionals at healthcare facilities and patient safety organizations interviewed report data to publicly available patient safety databases. Finally, most patient safety professionals at healthcare facilities and patient safety organizations stated that they evaluate the effectiveness of patient safety interventions; however, they mentioned methods that may be less rigorous including audits, self-reporting, and subjective judgment. CONCLUSIONS: Patient safety professionals often utilize similar methods and resources to develop and evaluate patient safety interventions; however, many of these efforts are not coordinated across healthcare organizations and could benefit from working collectively in a systematic fashion. Additionally, healthcare facilities and patient safety organizations face similar challenges and there are several opportunities for optimization on a national level that may improve patient safety.

A natural language processing approach to categorise contributing factors from patient safety event reports.

OBJECTIVES: The objective of this study was to explore the use of natural language processing (NLP) algorithm to categorise contributing factors from patient safety event (PSE). Contributing factors are elements in the healthcare process (eg, communication failures) that instigate an event or allow an event to occur. Contributing factors can be used to further investigate why safety events occurred. METHODS: We used 10 years of self-reported PSE reports from a multihospital healthcare system in the USA. Reports were first selected by event date. We calculated χ2 values for each ngram in the bag-of-words then selected N ngrams with the highest χ2 values. Then, PSE reports were filtered to only include the sentences containing the selected ngrams. Such sentences were called information-rich sentences. We compared two feature extraction techniques from free-text data: (1) baseline bag-of-words features and (2) features from information-rich sentences. Three machine learning algorithms were used to categorise five contributing factors representing sociotechnical errors: communication/hand-off failure, technology issue, policy/procedure issue, distractions/interruptions and lapse/slip. We trained 15 binary classifiers (five contributing factors * three machine learning models). The models' performances were evaluated according to the area under the precision-recall curve (AUPRC), precision, recall, and F1-score. RESULTS: Applying the information-rich sentence selection algorithm boosted the contributing factor categorisation performance. Comparing the AUPRCs, the proposed NLP approach improved the categorisation performance of two and achieved comparable results with baseline in categorising three contributing factors. CONCLUSIONS: Information-rich sentence selection can be incorporated to extract the sentences in free-text event narratives in which the contributing factor information is embedded.

"Are we there yet?" Ten persistent hazards and inefficiencies with the use of medication administration technology from the perspective of practicing nurses.

OBJECTIVES: (1) Characterize persistent hazards and inefficiencies in inpatient medication administration; (2) Explore cognitive attributes of medication administration tasks; and (3) Discuss strategies to reduce medication administration technology-related hazards. MATERIALS AND METHODS: Interviews were conducted with 32 nurses practicing at 2 urban, eastern and western US health systems. Qualitative analysis using inductive and deductive coding included consensus discussion, iterative review, and coding structure revision. We abstracted hazards and inefficiencies through the lens of risks to patient safety and the cognitive perception-action cycle (PAC). RESULTS: Persistent safety hazards and inefficiencies related to MAT organized around the PAC cycle included: (1) Compatibility constraints create information silos; (2) Missing action cues; (3) Intermittent communication flow between safety monitoring systems and nurses; (4) Occlusion of important alerts by other, less helpful alerts; (5) Dispersed information: Information required for tasks is not collocated; (6) Inconsistent data organization: Mismatch of the display and the user's mental model; (7) Hidden medication administration technologies (MAT) limitations: Inaccurate beliefs about MAT functionality contribute to overreliance on the technology; (8) Software rigidity causes workarounds; (9) Cumbersome dependencies between technology and the physical environment; and (10) Technology breakdowns require adaptive actions. DISCUSSION: Errors might persist in medication administration despite successful Bar Code Medication Administration and Electronic Medication Administration Record deployment for reducing errors. Opportunities to improve MAT require a deeper understanding of high-level reasoning in medication administration, including control over the information space, collaboration tools, and decision support. CONCLUSION: Future medication administration technology should consider a deeper understanding of nursing knowledge work for medication administration.

A Systematic Review of Quantitative Methods for Evaluating Electronic Medication Administration Record and Bar-Coded Medication Administration Usability.

BACKGROUND: Although electronic medication administration records (eMARs) and bar-coded medication administration (BCMA) have improved medication safety, poor usability of these technologies can increase patient safety risks. OBJECTIVES: The objective of our systematic review was to identify the impact of eMAR and BCMA design on usability, operationalized as efficiency, effectiveness, and satisfaction. METHODS: We retrieved peer-reviewed journal articles on BCMA and eMAR quantitative usability measures from PsycInfo and MEDLINE (1946-August 20, 2019), and EMBASE (1976-October 23, 2019). Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, we screened articles, extracted and categorized data into the usability categories of effectiveness, efficiency, and satisfaction, and evaluated article quality. RESULTS: We identified 1,922 articles and extracted data from 41 articles. Twenty-four articles (58.5%) investigated BCMA only, 10 (24.4%) eMAR only, and seven (17.1%) both BCMA and eMAR. Twenty-four articles (58.5%) measured effectiveness, 8 (19.5%) efficiency, and 17 (41.5%) satisfaction. Study designs included randomized controlled trial (n = 1; 2.4%), interrupted time series (n = 1; 2.4%), pretest/posttest (n = 21; 51.2%), posttest only (n = 14; 34.1%), and pretest/posttest and posttest only for different dependent variables (n = 4; 9.8%). Data collection occurred through observations (n = 19, 46.3%), surveys (n = 17, 41.5%), patient safety event reports (n = 9, 22.0%), surveillance (n = 6, 14.6%), and audits (n = 3, 7.3%). CONCLUSION: Of the 100 measures across the 41 articles, implementing BCMA and/or eMAR broadly resulted in an increase in measures of effectiveness (n = 23, 52.3%) and satisfaction (n = 28, 62.2%) compared to measures of efficiency (n = 3, 27.3%). Future research should focus on eMAR efficiency measures, utilize rigorous study designs, and generate specific design requirements.

Development and pilot evaluation of an electronic health record usability and safety self-assessment tool.

Objective: Poor electronic health record (EHR) usability contributes to clinician burnout and poses patent safety risks. Site-specific customization and configuration of EHRs require individual EHR system usability and safety testing which is resource intensive. We developed and pilot-tested a self-administered EHR usability and safety assessment tool, focused on computerized provider order entry (CPOE), which can be used by any facility to identify specific issues. In addition, the tool provides recommendations for improvement. Materials and Methods: An assessment tool consisting of 104 questions was developed and pilot-tested at 2 hospitals, one using a Cerner EHR and the other using Epic. Five physicians at each site participated in and completed the assessment. Participant response accuracy compared to actual EHR interactions, consistency across participants, and usability issues identified through the tool were measured at each site. Results: Across sites, participants answered an average of 46 questions in 23 min with 89.9% of responses either correct or partially correct. The tool identified 8 usability and safety issues at one site and 7 at the other site across medication, laboratory, and radiology CPOE functions. Discussion: The tool shows promise as a method to rapidly evaluate EHR usability and safety and provide guidance on specific areas for improvement. Important improvements to the evaluation tool were identified including the need to clarify certain questions and provide definitions for usability terminology. Conclusion: A self-administered usability and safety assessment tool can serve to identify specific usability and safety issues in the EHR and provide guidance for improvements.

A novel application of SMART on FHIR architecture for interoperable and scalable integration of patient-reported outcome data with electronic health records.

OBJECTIVE: Despite a proliferation of applications (apps) to conveniently collect patient-reported outcomes (PROs) from patients, PRO data are yet to be seamlessly integrated with electronic health records (EHRs) in a way that improves interoperability and scalability. We applied the newly created PRO standards from the Office of the National Coordinator for Health Information Technology to facilitate the collection and integration of standardized PRO data. A novel multitiered architecture was created to enable seamless integration of PRO data via Substitutable Medical Apps and Reusable Technologies on Fast Healthcare Interoperability Resources apps and scaled to different EHR platforms in multiple ambulatory settings. MATERIALS AND METHODS: We used a standards-based approach to deploy 2 apps that source and surface PRO data in real-time for provider use within the EHR and which rely on PRO assessments from an external center to streamline app and EHR integration. RESULTS: The apps were developed to enable patients to answer validated assessments (eg, a Patient-Reported Outcomes Measurement Information System including using a Computer Adaptive Test format). Both apps were developed to populate the EHR in real time using the Health Level Seven FHIR standard allowing providers to view patients' data during the clinical encounter. The process of implementing this architecture with 2 different apps across 18 ambulatory care sites and 3 different EHR platforms is described. CONCLUSION: Our approach and solution proved feasible, secure, and time- and resource-efficient. We offer actionable guidance for this technology to be scaled and adapted to promote adoption in diverse ambulatory care settings and across different EHRs.

Clinician Perceptions on the Use of Free-Text Communication Orders.

OBJECTIVE: The aim of this study was to investigate (1) why ordering clinicians use free-text orders to communicate medication information; (2) what risks physicians and nurses perceive when free-text orders are used for communicating medication information; and (3) how electronic health records (EHRs) could be improved to encourage the safe communication of medication information. METHODS: We performed semi-structured, scenario-based interviews with eight physicians and eight nurses. Interview responses were analyzed and grouped into common themes. RESULTS: Participants described eight reasons why clinicians use free-text medication orders, five risks relating to the use of free-text medication orders, and five recommendations for improving EHR medication-related communication. Poor usability, including reduced efficiency and limited functionality associated with structured order entry, was the primary reason clinicians used free-text orders to communicate medication information. Common risks to using free-text orders for medication communication included the increased likelihood of missing orders and the increased workload on nurses responsible for executing orders. DISCUSSION: Clinicians' use of free-text orders is primarily due to limitations in the current structured order entry design. To encourage the safe communication of medication information between clinicians, the EHR's structured order entry must be redesigned to support clinicians' cognitive and workflow needs that are currently being addressed via the use of free-text orders. CONCLUSION: Clinicians' use of free-text orders as a workaround to insufficient structured order entry can create unintended patient safety risks. Thoughtful solutions designed to address these workarounds can improve the medication ordering process and the subsequent medication administration process.

Identifying Health Information Technology Usability Issues Contributing to Medication Errors Across Medication Process Stages.

OBJECTIVE: Different health information technology (health IT) systems are intended to support medication ordering, reviewing, and administration. We sought to identify the types of medication errors associated with health IT use, whether they reached the patient, where in the medication process those errors occurred, and the specific usability issues contributing to those errors. METHODS: Patient safety event reports from more than 595 healthcare facilities entered between January 2013 and September 2018 were analyzed. We computationally identified reports associated with health IT intended to support the medication process, including computerized provider order entry, electronic medication administration record, and barcode medication administration. From these, 2700 reports were manually reviewed to determine the type of medication error, medication process stage, and health IT usability issue. RESULTS: Of the 2700 manually reviewed reports, 1508 (55.9%) described a medication error that was associated with health IT use and 750 (49.7%) reached the patient. Improper dose errors were frequent (1214 of 1508, 80.5%) with most errors during ordering (673 of 1508, 44.6%) and reviewing medications (639 of 1508, 42.4%). Most health IT-associated medication error reports described usability issues (n = 1468 of 1508, 97.3%) including data entry, workflow support, and alerting. Data entry usability issues impacted few medication process stages, whereas workflow support and alerting impacted several stages. CONCLUSIONS: Health IT usability issues are a prevalent contributing factor to medication errors, many of which reach the patient. Data entry, workflow support, and alerting should be prioritized during usability and safety optimization efforts.

Emergency Physician Perceptions of Electronic Health Record Usability and Safety.

OBJECTIVES: Despite requirements for electronic health record (EHR) vendor usability testing, usability challenges persist, contributing to patient safety concerns. We sought to identify emergency physicians' perceived EHR usability and safety strengths and shortcomings across major EHR vendor products. METHODS: Fifty-five emergency physicians from 4 different hospitals were interviewed. The interviews were qualitatively analyzed, and physician comments were aligned with a usability taxonomy to identify emerging themes by vendor and hospital. RESULTS: Of the 194 comments about usability, the 3 most commonly discussed usability topics were Workflow Support (33.5% of comments), Visual Display (20.1%), and Data Entry (14.4%). Electronic health record usability strengths were centered on Visual Display, and the most common shortcoming was the lack of Workflow Support. Fourteen cross-hospital/cross-vendor themes, 6 vendor-specific themes, and 4 hospital-specific themes were identified. CONCLUSIONS: Usability shortcomings that spanned across hospitals and vendors may suggest a need for more applied research and improved design to resolve these issues. Shortcomings that are localized to a specific product or hospital may be due to customization and may be addressable by learning from other organizations.

A Machine Learning Approach to Reclassifying Miscellaneous Patient Safety Event Reports.

BACKGROUND AND OBJECTIVES: Medical errors are a leading cause of death in the United States. Despite widespread adoption of patient safety reporting systems to address medical errors, making sense of the reports collected in these systems is challenging in practice. Event classification taxonomies used in many reporting systems can be complex and difficult to understand by frontline reporters, leading reporters to classify reports as "miscellaneous" as opposed to assigning a specific event-type category, which may facilitate analysis. METHODS: To assist patient safety analysts in their analysis of "miscellaneous" reports, we developed an ensemble machine learning natural language processing model to reclassify these reports. We integrated the model into a clinical workflow dashboard, evaluated user feedback, and compared differences in user thresholds for model performance. RESULTS AND CONCLUSIONS: Integrating an ensemble model to classify "miscellaneous" event reports with an interactive visualization was helpful to patient safety analysts review "miscellaneous" reports. However, patient safety analysts have different thresholds for model reclassification depending on their role and experience with "miscellaneous" event reports.

Temporal dynamics of visual representations in the infant brain.

Tools from computational neuroscience have facilitated the investigation of the neural correlates of mental representations. However, access to the representational content of neural activations early in life has remained limited. We asked whether patterns of neural activity elicited by complex visual stimuli (animals, human body) could be decoded from EEG data gathered from 12-15-month-old infants and adult controls. We assessed pairwise classification accuracy at each time-point after stimulus onset, for individual infants and adults. Classification accuracies rose above chance in both groups, within 500 ms. In contrast to adults, neural representations in infants were not linearly separable across visual domains. Representations were similar within, but not across, age groups. These findings suggest a developmental reorganization of visual representations between the second year of life and adulthood and provide a promising proof-of-concept for the feasibility of decoding EEG data within-subject to assess how the infant brain dynamically represents visual objects.

Varying rates of patient identity verification when using computerized provider order entry.

OBJECTIVE: We sought to determine rates of computerized provider order entry (CPOE) patient identity verification and when and where in the ordering process verification occurred. MATERIALS AND METHODS: Fifty-five physicians from 4 healthcare systems completed simulated patient scenarios using their respective CPOE system (Epic or Cerner). Eye movements were recorded and analyzed. RESULTS: Across all participants patient id was verified significantly more often than not (62.4% vs 37.6%). Vendor A had significantly higher verification rates than not; vendor B had no difference. Participants using vendor A verified information significantly more often before signing the order than after (88.4% vs 11.6%); there was no difference in vendor B. The banner bar was the most frequent verification location. DISCUSSION: Factors such as CPOE design, physician training, and the use of a simulated methodology may be impacting verification rates. CONCLUSIONS: Verification rates vary by CPOE product, and this can have patient safety consequences.

Dynamics of neural representations when searching for exemplars and categories of human and non-human faces.

Face perception abilities in humans exhibit a marked expertise in distinguishing individual human faces at the expense of individual faces from other species (the other-species effect). In particular, one behavioural effect of such specialization is that human adults search for and find categories of non-human faces faster and more accurately than a specific non-human face, and vice versa for human faces. However, a recent visual search study showed that neural responses (event-related potentials, ERPs) were identical when finding either a non-human or human face. We used time-resolved multivariate pattern analysis of the EEG data from that study to investigate the dynamics of neural representations during a visual search for own-species (human) or other-species (non-human ape) faces, with greater sensitivity than traditional ERP analyses. The location of each target (i.e., right or left) could be decoded from the EEG, with similar accuracy for human and non-human faces. However, the neural patterns associated with searching for an exemplar versus a category target differed for human faces compared to non-human faces: Exemplar representations could be more reliably distinguished from category representations for human than non-human faces. These findings suggest that the other-species effect modulates the nature of representations, but preserves the attentional selection of target items based on these representations.

Emergence of the benefits and costs of grouping for visual search.

The present study investigated how grouping related items leads to the emergence of benefits (facilitation when related items are search targets) and costs (interference when related items are distractors) in visual search. Participants integrated different views (related items) of a novel Lego object via (a) assembling the object, (b) disassembling the object, or (c) sitting quietly without explicit instructions. An omnibus ANOVA revealed that neural responses (N2pc ERP) for attentional selection increased between pretest to posttest regardless of the training condition when a specific target view appeared (benefit) and when a nontarget view from the same object as the target view appeared (cost). Bonferroni-corrected planned comparisons revealed that assembling the object (but not disassembling the object or no training) had a significant impact from pretest to posttest, although the ANOVA did not reveal any interaction effects, suggesting that the effects might not differ across training conditions. This study is one of the first to demonstrate the emergence of the costs and benefits of grouping novel targets on visual search efficiency.

Increased experience amplifies the activation of task-irrelevant category representations.

Prior research has demonstrated the benefits (i.e., task-relevant attentional selection) and costs (i.e., task-irrelevant attentional capture) of prior knowledge on search for an individual target or multiple targets from a category. This study investigated whether the level of experience with particular categories predicts the degree of task-relevant and task-irrelevant activation of item and category representations. Adults with varying levels of dieting experience (measured via 3 subscales of Disinhibition, Restraint, Hunger; Stunkard & Messick, Journal of Psychosomatic Research, 29(1), 71-83, 1985) searched for targets defined as either a specific food item (e.g., carrots), or a category (i.e., any healthy or unhealthy food item). Apart from the target-present trials, in the target-absent "foil" trials, when searching for a specific item (e.g., carrots), irrelevant items from the target's category (e.g., squash) were presented. The ERP (N2pc) results revealed that the activation of task-relevant representations (measured via Exemplar and Category N2pc amplitudes) did not differ based on the degree of experience. Critically, however, increased dieting experience, as revealed by lower Disinhibition scores, predicted activation of task-irrelevant representations (i.e., attentional capture of foils from the target item category). Our results suggest that increased experience with particular categories encourages the rapid activation of category representations even when category information is task irrelevant, and that the N2pc in foil trials could potentially serve as an indication of experience level in future studies on categorization.

A neural signature of rapid category-based target selection as a function of intra-item perceptual similarity, despite inter-item dissimilarity.

Previous work on visual search has suggested that only a single attentional template can be prioritized at any given point in time. Grouping features into objects and objects into categories can facilitate search performance by maximizing the amount of information carried by an attentional template. From infancy to adulthood, earlier studies on perceptual similarity have shown that consistent features increase the likelihood of grouping features into objects (e.g., Quinn & Bhatt, Psychological Science. 20:933-938, 2009) and objects into categories (e.g., shape bias; Landau, Smith, & Jones, Cognitive Development. 3:299-321, 1988). Here we asked whether lower-level, intra-item similarity facilitates higher-level categorization, despite inter-item dissimilarity. Adults participated in four visual search tasks in which targets were defined as either one item (a specific alien) or a category (any alien) with either similar features (e.g., circle belly shape and circle back spikes) or dissimilar features (e.g., circle belly shape and triangle back spikes). Using behavioral and neural measures (i.e., the N2pc event-related potential component, which typically emerges 200 ms poststimulus), we found that intra-item feature similarity facilitated categorization, despite dissimilar features across the category items. Our results demonstrate that feature similarity builds novel categories and activates a task-appropriate abstract categorical search template. In other words, grouping at the lower, item level facilitates grouping at the higher, category level, which allows us to overcome efficiency limitations in visual search.