Defining AMIA's artificial intelligence principles.

Recent advances in the science and technology of artificial intelligence (AI) and growing numbers of deployed AI systems in healthcare and other services have called attention to the need for ethical principles and governance. We define and provide a rationale for principles that should guide the commission, creation, implementation, maintenance, and retirement of AI systems as a foundation for governance throughout the lifecycle. Some principles are derived from the familiar requirements of practice and research in medicine and healthcare: beneficence, nonmaleficence, autonomy, and justice come first. A set of principles follow from the creation and engineering of AI systems: explainability of the technology in plain terms; interpretability, that is, plausible reasoning for decisions; fairness and absence of bias; dependability, including "safe failure"; provision of an audit trail for decisions; and active management of the knowledge base to remain up to date and sensitive to any changes in the environment. In organizational terms, the principles require benevolence-aiming to do good through the use of AI; transparency, ensuring that all assumptions and potential conflicts of interest are declared; and accountability, including active oversight of AI systems and management of any risks that may arise. Particular attention is drawn to the case of vulnerable populations, where extreme care must be exercised. Finally, the principles emphasize the need for user education at all levels of engagement with AI and for continuing research into AI and its biomedical and healthcare applications.

Multi-Institutional Implementation of Clinical Decision Support for APOL1, NAT2, and YEATS4 Genotyping in Antihypertensive Management.

(1) Background: Clinical decision support (CDS) is a vitally important adjunct to the implementation of pharmacogenomic-guided prescribing in clinical practice. A novel CDS was sought for the APOL1, NAT2, and YEATS4 genes to guide optimal selection of antihypertensive medications among the African American population cared for at multiple participating institutions in a clinical trial. (2) Methods: The CDS committee, made up of clinical content and CDS experts, developed a framework and contributed to the creation of the CDS using the following guiding principles: 1. medical algorithm consensus; 2. actionability; 3. context-sensitive triggers; 4. workflow integration; 5. feasibility; 6. interpretability; 7. portability; and 8. discrete reporting of lab results. (3) Results: Utilizing the principle of discrete patient laboratory and vital information, a novel CDS for APOL1, NAT2, and YEATS4 was created for use in a multi-institutional trial based on a medical algorithm consensus. The alerts are actionable and easily interpretable, clearly displaying the purpose and recommendations with pertinent laboratory results, vitals and links to ordersets with suggested antihypertensive dosages. Alerts were either triggered immediately once a provider starts to order relevant antihypertensive agents or strategically placed in workflow-appropriate general CDS sections in the electronic health record (EHR). Detailed implementation instructions were shared across institutions to achieve maximum portability. (4) Conclusions: Using sound principles, the created genetic algorithms were applied across multiple institutions. The framework outlined in this study should apply to other disease-gene and pharmacogenomic projects employing CDS.

Augmented intelligence with natural language processing applied to electronic health records for identifying patients with non-alcoholic fatty liver disease at risk for disease progression.

OBJECTIVE: Electronic health record (EHR) systems contain structured data (such as diagnostic codes) and unstructured data (clinical documentation). Clinical insights can be derived from analyzing both. The use of natural language processing (NLP) algorithms to effectively analyze unstructured data has been well demonstrated. Here we examine the utility of NLP for the identification of patients with non-alcoholic fatty liver disease, assess patterns of disease progression, and identify gaps in care related to breakdown in communication among providers. MATERIALS AND METHODS: All clinical notes available on the 38,575 patients enrolled in the Mount Sinai BioMe cohort were loaded into the NLP system. We compared analysis of structured and unstructured EHR data using NLP, free-text search, and diagnostic codes with validation against expert adjudication. We then used the NLP findings to measure physician impression of progression from early-stage NAFLD to NASH or cirrhosis. Similarly, we used the same NLP findings to identify mentions of NAFLD in radiology reports that did not persist into clinical notes. RESULTS: Out of 38,575 patients, we identified 2,281 patients with NAFLD. From the remainder, 10,653 patients with similar data density were selected as a control group. NLP outperformed ICD and text search in both sensitivity (NLP: 0.93, ICD: 0.28, text search: 0.81) and F2 score (NLP: 0.92, ICD: 0.34, text search: 0.81). Of 2281 NAFLD patients, 673 (29.5%) were believed to have progressed to NASH or cirrhosis. Among 176 where NAFLD was noted prior to NASH, the average progression time was 410 days. 619 (27.1%) NAFLD patients had it documented only in radiology notes and not acknowledged in other forms of clinical documentation. Of these, 170 (28.4%) were later identified as having likely developed NASH or cirrhosis after a median 1057.3 days. DISCUSSION: NLP-based approaches were more accurate at identifying NAFLD within the EHR than ICD/text search-based approaches. Suspected NAFLD on imaging is often not acknowledged in subsequent clinical documentation. Many such patients are later found to have more advanced liver disease. Analysis of information flows demonstrated loss of key information that could have been used to help prevent the progression of early NAFLD (NAFL) to NASH or cirrhosis. CONCLUSION: For identification of NAFLD, NLP performed better than alternative selection modalities. It then facilitated analysis of knowledge flow between physician and enabled the identification of breakdowns where key information was lost that could have slowed or prevented later disease progression.

Barriers and facilitators affecting the implementation of substance use screening in primary care clinics: a qualitative study of patients, providers, and staff.

BACKGROUND: Alcohol and drug use are leading causes of morbidity and mortality that frequently go unidentified in medical settings. As part of a multi-phase study to implement electronic health record-integrated substance use screening in primary care clinics, we interviewed key clinical stakeholders to identify current substance use screening practices, barriers to screening, and recommendations for its implementation. METHODS: Focus groups and individual interviews were conducted with 67 stakeholders, including patients, primary care providers (faculty and resident physicians), nurses, and medical assistants, in two urban academic health systems. Themes were identified using an inductive approach, revised through an iterative process, and mapped to the Knowledge to Action (KTA) framework, which guides the implementation of new clinical practices (Graham et al. in J Contin Educ Health Prof 26(1):13-24, 2006). RESULTS: Factors affecting implementation based on KTA elements were identified from participant narratives. Identifying the problem: Participants consistently agreed that having knowledge of a patient's substance use is important because of its impacts on health and medical care, that substance use is not properly identified in medical settings currently, and that universal screening is the best approach. Assessing barriers: Patients expressed concerns about consequences of disclosing substance use, confidentiality, and the individual's own reluctance to acknowledge a substance use problem. Barriers identified by providers included individual-level factors such as lack of clinical knowledge and training, as well as systems-level factors including time pressure, resources, lack of space, and difficulty accessing addiction treatment. Adapting to the local context: Most patients and providers stated that the primary care provider should play a key role in substance use screening and interventions. Opinions diverged regarding the optimal approach to delivering screening, although most preferred a patient self-administered approach. Many providers reported that taking effective action once unhealthy substance use is identified is crucial. CONCLUSIONS: Participants expressed support for substance use screening as a valuable part of medical care, and identified individual-level as well as systems-level barriers to its implementation. These findings suggest that screening programs should clearly communicate the goals of screening to patients and proactively counteract stigma, address staff concerns regarding time and workflow, and provide education as well as treatment resources to primary care providers.

Physician Attitudes toward Adopting Genome-Guided Prescribing through Clinical Decision Support.

This study assessed physician attitudes toward adopting genome-guided prescribing through clinical decision support (CDS), prior to enlisting in the Clinical Implementation of Personalized Medicine through Electronic Health Records and Genomics pilot pharmacogenomics project (CLIPMERGE PGx). We developed a survey instrument that includes the Evidence Based Practice Attitude Scale, adapted to measure attitudes toward adopting genome-informed interventions (EBPAS-GII). The survey also includes items to measure physicians' characteristics (awareness, experience, and perceived usefulness), attitudes about personal genome testing (PGT) services, and comfort using technology. We surveyed 101 General Internal Medicine physicians from the Icahn School of Medicine at Mount Sinai (ISMMS). The majority were residency program trainees (~88%). Prior to enlisting into CLIPMERGE PGx, most physicians were aware of and had used decision support aids. Few physicians, however, were aware of and had used genome-guided prescribing. The majority of physicians viewed decision support aids and genotype data as being useful for making prescribing decisions. Most physicians had not heard of, but were willing to use, PGT services and felt comfortable interpreting PGT results. Most physicians were comfortable with technology. Physicians who perceived genotype data to be useful in making prescribing decisions, had more positive attitudes toward adopting genome-guided prescribing through CDS. Our findings suggest that internal medicine physicians have a deficit in their familiarity and comfort interpreting and using genomic information. This has reinforced the importance of gathering feedback and guidance from our enrolled physicians when designing genome-guided CDS and the importance of prioritizing genomic medicine education at our institutions.

Integrating usability testing and think-aloud protocol analysis with "near-live" clinical simulations in evaluating clinical decision support.

PURPOSE: Usability evaluations can improve the usability and workflow integration of clinical decision support (CDS). Traditional usability testing using scripted scenarios with think-aloud protocol analysis provide a useful but incomplete assessment of how new CDS tools interact with users and clinical workflow. "Near-live" clinical simulations are a newer usability evaluation tool that more closely mimics clinical workflow and that allows for a complementary evaluation of CDS usability as well as impact on workflow. METHODS: This study employed two phases of testing a new CDS tool that embedded clinical prediction rules (an evidence-based medicine tool) into primary care workflow within a commercial electronic health record. Phase I applied usability testing involving "think-aloud" protocol analysis of 8 primary care providers encountering several scripted clinical scenarios. Phase II used "near-live" clinical simulations of 8 providers interacting with video clips of standardized trained patient actors enacting the clinical scenario. In both phases, all sessions were audiotaped and had screen-capture software activated for onscreen recordings. Transcripts were coded using qualitative analysis methods. RESULTS: In Phase I, the impact of the CDS on navigation and workflow were associated with the largest volume of negative comments (accounting for over 90% of user raised issues) while the overall usability and the content of the CDS were associated with the most positive comments. However, usability had a positive-to-negative comment ratio of only 0.93 reflecting mixed perceptions about the usability of the CDS. In Phase II, the duration of encounters with simulated patients was approximately 12 min with 71% of the clinical prediction rules being activated after half of the visit had already elapsed. Upon activation, providers accepted the CDS tool pathway 82% of times offered and completed all of its elements in 53% of all simulation cases. Only 12.2% of encounter time was spent using the CDS tool. Two predominant clinical workflows, accounting for 75% of all cases simulations, were identified that characterized the sequence of provider interactions with the CDS. These workflows demonstrated a significant variation in temporal sequence of potential activation of the CDS. CONCLUSIONS: This study successfully combined "think-aloud" protocol analysis with "near-live" clinical simulations in a usability evaluation of a new primary care CDS tool. Each phase of the study provided complementary observations on problems with the new onscreen tool and was used to refine both its usability and workflow integration. Synergistic use of "think-aloud" protocol analysis and "near-live" clinical simulations provide a robust assessment of how CDS tools would interact in live clinical environments and allows for enhanced early redesign to augment clinician utilization. The findings suggest the importance of using complementary testing methods before releasing CDS for live use.

Rationale, design, and implementation protocol of an electronic health record integrated clinical prediction rule (iCPR) randomized trial in primary care.

BACKGROUND: Clinical prediction rules (CPRs) represent well-validated but underutilized evidence-based medicine tools at the point-of-care. To date, an inability to integrate these rules into an electronic health record (EHR) has been a major limitation and we are not aware of a study demonstrating the use of CPR's in an ambulatory EHR setting. The integrated clinical prediction rule (iCPR) trial integrates two CPR's in an EHR and assesses both the usability and the effect on evidence-based practice in the primary care setting. METHODS: A multi-disciplinary design team was assembled to develop a prototype iCPR for validated streptococcal pharyngitis and bacterial pneumonia CPRs. The iCPR tool was built as an active Clinical Decision Support (CDS) tool that can be triggered by user action during typical workflow. Using the EHR CDS toolkit, the iCPR risk score calculator was linked to tailored ordered sets, documentation, and patient instructions. The team subsequently conducted two levels of 'real world' usability testing with eight providers per group. Usability data were used to refine and create a production tool. Participating primary care providers (n = 149) were randomized and intervention providers were trained in the use of the new iCPR tool. Rates of iCPR tool triggering in the intervention and control (simulated) groups are monitored and subsequent use of the various components of the iCPR tool among intervention encounters is also tracked. The primary outcome is the difference in antibiotic prescribing rates (strep and pneumonia iCPR's encounters) and chest x-rays (pneumonia iCPR only) between intervention and control providers. DISCUSSION: Using iterative usability testing and development paired with provider training, the iCPR CDS tool leverages user-centered design principles to overcome pervasive underutilization of EBM and support evidence-based practice at the point-of-care. The ongoing trial will determine if this collaborative process will lead to higher rates of utilization and EBM guided use of antibiotics and chest x-ray's in primary care. TRIAL REGISTRATION: ClinicalTrials.gov Identifier NCT01386047.

Technology induced error and usability: the relationship between usability problems and prescription errors when using a handheld application.

This paper describes an innovative approach to the evaluation of a handheld prescription writing application. Participants (10 physicians) were asked to perform a series of tasks involving entering prescriptions into the application from a medication list. The study procedure involved the collection of data consisting of transcripts of the subjects who were asked to "think aloud" while interacting with the prescription writing program to enter medications. All user interactions with the device were video and audio recorded. Analysis of the protocols was conducted in two phases: (1) usability problems were identified from coding of the transcripts and video data, (2) actual errors in entering prescription data were also identified. The results indicated that there were a variety of usability problems, with most related to interface design issues. In examining the relationship between usability problems and errors, it was found that certain types of usability problems were closely associated with the occurrence of specific types of errors in prescription of medications. Implications for identifying and predicting technology-induced error are discussed in the context of improving the safety of health care information systems.

Comparison of three methods of entering clinical information into a prototype triage system.

This research study is a prospective, randomized trial evaluating varied means of inputting structured clinical data into a prototype triage system. Three different methods of historical data entry were evaluated for efficiency, quality of obtained data and ease of use. The results show that using a "customized" form containing check-box entries for only the most commonly seen historical data and space for text entry of other information was the most accurate method of entering triage information and was nearly as fast and easy as free text entry.