Correction: Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson.

[This corrects the article DOI: 10.2196/43960.].

Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson.

BACKGROUND: Evidence-based point-of-care information (POCI) tools can facilitate patient safety and care by helping clinicians to answer disease state and drug information questions in less time and with less effort. However, these tools may also be visually challenging to navigate or lack the comprehensiveness needed to sufficiently address a medical issue. OBJECTIVE: This study aimed to collect clinicians' feedback and directly observe their use of the combined POCI tool DynaMed and Micromedex with Watson, now known as DynaMedex. EBSCO partnered with IBM Watson Health, now known as Merative, to develop the combined tool as a resource for clinicians. We aimed to identify areas for refinement based on participant feedback and examine participant perceptions to inform further development. METHODS: Participants (N=43) within varying clinical roles and specialties were recruited from Brigham and Women's Hospital and Massachusetts General Hospital in Boston, Massachusetts, United States, between August 10, 2021, and December 16, 2021, to take part in usability sessions aimed at evaluating the efficiency and effectiveness of, as well as satisfaction with, the DynaMed and Micromedex with Watson tool. Usability testing methods, including think aloud and observations of user behavior, were used to identify challenges regarding the combined tool. Data collection included measurements of time on task; task ease; satisfaction with the answer; posttest feedback on likes, dislikes, and perceived reliability of the tool; and interest in recommending the tool to a colleague. RESULTS: On a 7-point Likert scale, pharmacists rated ease (mean 5.98, SD 1.38) and satisfaction (mean 6.31, SD 1.34) with the combined POCI tool higher than the physicians, nurse practitioner, and physician's assistants (ease: mean 5.57, SD 1.64, and satisfaction: mean 5.82, SD 1.60). Pharmacists spent longer (mean 2 minutes, 26 seconds, SD 1 minute, 41 seconds) on average finding an answer to their question than the physicians, nurse practitioner, and physician's assistants (mean 1 minute, 40 seconds, SD 1 minute, 23 seconds). CONCLUSIONS: Overall, the tool performed well, but this usability evaluation identified multiple opportunities for improvement that would help inexperienced users.

The Development and Piloting of the Ambulatory Electronic Health Record Evaluation Tool: Lessons Learned.

BACKGROUND: Substantial research has been performed about the impact of computerized physician order entry on medication safety in the inpatient setting; however, relatively little has been done in ambulatory care, where most medications are prescribed. OBJECTIVE: To outline the development and piloting process of the Ambulatory Electronic Health Record (EHR) Evaluation Tool and to report the quantitative and qualitative results from the pilot. METHODS: The Ambulatory EHR Evaluation Tool closely mirrors the inpatient version of the tool, which is administered by The Leapfrog Group. The tool was piloted with seven clinics in the United States, each using a different EHR. The tool consists of a medication safety test and a medication reconciliation module. For the medication test, clinics entered test patients and associated test orders into their EHR and recorded any decision support they received. An overall percentage score of unsafe orders detected, and order category scores were provided to clinics. For the medication reconciliation module, clinics demonstrated how their EHR electronically detected discrepancies between two medication lists. RESULTS: For the medication safety test, the clinics correctly alerted on 54.6% of unsafe medication orders. Clinics scored highest in the drug allergy (100%) and drug-drug interaction (89.3%) categories. Lower scoring categories included drug age (39.3%) and therapeutic duplication (39.3%). None of the clinics alerted for the drug laboratory or drug monitoring orders. In the medication reconciliation module, three (42.8%) clinics had an EHR-based medication reconciliation function; however, only one of those clinics could demonstrate it during the pilot. CONCLUSION: Clinics struggled in areas of advanced decision support such as drug age, drug laboratory, and drub monitoring. Most clinics did not have an EHR-based medication reconciliation function and this process was dependent on accessing patients' medication lists. Wider use of this tool could improve outpatient medication safety and can inform vendors about areas of improvement.

Design and testing of a mobile health application rating tool.

Mobile health applications ("apps") have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps' potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with ≥4 stars and <4-stars and then rated them with THESIS to examine the tool's interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96-3.09]), but especially poorly for privacy/security (2.21 out of 5 [95% CI, 2.11-2.32]), interoperability (1.75 [95% CI, 1.59-1.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30-1.56]). Ratings using THESIS had fair interrater reliability (κ = 0.3-0.6) and excellent scale reliability (ɑ = 0.85). Correlation with traditional star ratings was low (r = 0.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.

National Trends in the Safety Performance of Electronic Health Record Systems From 2009 to 2018.

Importance: Despite the broad adoption of electronic health record (EHR) systems across the continuum of care, safety problems persist. Objective: To measure the safety performance of operational EHRs in hospitals across the country during a 10-year period. Design, Setting, and Participants: This case series included all US adult hospitals nationwide that used the National Quality Forum Health IT Safety Measure EHR computerized physician order entry safety test administered by the Leapfrog Group between 2009 and 2018. Data were analyzed from July 1, 2018 to December 1, 2019. Exposure: The Health IT Safety Measure test, which uses simulated medication orders that have either injured or killed patients previously to evaluate how well hospital EHRs could identify medication errors with potential for patient harm. Main Outcomes and Measures: Descriptive statistics for performance on the assessment test over time were calculated at the overall test score level, type of decision support category level, and EHR vendor level. Results: Among 8657 hospital-years observed during the study, mean (SD) scores on the overall test increased from 53.9% (18.3%) in 2009 to 65.6% (15.4%) in 2018. Mean (SD) hospital score for the categories representing basic clinical decision support increased from 69.8% (20.8%) in 2009 to 85.6% (14.9%) in 2018. For the categories representing advanced clinical decision support, the mean (SD) score increased from 29.6% (22.4%) in 2009 to 46.1% (21.6%) in 2018. There was considerable variation in test performance by EHR. Conclusions and Relevance: These findings suggest that despite broad adoption and optimization of EHR systems in hospitals, wide variation in the safety performance of operational EHR systems remains across a large sample of hospitals and EHR vendors. Hospitals using some EHR vendors had significantly higher test scores. Overall, substantial safety risk persists in current hospital EHR systems.

The Impact of Alternative Payment in Chronically Ill and Older Patients in the Patient-centered Medical Home.

BACKGROUND: Patient-centered medical home (PCMH) has gained prominence as a promising model to encourage improved primary care delivery. There is a paucity of studies that evaluate the impact of payment models in the PCMH. OBJECTIVES: We sought to examine whether coupling coordinated, team-based care transformation plan with a novel reimbursement model affects outcomes related to expenditures and utilization. RESEARCH DESIGN: Interrupted time-series model with a difference-in-differences approach to assess differences between intervention and control groups, across time periods attributable to PCMH transformation and/or payment change. RESULTS: Although results were modest and mixed overall, PCMH with payment reform is associated with a reduction of $1.04 (P=0.0347) per member per month (PMPM) in pharmacy expenditures. Patients with hypertension, hyperlipidemia, diabetes, and coronary atherosclerosis enrolled in PCMH without payment reform experienced reductions in emergency department visits of 2.16 (P<0.0001), 2.42 (P<0.0001), 3.98 (P<0.0001), and 3.61 (P<0.0001) per 1000 per month. Modest increases in inpatient admission were seen among these patients in PCMH either with or without payment reform. Patients 65 and older enrolled in PMCH without payment reform experienced reductions in pharmacy expenditures $2.35 (P=0.0077) PMPM with a parallel reduction in pharmacy standardized cost of $2.81 (P=0.0174) PMPM indicative of a reduction in the intensity of drug utilization. CONCLUSIONS: We conclude that PCMH implementation coupled with an innovative payment arrangement generated mixed results with modest improvements with respect to pharmacy expenditures, but no overall financial improvement. However, we did see improvement within specific groups, especially older patients and those with chronic conditions.

Patient-Facing Mobile Apps to Treat High-Need, High-Cost Populations: A Scoping Review.

BACKGROUND: Self-management is essential to caring for high-need, high-cost (HNHC) populations. Advances in mobile phone technology coupled with increased availability and adoption of health-focused mobile apps have made self-management more achievable, but the extent and quality of the literature supporting their use is not well defined. OBJECTIVE: The purpose of this review was to assess the breadth, quality, bias, and types of outcomes measured in the literature supporting the use of apps targeting HNHC populations. METHODS: Data sources included articles in PubMed and MEDLINE (National Center for Biotechnology Information), EMBASE (Elsevier), the Cochrane Central Register of Controlled Trials (EBSCO), Web of Science (Thomson Reuters), and the NTIS (National Technical Information Service) Bibliographic Database (EBSCO) published since 2008. We selected studies involving use of patient-facing iOS or Android mobile health apps. Extraction was performed by 1 reviewer; 40 randomly selected articles were evaluated by 2 reviewers to assess agreement. RESULTS: Our final analysis included 175 studies. The populations most commonly targeted by apps included patients with obesity, physical handicaps, diabetes, older age, and dementia. Only 30.3% (53/175) of the apps studied in the reviewed literature were identifiable and available to the public through app stores. Many of the studies were cross-sectional analyses (42.9%, 75/175), small (median number of participants=31, interquartile range 11.0-207.2, maximum 11,690), or performed by an app's developers (61.1%, 107/175). Of the 175 studies, only 36 (20.6%, 36/175) studies evaluated a clinical outcome. CONCLUSIONS: Most apps described in the literature could not be located on the iOS or Android app stores, and existing research does not robustly evaluate the potential of mobile apps. Whereas apps may be useful in patients with chronic conditions, data do not support this yet. Although we had 2-3 reviewers to screen and assess abstract eligibility, only 1 reviewer abstracted the data. This is one limitation of our study. With respect to the 40 articles (22.9%, 40/175) that were assigned to 2 reviewers (of which 3 articles were excluded), inter-rater agreement was significant on the majority of items (17 of 30) but fair-to-moderate on others.

Many Mobile Health Apps Target High-Need, High-Cost Populations, But Gaps Remain.

With rising smartphone ownership, mobile health applications (mHealth apps) have the potential to support high-need, high-cost populations in managing their health. While the number of available mHealth apps has grown substantially, no clear strategy has emerged on how providers should evaluate and recommend such apps to patients. Key stakeholders, including medical professional societies, insurers, and policy makers, have largely avoided formally recommending apps, which forces patients to obtain recommendations from other sources. To help stakeholders overcome barriers to reviewing and recommending apps, we evaluated 137 patient-facing mHealth apps-those intended for use by patients to manage their health-that were highly rated by consumers and recommended by experts and that targeted high-need, high-cost populations. We found that there is a wide variety of apps in the marketplace but that few apps address the needs of the patients who could benefit the most. We also found that consumers' ratings were poor indications of apps' clinical utility or usability and that most apps did not respond appropriately when a user entered potentially dangerous health information. Going forward, data privacy and security will continue to be major concerns in the dissemination of mHealth apps.

Evaluation: A Qualitative Pilot Study of Novel Information Technology Infrastructure to Communicate Genetic Variant Updates.

BACKGROUND: Partners HealthCare Personalized Medicine developed GeneInsight Clinic (GIC), a tool designed to communicate updated variant information from laboratory geneticists to treating clinicians through automated alerts, categorized by level of variant interpretation change. OBJECTIVES: The study aimed to evaluate feedback from the initial users of the GIC, including the advantages and challenges to receiving this variant information and using this technology at the point of care. METHODS: Healthcare professionals from two clinics that ordered genetic testing for cardiomyopathy and related disorders were invited to participate in one-hour semi-structured interviews and/ or a one-hour focus group. Using a Grounded Theory approach, transcript concepts were coded and organized into themes. RESULTS: Two genetic counselors and two physicians from two treatment clinics participated in individual interviews. Focus group participants included one genetic counselor and four physicians. Analysis resulted in 8 major themes related to structuring and communicating variant knowledge, GIC's impact on the clinic, and suggestions for improvements. The interview analysis identified longitudinal patient care, family data, and growth in genetic testing content as potential challenges to optimization of the GIC infrastructure. DISCUSSION: Participants agreed that GIC implementation increased efficiency and effectiveness of the clinic through increased access to genetic variant information at the point of care. CONCLUSION: Development of information technology (IT) infrastructure to aid in the organization and management of genetic variant knowledge will be critical as the genetic field moves towards whole exome and whole genome sequencing. Findings from this study could be applied to future development of IT support for genetic variant knowledge management that would serve to improve clinicians' ability to manage and care for patients.

Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications.

Rising ownership of smartphones and tablets across social and demographic groups has made mobile applications, or apps, a potentially promising tool for engaging patients in their health care, particularly those with high health care needs. Through a systematic search of iOS (Apple) and Android app stores and an analysis of apps targeting individuals with chronic illnesses, we assessed the degree to which apps are likely to be useful in patient engagement efforts. Usefulness was determined based on the following criteria: description of engagement, relevance to the targeted patient population, consumer ratings and reviews, and most recent app update. Among the 1,046 health care-related, patient-facing applications identified by our search, 43 percent of iOS apps and 27 percent of Android apps appeared likely to be useful. We also developed criteria for evaluating the patient engagement, quality, and safety of mobile apps.

Linking acknowledgement to action: closing the loop on non-urgent, clinically significant test results in the electronic health record.

Failure to follow-up nonurgent, clinically significant test results (CSTRs) is an ambulatory patient safety concern. Tools within electronic health records (EHRs) may facilitate test result acknowledgment, but their utility with regard to nonurgent CSTRs is unclear. We measured use of an acknowledgment tool by 146 primary care physicians (PCPs) at 13 network-affiliated practices that use the same EHR. We then surveyed PCPs to assess use of, satisfaction with, and desired enhancements to the acknowledgment tool. The rate of acknowledgment of non-urgent CSTRs by PCPs was 78%. Of 73 survey respondents, 72 reported taking one or more actions after reviewing a CSTR; fewer (40-75%) reported that using the acknowledgment tool was helpful for a specific purpose. Forty-six (64%) were satisfied with the tool. Both satisfied and nonsatisfied PCPs reported that enhancements linking acknowledgment to routine actions would be useful. EHR vendors should consider enhancements to acknowledgment functionality to ensure follow-up of nonurgent CSTRs.

A novel clinician interface to improve clinician access to up-to-date genetic results.

OBJECTIVES: To understand the impact of GeneInsight Clinic (GIC), a web-based tool designed to manage genetic information and facilitate communication of test results and variant updates from the laboratory to the clinics, we measured the use of GIC and the time it took for new genetic knowledge to be available to clinicians. METHODS: Usage data were collected across four study sites for the GIC launch and post-GIC implementation time periods. The primary outcome measures were the time (average number of days) between variant change approval and notification of clinic staff, and the time between notification and viewing the patient record. RESULTS: Post-GIC, time between a variant change approval and provider notification was shorter than at launch (average days at launch 503.8, compared to 4.1 days post-GIC). After e-mail alerts were sent at launch, providers clicked into the patient record associated with 91% of these alerts. In the post period, clinic providers clicked into the patient record associated with 95% of the alerts, on average 12 days after the e-mail was sent. DISCUSSION: We found that GIC greatly increased the likelihood that a provider would receive updated variant information as well as reduced the time associated with distributing that variant information, thus providing a more efficient process for incorporating new genetic knowledge into clinical care. CONCLUSIONS: Our study results demonstrate that health information technology systems have the potential effectively to assist providers in utilizing genetic information in patient care.

Usability of a novel clinician interface for genetic results.

The complexity and rapid growth of genetic data demand investment in information technology to support effective use of this information. Creating infrastructure to communicate genetic information to healthcare providers and enable them to manage that data can positively affect a patient's care in many ways. However, genetic data are complex and present many challenges. We report on the usability of a novel application designed to assist providers in receiving and managing a patient's genetic profile, including ongoing updated interpretations of the genetic variants in those patients. Because these interpretations are constantly evolving, managing them represents a challenge. We conducted usability tests with potential users of this application and reported findings to the application development team, many of which were addressed in subsequent versions. Clinicians were excited about the value this tool provides in pushing out variant updates to providers and overall gave the application high usability ratings, but had some difficulty interpreting elements of the interface. Many issues identified required relatively little development effort to fix suggesting that consistently incorporating this type of analysis in the development process can be highly beneficial. For genetic decision support applications, our findings suggest the importance of designing a system that can deliver the most current knowledge and highlight the significance of new genetic information for clinical care. Our results demonstrate that using a development and design process that is user focused helped optimize the value of this application for personalized medicine.

Actionable reminders did not improve performance over passive reminders for overdue tests in the primary care setting.

Actionable reminders (electronic reminders linked to computerized order entry) might improve care by facilitating direct ordering of recommended tests. The authors implemented four enhanced actionable reminders targeting performance of annual mammography, one-time bone-density screening, and diabetic testing. There was no difference in rates of appropriate testing between the four intervention and four matched, control primary care clinics for screening mammography (OR 0.81, 95% CI 0.64 to 1.02), bone-density exams (OR 1.29, 95% CI 0.82 to 2.02), HbA1c monitoring (OR 0.91, 95% CI 0.58 to 1.42) and LDL cholesterol monitoring (OR 1.40, 95% CI 0.76 to 2.59). Of the survey respondents, 79% almost never used the system or were unaware of the functionality. In the 9/228 (3.9%) cases with indirect evidence of mammography reminder use, there was a significantly lower proportion with test performance. Our actionable reminders did not improve receipt of overdue testing, potentially due to limitations of workflow integration.

Trends in primary care clinician perceptions of a new electronic health record.

BACKGROUND: Clinician perceptions of a newly implemented electronic health record play an important role in its success or failure. OBJECTIVE: To measure changes in primary care clinician attitudes toward an electronic health record during the first year following implementation. DESIGN: Longitudinal survey. PARTICIPANTS: 86 primary care clinicians surveyed between December 2006 and January 2008. MEASUREMENTS: Perceived impact on overall quality of care, patient safety, communication, and efficiency at 1, 3, 6, and 12 months following implementation. RESULTS: Response rates for months 1, 3, 6, and 12 were 92%, 95%, 90%, and 82%, respectively. The proportion of clinicians agreeing that the EHR improved the overall quality of care (63% to 86%; p < 0.001), reduced medication-related errors (72% to 81%; p = 0.03), improved follow-up of test results (62% to 87%; p < 0.001), and improved communication among clinicians (72% to 93%; p < 0.001) increased from month 1 to month 12. During the same time period, a decreasing proportion of clinicians agreed that the EHR reduced the quality of patient interactions (49% to 33%; p = 0.001), resulted in longer patient visits (68% to 51%; p = 0.001), and increased time spent on medical documentation (78% to 68%; p = 0.006). Significant improvements in perceptions related to test result follow-up were first detected at 6 months, while those related to overall quality, efficiency, and communication were first identified at 12 months. CONCLUSIONS: Primary care clinicians report increasingly positive perceptions of a new electronic health record within 1 year of implementation across a spectrum of domains of care.

Trends in clinician perceptions of a new electronic health record.

Clinician perceptions of a newly implemented electronic health record (EHR) play an important role in its success or failure. Limited data exist describing how these attitudes change over time. We measured attitudes of primary care clinicians toward a new EHR during the first year following implementation. We found that perceptions of the new system became significantly more favorable over time with variable rates of change for different aspects of the EHR.

Impact of ambulatory computerized physician order entry on clinicians' time.

While Ambulatory Computerized Physician Order Entry (ACPOE) may improve health care quality, the impact on clinician time may impose a barrier to implementation. We performed a time-motion analysis to evaluate this potential impact. The results indicate that there is no significant impact on clinicians' time with patients. Further analyses will assess ACPOE's impact on clinic time between patient visits.

An effort to improve electronic health record medication list accuracy between visits: patients' and physicians' response.

OBJECTIVE: To evaluate the efficacy of a secure web-based patient portal called Patient Gateway (PG) in producing more accurate medication lists in the electronic health record (EHR), and whether sending primary care physicians (PCPs) a clinical message updating them on the information their patients provided caused physicians to update the EHR medication list. METHODS: We compared the medication list accuracy of 84 patients using PG with that of 79 who were not. Patient-reported medication discrepancies were noted in the EHR in a clinical note by research staff and a message was sent to the participants' PCPs notifying them of the updated information. RESULTS: Participants were taking 665 medications according to the EHR, and reported 273 additional medications. A lower percentage of PG users' drug regimens (54% versus 61%, p=0.07) were reported to be correct than those of PG non-users, although PG users took significantly more medications than their non-user counterparts (5.0 versus 3.1 medications, p=0.0001). Providing patient-reported information in a clinical note and sending a clinical message to the primary care doctor did not result in PCPs updating their patients' EHR medication lists. CONCLUSIONS: Medication lists in EHRs were frequently inaccurate and most frequently overlooked over-the-counter (OTC) and non-prescription drugs. Patients using a secure portal had just as many discrepancies between medication lists and self-report as those who did not, and notifying physicians of discrepancies via e-mail had no effect.

Do physicians take action on high risk family history information provided by patients outside of a clinic visit?

Clinically relevant family history information is frequently missing or not readily available in electronic health records. Improving the availability of family history information is important for optimum care of many patients. Family history information on five conditions was collected in a survey from 163 primary care patients. Overall, 53% of patients had no family history information in the electronic health record (EHR) either on the patient's problem list or within a templated family history note. New information provided by patients resulted in an increase in the patient's risk level for 32% of patients with a positive family history of breast cancer, 40% for coronary artery disease, 50% for colon cancer, 74% for diabetes, and 95% each for osteoporosis and glaucoma. Informing physicians of new family history information outside of a clinic visit through an electronic clinical message and note in the EHR was not sufficient to achieve recommended follow-up care. Better tools need to be developed to facilitate the collection of family history information and to support clinical decision-making and action.

Electronic health records in specialty care: a time-motion study.

BACKGROUND: Electronic health records (EHRs) have great potential to improve safety, quality, and efficiency in medicine. However, adoption has been slow, and a key concern has been that clinicians will require more time to complete their work using EHRs. Most previous studies addressing this issue have been done in primary care. OBJECTIVE: To assess the impact of using an EHR on specialists' time. DESIGN Prospective, before-after trial of the impact of an EHR on attending physician time in four specialty clinics at an integrated delivery system: cardiology, dermatology, endocrine, and pain. MEASUREMENTS: We used a time-motion method to measure physician time spent in one of 85 designated activities. RESULTS: Attending physicians were monitored before and after the switch from paper records to a web-based ambulatory EHR. Across all specialties, 15 physicians were observed treating 157 patients while still using paper-based records, and 15 physicians were observed treating 146 patients after adoption. Following EHR implementation, the average adjusted total time spent per patient across all specialties increased slightly but not significantly (Delta = 0.94 min., p = 0.83) from 28.8 (SE = 3.6) to 29.8 (SE = 3.6) min. CONCLUSION: These data suggest that implementation of an EHR had little effect on overall visit time in specialty clinics.