Correction: Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson.

[This corrects the article DOI: 10.2196/43960.].

Performance of a Web-Based Reference Database With Natural Language Searching Capabilities: Usability Evaluation of DynaMed and Micromedex With Watson.

BACKGROUND: Evidence-based point-of-care information (POCI) tools can facilitate patient safety and care by helping clinicians to answer disease state and drug information questions in less time and with less effort. However, these tools may also be visually challenging to navigate or lack the comprehensiveness needed to sufficiently address a medical issue. OBJECTIVE: This study aimed to collect clinicians' feedback and directly observe their use of the combined POCI tool DynaMed and Micromedex with Watson, now known as DynaMedex. EBSCO partnered with IBM Watson Health, now known as Merative, to develop the combined tool as a resource for clinicians. We aimed to identify areas for refinement based on participant feedback and examine participant perceptions to inform further development. METHODS: Participants (N=43) within varying clinical roles and specialties were recruited from Brigham and Women's Hospital and Massachusetts General Hospital in Boston, Massachusetts, United States, between August 10, 2021, and December 16, 2021, to take part in usability sessions aimed at evaluating the efficiency and effectiveness of, as well as satisfaction with, the DynaMed and Micromedex with Watson tool. Usability testing methods, including think aloud and observations of user behavior, were used to identify challenges regarding the combined tool. Data collection included measurements of time on task; task ease; satisfaction with the answer; posttest feedback on likes, dislikes, and perceived reliability of the tool; and interest in recommending the tool to a colleague. RESULTS: On a 7-point Likert scale, pharmacists rated ease (mean 5.98, SD 1.38) and satisfaction (mean 6.31, SD 1.34) with the combined POCI tool higher than the physicians, nurse practitioner, and physician's assistants (ease: mean 5.57, SD 1.64, and satisfaction: mean 5.82, SD 1.60). Pharmacists spent longer (mean 2 minutes, 26 seconds, SD 1 minute, 41 seconds) on average finding an answer to their question than the physicians, nurse practitioner, and physician's assistants (mean 1 minute, 40 seconds, SD 1 minute, 23 seconds). CONCLUSIONS: Overall, the tool performed well, but this usability evaluation identified multiple opportunities for improvement that would help inexperienced users.

The Development and Piloting of the Ambulatory Electronic Health Record Evaluation Tool: Lessons Learned.

BACKGROUND: Substantial research has been performed about the impact of computerized physician order entry on medication safety in the inpatient setting; however, relatively little has been done in ambulatory care, where most medications are prescribed. OBJECTIVE: To outline the development and piloting process of the Ambulatory Electronic Health Record (EHR) Evaluation Tool and to report the quantitative and qualitative results from the pilot. METHODS: The Ambulatory EHR Evaluation Tool closely mirrors the inpatient version of the tool, which is administered by The Leapfrog Group. The tool was piloted with seven clinics in the United States, each using a different EHR. The tool consists of a medication safety test and a medication reconciliation module. For the medication test, clinics entered test patients and associated test orders into their EHR and recorded any decision support they received. An overall percentage score of unsafe orders detected, and order category scores were provided to clinics. For the medication reconciliation module, clinics demonstrated how their EHR electronically detected discrepancies between two medication lists. RESULTS: For the medication safety test, the clinics correctly alerted on 54.6% of unsafe medication orders. Clinics scored highest in the drug allergy (100%) and drug-drug interaction (89.3%) categories. Lower scoring categories included drug age (39.3%) and therapeutic duplication (39.3%). None of the clinics alerted for the drug laboratory or drug monitoring orders. In the medication reconciliation module, three (42.8%) clinics had an EHR-based medication reconciliation function; however, only one of those clinics could demonstrate it during the pilot. CONCLUSION: Clinics struggled in areas of advanced decision support such as drug age, drug laboratory, and drub monitoring. Most clinics did not have an EHR-based medication reconciliation function and this process was dependent on accessing patients' medication lists. Wider use of this tool could improve outpatient medication safety and can inform vendors about areas of improvement.

The tradeoffs between safety and alert fatigue: Data from a national evaluation of hospital medication-related clinical decision support.

OBJECTIVE: The study sought to evaluate the overall performance of hospitals that used the Computerized Physician Order Entry Evaluation Tool in both 2017 and 2018, along with their performance against fatal orders and nuisance orders. MATERIALS AND METHODS: We evaluated 1599 hospitals that took the test in both 2017 and 2018 by using their overall percentage scores on the test, along with the percentage of fatal orders appropriately alerted on, and the percentage of nuisance orders incorrectly alerted on. RESULTS: Hospitals showed overall improvement; the mean score in 2017 was 58.1%, and this increased to 66.2% in 2018. Fatal order performance improved slightly from 78.8% to 83.0% (P < .001), though there was almost no change in nuisance order performance (89.0% to 89.7%; P = .43). Hospitals alerting on one or more nuisance orders had a 3-percentage-point increase in their overall score. DISCUSSION: Despite the improvement of overall scores in 2017 and 2018, there was little improvement in fatal order performance, suggesting that hospitals are not targeting the deadliest orders first. Nuisance order performance showed almost no improvement, and some hospitals may be achieving higher scores by overalerting, suggesting that the thresholds for which alerts are fired from are too low. CONCLUSIONS: Although hospitals improved overall from 2017 to 2018, there is still important room for improvement for both fatal and nuisance orders. Hospitals that incorrectly alerted on one or more nuisance orders had slightly higher overall performance, suggesting that some hospitals may be achieving higher scores at the cost of overalerting, which has the potential to cause clinician burnout and even worsen safety.

Design and testing of a mobile health application rating tool.

Mobile health applications ("apps") have rapidly proliferated, yet their ability to improve outcomes for patients remains unclear. A validated tool that addresses apps' potentially important dimensions has not been available to patients and clinicians. The objective of this study was to develop and preliminarily assess a usable, valid, and open-source rating tool to objectively measure the risks and benefits of health apps. We accomplished this by using a Delphi process, where we constructed an app rating tool called THESIS that could promote informed app selection. We used a systematic process to select chronic disease apps with ≥4 stars and <4-stars and then rated them with THESIS to examine the tool's interrater reliability and internal consistency. We rated 211 apps, finding they performed fair overall (3.02 out of 5 [95% CI, 2.96-3.09]), but especially poorly for privacy/security (2.21 out of 5 [95% CI, 2.11-2.32]), interoperability (1.75 [95% CI, 1.59-1.91]), and availability in multiple languages (1.43 out of 5 [95% CI, 1.30-1.56]). Ratings using THESIS had fair interrater reliability (κ = 0.3-0.6) and excellent scale reliability (ɑ = 0.85). Correlation with traditional star ratings was low (r = 0.24), suggesting THESIS captures issues beyond general user acceptance. Preliminary testing of THESIS suggests apps that serve patients with chronic disease could perform much better, particularly in privacy/security and interoperability. THESIS warrants further testing and may guide software and policymakers to further improve app performance, so apps can more consistently improve patient outcomes.

National Trends in the Safety Performance of Electronic Health Record Systems From 2009 to 2018.

Importance: Despite the broad adoption of electronic health record (EHR) systems across the continuum of care, safety problems persist. Objective: To measure the safety performance of operational EHRs in hospitals across the country during a 10-year period. Design, Setting, and Participants: This case series included all US adult hospitals nationwide that used the National Quality Forum Health IT Safety Measure EHR computerized physician order entry safety test administered by the Leapfrog Group between 2009 and 2018. Data were analyzed from July 1, 2018 to December 1, 2019. Exposure: The Health IT Safety Measure test, which uses simulated medication orders that have either injured or killed patients previously to evaluate how well hospital EHRs could identify medication errors with potential for patient harm. Main Outcomes and Measures: Descriptive statistics for performance on the assessment test over time were calculated at the overall test score level, type of decision support category level, and EHR vendor level. Results: Among 8657 hospital-years observed during the study, mean (SD) scores on the overall test increased from 53.9% (18.3%) in 2009 to 65.6% (15.4%) in 2018. Mean (SD) hospital score for the categories representing basic clinical decision support increased from 69.8% (20.8%) in 2009 to 85.6% (14.9%) in 2018. For the categories representing advanced clinical decision support, the mean (SD) score increased from 29.6% (22.4%) in 2009 to 46.1% (21.6%) in 2018. There was considerable variation in test performance by EHR. Conclusions and Relevance: These findings suggest that despite broad adoption and optimization of EHR systems in hospitals, wide variation in the safety performance of operational EHR systems remains across a large sample of hospitals and EHR vendors. Hospitals using some EHR vendors had significantly higher test scores. Overall, substantial safety risk persists in current hospital EHR systems.

Assessing the safety of electronic health records: a national longitudinal study of medication-related decision support.

BACKGROUND: Electronic health records (EHR) can improve safety via computerised physician order entry with clinical decision support, designed in part to alert providers and prevent potential adverse drug events at entry and before they reach the patient. However, early evidence suggested performance at preventing adverse drug events was mixed. METHODS: We used data from a national, longitudinal sample of 1527 hospitals in the USA from 2009 to 2016 who took a safety performance assessment test using simulated medication orders to test how well their EHR prevented medication errors with potential for patient harm. We calculated the descriptive statistics on performance on the assessment over time, by years of hospital experience with the test and across hospital characteristics. Finally, we used ordinary least squares regression to identify hospital characteristics associated with higher test performance. RESULTS: The average hospital EHR system correctly prevented only 54.0% of potential adverse drug events tested on the 44-order safety performance assessment in 2009; this rose to 61.6% in 2016. Hospitals that took the assessment multiple times performed better in subsequent years than those taking the test the first time, from 55.2% in the first year of test experience to 70.3% in the eighth, suggesting efforts to participate in voluntary self-assessment and improvement may be helpful in improving medication safety performance. CONCLUSION: Hospital medication order safety performance has improved over time but is far from perfect. The specifics of EHR medication safety implementation and improvement play a key role in realising the benefits of computerising prescribing, as organisations have substantial latitude in terms of what they implement. Intentional quality improvement efforts appear to be a critical part of high safety performance and may indicate the importance of a culture of safety.

The Impact of Alternative Payment in Chronically Ill and Older Patients in the Patient-centered Medical Home.

BACKGROUND: Patient-centered medical home (PCMH) has gained prominence as a promising model to encourage improved primary care delivery. There is a paucity of studies that evaluate the impact of payment models in the PCMH. OBJECTIVES: We sought to examine whether coupling coordinated, team-based care transformation plan with a novel reimbursement model affects outcomes related to expenditures and utilization. RESEARCH DESIGN: Interrupted time-series model with a difference-in-differences approach to assess differences between intervention and control groups, across time periods attributable to PCMH transformation and/or payment change. RESULTS: Although results were modest and mixed overall, PCMH with payment reform is associated with a reduction of $1.04 (P=0.0347) per member per month (PMPM) in pharmacy expenditures. Patients with hypertension, hyperlipidemia, diabetes, and coronary atherosclerosis enrolled in PCMH without payment reform experienced reductions in emergency department visits of 2.16 (P<0.0001), 2.42 (P<0.0001), 3.98 (P<0.0001), and 3.61 (P<0.0001) per 1000 per month. Modest increases in inpatient admission were seen among these patients in PCMH either with or without payment reform. Patients 65 and older enrolled in PMCH without payment reform experienced reductions in pharmacy expenditures $2.35 (P=0.0077) PMPM with a parallel reduction in pharmacy standardized cost of $2.81 (P=0.0174) PMPM indicative of a reduction in the intensity of drug utilization. CONCLUSIONS: We conclude that PCMH implementation coupled with an innovative payment arrangement generated mixed results with modest improvements with respect to pharmacy expenditures, but no overall financial improvement. However, we did see improvement within specific groups, especially older patients and those with chronic conditions.

Patient-Facing Mobile Apps to Treat High-Need, High-Cost Populations: A Scoping Review.

BACKGROUND: Self-management is essential to caring for high-need, high-cost (HNHC) populations. Advances in mobile phone technology coupled with increased availability and adoption of health-focused mobile apps have made self-management more achievable, but the extent and quality of the literature supporting their use is not well defined. OBJECTIVE: The purpose of this review was to assess the breadth, quality, bias, and types of outcomes measured in the literature supporting the use of apps targeting HNHC populations. METHODS: Data sources included articles in PubMed and MEDLINE (National Center for Biotechnology Information), EMBASE (Elsevier), the Cochrane Central Register of Controlled Trials (EBSCO), Web of Science (Thomson Reuters), and the NTIS (National Technical Information Service) Bibliographic Database (EBSCO) published since 2008. We selected studies involving use of patient-facing iOS or Android mobile health apps. Extraction was performed by 1 reviewer; 40 randomly selected articles were evaluated by 2 reviewers to assess agreement. RESULTS: Our final analysis included 175 studies. The populations most commonly targeted by apps included patients with obesity, physical handicaps, diabetes, older age, and dementia. Only 30.3% (53/175) of the apps studied in the reviewed literature were identifiable and available to the public through app stores. Many of the studies were cross-sectional analyses (42.9%, 75/175), small (median number of participants=31, interquartile range 11.0-207.2, maximum 11,690), or performed by an app's developers (61.1%, 107/175). Of the 175 studies, only 36 (20.6%, 36/175) studies evaluated a clinical outcome. CONCLUSIONS: Most apps described in the literature could not be located on the iOS or Android app stores, and existing research does not robustly evaluate the potential of mobile apps. Whereas apps may be useful in patients with chronic conditions, data do not support this yet. Although we had 2-3 reviewers to screen and assess abstract eligibility, only 1 reviewer abstracted the data. This is one limitation of our study. With respect to the 40 articles (22.9%, 40/175) that were assigned to 2 reviewers (of which 3 articles were excluded), inter-rater agreement was significant on the majority of items (17 of 30) but fair-to-moderate on others.

Many Mobile Health Apps Target High-Need, High-Cost Populations, But Gaps Remain.

With rising smartphone ownership, mobile health applications (mHealth apps) have the potential to support high-need, high-cost populations in managing their health. While the number of available mHealth apps has grown substantially, no clear strategy has emerged on how providers should evaluate and recommend such apps to patients. Key stakeholders, including medical professional societies, insurers, and policy makers, have largely avoided formally recommending apps, which forces patients to obtain recommendations from other sources. To help stakeholders overcome barriers to reviewing and recommending apps, we evaluated 137 patient-facing mHealth apps-those intended for use by patients to manage their health-that were highly rated by consumers and recommended by experts and that targeted high-need, high-cost populations. We found that there is a wide variety of apps in the marketplace but that few apps address the needs of the patients who could benefit the most. We also found that consumers' ratings were poor indications of apps' clinical utility or usability and that most apps did not respond appropriately when a user entered potentially dangerous health information. Going forward, data privacy and security will continue to be major concerns in the dissemination of mHealth apps.

Evaluation: A Qualitative Pilot Study of Novel Information Technology Infrastructure to Communicate Genetic Variant Updates.

BACKGROUND: Partners HealthCare Personalized Medicine developed GeneInsight Clinic (GIC), a tool designed to communicate updated variant information from laboratory geneticists to treating clinicians through automated alerts, categorized by level of variant interpretation change. OBJECTIVES: The study aimed to evaluate feedback from the initial users of the GIC, including the advantages and challenges to receiving this variant information and using this technology at the point of care. METHODS: Healthcare professionals from two clinics that ordered genetic testing for cardiomyopathy and related disorders were invited to participate in one-hour semi-structured interviews and/ or a one-hour focus group. Using a Grounded Theory approach, transcript concepts were coded and organized into themes. RESULTS: Two genetic counselors and two physicians from two treatment clinics participated in individual interviews. Focus group participants included one genetic counselor and four physicians. Analysis resulted in 8 major themes related to structuring and communicating variant knowledge, GIC's impact on the clinic, and suggestions for improvements. The interview analysis identified longitudinal patient care, family data, and growth in genetic testing content as potential challenges to optimization of the GIC infrastructure. DISCUSSION: Participants agreed that GIC implementation increased efficiency and effectiveness of the clinic through increased access to genetic variant information at the point of care. CONCLUSION: Development of information technology (IT) infrastructure to aid in the organization and management of genetic variant knowledge will be critical as the genetic field moves towards whole exome and whole genome sequencing. Findings from this study could be applied to future development of IT support for genetic variant knowledge management that would serve to improve clinicians' ability to manage and care for patients.

Usability of Commercially Available Mobile Applications for Diverse Patients.

BACKGROUND: Mobile applications or 'apps' intended to help people manage their health and chronic conditions are widespread and gaining in popularity. However, little is known about their acceptability and usability for low-income, racially/ethnically diverse populations who experience a disproportionate burden of chronic disease and its complications. OBJECTIVE: The objective of this study was to investigate the usability of existing mobile health applications ("apps") for diabetes, depression, and caregiving, in order to facilitate development and tailoring of patient-facing apps for diverse populations. DESIGN: Usability testing, a mixed-methods approach that includes interviewing and direct observation of participant technology use, was conducted with participants (n = 9 caregivers; n = 10 patients with depression; and n = 10 patients with diabetes) on a total of 11 of the most popular health apps (four diabetes apps, four depression apps, and three caregiver apps) on both iPad and Android tablets. PARTICIPANTS: The participants were diverse: 15 (58 %) African Americans, seven (27 %) Whites, two (8 %) Asians, two (8 %) Latinos with either diabetes, depression, or who were caregivers. MAIN MEASURES: Participants were given condition-specific tasks, such as entering a blood glucose value into a diabetes app. Participant interviews were video recorded and were coded using standard methods to evaluate attempts and completions of tasks. We performed inductive coding of participant comments to identify emergent themes. KEY RESULTS: Participants completed 79 of 185 (43 %) tasks across 11 apps without assistance. Three themes emerged from participant comments: lack of confidence with technology, frustration with design features and navigation, and interest in having technology to support their self-management. CONCLUSIONS: App developers should employ participatory design strategies in order to have an impact on chronic conditions such as diabetes and depression that disproportionately affect vulnerable populations. While patients express interest in using technologies for self-management, current tools are not consistently usable for diverse patients.

Developing a Framework for Evaluating the Patient Engagement, Quality, and Safety of Mobile Health Applications.

Rising ownership of smartphones and tablets across social and demographic groups has made mobile applications, or apps, a potentially promising tool for engaging patients in their health care, particularly those with high health care needs. Through a systematic search of iOS (Apple) and Android app stores and an analysis of apps targeting individuals with chronic illnesses, we assessed the degree to which apps are likely to be useful in patient engagement efforts. Usefulness was determined based on the following criteria: description of engagement, relevance to the targeted patient population, consumer ratings and reviews, and most recent app update. Among the 1,046 health care-related, patient-facing applications identified by our search, 43 percent of iOS apps and 27 percent of Android apps appeared likely to be useful. We also developed criteria for evaluating the patient engagement, quality, and safety of mobile apps.

Linking acknowledgement to action: closing the loop on non-urgent, clinically significant test results in the electronic health record.

Failure to follow-up nonurgent, clinically significant test results (CSTRs) is an ambulatory patient safety concern. Tools within electronic health records (EHRs) may facilitate test result acknowledgment, but their utility with regard to nonurgent CSTRs is unclear. We measured use of an acknowledgment tool by 146 primary care physicians (PCPs) at 13 network-affiliated practices that use the same EHR. We then surveyed PCPs to assess use of, satisfaction with, and desired enhancements to the acknowledgment tool. The rate of acknowledgment of non-urgent CSTRs by PCPs was 78%. Of 73 survey respondents, 72 reported taking one or more actions after reviewing a CSTR; fewer (40-75%) reported that using the acknowledgment tool was helpful for a specific purpose. Forty-six (64%) were satisfied with the tool. Both satisfied and nonsatisfied PCPs reported that enhancements linking acknowledgment to routine actions would be useful. EHR vendors should consider enhancements to acknowledgment functionality to ensure follow-up of nonurgent CSTRs.

A novel clinician interface to improve clinician access to up-to-date genetic results.

OBJECTIVES: To understand the impact of GeneInsight Clinic (GIC), a web-based tool designed to manage genetic information and facilitate communication of test results and variant updates from the laboratory to the clinics, we measured the use of GIC and the time it took for new genetic knowledge to be available to clinicians. METHODS: Usage data were collected across four study sites for the GIC launch and post-GIC implementation time periods. The primary outcome measures were the time (average number of days) between variant change approval and notification of clinic staff, and the time between notification and viewing the patient record. RESULTS: Post-GIC, time between a variant change approval and provider notification was shorter than at launch (average days at launch 503.8, compared to 4.1 days post-GIC). After e-mail alerts were sent at launch, providers clicked into the patient record associated with 91% of these alerts. In the post period, clinic providers clicked into the patient record associated with 95% of the alerts, on average 12 days after the e-mail was sent. DISCUSSION: We found that GIC greatly increased the likelihood that a provider would receive updated variant information as well as reduced the time associated with distributing that variant information, thus providing a more efficient process for incorporating new genetic knowledge into clinical care. CONCLUSIONS: Our study results demonstrate that health information technology systems have the potential effectively to assist providers in utilizing genetic information in patient care.

Usability of a novel clinician interface for genetic results.

The complexity and rapid growth of genetic data demand investment in information technology to support effective use of this information. Creating infrastructure to communicate genetic information to healthcare providers and enable them to manage that data can positively affect a patient's care in many ways. However, genetic data are complex and present many challenges. We report on the usability of a novel application designed to assist providers in receiving and managing a patient's genetic profile, including ongoing updated interpretations of the genetic variants in those patients. Because these interpretations are constantly evolving, managing them represents a challenge. We conducted usability tests with potential users of this application and reported findings to the application development team, many of which were addressed in subsequent versions. Clinicians were excited about the value this tool provides in pushing out variant updates to providers and overall gave the application high usability ratings, but had some difficulty interpreting elements of the interface. Many issues identified required relatively little development effort to fix suggesting that consistently incorporating this type of analysis in the development process can be highly beneficial. For genetic decision support applications, our findings suggest the importance of designing a system that can deliver the most current knowledge and highlight the significance of new genetic information for clinical care. Our results demonstrate that using a development and design process that is user focused helped optimize the value of this application for personalized medicine.

Actionable reminders did not improve performance over passive reminders for overdue tests in the primary care setting.

Actionable reminders (electronic reminders linked to computerized order entry) might improve care by facilitating direct ordering of recommended tests. The authors implemented four enhanced actionable reminders targeting performance of annual mammography, one-time bone-density screening, and diabetic testing. There was no difference in rates of appropriate testing between the four intervention and four matched, control primary care clinics for screening mammography (OR 0.81, 95% CI 0.64 to 1.02), bone-density exams (OR 1.29, 95% CI 0.82 to 2.02), HbA1c monitoring (OR 0.91, 95% CI 0.58 to 1.42) and LDL cholesterol monitoring (OR 1.40, 95% CI 0.76 to 2.59). Of the survey respondents, 79% almost never used the system or were unaware of the functionality. In the 9/228 (3.9%) cases with indirect evidence of mammography reminder use, there was a significantly lower proportion with test performance. Our actionable reminders did not improve receipt of overdue testing, potentially due to limitations of workflow integration.

Trends in primary care clinician perceptions of a new electronic health record.

BACKGROUND: Clinician perceptions of a newly implemented electronic health record play an important role in its success or failure. OBJECTIVE: To measure changes in primary care clinician attitudes toward an electronic health record during the first year following implementation. DESIGN: Longitudinal survey. PARTICIPANTS: 86 primary care clinicians surveyed between December 2006 and January 2008. MEASUREMENTS: Perceived impact on overall quality of care, patient safety, communication, and efficiency at 1, 3, 6, and 12 months following implementation. RESULTS: Response rates for months 1, 3, 6, and 12 were 92%, 95%, 90%, and 82%, respectively. The proportion of clinicians agreeing that the EHR improved the overall quality of care (63% to 86%; p < 0.001), reduced medication-related errors (72% to 81%; p = 0.03), improved follow-up of test results (62% to 87%; p < 0.001), and improved communication among clinicians (72% to 93%; p < 0.001) increased from month 1 to month 12. During the same time period, a decreasing proportion of clinicians agreed that the EHR reduced the quality of patient interactions (49% to 33%; p = 0.001), resulted in longer patient visits (68% to 51%; p = 0.001), and increased time spent on medical documentation (78% to 68%; p = 0.006). Significant improvements in perceptions related to test result follow-up were first detected at 6 months, while those related to overall quality, efficiency, and communication were first identified at 12 months. CONCLUSIONS: Primary care clinicians report increasingly positive perceptions of a new electronic health record within 1 year of implementation across a spectrum of domains of care.

Trends in clinician perceptions of a new electronic health record.

Clinician perceptions of a newly implemented electronic health record (EHR) play an important role in its success or failure. Limited data exist describing how these attitudes change over time. We measured attitudes of primary care clinicians toward a new EHR during the first year following implementation. We found that perceptions of the new system became significantly more favorable over time with variable rates of change for different aspects of the EHR.

Linking test ordering with order tracking: closing the loop in ambulatory care.

Test non-completion decreases quality of care and accounts for many diagnosis-related malpractice claims. Currently, clinicians using Partners' electronic Longitudinal Medical Record (LMR) can track results but lack a mechanism for tracking non-completed tests. This pilot intervention will study an "order tracking" functionality that flags newly-ordered tests and will lead to generation of written patient reminders if tests are not completed within pre-specified timeframes. If test completion rates improve, we will pursue development of a dedicated LMR application.