DRRisk: A Web-based tool to Assess the Risk of Diabetic Retinopathy through Machine Learning on Electronic Health Records.

Objective: We developed a web-based tool for diabetic retinopathy (DR) risk assessment called DRRisk (https://drandml.cdrewu.edu/) using machine learning on electronic health record (EHR) data, with a goal of preventing vision loss in persons with diabetes, especially in underserved settings. Methods: DRRisk uses Python's Flask framework. Its user-interface is implemented using HTML, CSS and Javascript. Clinical experts were consulted on the tool's design. Results: DRRisk assesses current DR risk for people with diabetes, categorizing their risk level as low, moderate, or high, depending on the percentage of DR risk assigned by the underlying machine learning model. Discussion: A goal of our tool is to help providers prioritize patients at high risk for DR in order to prevent blindness. Conclusion: Our tool uses DR risk factors from EHR data to calculate a diabetic person's current DR risk. It may be useful for identifying unscreened diabetic patients who have undiagnosed DR.

Detecting diabetic retinopathy through machine learning on electronic health record data from an urban, safety net healthcare system.

OBJECTIVE: Clinical guidelines recommend annual eye examinations to detect diabetic retinopathy (DR) in patients with diabetes. However, timely DR detection remains a problem in medically underserved and under-resourced settings in the United States. Machine learning that identifies patients with latent/undiagnosed DR could help to address this problem. MATERIALS AND METHODS: Using electronic health record data from 40 631 unique diabetic patients seen at Los Angeles County Department of Health Services healthcare facilities between January 1, 2015 and December 31, 2017, we compared ten machine learning environments, including five classifier models, for assessing the presence or absence of DR. We also used data from a distinct set of 9300 diabetic patients seen between January 1, 2018 and December 31, 2018 as an external validation set. RESULTS: Following feature subset selection, the classifier with the best AUC on the external validation set was a deep neural network using majority class undersampling, with an AUC of 0.8, the sensitivity of 72.17%, and specificity of 74.2%. DISCUSSION: A deep neural network produced the best AUCs and sensitivity results on the test set and external validation set. Models are intended to be used to screen guideline noncompliant diabetic patients in an urban safety-net setting. CONCLUSION: Machine learning on diabetic patients' routinely collected clinical data could help clinicians in safety-net settings to identify and target unscreened diabetic patients who potentially have undiagnosed DR.

Predictive Models for Diabetic Retinopathy from Non-Image Teleretinal Screening Data.

Introduction: Timely diabetic retinopathy detection remains a problem in medically underserved settings in the US; diabetic patients in these locales have limited access to eye specialists. Teleretinal screening programs have been introduced to address this problem. Methods: Using data on ethnicity, gender, age, hemoglobin A1C, insulin dependence, time since last eye examination, subjective diabetes control, and years with diabetes from 27,116 diabetic patients participating in a Los Angeles County teleretinal screening program, we compared different machine learning methods for predicting retinopathy. The dataset exhibited a class imbalance. Results: Six classifiers learned on the data were predictive of retinopathy. The best model had an AUC of 0.754, sensitivity of 58% and specificity of 80%. Discussion: Successfully detecting retinopathy from diabetic patients' routinely collected clinical data could help clinicians in medically underserved areas identify unscreened diabetic patients who are at risk of developing retinopathy. This work is a step towards that goal.

2018 Salary Survey of AMIA Members: Factors Associated with Higher Salaries.

Greater transparency in salaries overall and in factors associated with differing salaries can help students and professionals plan their careers, discover biases and obstacles, and help advance professional disciplines broadly. In March 2018, we conducted the first salary survey of American Medical Informatics Association members. Our goal was to summarize salary information and provide a nuanced view pertaining to the diverse biomedical informatics community. To identify factors associated with higher salaries, we reviewed average salaries for different groups (physician status, academic status, and different leadership positions) by gender. We also fitted multiple linear regression models for all participants (N = 201) and for gender, physician- and academic-status subgroup. The mean (standard deviation) salary was $181,774 ($99,566). Men earned more than women on average, and especially among professionals from academic settings. More years working in informatics and full-time employment were two factors that were consistently associated with higher salary.

Understanding the Knowledge Gap Experienced by U.S. Safety Net Patients in Teleretinal Screening.

Safety-net patients' socioeconomic barriers interact with limited digital and health literacies to produce a "knowledge gap" that impacts the delivery of healthcare via telehealth technologies. Six focus groups (2 African- American and 4 Latino) were conducted with patients who received teleretinal screening in a U.S. urban safety-net setting. Focus groups were analyzed using a modified grounded theory methodology. Findings indicate that patients' knowledge gap is primarily produced at three points during the delivery of care: (1) exacerbation of patients' pre-existing personal barriers in the clinical setting; (2) encounters with technology during screening; and (3) lack of follow up after the visit. This knowledge gap produces confusion, potentially limiting patients' perceptions of care and their ability to manage their own care. It may be ameliorated through delivery of patient education focused on both disease pathology and specific role of telehealth technologies in disease management.

Machine Learning Approaches for Detecting Diabetic Retinopathy from Clinical and Public Health Records.

INTRODUCTION: Annual eye examinations are recommended for diabetic patients in order to detect diabetic retinopathy and other eye conditions that arise from diabetes. Medically underserved urban communities in the US have annual screening rates that are much lower than the national average and could benefit from informatics approaches to identify unscreened patients most at risk of developing retinopathy. METHODS: Using clinical data from urban safety net clinics as well as public health data from the CDC's National Health and Nutrition Examination Survey, we examined different machine learning approaches for predicting retinopathy from clinical or public health data. All datasets utilized exhibited a class imbalance. RESULTS: Classifiers learned on the clinical data were modestly predictive of retinopathy with the best model having an AUC of 0.72, sensitivity of 69.2% and specificity of 55.9%. Classifiers learned on public health data were not predictive of retinopathy. DISCUSSION: Successful approaches to detecting latent retinopathy using machine learning could help safety net and other clinics identify unscreened patients who are most at risk of developing retinopathy and the use of ensemble classifiers on clinical data shows promise for this purpose.

Evaluating predictive modeling's potential to improve teleretinal screening participation in urban safety net clinics.

INTRODUCTION: Screening guidelines for diabetic patients recommend yearly eye examinations to detect diabetic retinopathy and other forms of diabetic eye disease. However, annual screening rates for retinopathy in US urban safety net settings remain low. METHODS: Using data gathered from a study of teleretinal screening in six urban safety net clinics, we assessed whether predictive modeling could be of value in identifying patients at risk of developing retinopathy. We developed and examined the accuracy of two predictive modeling approaches for diabetic retinopathy in a sample of 513 diabetic individuals, using routinely available clinical variables from retrospective medical record reviews. Bayesian networks and radial basis function (neural) networks were learned using ten-fold cross-validation. RESULTS: The predictive models were modestly predictive with the best model having an AUC of 0.71. DISCUSSION: Using routinely available clinical variables to predict patients at risk of developing retinopathy and to target them for annual eye screenings may be of some usefulness to safety net clinics.

Autonomy versus automation: perceptions of nonmydriatic camera choice for teleretinal screening in an urban safety net clinic.

OBJECTIVE: Teleretinal screening with nonmydriatic cameras has been presented as a means of increasing the number of patients assessed for diabetic retinopathy in urban safety net clinics. It has been hypothesized that automated nonmydriatic cameras may improve screening rates by reducing the learning curve for camera use. In this article, we examine the impact of introducing automated nonmydriatic cameras to urban safety net clinics whose photographers had previously used manual cameras. MATERIALS AND METHODS: We evaluated the impact of manual and automated digital nonmydriatic cameras on teleretinal screening using a quantitative analysis of readers' image quality ratings as well as a qualitative analysis, through in-depth interviews, of photographers' experiences. RESULTS: With the manual camera, 68% of images were rated "adequate" or better, including 24% rated "good" and 20% rated "excellent." With the automated camera, 61% were rated "adequate" or better, including 9% rated "good" and 0% rated "excellent." Photographers expressed frustration with their inability to control image-taking settings with the automated camera, which led to unexpected delays. CONCLUSIONS: For safety net clinics in which medical assistants are already trained to take photographs for diabetic retinopathy screening with a manual camera, the introduction of automated cameras may lead to frustration and paradoxically contribute to increased patient wait times. When photographers have achieved a high degree of aptitude with manual cameras and value the control they have over camera features, the introduction of automated cameras should be approached with caution and may require extensive training to increase user acceptability.

Identifying appropriate reference data models for comparative effectiveness research (CER) studies based on data from clinical information systems.

INTRODUCTION: The need for a common format for electronic exchange of clinical data prompted federal endorsement of applicable standards. However, despite obvious similarities, a consensus standard has not yet been selected in the comparative effectiveness research (CER) community. METHODS: Using qualitative metrics for data retrieval and information loss across a variety of CER topic areas, we compare several existing models from a representative sample of organizations associated with clinical research: the Observational Medical Outcomes Partnership (OMOP), Biomedical Research Integrated Domain Group, the Clinical Data Interchange Standards Consortium, and the US Food and Drug Administration. RESULTS: While the models examined captured a majority of the data elements that are useful for CER studies, data elements related to insurance benefit design and plans were most detailed in OMOP's CDM version 4.0. Standardized vocabularies that facilitate semantic interoperability were included in the OMOP and US Food and Drug Administration Mini-Sentinel data models, but are left to the discretion of the end-user in Biomedical Research Integrated Domain Group and Analysis Data Model, limiting reuse opportunities. Among the challenges we encountered was the need to model data specific to a local setting. This was handled by extending the standard data models. DISCUSSION: We found that the Common Data Model from the OMOP met the broadest complement of CER objectives. Minimal information loss occurred in mapping data from institution-specific data warehouses onto the data models from the standards we assessed. However, to support certain scenarios, we found a need to enhance existing data dictionaries with local, institution-specific information.

Teleretinal screening for diabetic retinopathy in six Los Angeles urban safety-net clinics: final study results.

In a previous paper, we presented initial findings from a study on the feasibility and challenges of implementing teleretinal screening for diabetic retinopathy in an urban safety net setting facing eyecare specialist shortages. This paper presents some final results from that study, which involved six South Los Angeles safety net clinics. A total of 2,732 unique patients were screened for diabetic retinopathy by three ophthalmologist readers, with 1035 receiving a recommendation for referral to specialty care. Referrals included 48 for proliferative diabetic retinopathy, 115 for severe non-proliferative diabetic retinopathy (NPDR), 247 for moderate NPDR, 246 for mild NPDR, 97 for clinically significant macular edema, and 282 for a non-diabetic condition, such as glaucoma. Image quality was also assessed, with ophthalmologist readers grading 4% to 13% of retinal images taken at the different clinics as being inadequate for any diagnostic interpretation.

Workflow concerns and workarounds of readers in an urban safety net teleretinal screening study.

Telemedicine holds great promise for increased access to specialty care services for safety net clinic patients. However, the adoption of these technologies is not a seamless transition for clinicians working in resource-poor settings. Previous research has analyzed workflow issues that arise in primary care settings when adopting telehealth tools but has not examined the unique workflow challenges facing specialists who provide assessments to safety net clinics. Findings are presented from a case study that employed qualitative methodologies as part of an assessment of a teleretinal screening program in Los Angeles urban safety net clinics. The program utilizes external ophthalmologists to perform retinal readings. The case study provides insights into how difficulties that arise in reader workflow are resolved and identifies unique factors requiring consideration when highly trained specialists perform teleretinal readings. The discussion outlines important issues to address when developing telehealth workflow protocols for the safety net, specifically, and their broader applicability in telemedicine.

Teleretinal screening for diabetic retinopathy in six Los Angeles urban safety-net clinics: initial findings.

Diabetic retinopathy is a leading cause of blindness in US adults. This paper presents initial results of a teleretinal screening project for diabetic retinopathy involving six Los Angeles safety net clinics. A total of 1,943 patients have been screened for diabetic retinopathy by three ophthalmologist readers, with 416 receiving a recommendation for referral to specialty care. Of the cases recommended for referral, 24 had proliferative diabetic retinopathy, 62 had severe non-proliferative diabetic retinopathy (NPDR), 60 had moderate NPDR, 19 had mild NPDR, 138 had a non-diabetic condition, such as glaucoma, 63 had clinically significant macular edema without retinopathy and 50 had non-gradable images. Between 3% and 12.2% of retinal images taken at the clinics were assessed by readers as inadequate for any interpretation. The study shows the feasibility and challenges of teleretinal screening for diabetic retinopathy in urban areas facing specialist shortages and an overburdened, under-resourced safety net care-delivery system.

CEDRIC: a computerized chronic disease management system for urban, safety net clinics.

To meet the challenge of improving health care quality in urban, medically underserved areas of the US that have a predominance of chronic diseases such as diabetes, we have developed a new information system called CEDRIC for managing chronic diseases. CEDRIC was developed in collaboration with clinicians at an urban safety net clinic, using a community-participatory partnered research approach, with a view to addressing the particular needs of urban clinics with a high physician turnover and large uninsured/underinsured patient population. The pilot implementation focuses on diabetes management. In this paper, we describe the system's architecture and features.

Renal dysfunction, metabolic syndrome and cardiovascular disease mortality.

Background. Renal disease is commonly described as a complication of metabolic syndrome (MetS) but some recent studies suggest that Chronic Kidney disease (CKD) may actually antecede MetS. Few studies have explored the predictive utility of co-clustering CKD with MetS for cardiovascular disease (CVD) mortality. Methods. Data from a nationally representative sample of United States adults (NHANES) was utilized. A sample of 13115 non-pregnant individuals aged >/=35 years, with available follow-up mortality assessment was selected. Multivariable Cox Proportional hazard regression analysis techniques explored the relationship between co-clustered CKD, MetS and CVD mortality. Bayesian analysis techniques tested the predictive accuracy for CVD Mortality of two models using co-clustered MetS and CKD and MetS alone. Results. Co-clustering early and late CKD respectively resulted in statistically significant higher hazard for CVD mortality (HR = 1.80, CI = 1.45-2.23, and HR = 3.23, CI = 2.56-3.70) when compared with individuals with no MetS and no CKD. A model with early CKD and MetS has a higher predictive accuracy (72.0% versus 67.6%), area under the ROC (0.74 versus 0.66), and Cohen's kappa (0.38 versus 0.21) than that with MetS alone. Conclusion. The study findings suggest that the co-clustering of early CKD with MetS increases the accuracy of risk prediction for CVD mortality.

A comparison of methods for assessing penetrating trauma on retrospective multi-center data.

OBJECTIVE: TraumaSCAN-Web (TSW) is a computerized decision support system for assessing chest and abdominal penetrating trauma which utilizes 3D geometric reasoning and a Bayesian network with subjective probabilities obtained from an expert. The goal of the present study is to determine whether a trauma risk prediction approach using a Bayesian network with a predefined structure and probabilities learned from penetrating trauma data is comparable in diagnostic accuracy to TSW. METHODS: Parameters for two Bayesian networks with expert-defined structures were learned from 637 gunshot and stab wound cases from three hospitals, and diagnostic accuracy was assessed using 10-fold cross-validation. The first network included information on external wound locations, while the second network did not. Diagnostic accuracy of learned networks was compared to that of TSW on 194 previously evaluated cases. RESULTS: For 23 of the 24 conditions modeled by TraumaSCAN-Web, 16 conditions had Areas Under the ROC Curve (AUCs) greater than 0.90 while 21 conditions had AUCs greater than 0.75 for the first network. For the second network, 16 and 20 conditions had AUCs greater than 0.90 and 0.75, respectively. AUC results for learned networks on 194 previously evaluated cases were better than or equal to AUC results for TSW for all diagnoses evaluated except diaphragm and heart injuries. CONCLUSIONS: For 23 of the 24 penetrating trauma conditions studied, a trauma diagnosis approach using Bayesian networks with predefined structure and probabilities learned from penetrating trauma data was better than or equal in diagnostic accuracy to TSW. In many cases, information on wound location in the first network did not significantly add to predictive accuracy. The study suggests that a decision support approach that uses parameter-learned Bayesian networks may be sufficient for assessing some penetrating trauma conditions.

Assessing a Bayesian risk prediction model in a high-risk breast cancer population.

The purpose of this study was to utilize a Bayesian risk prediction model to predict the incidence of breast cancer in a high risk population. 10-fold cross-validation was performed using a Naïve Bayes classifier. The area under the ROC curve (AUC) was used to measure prediction accuracy. These results were then compared to the ROC curve (AUC) results of the Gail Model Risk Assessment Tool.

Methods for reasoning from geometry about anatomic structures injured by penetrating trauma.

This paper presents the methods used for three-dimensional (3D) reasoning about anatomic structures affected by penetrating trauma in TraumaSCAN-Web, a platform-independent decision support system for evaluating the effects of penetrating trauma to the chest and abdomen. In assessing outcomes for an injured patient, TraumaSCAN-Web utilizes 3D models of anatomic structures and 3D models of the regions of damage associated with stab and gunshot wounds to determine the probability of injury to anatomic structures. Probabilities estimated from 3D reasoning about affected anatomic structures serve as input to a Bayesian network which calculates posterior probabilities of injury based on these initial probabilities together with available information about patient signs, symptoms and test results. In addition to displaying textual descriptions of conditions arising from penetrating trauma to a patient, TraumaSCAN-Web allows users to visualize the anatomy suspected of being injured in 3D, in this way providing a guide to its reasoning process.

Evaluating the discriminatory power of a computer-based system for assessing penetrating trauma on retrospective multi-center data.

OBJECTIVE: To evaluate the discriminatory power of TraumaSCAN-Web, a system for assessing penetrating trauma, using retrospective multi-center case data for gunshot and stab wounds to the thorax and abdomen. METHODS: 80 gunshot and 114 stab cases were evaluated using TraumaSCAN-Web. Areas under the Receiver Operator Characteristic Curves (AUC) were calculated for each condition modeled in TraumaSCAN-Web. RESULTS: Of the 23 conditions modeled by TraumaSCAN-Web, 19 were present in either the gunshot or stab case data. The gunshot AUCs ranged from 0.519 (pericardial tamponade) to 0.975 (right renal injury). The stab AUCs ranged from 0.701 (intestinal injury) to 1.000 (tracheal injury).

Design and implementation of the GLIF3 guideline execution engine.

We have developed the GLIF3 Guideline Execution Engine (GLEE) as a tool for executing guidelines encoded in the GLIF3 format. In addition to serving as an interface to the GLIF3 guideline representation model to support the specified functions, GLEE provides defined interfaces to electronic medical records (EMRs) and other clinical applications to facilitate its integration with the clinical information system at a local institution. The execution model of GLEE takes the "system suggests, user controls" approach. A tracing system is used to record an individual patient's state when a guideline is applied to that patient. GLEE can also support an event-driven execution model once it is linked to the clinical event monitor in a local environment. Evaluation has shown that GLEE can be used effectively for proper execution of guidelines encoded in the GLIF3 format. When using it to execute each guideline in the evaluation, GLEE's performance duplicated that of the reference systems implementing the same guideline but taking different approaches. The execution flexibility and generality provided by GLEE, and its integration with a local environment, need to be further evaluated in clinical settings. Integration of GLEE with a specific event-monitoring and order-entry environment is the next step of our work to demonstrate its use for clinical decision support. Potential uses of GLEE also include quality assurance, guideline development, and medical education.

Description and status update on GELLO: a proposed standardized object-oriented expression language for clinical decision support.

A major obstacle to sharing computable clinical knowledge is the lack of a common language for specifying expressions and criteria. Such a language could be used to specify decision criteria, formulae, and constraints on data and action. Al-though the Arden Syntax addresses this problem for clinical rules, its generalization to HL7's object-oriented data model is limited. The GELLO Expression language is an object-oriented language used for expressing logical conditions and computations in the GLIF3 (GuideLine Interchange Format, v. 3) guideline modeling language. It has been further developed under the auspices of the HL7 Clinical Decision Support Technical Committee, as a proposed HL7 standard., GELLO is based on the Object Constraint Language (OCL), because it is vendor-independent, object-oriented, and side-effect-free. GELLO expects an object-oriented data model. Although choice of model is arbitrary, standardization is facilitated by ensuring that the data model is compatible with the HL7 Reference Information Model (RIM).