Prevalence of Short Peer Reviews in 3 Leading General Medical Journals.

Importance: High-quality peer reviews are often thought to be essential to ensuring the integrity of the scientific publication process, but measuring peer review quality is challenging. Although imperfect, review word count could potentially serve as a simple, objective metric of review quality. Objective: To determine the prevalence of very short peer reviews and how often they inform editorial decisions on research articles in 3 leading general medical journals. Design, Setting, and Participants: This cross-sectional study compiled a data set of peer reviews from published, full-length original research articles from 3 general medical journals (The BMJ, PLOS Medicine, and BMC Medicine) between 2003 and 2022. Eligible articles were those with peer review data; all peer reviews used to make the first editorial decision (ie, accept vs revise and resubmit) were included. Main Outcomes and Measures: Prevalence of very short reviews was the primary outcome, which was defined as a review of fewer than 200 words. In secondary analyses, thresholds of fewer than 100 words and fewer than 300 words were used. Results were disaggregated by journal and year. The proportion of articles for which the first editorial decision was made based on a set of peer reviews in which very short reviews constituted 100%, 50% or more, 33% or more, and 20% or more of the reviews was calculated. Results: In this sample of 11 466 reviews (including 6086 in BMC Medicine, 3816 in The BMJ, and 1564 in PLOS Medicine) corresponding to 4038 published articles, the median (IQR) word count per review was 425 (253-575) words, and the mean (SD) word count was 520.0 (401.0) words. The overall prevalence of very short (<200 words) peer reviews was 1958 of 11 466 reviews (17.1%). Across the 3 journals, 843 of 4038 initial editorial decisions (20.9%) were based on review sets containing 50% or more very short reviews. The prevalence of very short reviews and share of editorial decisions based on review sets containing 50% or more very short reviews was highest for BMC Medicine (693 of 2585 editorial decisions [26.8%]) and lowest for The BMJ (76 of 1040 editorial decisions [7.3%]). Conclusion and Relevance: In this study of 3 leading general medical journals, one-fifth of initial editorial decisions for published articles were likely based at least partially on reviews of such short length that they were unlikely to be of high quality. Future research could determine whether monitoring peer review length improves the quality of peer reviews and which interventions, such as incentives and norm-based interventions, may elicit more detailed reviews.

A multisite comparison using electronic health records and natural language processing to identify the association between suicidality and hospital readmission amongst patients with eating disorders.

OBJECTIVES: To describe and compare the association between suicidality and subsequent readmission for patients hospitalized for eating disorder treatment, within 2 years of discharge, at two large academic medical centers in two different countries. METHODS: Over an 8-year study window from January 2009 to March 2017, we identified all inpatient eating disorder admissions at Weill Cornell Medicine, New York, USA (WCM) and South London and Maudsley Foundation NHS Trust, London, UK (SLaM). To establish each patient's-suicidality profile, we applied two natural language processing (NLP) algorithms, independently developed at the two institutions, and detected suicidality in clinical notes documented in the first week of admission. We calculated the odds ratios (OR) for any subsequent readmission within 2 years postdischarge and determined whether this was to another eating disorder unit, other psychiatric unit, a general medical hospital admission or emergency room attendance. RESULTS: We identified 1126 and 420 eating disorder inpatient admissions at WCM and SLaM, respectively. In the WCM cohort, evidence of above average suicidality during the first week of admission was significantly associated with an increased risk of noneating disorder-related psychiatric readmission (OR 3.48 95% CI = 2.03-5.99, p-value < .001), but a similar pattern was not observed in the SLaM cohort (OR 1.34, 95% CI = 0.75-2.37, p = .32), there was no significant increase in risk of admission. In both cohorts, personality disorder increased the risk of any psychiatric readmission within 2 years. DISCUSSION: Patterns of increased risk of psychiatric readmission from above average suicidality detected via NLP during inpatient eating disorder admissions differed in our two patient cohorts. However, comorbid diagnoses such as personality disorder increased the risk of any psychiatric readmission across both cohorts. PUBLIC SIGNIFICANCE: Suicidality amongst is eating disorders is an extremely common presentation and it is important we further our understanding of identifying those most at risk. This research also provides a novel study design, comparing two NLP algorithms on electronic health record data based in the United States and United Kingdom on eating disorder inpatients. Studies researching both UK and US mental health patients are sparse therefore this study provides novel data.

Population-Wide Screening for Chronic Kidney Disease : A Cost-Effectiveness Analysis.

BACKGROUND: Sodium-glucose cotransporter-2 (SGLT2) inhibitors have the potential to alter the natural history of chronic kidney disease (CKD), and they should be included in cost-effectiveness analyses of screening for CKD. OBJECTIVE: To determine the cost-effectiveness of adding population-wide screening for CKD. DESIGN: Markov cohort model. DATA SOURCES: NHANES (National Health and Nutrition Examination Survey), U.S. Centers for Medicare & Medicaid Services data, cohort studies, and randomized clinical trials, including the DAPA-CKD (Dapagliflozin and Prevention of Adverse Outcomes in Chronic Kidney Disease) trial. TARGET POPULATION: Adults. TIME HORIZON: Lifetime. PERSPECTIVE: Health care sector. INTERVENTION: Screening for albuminuria with and without adding SGLT2 inhibitors to the current standard of care for CKD. OUTCOME MEASURES: Costs, quality-adjusted life-years (QALYs), and incremental cost-effectiveness ratios (ICERs), all discounted at 3% annually. RESULTS OF BASE-CASE ANALYSIS: One-time CKD screening at age 55 years had an ICER of $86 300 per QALY gained by increasing costs from $249 800 to $259 000 and increasing QALYs from 12.61 to 12.72; this was accompanied by a decrease in the incidence of kidney failure requiring dialysis or kidney transplant of 0.29 percentage points and an increase in life expectancy from 17.29 to 17.45 years. Other options were also cost-effective. During ages 35 to 75 years, screening once prevented dialysis or transplant in 398 000 people and screening every 10 years until age 75 years cost less than $100 000 per QALY gained. RESULTS OF SENSITIVITY ANALYSIS: When SGLT2 inhibitors were 30% less effective, screening every 10 years during ages 35 to 75 years cost between $145 400 and $182 600 per QALY gained, and price reductions would be required for screening to be cost-effective. LIMITATION: The efficacy of SGLT2 inhibitors was derived from a single randomized controlled trial. CONCLUSION: Screening adults for albuminuria to identify CKD could be cost-effective in the United States. PRIMARY FUNDING SOURCE: Agency for Healthcare Research and Quality, Veterans Affairs Office of Academic Affiliations, and National Institute of Diabetes and Digestive and Kidney Diseases.

Measuring concordance of data sources used for infectious disease research in the USA: a retrospective data analysis.

OBJECTIVES: As highlighted by the COVID-19 pandemic, researchers are eager to make use of a wide variety of data sources, both government-sponsored and alternative, to characterise the epidemiology of infectious diseases. The objective of this study is to investigate the strengths and limitations of sources currently being used for research. DESIGN: Retrospective descriptive analysis. PRIMARY AND SECONDARY OUTCOME MEASURES: Yearly number of national-level and state-level disease-specific case counts and disease clusters for three diseases (measles, mumps and varicella) during a 5-year study period (2013-2017) across four different data sources: Optum (health insurance billing claims data), HealthMap (online news surveillance data), Morbidity and Mortality Weekly Reports (official government reports) and National Notifiable Disease Surveillance System (government case surveillance data). RESULTS: Our study demonstrated drastic differences in reported infectious disease incidence across data sources. When compared with the other three sources of interest, Optum data showed substantially higher, implausible standardised case counts for all three diseases. Although there was some concordance in identified state-level case counts and disease clusters, all four sources identified variations in state-level reporting. CONCLUSIONS: Researchers should consider data source limitations when attempting to characterise the epidemiology of infectious diseases. Some data sources, such as billing claims data, may be unsuitable for epidemiological research within the infectious disease context.

Impact of air pollution on running performance.

Air pollution exposures during training may impact race preformances. We aggregated data on 334 collegiate male track & field athletes from 46 universities across the United States over 2010-2014. Using distributed lag non-linear models, we analyzed the relationship between race time and PM2.5, ozone, and two versions of the Air Quality Index (AQI) exposures up to 21 days prior to the race. We observed a 12.8 (95% CI: 1.3, 24.2) second and 11.5 (95% CI: 0.8, 22.1) second increase in race times from 21 days of PM2.5 exposure (10.0 versus 5.0 µg/m3) and ozone exposure (54.9 versus 36.9 ppm), respectively. Exposure measured by the two-pollutant threshold (PM2.5 and ozone) AQI was not significantly associated with race time; however, the association for summed two-pollutant AQI (PM2.5 plus ozone) was similar to associations observed for the individual pollutants (12.4, 95% CI: 1.8, 23.0 s). Training and competing at elevated air pollution levels, even at exposures within AQI's good-to-moderate classifications, was associated with slower race times. This work provides an initial characterization of the effect of air pollution on running performance and a justification for why coaches should consider approaches to reduce air pollution exposures while training.

Comparative Merits of Available Mortality Data Sources for Clinical Research.

Obtaining reliable data on patient mortality is a critical challenge facing observational researchers seeking to conduct studies using real-world data. As these analyses are conducted more broadly using newly-available sources of real-world evidence, missing data can serve as a rate-limiting factor. We conducted a comparison of mortality data sources from different stakeholder perspectives - academic medical center (AMC) informatics service providers, AMC research coordinators, industry analytics professionals, and academics - to understand the strengths and limitations of differing mortality data sources: locally generated data from sites conducting research, data provided by governmental sources, and commercially available data sets. Researchers seeking to conduct observational studies using extant data should consider these factors in sourcing outcomes data for their populations of interest.

Data Source Concordance for Infectious Disease Epidemiology.

Background: As highlighted by the COVID-19 pandemic, researchers are eager to make use of a wide variety of data sources, both government-sponsored and alternative, to characterize the epidemiology of infectious diseases. To date, few studies have investigated the strengths and limitations of sources currently being used for such research. These are critical for policy makers to understand when interpreting study findings. Methods: To fill this gap in the literature, we compared infectious disease reporting for three diseases (measles, mumps, and varicella) across four different data sources: Optum (health insurance billing claims data), HealthMap (online news surveillance data), Morbidity and Mortality Weekly Reports (official government reports), and National Notifiable Disease Surveillance System (government case surveillance data). We reported the yearly number of national- and state-level disease-specific case counts and disease clusters according to each of our sources during a five-year study period (2013-2017). Findings: Our study demonstrated drastic differences in reported infectious disease incidence across data sources. When compared against the other three sources of interest, Optum data showed substantially higher, implausible standardized case counts for all three diseases. Although there was some concordance in identified state-level case counts and disease clusters, all four sources identified variations in state-level reporting. Interpretation: Researchers should consider data source limitations when attempting to characterize the epidemiology of infectious diseases. Some data sources, such as billing claims data, may be unsuitable for epidemiological research within the infectious disease context.

Cost-Effectiveness of Dapagliflozin for Non-diabetic Chronic Kidney Disease.

BACKGROUND: In the USA, chronic kidney disease (CKD) affects 1 in 7 adults and costs $100 billion annually. The DAPA-CKD trial found dapagliflozin, a sodium glucose co-transporter 2 (SGLT2) inhibitor, to be effective in reducing CKD progression and mortality in patients with diabetic and non-diabetic CKD. Currently, SGLT2 inhibitors are not considered standard of care for patients with non-diabetic CKD. OBJECTIVE: Determine the cost-effectiveness of adding dapagliflozin to standard management of patients with non-diabetic CKD. DESIGN: Markov model with lifetime time horizon and US healthcare sector perspective. PATIENTS: Patients with non-diabetic CKD INTERVENTION: Dapagliflozin plus standard care versus standard care only. MAIN MEASURES: Quality-adjusted life years (QALYs), costs, and incremental cost-effectiveness ratios (ICERs), all discounted at 3% annually; total incidence of kidney failure on kidney replacement therapy; average years on kidney replacement therapy. KEY RESULTS: Adding dapagliflozin to standard care improved life expectancy by 2 years, increased discounted QALYS (from 6.75 to 8.06), and reduced the total incidence of kidney failure on kidney replacement therapy (KRT) (from 17.4 to 11.0%) and average years on KRT (from 0.77 to 0.43) over the lifetime of the cohort. Dapagliflozin plus standard care was more effective than standard care alone while increasing lifetime costs (from $245,900 to $324,8900, or $60,000 per QALY gained). Results were robust to variations in assumptions about dapagliflozin's efficacy over time and by CKD stage, added costs of kidney replacement therapy, and expected population annual CKD progression rates and sensitive to the cost of dapagliflozin. The net 1-year budgetary implication of treating all US patients with non-diabetic CKD could be up to $21 billion. CONCLUSIONS: Dapagliflozin improved life expectancy and reduced progression of CKD, the proportion of patients requiring kidney replacement therapy, and time on kidney replacement therapy in patients with non-diabetic CKD. Use of dapagliflozin meets conventional criteria for cost-effectiveness.

Portability of natural language processing methods to detect suicidality from clinical text in US and UK electronic health records.

Background: In the global effort to prevent death by suicide, many academic medical institutions are implementing natural language processing (NLP) approaches to detect suicidality from unstructured clinical text in electronic health records (EHRs), with the hope of targeting timely, preventative interventions to individuals most at risk of suicide. Despite the international need, the development of these NLP approaches in EHRs has been largely local and not shared across healthcare systems. Methods: In this study, we developed a process to share NLP approaches that were individually developed at King's College London (KCL), UK and Weill Cornell Medicine (WCM), US - two academic medical centers based in different countries with vastly different healthcare systems. We tested and compared the algorithms' performance on manually annotated clinical notes (KCL: n = 4,911 and WCM = 837). Results: After a successful technical porting of the NLP approaches, our quantitative evaluation determined that independently developed NLP approaches can detect suicidality at another healthcare organization with a different EHR system, clinical documentation processes, and culture, yet do not achieve the same level of success as at the institution where the NLP algorithm was developed (KCL approach: F1-score 0.85 vs. 0.68, WCM approach: F1-score 0.87 vs. 0.72). Limitations: Independent NLP algorithm development and patient cohort selection at the two institutions comprised direct comparability. Conclusions: Shared use of these NLP approaches is a critical step forward towards improving data-driven algorithms for early suicide risk identification and timely prevention.

Automated Extraction of Tumor Staging and Diagnosis Information From Surgical Pathology Reports.

PURPOSE: Typically stored as unstructured notes, surgical pathology reports contain data elements valuable to cancer research that require labor-intensive manual extraction. Although studies have described natural language processing (NLP) of surgical pathology reports to automate information extraction, efforts have focused on specific cancer subtypes rather than across multiple oncologic domains. To address this gap, we developed and evaluated an NLP method to extract tumor staging and diagnosis information across multiple cancer subtypes. METHODS: The NLP pipeline was implemented on an open-source framework called Leo. We used a total of 555,681 surgical pathology reports of 329,076 patients to develop the pipeline and evaluated our approach on subsets of reports from patients with breast, prostate, colorectal, and randomly selected cancer subtypes. RESULTS: Averaged across all four cancer subtypes, the NLP pipeline achieved an accuracy of 1.00 for International Classification of Diseases, Tenth Revision codes, 0.89 for T staging, 0.90 for N staging, and 0.97 for M staging. It achieved an F1 score of 1.00 for International Classification of Diseases, Tenth Revision codes, 0.88 for T staging, 0.90 for N staging, and 0.24 for M staging. CONCLUSION: The NLP pipeline was developed to extract tumor staging and diagnosis information across multiple cancer subtypes to support the research enterprise in our institution. Although it was not possible to demonstrate generalizability of our NLP pipeline to other institutions, other institutions may find value in adopting a similar NLP approach-and reusing code available at GitHub-to support the oncology research enterprise with elements extracted from surgical pathology reports.

Critical carE Database for Advanced Research (CEDAR): An automated method to support intensive care units with electronic health record data.

Patients treated in an intensive care unit (ICU) are critically ill and require life-sustaining organ failure support. Existing critical care data resources are limited to a select number of institutions, contain only ICU data, and do not enable the study of local changes in care patterns. To address these limitations, we developed the Critical carE Database for Advanced Research (CEDAR), a method for automating extraction and transformation of data from an electronic health record (EHR) system. Compared to an existing gold standard of manually collected data at our institution, CEDAR was statistically similar in most measures, including patient demographics and sepsis-related organ failure assessment (SOFA) scores. Additionally, CEDAR automated data extraction obviated the need for manual collection of 550 variables. Critically, during the spring 2020 COVID-19 surge in New York City, a modified version of CEDAR supported pandemic response efforts, including clinical operations and research. Other academic medical centers may find value in using the CEDAR method to automate data extraction from EHR systems to support ICU activities.

Using weak supervision and deep learning to classify clinical notes for identification of current suicidal ideation.

Mental health concerns, such as suicidal thoughts, are frequently documented by providers in clinical notes, as opposed to structured coded data. In this study, we evaluated weakly supervised methods for detecting "current" suicidal ideation from unstructured clinical notes in electronic health record (EHR) systems. Weakly supervised machine learning methods leverage imperfect labels for training, alleviating the burden of creating a large manually annotated dataset. After identifying a cohort of 600 patients at risk for suicidal ideation, we used a rule-based natural language processing approach (NLP) approach to label the training and validation notes (n = 17,978). Using this large corpus of clinical notes, we trained several statistical machine learning models-logistic classifier, support vector machines (SVM), Naive Bayes classifier-and one deep learning model, namely a text classification convolutional neural network (CNN), to be evaluated on a manually-reviewed test set (n = 837). The CNN model outperformed all other methods, achieving an overall accuracy of 94% and a F1-score of 0.82 on documents with "current" suicidal ideation. This algorithm correctly identified an additional 42 encounters and 9 patients indicative of suicidal ideation but missing a structured diagnosis code. When applied to a random subset of 5,000 clinical notes, the algorithm classified 0.46% (n = 23) for "current" suicidal ideation, of which 87% were truly indicative via manual review. Implementation of this approach for large-scale document screening may play an important role in point-of-care clinical information systems for targeted suicide prevention interventions and improve research on the pathways from ideation to attempt.

A Method to Improve Availability and Quality of Patient Race Data in an Electronic Health Record System.

BACKGROUND: Although federal regulations mandate documentation of structured race data according to Office of Management and Budget (OMB) categories in electronic health record (EHR) systems, many institutions have reported gaps in EHR race data that hinder secondary use for population-level research focused on underserved populations. When evaluating race data available for research purposes, we found our institution's enterprise EHR contained structured race data for only 51% (1.6 million) of patients. OBJECTIVES: We seek to improve the availability and quality of structured race data available to researchers by integrating values from multiple local sources. METHODS: To address the deficiency in race data availability, we implemented a method to supplement OMB race values from four local sources-inpatient EHR, inpatient billing, natural language processing, and coded clinical observations. We evaluated this method by measuring race data availability and data quality with respect to completeness, concordance, and plausibility. RESULTS: The supplementation method improved race data availability in the enterprise EHR up to 10% for some minority groups and 4% overall. We identified structured OMB race values for more than 142,000 patients, nearly a third of whom were from racial minority groups. Our data quality evaluation indicated that the supplemented race values improved completeness in the enterprise EHR, originated from sources in agreement with the enterprise EHR, and were unbiased to the enterprise EHR. CONCLUSION: Implementation of this method can successfully increase OMB race data availability, potentially enhancing accrual of patients from underserved populations to research studies.

Risk of Ischemic Stroke in Patients With Coronavirus Disease 2019 (COVID-19) vs Patients With Influenza.

IMPORTANCE: It is uncertain whether coronavirus disease 2019 (COVID-19) is associated with a higher risk of ischemic stroke than would be expected from a viral respiratory infection. OBJECTIVE: To compare the rate of ischemic stroke between patients with COVID-19 and patients with influenza, a respiratory viral illness previously associated with stroke. DESIGN, SETTING, AND PARTICIPANTS: This retrospective cohort study was conducted at 2 academic hospitals in New York City, New York, and included adult patients with emergency department visits or hospitalizations with COVID-19 from March 4, 2020, through May 2, 2020. The comparison cohort included adults with emergency department visits or hospitalizations with influenza A/B from January 1, 2016, through May 31, 2018 (spanning moderate and severe influenza seasons). EXPOSURES: COVID-19 infection confirmed by evidence of severe acute respiratory syndrome coronavirus 2 in the nasopharynx by polymerase chain reaction and laboratory-confirmed influenza A/B. MAIN OUTCOMES AND MEASURES: A panel of neurologists adjudicated the primary outcome of acute ischemic stroke and its clinical characteristics, mechanisms, and outcomes. We used logistic regression to compare the proportion of patients with COVID-19 with ischemic stroke vs the proportion among patients with influenza. RESULTS: Among 1916 patients with emergency department visits or hospitalizations with COVID-19, 31 (1.6%; 95% CI, 1.1%-2.3%) had an acute ischemic stroke. The median age of patients with stroke was 69 years (interquartile range, 66-78 years); 18 (58%) were men. Stroke was the reason for hospital presentation in 8 cases (26%). In comparison, 3 of 1486 patients with influenza (0.2%; 95% CI, 0.0%-0.6%) had an acute ischemic stroke. After adjustment for age, sex, and race, the likelihood of stroke was higher with COVID-19 infection than with influenza infection (odds ratio, 7.6; 95% CI, 2.3-25.2). The association persisted across sensitivity analyses adjusting for vascular risk factors, viral symptomatology, and intensive care unit admission. CONCLUSIONS AND RELEVANCE: In this retrospective cohort study from 2 New York City academic hospitals, approximately 1.6% of adults with COVID-19 who visited the emergency department or were hospitalized experienced ischemic stroke, a higher rate of stroke compared with a cohort of patients with influenza. Additional studies are needed to confirm these findings and to investigate possible thrombotic mechanisms associated with COVID-19.

Characterizing Basic and Complex Usage of i2b2 at an Academic Medical Center.

Developed to enable basic queries for cohort discovery, i2b2 has evolved to support complex queries. Little is known whether query sophistication - and the informatics resources required to support it - addresses researcher needs. In three years at our institution, 609 researchers ran 6,662 queries and requested re-identification of 80 patient cohorts to support specific studies. After characterizing all queries as "basic" or "complex" with respect to use of sophisticated query features, we found that the majority of all queries, and the majority of queries resulting in a request for cohort re-identification, did not use complex i2b2 features. Data domains that required extensive effort to implement saw relatively little use compared to common domains (e.g., diagnoses). These findings suggest that efforts to ensure the performance of basic queries using common data domains may better serve the needs of the research community than efforts to integrate novel domains or introduce complex new features.