Performance of two large language models for data extraction in evidence synthesis.

Accurate data extraction is a key component of evidence synthesis and critical to valid results. The advent of publicly available large language models (LLMs) has generated interest in these tools for evidence synthesis and created uncertainty about the choice of LLM. We compare the performance of two widely available LLMs (Claude 2 and GPT-4) for extracting pre-specified data elements from 10 published articles included in a previously completed systematic review. We use prompts and full study PDFs to compare the outputs from the browser versions of Claude 2 and GPT-4. GPT-4 required use of a third-party plugin to upload and parse PDFs. Accuracy was high for Claude 2 (96.3%). The accuracy of GPT-4 with the plug-in was lower (68.8%); however, most of the errors were due to the plug-in. Both LLMs correctly recognized when prespecified data elements were missing from the source PDF and generated correct information for data elements that were not reported explicitly in the articles. A secondary analysis demonstrated that, when provided selected text from the PDFs, Claude 2 and GPT-4 accurately extracted 98.7% and 100% of the data elements, respectively. Limitations include the narrow scope of the study PDFs used, that prompt development was completed using only Claude 2, and that we cannot guarantee the open-source articles were not used to train the LLMs. This study highlights the potential for LLMs to revolutionize data extraction but underscores the importance of accurate PDF parsing. For now, it remains essential for a human investigator to validate LLM extractions.

Data extraction for evidence synthesis using a large language model: A proof-of-concept study.

Data extraction is a crucial, yet labor-intensive and error-prone part of evidence synthesis. To date, efforts to harness machine learning for enhancing efficiency of the data extraction process have fallen short of achieving sufficient accuracy and usability. With the release of large language models (LLMs), new possibilities have emerged to increase efficiency and accuracy of data extraction for evidence synthesis. The objective of this proof-of-concept study was to assess the performance of an LLM (Claude 2) in extracting data elements from published studies, compared with human data extraction as employed in systematic reviews. Our analysis utilized a convenience sample of 10 English-language, open-access publications of randomized controlled trials included in a single systematic review. We selected 16 distinct types of data, posing varying degrees of difficulty (160 data elements across 10 studies). We used the browser version of Claude 2 to upload the portable document format of each publication and then prompted the model for each data element. Across 160 data elements, Claude 2 demonstrated an overall accuracy of 96.3% with a high test-retest reliability (replication 1: 96.9%; replication 2: 95.0% accuracy). Overall, Claude 2 made 6 errors on 160 data items. The most common errors (n = 4) were missed data items. Importantly, Claude 2's ease of use was high; it required no technical expertise or labeled training data for effective operation (i.e., zero-shot learning). Based on findings of our proof-of-concept study, leveraging LLMs has the potential to substantially enhance the efficiency and accuracy of data extraction for evidence syntheses.

The smashHitCore Ontology for GDPR-Compliant Sensor Data Sharing in Smart Cities.

The adoption of the General Data Protection Regulation (GDPR) has resulted in a significant shift in how the data of European Union citizens is handled. A variety of data sharing challenges in scenarios such as smart cities have arisen, especially when attempting to semantically represent GDPR legal bases, such as consent, contracts and the data types and specific sources related to them. Most of the existing ontologies that model GDPR focus mainly on consent. In order to represent other GDPR bases, such as contracts, multiple ontologies need to be simultaneously reused and combined, which can result in inconsistent and conflicting knowledge representation. To address this challenge, we present the smashHitCore ontology. smashHitCore provides a unified and coherent model for both consent and contracts, as well as the sensor data and data processing associated with them. The ontology was developed in response to real-world sensor data sharing use cases in the insurance and smart city domains. The ontology has been successfully utilised to enable GDPR-complaint data sharing in a connected car for insurance use cases and in a city feedback system as part of a smart city use case.

A New Type 2 Diabetes Microsimulation Model to Estimate Long-Term Health Outcomes, Costs, and Cost-Effectiveness.

OBJECTIVES: This study aimed to develop a microsimulation model to estimate the health effects, costs, and cost-effectiveness of public health and clinical interventions for preventing/managing type 2 diabetes. METHODS: We combined newly developed equations for complications, mortality, risk factor progression, patient utility, and cost-all based on US studies-in a microsimulation model. We performed internal and external validation of the model. To demonstrate the model's utility, we predicted remaining life-years, quality-adjusted life-years (QALYs), and lifetime medical cost for a representative cohort of 10 000 US adults with type 2 diabetes. We then estimated the cost-effectiveness of reducing hemoglobin A1c from 9% to 7% among adults with type 2 diabetes, using low-cost, generic, oral medications. RESULTS: The model performed well in internal validation; the average absolute difference between simulated and observed incidence for 17 complications was < 8%. In external validation, the model was better at predicting outcomes in clinical trials than in observational studies. The cohort of US adults with type 2 diabetes was projected to have an average of 19.95 remaining life-years (from mean age 61), incur $187 729 in discounted medical costs, and accrue 8.79 discounted QALYs. The intervention to reduce hemoglobin A1c increased medical costs by $1256 and QALYs by 0.39, yielding an incremental cost-effectiveness ratio of $9103 per QALY. CONCLUSIONS: Using equations exclusively derived from US studies, this new microsimulation model achieves good prediction accuracy in US populations. The model can be used to estimate the long-term health impact, costs, and cost-effectiveness of interventions for type 2 diabetes in the United States.

Data Protection by Design Tool for Automated GDPR Compliance Verification Based on Semantically Modeled Informed Consent.

The enforcement of the GDPR in May 2018 has led to a paradigm shift in data protection. Organizations face significant challenges, such as demonstrating compliance (or auditability) and automated compliance verification due to the complex and dynamic nature of consent, as well as the scale at which compliance verification must be performed. Furthermore, the GDPR's promotion of data protection by design and industrial interoperability requirements has created new technical challenges, as they require significant changes in the design and implementation of systems that handle personal data. We present a scalable data protection by design tool for automated compliance verification and auditability based on informed consent that is modeled with a knowledge graph. Automated compliance verification is made possible by implementing a regulation-to-code process that translates GDPR regulations into well-defined technical and organizational measures and, ultimately, software code. We demonstrate the effectiveness of the tool in the insurance and smart cities domains. We highlight ways in which our tool can be adapted to other domains.

Rural and urban differences in undersupply of buprenorphine provider availability in the United States, 2018.

BACKGROUND: Medications to treat opioid use disorder (OUD) including buprenorphine products are evidence-based and cost-effective tools for combating the opioid crisis. However, limited availability to buprenorphine is pervasive in the United States (US) and may serve to exacerbate the deadly epidemic. Although prior research points to rural counties as especially needy of strategies that improve buprenorphine availability, it is important to investigate the availability of waivered providers according to treatment need as defined by the county-level rate of opioid-overdose deaths (OOD). This study examined differences in buprenorphine provider availability relative to treatment need among rural and urban counties in the US. METHODS: Buprenorphine provider availability relative to need in each county was defined as the number of waivered providers divided by the rate of OODs (i.e., number of OODs/100,000 population), according to 2018 data. Counties with ratios in the bottom tertile of their state were classified as buprenorphine undersupplied. We estimated logit models to statistically test the association of rurality and state main effects and their interaction terms (independent variables) and the county classified as buprenorphine undersupplied (dependent variable). RESULTS: A total of 38 states and 2595 counties had sufficient non-suppressed data to remain in the analysis. A larger percent of urban counties (36.43%) than rural counties (32.01%) were classified as buprenorphine undersupplied (p = 0.001). The likelihood of a rural county being undersupplied varied considerably by state (Chi Square = 82.88, p = 0.000). All states with significant (p < 0.05 or p < 0.10) interaction terms showed lower likelihood of buprenorphine undersupply in rural counties. CONCLUSIONS: The rural-urban distribution in undersupply of waivered buprenorphine providers relative to need varied markedly by state. Strategies for improving access to buprenorphine-waivered providers should be state-centric and informed by county-specific indicators of need.

Modeling inpatient and outpatient antibiotic stewardship interventions to reduce the burden of Clostridioides difficile infection in a regional healthcare network.

Antibiotic exposure can lead to unintended outcomes, including drug-drug interactions, adverse drug events, and healthcare-associated infections like Clostridioides difficile infection (CDI). Improving antibiotic use is critical to reduce an individual's CDI risk. Antibiotic stewardship initiatives can reduce inappropriate antibiotic prescribing (e.g., unnecessary antibiotic prescribing, inappropriate antibiotic selection), impacting both hospital (healthcare)-onset (HO)-CDI and community-associated (CA)-CDI. Previous computational and mathematical modeling studies have demonstrated a reduction in CDI incidence associated with antibiotic stewardship initiatives in hospital settings. Although the impact of antibiotic stewardship initiatives in long-term care facilities (LTCFs), including nursing homes, and in outpatient settings have been documented, the effects of specific interventions on CDI incidence are not well understood. We examined the relative effectiveness of antibiotic stewardship interventions on CDI incidence using a geospatially explicit agent-based model of a regional healthcare network in North Carolina. We simulated reductions in unnecessary antibiotic prescribing and inappropriate antibiotic selection with intervention scenarios at individual and network healthcare facilities, including short-term acute care hospitals (STACHs), nursing homes, and outpatient locations. Modeled antibiotic prescription rates were calculated using patient-level data on antibiotic length of therapy for the 10 modeled network STACHs. By simulating a 30% reduction in antibiotics prescribed across all inpatient and outpatient locations, we found the greatest reductions on network CDI incidence among tested scenarios, namely a 17% decrease in HO-CDI incidence and 7% decrease in CA-CDI. Among intervention scenarios of reducing inappropriate antibiotic selection, we found a greater impact on network CDI incidence when modeling this reduction in nursing homes alone compared to the same intervention in STACHs alone. These results support the potential importance of LTCF and outpatient antibiotic stewardship efforts on network CDI burden and add to the evidence that a coordinated approach to antibiotic stewardship across multiple facilities, including inpatient and outpatient settings, within a regional healthcare network could be an effective strategy to reduce network CDI burden.

Modeling Infectious Diseases in Healthcare Network (MInD-Healthcare) Framework for Describing and Reporting Multidrug-resistant Organism and Healthcare-Associated Infections Agent-based Modeling Methods.

Mathematical modeling of healthcare-associated infections and multidrug-resistant organisms improves our understanding of pathogen transmission dynamics and provides a framework for evaluating prevention strategies. One way of improving the communication among modelers is by providing a standardized way of describing and reporting models, thereby instilling confidence in the reproducibility and generalizability of such models. We updated the Overview, Design concepts, and Details protocol developed by Grimm et al [11] for describing agent-based models (ABMs) to better align with elements commonly included in healthcare-related ABMs. The Modeling Infectious Diseases in Healthcare Network (MInD-Healthcare) framework includes the following 9 key elements: (1) Purpose and scope; (2) Entities, state variables, and scales; (3) Initialization; (4) Process overview and scheduling; (5) Input data; (6) Agent interactions and organism transmission; (7) Stochasticity; (8) Submodels; and (9) Model verification, calibration, and validation. Our objective is that this framework will improve the quality of evidence generated utilizing these models.

Creation of a Geospatially Explicit, Agent-based Model of a Regional Healthcare Network with Application to Clostridioides difficile Infection.

Agent-based models (ABMs) describe and simulate complex systems comprising unique agents, or individuals, while accounting for geospatial and temporal variability among dynamic processes. ABMs are increasingly used to study healthcare-associated infections (ie, infections acquired during admission to a healthcare facility), including Clostridioides difficile infection, currently the most common healthcare-associated infection in the United States. The overall burden and transmission dynamics of healthcare-associated infections, including C difficile infection, may be influenced by community sources and movement of people among healthcare facilities and communities. These complex dynamics warrant geospatially explicit ABMs that extend beyond single healthcare facilities to include entire systems (eg, hospitals, nursing homes and extended care facilities, the community). The agents in ABMs can be built on a synthetic population, a model-generated representation of the actual population with associated spatial (eg, home residence), temporal (eg, change in location over time), and nonspatial (eg, sociodemographic features) attributes. We describe our methods to create a geospatially explicit ABM of a major regional healthcare network using a synthetic population as microdata input. We illustrate agent movement in the healthcare network and the community, informed by patient-level medical records, aggregate hospital discharge data, healthcare facility licensing data, and published literature. We apply the ABM output to visualize agent movement in the healthcare network and the community served by the network. We provide an application example of the ABM to C difficile infection using a natural history submodel. We discuss the ABM's potential to detect network areas where disease risk is high; simulate and evaluate interventions to protect public health; adapt to other geographic locations and healthcare-associated infections, including emerging pathogens; and meaningfully translate results to public health practitioners, healthcare providers, and policymakers.

Cost-effectiveness of risk stratified medication management for reducing premature cardiovascular mortality in Kenya.

BACKGROUND: Cardiovascular disease (CVD) is a major contributor to the burden from non-communicable diseases in Sub-Saharan Africa and hypertension is the leading risk factor for CVD. The objective of this modeling study is to assess the cost-effectiveness of a risk stratified approach to medication management in Kenya in order to achieve adequate blood pressure control to reduce CVD events. METHODS: We developed a microsimulation model to evaluate CVD risk over the lifetime of a cohort of individuals. Risk groups were assigned utilizing modified Framingham study distributions based on individual level risk factors from the Kenya STEPwise survey which collected details on blood pressure, blood glucose, tobacco and alcohol use and cholesterol levels. We stratified individuals into 4 risk groups: very low, low, moderate and high risk. Mortality could occur due to acute CVD events, subsequent future events (for individual who survive the initial event) and other causes. We present cost and DALYs gained due to medication management for men and women 25 to 69 years. RESULTS: Treating high risk individuals only was generally more cost-effective that treating high and moderate risk individuals. At the anticipated base levels of effectiveness, medication management was only cost-effective under the low cost scenario. The incremental cost per DALY gained with low cost ranged from $1,505 to $3,608, which is well under $4,785 (3 times GPD per capita) threshold for Kenya. Under the low cost scenario, even lower levels of effectiveness of medication management are likely to be cost-effective for high-risk men and women. CONCLUSIONS: In Kenya, our results indicate that the risk stratified approach to treating hypertension may be cost-effective especially for men and women at a high risk for CVD events, but these results are highly sensitive to the cost of medications. Medication management would require significant financial investment and therefore other interventions, including lifestyle changes, should be evaluated especially for those with elevated blood pressure and overall 10-year risk that is less than 20%.

Sexual content in video games: an analysis of the Entertainment Software Rating Board classification from 1994 to 2013.

Background Video games are widely used by children and adolescents and have become a significant source of exposure to sexual content. Despite evidence of the important role of media in the development of sexual attitudes and behaviours, little attention has been paid to monitor sexual content in video games. METHODS: Data was obtained about sexual content and rating for 23722 video games from 1994 to 2013 from the Entertainment Software Rating Board database; release dates and information on the top 100 selling video games was also obtained. A yearly prevalence of sexual content according to rating categories was calculated. Trends and comparisons were estimated using Joinpoint regression. RESULTS: Sexual content was present in 13% of the video games. Games rated 'Mature' had the highest prevalence of sexual content (34.5%) followed by 'Teen' (30.7%) and 'E10+' (21.3%). Over time, sexual content decreased in the 'Everyone' category, 'E10+' maintained a low prevalence and 'Teen' and 'Mature' showed a marked increase. Both top and non-top video games showed constant increases, with top selling video games having 10.1% more sexual content across the period of study. CONCLUSION: Over the last 20 years, the prevalence of sexual content has increased in video games with a 'Teen' or 'Mature' rating. Further studies are needed to quantify the potential association between sexual content in video games and sexual behaviour in children and adolescents.