Sample Size Considerations for Fine-Tuning Large Language Models for Named Entity Recognition Tasks: Methodological Study.

BACKGROUND: Large language models (LLMs) have the potential to support promising new applications in health informatics. However, practical data on sample size considerations for fine-tuning LLMs to perform specific tasks in biomedical and health policy contexts are lacking. OBJECTIVE: This study aims to evaluate sample size and sample selection techniques for fine-tuning LLMs to support improved named entity recognition (NER) for a custom data set of conflicts of interest disclosure statements. METHODS: A random sample of 200 disclosure statements was prepared for annotation. All "PERSON" and "ORG" entities were identified by each of the 2 raters, and once appropriate agreement was established, the annotators independently annotated an additional 290 disclosure statements. From the 490 annotated documents, 2500 stratified random samples in different size ranges were drawn. The 2500 training set subsamples were used to fine-tune a selection of language models across 2 model architectures (Bidirectional Encoder Representations from Transformers [BERT] and Generative Pre-trained Transformer [GPT]) for improved NER, and multiple regression was used to assess the relationship between sample size (sentences), entity density (entities per sentence [EPS]), and trained model performance (F1-score). Additionally, single-predictor threshold regression models were used to evaluate the possibility of diminishing marginal returns from increased sample size or entity density. RESULTS: Fine-tuned models ranged in topline NER performance from F1-score=0.79 to F1-score=0.96 across architectures. Two-predictor multiple linear regression models were statistically significant with multiple R2 ranging from 0.6057 to 0.7896 (all P<.001). EPS and the number of sentences were significant predictors of F1-scores in all cases ( P<.001), except for the GPT-2_large model, where EPS was not a significant predictor (P=.184). Model thresholds indicate points of diminishing marginal return from increased training data set sample size measured by the number of sentences, with point estimates ranging from 439 sentences for RoBERTa_large to 527 sentences for GPT-2_large. Likewise, the threshold regression models indicate a diminishing marginal return for EPS with point estimates between 1.36 and 1.38. CONCLUSIONS: Relatively modest sample sizes can be used to fine-tune LLMs for NER tasks applied to biomedical text, and training data entity density should representatively approximate entity density in production data. Training data quality and a model architecture's intended use (text generation vs text processing or classification) may be as, or more, important as training data volume and model parameter size.

A Content Analysis of Self-Reported Financial Relationships in Biomedical Research.

INTRODUCTION: Financial conflicts of interest (fCOI) present well documented risks to the integrity of biomedical research. However, few studies differentiate among fCOI types in their analyses, and those that do tend to use preexisting taxonomies for fCOI identification. Research on fCOI would benefit from an empirically-derived taxonomy of self-reported fCOI and data on fCOI type and payor prevalence. METHODS: We conducted a content analysis of 6,165 individual self-reported relationships from COI statements distributed across 378 articles indexed with PubMed. Two coders used an iterative coding process to identify and classify individual fCOI types and payors. Inter-rater reliability was κ = 0.935 for fCOI type and κ = 0.884 for payor identification. RESULTS: Our analysis identified 21 fCOI types, 9 of which occurred at prevalences greater than 1%. These included research funding (24.8%), speaking fees (20.8%), consulting fees (18.8%), advisory relationships (11%), industry employment (7.6%), unspecified fees (4.8%), travel fees (3.2%), stock holdings (3.1%), and patent ownership (1%). Reported fCOI were held with 1,077 unique payors, 22 of which were present in more than 1% of financial relationships. The ten most common payors included Pfizer (4%), Novartis (3.9%), MSD (3.8%), Bristol Myers Squibb (3.2%), AstraZeneca (3.1%), GSK (3%), Boehringer Ingelheim (2.9%), Roche (2.8%), Eli LIlly (2.5%), and AbbVie (2.4%). CONCLUSIONS: These results provide novel multi-domain prevalence data on self-reported fCOI and payors in biomedical research. As such, they have the potential to catalyze future research that can assess the differential effects of various types of fCOI. Specifically, the data suggest that comparative analyses of the effects of different fCOI types are needed and that special attention should be paid to the diversity of payor types for research relationships.

Evidence for stratified conflicts of interest policies in research contexts: a methodological review.

OBJECTIVES: The purpose of this study was to conduct a methodological review of research on the effects of conflicts of interest (COIs) in research contexts. DESIGN: Methodological review. DATA SOURCES: Ovid. ELIGIBILITY CRITERIA: Studies published between 1986 and 2021 conducting quantitative assessments of relationships between industry funding or COI and four target outcomes: positive study results, methodological biases, reporting quality and results-conclusions concordance. DATA EXTRACTION AND SYNTHESIS: We assessed key facets of study design: our primary analysis identified whether studies stratified industry funding or COI variables by magnitude (ie, number of COI or disbursement amount), type (employment, travel fees, speaking fees) or if they assessed dichotomous variables (ie, conflict present or absent). Secondary analyses focused on target outcomes and available effects measures. RESULTS: Of the 167 articles included in this study, a substantial majority (98.2%) evaluated the effects of industry sponsorship. None evaluated associations between funding magnitude and outcomes of interest. Seven studies (4.3%) stratified industry funding based on the mechanism of disbursement or funder relationship to product (manufacturer or competitor). A fifth of the articles (19.8%) assessed the effects of author COI on target outcomes. None evaluated COI magnitude, and three studies (9.1%) stratified COI by disbursement type and/or reporting practices. Participation of an industry-employed author showed the most consistent effect on favourability of results across studies. CONCLUSIONS: Substantial evidence demonstrates that industry funding and COI can bias biomedical research. Evidence-based policies are essential for mitigating the risks associated with COI. Although most policies stratify guidelines for managing COI, differentiating COIs based on the type of relationship or monetary value, this review shows that the available research has generally not been designed to assess the differential risks of COI types or magnitudes. Targeted research is necessary to establish an evidence base that can effectively inform policy to manage COI.

Associations Between Aggregate NLP-Extracted Conflicts of Interest and Adverse Events by Drug Product.

This study evaluates associations between aggregate conflicts of interest (COI) and drug safety. We used a machine-learning system to extract and classify COI from PubMed-indexed disclosure statements. Individual conflicts were classified as Type 1 (personal fees, travel, board memberships, and non-financial support), Type 2 (grants and research support), or Type 3 (stock ownership and industry employment). COI were aggregated by type compared to adverse events by product. Type 1 COI are associated with a 1.1-1.8% increase in the number of adverse events, serious events, hospitalizations, and deaths. Type 2 COI are associated with a 1.7-2% decrease in adverse events across severity levels. Type 3 COI are associated with an approximately 1% increase in adverse events, serious events, and hospitalizations, but have no significant association with adverse events resulting in death. The findings suggest that COI policies might be adapted to account the relative risks of different types of financial relationships.

A dashboard for exploring clinical trials sponsorship and potential virtual monopolies.

OBJECTIVE: To create a data visualization dashboard to advance research related to clinical trials sponsorship and monopolistic practices in the pharmaceuticals industry. MATERIALS AND METHODS: This R Shiny application aggregates data from ClinicialTrials.gov resulting from user's queries by terms. Returned data are visualized through an interactive dashboard. RESULTS: The Clinical Trials Sponsorship Network Dashboard (CTSND) uses force-directed network mapping algorithms to visualize clinical trials sponsorship data. Interpretation of network visualization is further supported with data on sponsor classes, sponsorship timelines, evaluated products, and target conditions. The source code for the CTSND is available at https://github.com/sscottgraham/ConflictMetrics. DISCUSSION: Monopolistic practices have been identified as a likely contributor to high drug prices in the United States. CTSND data and visualizations support the analysis of clinical trials sponsorship networks and may aid in identifying current and emerging monopolistic practices. CONCLUSIONS: CTSND data can support more robust deliberation about an understudied area of drug pricing.

Relationships among commercial practices and author conflicts of interest in biomedical publishing.

Recently, concerns have been raised over the potential impacts of commercial relationships on editorial practices in biomedical publishing. Specifically, it has been suggested that certain commercial relationships may make editors more open to publishing articles with author conflicts of interest (aCOI). Using a data set of 128,781 articles published in 159 journals, we evaluated the relationships among commercial publishing practices and reported author conflicts of interest. The 159 journals were grouped according to commercial biases (reprint services, advertising revenue, and ownership by a large commercial publishing firm). 30.6% (39,440) of articles were published in journals showing no evidence of evaluated commercial publishing relationships. 33.9% (43,630) were published in journals accepting advertising and reprint fees; 31.7% (40,887) in journals owned by large publishing firms; 1.2% (1,589) in journals accepting reprint fees only; and 2.5% (3,235) in journals accepting only advertising fees. Journals with commercial relationships were more likely to publish articles with aCOI (9.2% (92/1000) vs. 6.4% (64/1000), p = 0.024). In the multivariate analysis, only a journal's acceptance of reprint fees served as a significant predictor (OR = 2.81 at 95% CI, 1.5 to 8.6). Shared control estimation was used to evaluate the relationships between commercial publishing practices and aCOI frequency in total and by type. BCa-corrected mean difference effect sizes ranged from -1.0 to 6.1, and confirm findings indicating that accepting reprint fees may constitute the most significant commercial bias. The findings indicate that concerns over the influence of industry advertising in medical journals may be overstated, and that accepting fees for reprints may constitute the largest risk of bias for editorial decision-making.

Immunized against science: Narrative community building among vaccine refusing/hesitant parents.

Recent outbreaks of measles have centered in specific communities, pointing to the influence of social ties on vaccination practices. This study adds to the conversation on public understanding of vaccine-related science, documenting how the individualist epistemologies highlighted in prior research are externalized and validated in communication with others, focusing on how the narrative strategies used to do so contribute to community building among vaccine refusing and hesitant parents. Through qualitative content analysis of testimonials given to the creators of the anti-vaccination documentary VaxXed, we identify how the common narrative strategies used to question the scientific consensus on vaccines-distrust of doctors, self-diagnosis, building credibility, advocacy, and community building-build a competing consensus based on personal expertise. With this approach, we are better able to understand how participation in online communities strengthens the privileging of individualist epistemologies among vaccine refusing and hesitant parents.