GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet.

BACKGROUND: Large language models (LLMs) that can efficiently screen and identify studies meeting specific criteria would streamline literature reviews. Additionally, those capable of extracting data from publications would enhance knowledge discovery by reducing the burden on human reviewers. METHODS: We created an automated pipeline utilizing OpenAI GPT-4 32 K API version "2023-05-15" to evaluate the accuracy of the LLM GPT-4 responses to queries about published papers on HIV drug resistance (HIVDR) with and without an instruction sheet. The instruction sheet contained specialized knowledge designed to assist a person trying to answer questions about an HIVDR paper. We designed 60 questions pertaining to HIVDR and created markdown versions of 60 published HIVDR papers in PubMed. We presented the 60 papers to GPT-4 in four configurations: (1) all 60 questions simultaneously; (2) all 60 questions simultaneously with the instruction sheet; (3) each of the 60 questions individually; and (4) each of the 60 questions individually with the instruction sheet. RESULTS: GPT-4 achieved a mean accuracy of 86.9% - 24.0% higher than when the answers to papers were permuted. The overall recall and precision were 72.5% and 87.4%, respectively. The standard deviation of three replicates for the 60 questions ranged from 0 to 5.3% with a median of 1.2%. The instruction sheet did not significantly increase GPT-4's accuracy, recall, or precision. GPT-4 was more likely to provide false positive answers when the 60 questions were submitted individually compared to when they were submitted together. CONCLUSIONS: GPT-4 reproducibly answered 3600 questions about 60 papers on HIVDR with moderately high accuracy, recall, and precision. The instruction sheet's failure to improve these metrics suggests that more sophisticated approaches are necessary. Either enhanced prompt engineering or finetuning an open-source model could further improve an LLM's ability to answer questions about highly specialized HIVDR papers.

HIV-1 Group M Capsid Amino Acid Variability: Implications for Sequence Quality Control of Genotypic Resistance Testing.

BACKGROUND: With the approval of the HIV-1 capsid inhibitor, lenacapavir, capsid sequencing will be required for managing lenacapavir-experienced individuals with detectable viremia. Successful sequence interpretation will require examining new capsid sequences in the context of previously published sequence data. METHODS: We analyzed published HIV-1 group M capsid sequences from 21,012 capsid-inhibitor naïve individuals to characterize amino acid variability at each position and influence of subtype and cytotoxic T lymphocyte (CTL) selection pressure. We determined the distributions of usual mutations, defined as amino acid differences from the group M consensus, with a prevalence ≥ 0.1%. Co-evolving mutations were identified using a phylogenetically-informed Bayesian graphical model method. RESULTS: 162 (70.1%) positions had no usual mutations (45.9%) or only conservative usual mutations with a positive BLOSUM62 score (24.2%). Variability correlated independently with subtype-specific amino acid occurrence (Spearman rho = 0.83; p < 1 × 10-9) and the number of times positions were reported to contain an HLA-associated polymorphism, an indicator of CTL pressure (rho = 0.43; p = 0.0002). CONCLUSIONS: Knowing the distribution of usual capsid mutations is essential for sequence quality control. Comparing capsid sequences from lenacapavir-treated and lenacapavir-naïve individuals will enable the identification of additional mutations potentially associated with lenacapavir therapy.

Part II. Comparison of Neurodevelopmental Outcomes Between Normothermic and Hypothermic Pediatric Cardiopulmonary Bypass.

Objectives: In the previous study we demonstrated that normothermic cardiopulmonary bypass (N-CPB, ≥35°C) provided better early clinical outcomes compared to mild/moderate hypothermic cardiopulmonary bypass (H-CPB, 28-34°C) for congenital heart surgery. In this follow-up study we compare early neurodevelopmental outcomes 2-3 years post-surgery. Methods: In this retrospective, non-randomized observational study, the medical notes of children from our previous cohort were reviewed after 2-3 years. Demographic and neurodevelopmental outcomes were tabulated to enable blinded statistical analysis comparing outcomes between N-CPB and H-CPB surgery for congenital heart defects. Multivariate logistic regression models were developed to identify any differences in outcomes after adjustment for confounders. Results: Ninety-five children who underwent H-CPB (n = 50) or N-CPB (n = 45) were included. The proportions of patients with one or more adverse neurodevelopmental outcomes 2-3 years later were 14/50 (28.0%) in the H-CPB group and 11/45 (24.4%) in N-CPB, which was not significantly different between groups (p = 0.47). The two CPB groups were balanced for demographic and surgical risk factors, with the exception of genetic conditions. A higher incidence of H-CPB patients acquired learning difficulties [23.1% compared to 2.56% for N-CPB (p = 0.014)] and neurological deficits [30.8% compared to 7.69% for N-CPB (p = 0.019)], but these differences were not robust to adjustment for genetic syndromes. Conclusions: Our study did not reveal any significant differences in early neurodevelopmental outcomes between H-CPB or N-CPB surgery for congenital heart defects. The most important factor in predicting outcomes was, as expected, the presence of a genetic syndrome. We found no evidence that CPB temperature affects early neurodevelopmental outcomes.