GPT-4 performance on querying scientific publications: reproducibility, accuracy, and impact of an instruction sheet.

BACKGROUND: Large language models (LLMs) that can efficiently screen and identify studies meeting specific criteria would streamline literature reviews. Additionally, those capable of extracting data from publications would enhance knowledge discovery by reducing the burden on human reviewers. METHODS: We created an automated pipeline utilizing OpenAI GPT-4 32 K API version "2023-05-15" to evaluate the accuracy of the LLM GPT-4 responses to queries about published papers on HIV drug resistance (HIVDR) with and without an instruction sheet. The instruction sheet contained specialized knowledge designed to assist a person trying to answer questions about an HIVDR paper. We designed 60 questions pertaining to HIVDR and created markdown versions of 60 published HIVDR papers in PubMed. We presented the 60 papers to GPT-4 in four configurations: (1) all 60 questions simultaneously; (2) all 60 questions simultaneously with the instruction sheet; (3) each of the 60 questions individually; and (4) each of the 60 questions individually with the instruction sheet. RESULTS: GPT-4 achieved a mean accuracy of 86.9% - 24.0% higher than when the answers to papers were permuted. The overall recall and precision were 72.5% and 87.4%, respectively. The standard deviation of three replicates for the 60 questions ranged from 0 to 5.3% with a median of 1.2%. The instruction sheet did not significantly increase GPT-4's accuracy, recall, or precision. GPT-4 was more likely to provide false positive answers when the 60 questions were submitted individually compared to when they were submitted together. CONCLUSIONS: GPT-4 reproducibly answered 3600 questions about 60 papers on HIVDR with moderately high accuracy, recall, and precision. The instruction sheet's failure to improve these metrics suggests that more sophisticated approaches are necessary. Either enhanced prompt engineering or finetuning an open-source model could further improve an LLM's ability to answer questions about highly specialized HIVDR papers.

Prevalence of Emergent Dolutegravir Resistance Mutations in People Living with HIV: A Rapid Scoping Review.

BACKGROUND: Dolutegravir (DTG) is a cornerstone of global antiretroviral (ARV) therapy (ART) due to its high efficacy and favorable tolerability. However, limited data exist regarding the risk of emergent integrase strand transfer inhibitor (INSTI) drug-resistance mutations (DRMs) in individuals receiving DTG-containing ART. METHODS: We performed a PubMed search using the term "Dolutegravir", last updated 18 December 2023, to estimate the prevalence of VF with emergent INSTI DRMs in people living with HIV (PLWH) without previous VF on an INSTI who received DTG-containing ART. RESULTS: Of 2131 retrieved records, 43 clinical trials, 39 cohorts, and 6 cross-sectional studies provided data across 6 clinical scenarios based on ART history, virological status, and co-administered ARVs: (1) ART-naïve PLWH receiving DTG plus two NRTIs; (2) ART-naïve PLWH receiving DTG plus lamivudine; (3) ART-experienced PLWH with VF on a previous regimen receiving DTG plus two NRTIs; (4) ART-experienced PLWH with virological suppression receiving DTG plus two NRTIs; (5) ART-experienced PLWH with virological suppression receiving DTG and a second ARV; and (6) ART-experienced PLWH with virological suppression receiving DTG monotherapy. The median proportion of PLWH in clinical trials with emergent INSTI DRMs was 1.5% for scenario 3 and 3.4% for scenario 6. In the remaining four trial scenarios, VF prevalence with emergent INSTI DRMs was ≤0.1%. Data from cohort studies minimally influenced prevalence estimates from clinical trials, whereas cross-sectional studies yielded prevalence data lacking denominator details. CONCLUSIONS: In clinical trials, the prevalence of VF with emergent INSTI DRMs in PLWH receiving DTG-containing regimens has been low. Novel approaches are required to assess VF prevalence with emergent INSTI DRMs in PLWH receiving DTG in real-world settings.

Treatment Emergent Dolutegravir Resistance Mutations in Individuals Naïve to HIV-1 Integrase Inhibitors: A Rapid Scoping Review.

Background: Dolutegravir (DTG)-based antiretroviral therapy (ART) rarely leads to virological failure (VF) and drug resistance in integrase strand transfer inhibitor (INSTI)-naïve persons living with HIV (PLWH). As a result, limited data are available on INSTI-associated drug resistance mutations (DRMs) selected by DTG-containing ART regimens. Methods: We reviewed studies published through July 2023 to identify those reporting emergent major INSTI-associated DRMs in INSTI-naïve PLWH receiving DTG and those containing in vitro DTG susceptibility results using a standardized assay. Results: We identified 36 publications reporting 99 PLWH in whom major nonpolymorphic INSTI-associated DRMs developed on a DTG-containing regimen and 21 publications containing 269 in vitro DTG susceptibility results. DTG-selected DRMs clustered into four largely non-overlapping mutational pathways characterized by mutations at four signature positions: R263K, G118R, N155H, and Q148H/R/K. Eighty-two (82.8%) viruses contained just one signature DRM, including R263K (n = 40), G118R (n = 24), N155H (n = 9), and Q148H/R/K (n = 9). Nine (9.1%) contained ≥1 signature DRM, and eight (8.1%) contained just other DRMs. R263K and G118R were negatively associated with one another and with N155H and Q148H/K/R. R263K alone conferred a median 2.0-fold (IQR: 1.8-2.2) reduction in DTG susceptibility. G118R alone conferred a median 18.8-fold (IQR:14.2-23.4) reduction in DTG susceptibility. N155H alone conferred a median 1.4-fold (IQR: 1.2-1.6) reduction in DTG susceptibility. Q148H/R/K alone conferred a median 0.8-fold (IQR: 0.7-1.1) reduction in DTG susceptibility. Considerably higher levels of reduced susceptibility often occurred when signature DRMs occurred with additional INSTI-associated DRMs. Conclusions: Among INSTI-naïve PLWH with VF and treatment emergent INSTI-associated DRMs, most developed one of four signature DRMs, most commonly R263K or G118R. G118R was associated with a much greater reduction in DTG susceptibility than R263K.

HIV-1 Group M Capsid Amino Acid Variability: Implications for Sequence Quality Control of Genotypic Resistance Testing.

BACKGROUND: With the approval of the HIV-1 capsid inhibitor, lenacapavir, capsid sequencing will be required for managing lenacapavir-experienced individuals with detectable viremia. Successful sequence interpretation will require examining new capsid sequences in the context of previously published sequence data. METHODS: We analyzed published HIV-1 group M capsid sequences from 21,012 capsid-inhibitor naïve individuals to characterize amino acid variability at each position and influence of subtype and cytotoxic T lymphocyte (CTL) selection pressure. We determined the distributions of usual mutations, defined as amino acid differences from the group M consensus, with a prevalence ≥ 0.1%. Co-evolving mutations were identified using a phylogenetically-informed Bayesian graphical model method. RESULTS: 162 (70.1%) positions had no usual mutations (45.9%) or only conservative usual mutations with a positive BLOSUM62 score (24.2%). Variability correlated independently with subtype-specific amino acid occurrence (Spearman rho = 0.83; p < 1 × 10-9) and the number of times positions were reported to contain an HLA-associated polymorphism, an indicator of CTL pressure (rho = 0.43; p = 0.0002). CONCLUSIONS: Knowing the distribution of usual capsid mutations is essential for sequence quality control. Comparing capsid sequences from lenacapavir-treated and lenacapavir-naïve individuals will enable the identification of additional mutations potentially associated with lenacapavir therapy.

Highly Ambiguous HIV-1 Pol Positions Encoding Multiple Amino Acids Usually Result from Antiviral or Immune Selection Pressure.

HIV-1 pol nucleotide ambiguities encoding amino acid mixtures occur commonly during population-based genotypic drug resistance testing. However, few studies have addressed the validity of sequences with fully ambiguous codons (FACs) containing codons translatable to more than four amino acids. We identified 839 published HIV-1 pol sequences with 846 FACs at 131 positions and determined their distribution relative to 215 HLA-associated pol positions (HAPs) and 84 drug-resistance positions. Among HIV-1 reverse transcriptase (RT) and protease sequences from antiretroviral therapy (ART)-naive and -experienced persons, there was a strong correlation between the likelihood a position was a FAC and that it was an HAP (Spearman's correlation coefficient rho >0.40; p < 1e-6). Among HIV-1 RT sequences from ART-experienced persons, there was a correlation between the likelihood that a position was a FAC and that it was a drug-resistance position (rho = 0.2; p = 8e-4). In the context of population-based genotypic resistance testing, FACs usually result from antiviral or immune selection pressure.

Sierra SARS-CoV-2 sequence and antiviral resistance analysis program.

INTRODUCTION: Although most laboratories are capable of employing established protocols to perform full-genome SARS-CoV-2 sequencing, many are unable to assess sequence quality, select appropriate mutation-detection thresholds, or report on the potential clinical significance of mutations in the targets of antiviral therapy METHODS: We describe the technical aspects and benchmark the performance of Sierra SARS-CoV-2, a program designed to perform these functions on user-submitted FASTQ and FASTA sequence files and lists of Spike mutations. Sierra SARS-CoV-2 indicates which sequences contain an unexpectedly large number of unusual mutations and which mutations are associated with reduced susceptibility to clinical stage mAbs, the RdRP inhibitor remdesivir, or the Mpro inhibitor nirmatrelvir RESULTS: To assess the performance of Sierra SARS-CoV-2 on FASTQ files, we applied it to 600 representative FASTQ sequences and compared the results to the COVID-19 EDGE program. To assess its performance on FASTA files, we applied it to nearly one million representative FASTA sequences and compared the results to the GISAID mutation annotation. To assess its performance on mutations lists, we applied it to 13,578 distinct Spike RBD mutation patterns and showed that exactly or partially matching annotations were available for 88% of patterns CONCLUSION: Sierra SARS-CoV-2 leverages previously published data to improve the quality control of submitted viral genomic data and to provide functional annotation on the impact of mutations in the targets of antiviral SARS-CoV-2 therapy. The program can be found at https://covdb.stanford.edu/sierra/sars2/ and its source code at https://github.com/hivdb/sierra-sars2.

Susceptibility of SARS-CoV-2 Omicron Variants to Therapeutic Monoclonal Antibodies: Systematic Review and Meta-analysis.

SARS-CoV-2 Omicron variants contain many mutations in its spike receptor-binding domain, the target of all authorized monoclonal antibodies (MAbs). Determining the extent to which Omicron variants reduced MAb susceptibility is critical to preventing and treating COVID-19. We systematically reviewed PubMed and three preprint servers, last updated 11 April 2022, for the in vitro activity of authorized MAbs against the Omicron variants. Fifty-one studies were eligible, including 50 containing Omicron BA.1 susceptibility data and 17 containing Omicron BA.2 susceptibility data. The first two authorized MAb combinations, bamlanivimab/etesevimab and casirivimab/imdevimab, were largely inactive against the Omicron BA.1 and BA.2 variants. In 34 studies, sotrovimab displayed a median 4.0-fold (interquartile range [IQR]: 2.6 to 6.9) reduction in activity against Omicron BA.1, and in 12 studies, it displayed a median 17-fold (IQR: 13 to 30) reduction in activity against Omicron BA.2. In 15 studies, the combination cilgavimab/tixagevimab displayed a median 86-fold (IQR: 27 to 151) reduction in activity against Omicron BA.1, and in six studies, it displayed a median 5.4-fold (IQR: 3.7 to 6.9) reduction in activity against Omicron BA.2. In eight studies against Omicron BA.1 and six studies against Omicron BA.2, bebtelovimab displayed no reduction in activity. Disparate results between assays were common. For authorized MAbs, 51/268 (19.0%) results for wild-type control variants and 78/348 (22.4%) results for Omicron BA.1 and BA.2 variants were more than 4-fold below or 4-fold above the median result for that MAb. Highly disparate results between published assays indicate a need for improved MAb susceptibility test standardization or interassay calibration. IMPORTANCE Monoclonal antibodies (MAbs) targeting the SARS-CoV-2 spike protein are among the most effective measures for preventing and treating COVID-19. However, SARS-CoV-2 Omicron variants contain many mutations in their spike receptor-binding domains, the target of all authorized MAbs. Therefore, determining the extent to which Omicron variants reduced MAb susceptibility is critical to preventing and treating COVID-19. We identified 51 studies that reported the in vitro susceptibility of the two main Omicron variants BA.1 and BA.2 to therapeutic MAbs in advanced clinical development, including eight authorized individual MAbs and three authorized MAb combinations. We estimated the degree to which different MAbs displayed reduced activity against Omicron variants. The marked loss of activity of many MAbs against Omicron variants underscores the importance of developing MAbs that target conserved regions of spike. Highly disparate results between assays indicate the need for improved MAb susceptibility test standardization.

Coronavirus Resistance Database (CoV-RDB): SARS-CoV-2 susceptibility to monoclonal antibodies, convalescent plasma, and plasma from vaccinated persons.

As novel SARS-CoV-2 variants with different patterns of spike protein mutations have emerged, the susceptibility of these variants to neutralization by antibodies has been rapidly assessed. However, neutralization data are generated using different approaches and are scattered across different publications making it difficult for these data to be located and synthesized. The Stanford Coronavirus Resistance Database (CoV-RDB; https://covdb.stanford.edu) is designed to house comprehensively curated published data on the neutralizing susceptibility of SARS-CoV-2 variants and spike mutations to monoclonal antibodies (mAbs), convalescent plasma (CP), and vaccinee plasma (VP). As of December 31, 2021, CoV-RDB encompassed 257 publications including 91 (35%) containing 9,070 neutralizing mAb susceptibility results, 131 (51%) containing 16,773 neutralizing CP susceptibility results, and 178 (69%) containing 33,540 neutralizing VP results. The database also records which spike mutations are selected during in vitro passage of SARS-CoV-2 in the presence of mAbs and which emerge in persons receiving mAbs as treatment. The CoV-RDB interface interactively displays neutralizing susceptibility data at different levels of granularity by filtering and/or aggregating query results according to one or more experimental conditions. The CoV-RDB website provides a companion sequence analysis program that outputs information about mutations present in a submitted sequence and that also assists users in determining the appropriate mutation-detection thresholds for identifying non-consensus amino acids. The most recent data underlying the CoV-RDB can be downloaded in its entirety from a GitHub repository in a documented machine-readable format.

The biological and clinical significance of emerging SARS-CoV-2 variants.

The past several months have witnessed the emergence of SARS-CoV-2 variants with novel spike protein mutations that are influencing the epidemiological and clinical aspects of the COVID-19 pandemic. These variants can increase rates of virus transmission and/or increase the risk of reinfection and reduce the protection afforded by neutralizing monoclonal antibodies and vaccination. These variants can therefore enable SARS-CoV-2 to continue its spread in the face of rising population immunity while maintaining or increasing its replication fitness. The identification of four rapidly expanding virus lineages since December 2020, designated variants of concern, has ushered in a new stage of the pandemic. The four variants of concern, the Alpha variant (originally identified in the UK), the Beta variant (originally identified in South Africa), the Gamma variant (originally identified in Brazil) and the Delta variant (originally identified in India), share several mutations with one another as well as with an increasing number of other recently identified SARS-CoV-2 variants. Collectively, these SARS-CoV-2 variants complicate the COVID-19 research agenda and necessitate additional avenues of laboratory, epidemiological and clinical research.

SARS-CoV-2 Antiviral Therapy.

The development of effective antiviral therapy for COVID-19 is critical for those awaiting vaccination, as well as for those who do not respond robustly to vaccination. This review summarizes 1 year of progress in the race to develop antiviral therapies for COVID-19, including research spanning preclinical and clinical drug development efforts, with an emphasis on antiviral compounds that are in clinical development or that are high priorities for clinical development. The review is divided into sections on compounds that inhibit SARS-CoV-2 enzymes, including its polymerase and proteases; compounds that inhibit virus entry, including monoclonal antibodies; interferons; and repurposed drugs that inhibit host processes required for SARS-CoV-2 replication. The review concludes with a summary of the lessons to be learned from SARS-CoV-2 drug development efforts and the challenges to continued progress.

Coronavirus Antiviral Research Database (CoV-RDB): An Online Database Designed to Facilitate Comparisons between Candidate Anti-Coronavirus Compounds.

BACKGROUND: To prioritize the development of antiviral compounds, it is necessary to compare their relative preclinical activity and clinical efficacy. METHODS: We reviewed in vitro, animal model, and clinical studies of candidate anti-coronavirus compounds and placed extracted data in an online relational database. RESULTS: As of August 2020, the Coronavirus Antiviral Research Database (CoV-RDB; covdb.stanford.edu) contained over 2800 cell culture, entry assay, and biochemical experiments, 259 animal model studies, and 73 clinical studies from over 400 published papers. SARS-CoV-2, SARS-CoV, and MERS-CoV account for 85% of the data. Approximately 75% of experiments involved compounds with known or likely mechanisms of action, including monoclonal antibodies and receptor binding inhibitors (21%), viral protease inhibitors (17%), miscellaneous host-acting inhibitors (10%), polymerase inhibitors (9%), interferons (7%), fusion inhibitors (5%), and host protease inhibitors (5%). Of 975 compounds with known or likely mechanism, 135 (14%) are licensed in the U.S. for other indications, 197 (20%) are licensed outside the U.S. or are in human trials, and 595 (61%) are pre-clinical investigational compounds. CONCLUSION: CoV-RDB facilitates comparisons between different candidate antiviral compounds, thereby helping scientists, clinical investigators, public health officials, and funding agencies prioritize the most promising compounds and repurposed drugs for further development.