Benchmarking the symptom-checking capabilities of ChatGPT for a broad range of diseases.

OBJECTIVE: This study evaluates ChatGPT's symptom-checking accuracy across a broad range of diseases using the Mayo Clinic Symptom Checker patient service as a benchmark. METHODS: We prompted ChatGPT with symptoms of 194 distinct diseases. By comparing its predictions with expectations, we calculated a relative comparative score (RCS) to gauge accuracy. RESULTS: ChatGPT's GPT-4 model achieved an average RCS of 78.8%, outperforming the GPT-3.5-turbo by 10.5%. Some specialties scored above 90%. DISCUSSION: The test set, although extensive, was not exhaustive. Future studies should include a more comprehensive disease spectrum. CONCLUSION: ChatGPT exhibits high accuracy in symptom checking for a broad range of diseases, showcasing its potential as a medical training tool in learning health systems to enhance care quality and address health disparities.

Accuracy of Chatbots in Citing Journal Articles.

This cross-sectional study quantifies the journal article citation error rate of an artificial intelligence chatbot.

Validation of a non-invasive prenatal test for fetal RhD, C, c, E, K and Fya antigens.

We developed and validated a next generation sequencing-(NGS) based NIPT assay using quantitative counting template (QCT) technology to detect RhD, C, c, E, K (Kell), and Fya (Duffy) fetal antigen genotypes from maternal blood samples in the ethnically diverse U.S. population. Quantitative counting template (QCT) technology is utilized to enable quantification and detection of paternally derived fetal antigen alleles in cell-free DNA with high sensitivity and specificity. In an analytical validation, fetal antigen status was determined for 1061 preclinical samples with a sensitivity of 100% (95% CI 99-100%) and specificity of 100% (95% CI 99-100%). Independent analysis of two duplicate plasma samples was conducted for 1683 clinical samples, demonstrating precision of 99.9%. Importantly, in clinical practice the no-results rate was 0% for 711 RhD-negative non-alloimmunized pregnant people and 0.1% for 769 alloimmunized pregnancies. In a clinical validation, NIPT results were 100% concordant with corresponding neonatal antigen genotype/serology for 23 RhD-negative pregnant individuals and 93 antigen evaluations in 30 alloimmunized pregnancies. Overall, this NGS-based fetal antigen NIPT assay had high performance that was comparable to invasive diagnostic assays in a validation study of a diverse U.S. population as early as 10 weeks of gestation, without the need for a sample from the biological partner. These results suggest that NGS-based fetal antigen NIPT may identify more fetuses at risk for hemolytic disease than current clinical practice, which relies on paternal genotyping and invasive diagnostics and therefore is limited by adherence rates and incorrect results due to non-paternity. Clinical adoption of NIPT for the detection of fetal antigens for both alloimmunized and RhD-negative non-alloimmunized pregnant individuals may streamline care and reduce unnecessary treatment, monitoring, and patient anxiety.

Simulation of a machine learning enabled learning health system for risk prediction using synthetic patient data.

When enabled by machine learning (ML), Learning Health Systems (LHS) hold promise for improving the effectiveness of healthcare delivery to patients. One major barrier to LHS research and development is the lack of access to EHR patient data. To overcome this challenge, this study demonstrated the feasibility of developing a simulated ML-enabled LHS using synthetic patient data. The ML-enabled LHS was initialized using a dataset of 30,000 synthetic Synthea patients and a risk prediction XGBoost base model for lung cancer. 4 additional datasets of 30,000 patients were generated and added to the previous updated dataset sequentially to simulate addition of new patients, resulting in datasets of 60,000, 90,000, 120,000 and 150,000 patients. New XGBoost models were built in each instance, and performance improved with data size increase, attaining 0.936 recall and 0.962 AUC (area under curve) in the 150,000 patients dataset. The effectiveness of the new ML-enabled LHS process was verified by implementing XGBoost models for stroke risk prediction on the same Synthea patient populations. By making the ML code and synthetic patient data publicly available for testing and training, this first synthetic LHS process paves the way for more researchers to start developing LHS with real patient data.