RESUMO
OBJECTIVES: Artificial intelligence (AI)-based chatbots have demonstrated accuracy in a variety of fields, including medicine, but research has yet to substantiate their accuracy and clinical relevance. We evaluated an AI chatbot's answers to questions posed during a treatment planning conference. METHODS: Pathology residents, pathology faculty, and an AI chatbot (OpenAI ChatGPT [January 30, 2023, release]) answered a questionnaire curated from a genitourinary subspecialty treatment planning conference. Results were evaluated by 2 blinded adjudicators: a clinician expert and a pathology expert. Scores were based on accuracy and clinical relevance. RESULTS: Overall, faculty scored highest (4.75), followed by the AI chatbot (4.10), research-prepared residents (3.50), and unprepared residents (2.87). The AI chatbot scored statistically significantly better than unprepared residents (P = .03) but not statistically significantly different from research-prepared residents (P = .33) or faculty (P = .30). Residents did not statistically significantly improve after research (P = .39), and faculty performed statistically significantly better than both resident categories (unprepared, P < .01; research prepared, P = .01). CONCLUSIONS: The AI chatbot gave answers to medical questions that were comparable in accuracy and clinical relevance to pathology faculty, suggesting promise for further development. Serious concerns remain, however, that without the ability to provide support with references, AI will face legitimate scrutiny as to how it can be integrated into medical decision-making.
RESUMO
PURPOSE: Obtaining remote patient-reported outcomes (PRO) is limited by low patient response rates and resource-intensive collection methods. We hypothesized that an e-mail-delivered Web-based data collection tool would outperform the traditional methods of telephone and standard mail for collecting long-term Boston Carpal Tunnel Questionnaire (BCTQ) scores at a minimum of 1 year following carpal tunnel release (CTR). METHODS: We conducted a randomized trial of 969 patients who underwent CTR at a tertiary medical center within the past 5 years. Participants were randomized to the PRO collection methods of mail, telephone, and e-mail. The primary outcome was survey response rate at 1 year after surgery. Secondary analyses included data completeness and the effect of time from surgery, mode effects, and patient modality preference. RESULTS: At 1 year from surgery, the response rates were 64% for telephone and 42% for both mail and e-mail. Ninety-nine percent of telephone surveys were complete compared with 88% and 83% for mail and e-mail, respectively. There was no significant difference in the overall response rate at 1 or 5 years after surgery, nor in the BCTQ score between the modalities. CONCLUSIONS: A higher response rate and increased survey completeness was achieved by telephone contact methods compared with standard mailings or Web-based methods for PRO collection after CTR 1 to 5 years after surgery. A Web-based method demonstrated response rates equivalent to those of standard mail, was the most preferred modality, and offered logistical advantages such as automation and immediate integration with outcome databases. CLINICAL RELEVANCE: Obtaining PRO routinely after treatment may increase in importance. A Web-based interface may assist clinicians in decreasing the resource utilization typically associated with more traditional methods used to obtain outcome data.
