Chatbot vs Medical Student Performance on Free-Response Clinical Reasoning Examinations.

This study compares performance on free-response clinical reasoning examinations of first- and second-year medical students vs 2 models of a popular chatbot.

Performance of ChatGPT on free-response, clinical reasoning exams.

Importance: Studies show that ChatGPT, a general purpose large language model chatbot, could pass the multiple-choice US Medical Licensing Exams, but the model's performance on open-ended clinical reasoning is unknown. Objective: To determine if ChatGPT is capable of consistently meeting the passing threshold on free-response, case-based clinical reasoning assessments. Design: Fourteen multi-part cases were selected from clinical reasoning exams administered to pre-clerkship medical students between 2019 and 2022. For each case, the questions were run through ChatGPT twice and responses were recorded. Two clinician educators independently graded each run according to a standardized grading rubric. To further assess the degree of variation in ChatGPT's performance, we repeated the analysis on a single high-complexity case 20 times. Setting: A single US medical school. Participants: ChatGPT. Main Outcomes and Measures: Passing rate of ChatGPT's scored responses and the range in model performance across multiple run throughs of a single case. Results: 12 out of the 28 ChatGPT exam responses achieved a passing score (43%) with a mean score of 69% (95% CI: 65% to 73%) compared to the established passing threshold of 70%. When given the same case 20 separate times, ChatGPT's performance on that case varied with scores ranging from 56% to 81%. Conclusions and Relevance: ChatGPT's ability to achieve a passing performance in nearly half of the cases analyzed demonstrates the need to revise clinical reasoning assessments and incorporate artificial intelligence (AI)-related topics into medical curricula and practice.

Lung Ultrasound Findings in Patients Hospitalized With COVID-19.

OBJECTIVES: Lung ultrasound (LUS) can accurately diagnose several pulmonary diseases, including pneumothorax, effusion, and pneumonia. LUS may be useful in the diagnosis and management of COVID-19. METHODS: This study was conducted at two United States hospitals from 3/21/2020 to 6/01/2020. Our inclusion criteria included hospitalized adults with COVID-19 (based on symptomatology and a confirmatory RT-PCR for SARS-CoV-2) who received a LUS. Providers used a 12-zone LUS scanning protocol. The images were interpreted by the researchers based on a pre-developed consensus document. Patients were stratified by clinical deterioration (defined as either ICU admission, invasive mechanical ventilation, or death within 28 days from the initial symptom onset) and time from symptom onset to their scan. RESULTS: N = 22 patients (N = 36 scans) were included. Eleven (50%) patients experienced clinical deterioration. Among N = 36 scans, only 3 (8%) were classified as normal. The remaining scans demonstrated B-lines (89%), consolidations (56%), pleural thickening (47%), and pleural effusion (11%). Scans from patients with clinical deterioration demonstrated higher percentages of bilateral consolidations (50 versus 15%; P = .033), anterior consolidations (47 versus 11%; P = .047), lateral consolidations (71 versus 29%; P = .030), pleural thickening (69 versus 30%; P = .045), but not B-lines (100 versus 80%; P = .11). Abnormal findings had similar prevalences between scans collected 0-6 days and 14-28 days from symptom onset. DISCUSSION: Certain LUS findings may be common in hospitalized COVID-19 patients, especially for those that experience clinical deterioration. These findings may occur anytime throughout the first 28 days of illness. Future efforts should investigate the predictive utility of these findings on clinical outcomes.

Point-of-Care Ultrasound Predicts Clinical Outcomes in Patients With COVID-19.

OBJECTIVES: Point-of-care ultrasound (POCUS) detects the pulmonary manifestations of COVID-19 and may predict patient outcomes. METHODS: We conducted a prospective cohort study at four hospitals from March 2020 to January 2021 to evaluate lung POCUS and clinical outcomes of COVID-19. Inclusion criteria included adult patients hospitalized for COVID-19 who received lung POCUS with a 12-zone protocol. Each image was interpreted by two reviewers blinded to clinical outcomes. Our primary outcome was the need for intensive care unit (ICU) admission versus no ICU admission. Secondary outcomes included intubation and supplemental oxygen usage. RESULTS: N = 160 patients were included. Among critically ill patients, B-lines (94 vs 76%; P < .01) and consolidations (70 vs 46%; P < .01) were more common. For scans collected within 24 hours of admission (N = 101 patients), early B-lines (odds ratio [OR] 4.41 [95% confidence interval, CI: 1.71-14.30]; P < .01) or consolidations (OR 2.49 [95% CI: 1.35-4.86]; P < .01) were predictive of ICU admission. Early consolidations were associated with oxygen usage after discharge (OR 2.16 [95% CI: 1.01-4.70]; P = .047). Patients with a normal scan within 24 hours of admission were less likely to require ICU admission (OR 0.28 [95% CI: 0.09-0.75]; P < .01) or supplemental oxygen (OR 0.26 [95% CI: 0.11-0.61]; P < .01). Ultrasound findings did not dynamically change over a 28-day scanning window after symptom onset. CONCLUSIONS: Lung POCUS findings detected within 24 hours of admission may provide expedient risk stratification for important COVID-19 clinical outcomes, including future ICU admission or need for supplemental oxygen. Conversely, a normal scan within 24 hours of admission appears protective. POCUS findings appeared stable over a 28-day scanning window, suggesting that these findings, regardless of their timing, may have clinical implications.

Handheld Ultrasound Device Usage and Image Acquisition Ability Among Internal Medicine Trainees: A Randomized Trial.

BACKGROUND: There is insufficient knowledge about how personal access to handheld ultrasound devices (HUDs) improves trainee learning with point-of-care ultrasound (POCUS). OBJECTIVE: To assess whether HUDs, alongside a yearlong lecture series, improved trainee POCUS usage and ability to acquire images. METHODS: Internal medicine intern physicians (n = 47) at a single institution from 2017 to 2018 were randomized 1:1 to receive personal HUDs (n = 24) for patient care/self-directed learning vs no-HUDs (n = 23). All interns received a repeated lecture series on cardiac, thoracic, and abdominal POCUS. Main outcome measures included self-reported HUD usage rates and post-intervention assessment scores using the Rapid Assessment of Competency in Echocardiography (RACE) scale between HUD and no-HUD groups. RESULTS: HUD interns reported performing POCUS assessments on patients a mean 6.8 (SD 2.2) times per week vs 6.4 (SD 2.9) times per week in non-HUD arm (P = .66). There was no relationship between the number of self-reported examinations per week and a trainee's post-intervention RACE score (rho = 0.022, P = .95). HUD interns did not have significantly higher post-intervention RACE scores (median HUD score 17.0 vs no-HUD score 17.8; P = .72). Trainee confidence with cardiac POCUS did not correlate with RACE scores. CONCLUSIONS: Personal HUDs without direct supervision did not increase the amount of POCUS usage or improve interns' acquisition abilities. Interns who reported performing more examinations per week did not have higher RACE scores. Improved HUD access and lectures without additional feedback may not improve POCUS mastery.

Interobserver Agreement of Lung Ultrasound Findings of COVID-19.

BACKGROUND: Lung ultrasound (LUS) has received considerable interest in the clinical evaluation of patients with COVID-19. Previously described LUS manifestations for COVID-19 include B-lines, consolidations, and pleural thickening. The interrater reliability (IRR) of these findings for COVID-19 is unknown. METHODS: This study was conducted between March and June 2020. Nine physicians (hospitalists: n = 4; emergency medicine: n = 5) from 3 medical centers independently evaluated n = 20 LUS scans (n = 180 independent observations) collected from patients with COVID-19, diagnosed via RT-PCR. These studies were randomly selected from an image database consisting of COVID-19 patients evaluated in the emergency department with portable ultrasound devices. Physicians were blinded to any patient information or previous LUS interpretation. Kappa values (κ) were used to calculate IRR. RESULTS: There was substantial IRR on the following items: normal LUS scan (κ = 0.79 [95% CI: 0.72-0.87]), presence of B-lines (κ = 0.79 [95% CI: 0.72-0.87]), ≥3 B-lines observed (κ = 0.72 [95% CI: 0.64-0.79]). Moderate IRR was observed for the presence of any consolidation (κ = 0.57 [95% CI: 0.50-0.64]), subpleural consolidation (κ = 0.49 [95% CI: 0.42-0.56]), and presence of effusion (κ = 0.49 [95% CI: 0.41-0.56]). Fair IRR was observed for pleural thickening (κ = 0.23 [95% CI: 0.15-0.30]). DISCUSSION: Many LUS manifestations for COVID-19 appear to have moderate to substantial IRR across providers from multiple specialties utilizing differing portable devices. The most reliable LUS findings with COVID-19 may include the presence/count of B-lines or determining if a scan is normal. Clinical protocols for LUS with COVID-19 may require additional observers for the confirmation of less reliable findings such as consolidations.

OrderRex clinical user testing: a randomized trial of recommender system decision support on simulated cases.

OBJECTIVE: To assess usability and usefulness of a machine learning-based order recommender system applied to simulated clinical cases. MATERIALS AND METHODS: 43 physicians entered orders for 5 simulated clinical cases using a clinical order entry interface with or without access to a previously developed automated order recommender system. Cases were randomly allocated to the recommender system in a 3:2 ratio. A panel of clinicians scored whether the orders placed were clinically appropriate. Our primary outcome included the difference in clinical appropriateness scores. Secondary outcomes included total number of orders, case time, and survey responses. RESULTS: Clinical appropriateness scores per order were comparable for cases randomized to the order recommender system (mean difference -0.11 order per score, 95% CI: [-0.41, 0.20]). Physicians using the recommender placed more orders (median 16 vs 15 orders, incidence rate ratio 1.09, 95%CI: [1.01-1.17]). Case times were comparable with the recommender system. Order suggestions generated from the recommender system were more likely to match physician needs than standard manual search options. Physicians used recommender suggestions in 98% of available cases. Approximately 95% of participants agreed the system would be useful for their workflows. DISCUSSION: User testing with a simulated electronic medical record interface can assess the value of machine learning and clinical decision support tools for clinician usability and acceptance before live deployments. CONCLUSIONS: Clinicians can use and accept machine learned clinical order recommendations integrated into an electronic order entry interface in a simulated setting. The clinical appropriateness of orders entered was comparable even when supported by automated recommendations.

Physician Usage and Acceptance of a Machine Learning Recommender System for Simulated Clinical Order Entry.

Clinical decision support tools that automatically disseminate patterns of clinical orders have the potential to improve patient care by reducing errors of omission and streamlining physician workflows. However, it is unknown if physicians will accept such tools or how their behavior will be affected. In this randomized controlled study, we exposed 34 licensed physicians to a clinical order entry interface and five simulated emergency cases, with randomized availability of a previously developed clinical order recommender system. With the recommender available, physicians spent similar time per case (6.7 minutes), but placed more total orders (17.1 vs. 15.8). The recommender demonstrated superior recall (59% vs 41%) and precision (25% vs 17%) compared to manual search results, and was positively received by physicians recognizing workflow benefits. Further studies must assess the potential clinical impact towards a future where electronic health records automatically anticipate clinical needs.

Timing and Predictors of Subspecialty Career Choice Among Internal Medicine Residents: A Retrospective Cohort Study.

BACKGROUND: Internal medicine residents face numerous career options after residency training. Little is known about when residents make their final career choice. OBJECTIVE: We assessed the timing and predictive factors of final career choices among internal medicine residents at graduation, including demographics, pre-residency career preferences, and rotation scheduling. METHODS: We conducted a retrospective study of graduates of an academic internal medicine residency program from 2014 to 2017. Main measures included demographics, rotation schedules, and self-reported career choices for residents at 5 time points: recruitment day, immediately after Match Day, end of postgraduate year 1 (PGY-1), end of PGY-2, and at graduation. RESULTS: Of the 138 residents eligible for the study, 5 were excluded based on participation in a fast-track program for an Accreditation Council for Graduate Medical Education subspecialty fellowship. Among the remaining 133 residents, 48 (36%) pursued general internal medicine fields and 78 (59%) pursued fellowship training. Career choices from recruitment day, Match Day, and PGY-1 were only weakly predictive of the career choice. Many choices demonstrated low concordance throughout training, and general medicine fields (primary care, hospital medicine) were frequently not decided until after PGY-2. Early clinical exposure to subspecialty rotations did not predict final career choice. CONCLUSIONS: Early career choices before and during residency training may have low predictability toward final career choices upon graduation in internal medicine. These choices may continue to have low predictability beyond PGY-2 for many specialties. Early clinical exposure may not predict final career choice for subspecialties.

Portable Ultrasound Device Usage and Learning Outcomes Among Internal Medicine Trainees: A Parallel-Group Randomized Trial.

BACKGROUND: Little is known about how to effectively train residents with point-of-care ultrasonography (POCUS) despite increasing usage. OBJECTIVE: This study aimed to assess whether handheld ultrasound devices (HUDs), alongside a year-long lecture series, improved trainee image interpretation skills with POCUS. METHODS: Internal medicine intern physicians (N = 149) at a single academic institution from 2016 to 2018 participated in the study. The 2017 interns (n = 47) were randomized 1:1 to receive personal HUDs (n = 24) for patient care vs no-HUDs (n = 23). All 2017 interns received a repeated lecture series regarding cardiac, thoracic, and abdominal POCUS. Interns were assessed on their ability to interpret POCUS images of normal/abnormal findings. The primary outcome was the difference in end-of-the-year assessment scores between interns randomized to receive HUDs vs not. Secondary outcomes included trainee scores after repeating lectures and confidence with POCUS. Intern scores were also compared with historical (2016, N = 50) and contemporaneous (2018, N = 52) controls who received no lectures. RESULTS: Interns randomized to HUDs did not have significantly higher image interpretation scores (median HUD score: 0.84 vs no-HUD score: 0.84; P = .86). However, HUD interns felt more confident in their abilities. The 2017 cohort had higher scores (median 0.84), compared with the 2016 historical control (median 0.71; P = .001) and 2018 contemporaneous control (median 0.48; P < .001). Assessment scores improved after first-time exposure to the lecture series, while repeated lectures did not improve scores. CONCLUSIONS: Despite feeling more confident, personalized HUDs did not improve interns' POCUS-related knowledge or interpretive ability. Repeated lecture exposure without further opportunities for deliberate practice may not be beneficial for mastering POCUS.

Thrombophilia testing in the inpatient setting: impact of an educational intervention.

BACKGROUND: Thrombophilia testing is frequently ordered in the inpatient setting despite its limited impact on clinical decision-making and unreliable results in the setting of acute thrombosis or ongoing anticoagulation. We sought to determine the effect of an educational intervention in reducing inappropriate thrombophilia testing for hospitalized patients. METHODS: During the 2014 academic year, we implemented an educational intervention with a phase implementation design for Internal Medicine interns at Stanford University Hospital. The educational session covering epidemiology, appropriate thrombophilia evaluation and clinical rationale behind these recommendations. Their ordering behavior was compared with a contemporaneous control (non-medicine and private services) and a historical control (interns from prior academic year). From the analyzed data, we determined the proportion of inappropriate thrombophilia testing of each group. Logistic generalized estimating equations were used to estimate odds ratios for inappropriate thrombophilia testing associated with the intervention. RESULTS: Of 2151 orders included, 934 were deemed inappropriate (43.4%). The two intervention groups placed 147 orders. A pooled analysis of ordering practices by intervention groups revealed a trend toward reduction of inappropriate ordering (p = 0.053). By the end of the study, the intervention groups had significantly lower rates of inappropriate testing compared to historical or contemporaneous controls. CONCLUSION: A brief educational intervention was associated with a trend toward reduction in inappropriate thrombophilia testing. These findings suggest that focused education on thrombophilia testing can positively impact inpatient ordering practices.

Evaluation of Trainee Competency with Point-of-Care Ultrasonography (POCUS): a Conceptual Framework and Review of Existing Assessments.

Point-of-care ultrasonography (POCUS) has the potential to transform healthcare delivery through its diagnostic expediency. Trainee competency with POCUS is now mandated for emergency medicine through the Accreditation Council for Graduate Medical Education (ACGME), and its use is expanding into other medical specialties, including internal medicine. However, a key question remains: how does one define "competency" with this emerging technology? As our trainees become more acquainted with POCUS, it is vital to develop validated methodology for defining and measuring competency amongst inexperienced users. As a framework, the assessment of competency should include evaluations that assess the acquisition and application of POCUS-related knowledge, demonstration of technical skill (e.g., proper probe selection, positioning, and image optimization), and effective integration into routine clinical practice. These assessments can be performed across a variety of settings, including web-based applications, simulators, standardized patients, and real clinical encounters. Several validated assessments regarding POCUS competency have recently been developed, including the Rapid Assessment of Competency in Echocardiography (RACE) or the Assessment of Competency in Thoracic Sonography (ACTS). However, these assessments focus mainly on technical skill and do not expand upon other areas of this framework, which represents a growing need. In this review, we explore the different methodologies for evaluating competency with POCUS as well as discuss current progress in the field of measuring trainee knowledge and technical skill.

CMS Billing Guidelines and Student Documentation: a New Era or New Burden?

The Centers for Medicare and Medicaid Services (CMS) recently revised their Medicare Claims Processing Manual with the addition of CR 10412, a provision that permits teaching providers to fully bill for medical student notes. This change will have significant implications on the documentation duties of teaching physicians and trainees. Potential benefits of this provision include reduced documentation burden on house officers, improved medical student empowerment, and the infusion of more original content into the electronic medical record. However, these benefits may be offset by shifting the burden of documentation onto medical students, which may compromise their time spent with patients and overall wellness. In this perspective, we review the changes that occurred with CR 10412 and their potential impact on documentation across the medical education spectrum.

The Role of Technology in the Bedside Encounter.

Technology has the potential to both distract and reconnect providers with their patients. The widespread adoption of electronic medical records in recent years pulls physicians away from time at the bedside. However, when used in conjunction with patients, technology has the potential to bring patients and physicians together. The increasing use of point-of-care ultrasound by physicians is changing the bedside encounter by allowing for real-time diagnosis with the treating physician. It is a powerful example of the way technology can be a force for refocusing on the bedside encounter.

Magnitude of Potentially Inappropriate Thrombophilia Testing in the Inpatient Hospital Setting.

Laboratory costs of thrombophilia testing exceed an estimated $650 million (in US dollars) annually. Quantifying the prevalence and financial impact of potentially inappropriate testing in the inpatient hospital setting represents an integral component of the effort to reduce healthcare expenditures. We conducted a retrospective analysis of our electronic medical record to evaluate 2 years' worth of inpatient thrombophilia testing measured against preformulated appropriateness criteria. Cost data were obtained from the Centers for Medicare and Medicaid Services 2016 Clinical Laboratory Fee Schedule. Of the 1817 orders analyzed, 777 (42.7%) were potentially inappropriate, with an associated cost of $40,422. The tests most frequently inappropriately ordered were Factor V Leiden, prothrombin gene mutation, protein C and S activity levels, antithrombin activity levels, and the lupus anticoagulant. Potentially inappropriate thrombophilia testing is common and costly. These data demonstrate a need for institution-wide changes in order to reduce unnecessary expenditures and improve patient care.

Time to Neurological Deterioration in Ischemic Stroke.

BACKGROUND: Neurological deterioration (ND) is common, with nearly one-half of ND patients deteriorating within the first 24 to 48 hours of stroke. The timing of ND with respect to ND etiology and reversibility has not been investigated. METHODS: At our center, we define ND as an increase of 2 or more points in the National Institutes of Health Stroke Scale (NIHSS) score within 24 hours and categorize etiologies of ND according to clinical reversibility. ND etiologies were considered non-reversible if such causes may have produced or extended any areas of ischemic neurologic injury due to temporary or permanent impairment in cerebral perfusion. RESULTS: Seventy-one of 350 ischemic stroke patients experienced ND. Over half (54.9%) of the patients who experienced ND did so within the 48 hours of last seen normal. The median time to ND for non-reversible causes was 1.5 days (IQR 0.9, 2.4 days) versus 2.6 days for reversible causes (IQR 1.4, 5.5 days, p=0.011). After adjusting for NIHSS and hematocrit on admission, the log-normal survival model demonstrated that for each 1-year increase in a patient's age, we expect a 3.9% shorter time to ND (p=0.0257). In addition, adjusting for age and hematocrit on admission, we found that that for each 1-point increase in the admission NIHSS, we expect a 3.1% shorter time to ND (p=0.0034). CONCLUSIONS: We found that despite having similar stroke severity and age, patients with nonreversible causes of ND had significantly shorter median time to ND when compared to patients with reversible causes of ND.