Creating virtual patients using large language models: scalable, global, and low cost.

Virtual patients (VPs) have long been used to teach and assess clinical reasoning. VPs can be programmed to simulate authentic patient-clinician interactions and to reflect a variety of contextual permutations. However, their use has historically been limited by the high cost and logistical challenges of large-scale implementation. We describe a novel globally-accessible approach to develop low-cost VPs at scale using artificial intelligence (AI) large language models (LLMs). We leveraged OpenAI Generative Pretrained Transformer (GPT) to create and implement two interactive VPs, and created permutations that differed in contextual features. We used systematic prompt engineering to refine a prompt instructing ChatGPT to emulate the patient for a given case scenario, and then provide feedback on clinician performance. We implemented the prompts using GPT-3.5-turbo and GPT-4.0, and created a simple text-only interface using the OpenAI API. GPT-4.0 was far superior. We also conducted limited testing using another LLM (Anthropic Claude), with promising results. We provide the final prompt, case scenarios, and Python code. LLM-VPs represent a 'disruptive innovation' - an innovation that is unmistakably inferior to existing products but substantially more accessible (due to low cost, global reach, or ease of implementation) and thereby able to reach a previously underserved market. LLM-VPs will lay the foundation for global democratization via low-cost-low-risk scalable development of educational and clinical simulations. These powerful tools could revolutionize the teaching, assessment, and research of management reasoning, shared decision-making, and AI evaluation (e.g. 'software as a medical device' evaluations).

Validation of the motivated strategies for learning questionnaire and instructional materials motivation survey.

PURPOSE: To validate the Motivated Strategies for Learning Questionnaire (MSLQ), which measures learner motivations; and the Instructional Materials Motivation Survey (IMMS), which measures the motivational properties of educational activities. METHODS: Participants (333 pharmacists, physicians, and advanced practice providers) completed the MSLQ, IMMS, Congruence-Personalization Questionnaire (CPQ), and a knowledge test immediately following an online learning module (April 2021). We randomly divided data for split-sample analysis using confirmatory factor analysis (CFA), exploratory factor analysis (EFA), and the multitrait-multimethod matrix. RESULTS: Cronbach alpha was ≥0.70 for most domains. CFA using sample 1 demonstrated suboptimal fit for both instruments, including 3 negatively-worded IMMS items with particularly low loadings. Revised IMMS (RIMMS) scores (which omit negatively-worded items) demonstrated better fit. Guided by EFA, we identified a novel 3-domain, 11-item 'MSLQ-Short Form-Revised' (MSLQ-SFR, with domains: Interest, Self-efficacy, and Attribution) and the 4-domain, 12-item RIMMS as the best models. CFA using sample 2 confirmed good fit. Correlations among MSLQ-SFR, RIMMS, and CPQ scores aligned with predictions; correlations with knowledge scores were small. CONCLUSIONS: Original MSLQ and IMMS scores show poor model fit, with negatively-worded items notably divergent. Revised, shorter models-the MSLQ-SFR and RIMMS-show satisfactory model fit (internal structure) and relations with other variables.

External vs Internal Focus of Attention in Procedural Skills Learning: A Randomized Study.

PURPOSE: The Optimizing Performance Through Intrinsic Motivation and Attention for Learning theory postulates that directing attention to the intended movement effect or outcome (external focus) is more effective than directing attention to the internal body motion or body part (internal focus). This study compared external vs internal focus of attention as novice students learned ultrasound-guided peripheral cannulation. The authors hypothesized that conditions promoting an external (vs internal) focus of attention would result in better performance on immediate and delayed (retention) testing. METHOD: The authors conducted a randomized, experimental study from October 2022 to February 2023 comparing external and internal focus using ultrasound-guided peripheral venous cannulation followed by practice using an ultrasound training block. Undergraduate health science students performed 4 practice trials with instructions to focus on the needle tip (external focus) or their hands (internal focus), followed by an immediate assessment and a retention assessment 2 weeks later. The primary outcome was time to successful cannulation. Results were compared using Kaplan-Meier curves and parametric survival analysis regression (accelerated failure time) models. RESULTS: Seventy participants completed the immediate assessment. The external focus group completed successful cannulation a mean of 2.33 times faster (95% CI = 1.47, 3.67; P < .001) than those in the internal focus group (median time, 12.0 vs 29.5 seconds). Results also favored the external focus group during practice, with the external focus group successfully completing the task a mean of 1.83 times faster (95% CI = 1.04, 3.21; P = .04) than the internal focus group. In the retention assessment, the external focus group successfully completed the task a mean of 2.37 times faster (95% CI = 1.19, 4.74; P = .02) than the internal focus group. CONCLUSIONS: External focus of attention compared with internal focus of attention improves medical motor skill learning in novice learners.

Validation of the Learner Engagement Instrument for Continuing Professional Development.

PURPOSE: Learner engagement is the energy learners exert to remain focused and motivated to learn. The Learner Engagement Instrument (LEI) was developed to measure learner engagement in a short continuing professional development (CPD) activity. The authors validated LEI scores using validity evidence of internal structure and relationships with other variables. METHOD: Participants attended 1 of 4 CPD courses (1 in-person, 2 online livestreamed, and 1 either in-person or online) in 2018, 2020, 2021, and 2022. Confirmatory factor analysis was used to examine model fit for several alternative structural models, separately for each course. The authors also conducted a generalizability study to estimate score reliability. Associations were evaluated between LEI scores and Continuing Medical Education Teaching Effectiveness (CMETE) scores and participant demographics. All statistical methods accounted for repeated measures by participants. RESULTS: 415 unique participants attended 203 different CPD presentations and completed the LEI 11,567 times. The originally hypothesized 4-domain model of learner engagement (domains: emotional, behavioral, cognitive in-class, cognitive out-of-class) demonstrated best model fit in all 4 courses, with Comparative Fit Index ≥0.99, Standardized Root Mean Square Residual ≤0.031, and Root Mean Square Error of Approximation ≤0.047. The reliability for overall scores and domain scores were all acceptable (50-rater G-coefficient ≥ 0.74) except for the cognitive-in-class domain (50-rater G-coefficient 0.55 to 0.66). All findings were similar for both in-person and online delivery modalities. Correlation of LEI scores with teaching effectiveness was confirmed (rho 0.58), and a small correlation was found with participant age (rho 0.19); other associations were small and not statistically significant. Using these findings, we generated a shortened 4-item instrument, the LEI Short Form. CONCLUSIONS: This study confirms a 4-domain model of learner engagement, and provides validity evidence that supports using LEI scores to measure learner engagement in both in-person and livestreamed CPD activities.

Assessment of management reasoning: Design considerations drawn from analysis of simulated outpatient encounters.

PURPOSE: Management reasoning is a distinct subset of clinical reasoning. We sought to explore features to be considered when designing assessments of management reasoning. METHODS: This is a hybrid empirical research study, narrative review, and expert perspective. In 2021, we reviewed and discussed 10 videos of simulated (staged) physician-patient encounters, actively seeking actions that offered insights into assessment of management reasoning. We analyzed our own observations in conjunction with literature on clinical reasoning assessment, using a constant comparative qualitative approach. RESULTS: Distinguishing features of management reasoning that will influence its assessment include management scripts, shared decision-making, process knowledge, illness-specific knowledge, and tailoring of the encounter and management plan. Performance domains that merit special consideration include communication, integration of patient preferences, adherence to the management script, and prognostication. Additional facets of encounter variation include the clinical problem, clinical and nonclinical patient characteristics (including preferences, values, and resources), team/system characteristics, and encounter features. We cataloged several relevant assessment approaches including written/computer-based, simulation-based, and workplace-based modalities, and a variety of novel response formats. CONCLUSIONS: Assessment of management reasoning could be improved with attention to the performance domains, facets of variation, and variety of approaches herein identified.

Gender Differences in Autonomy Granted to Residents and Fellows During Procedural Training: A Systematic Review and Meta-Analysis.

PURPOSE: Supervisors may be prone to implicit (unintentional) bias when granting procedural autonomy to trainees due to the subjectivity of autonomy decisions. The authors aimed to conduct a systematic review and meta-analysis to assess the differences in perceptions of procedural autonomy granted to physician trainees based on gender and/or race. METHOD: MEDLINE, Embase, CENTRAL, Scopus, and Web of Science were searched (search date: January 5, 2022) for studies reporting quantitative gender- or race-based differences in perceptions of procedural autonomy of physician trainees. Reviewers worked in duplicate for article selection and data abstraction. Primary measures of interest were self-reported and observer-rated procedural autonomy. Meta-analysis pooled differences in perceptions of procedural autonomy based on trainee gender. RESULTS: The search returned 2,714 articles, of which 16 were eligible for inclusion. These reported data for 6,109 trainees (median 90 per study) and 2,763 supervisors (median 54 per study). No studies investigated differences in perceptions of autonomy based on race. In meta-analysis of disparities between genders in autonomy ratings (positive number favoring female trainees), pooled standardized mean differences were -0.12 (95% confidence interval [CI] = -0.19, -0.04; P = .003; n = 10 studies) for trainee self-rated autonomy and -0.05 (95% CI = -0.11, 0.01; P = .07; n = 9 studies) for supervisor ratings of autonomy. CONCLUSIONS: Limited evidence suggests that female trainees perceived that they received less procedural autonomy than did males. Further research exploring the degree of gender- and race-based differences in procedural autonomy, and factors that influence these differences, is warranted.

Educational and Patient Care Impacts of In Situ Simulation in Healthcare: A Systematic Review.

ABSTRACT: This systematic review was performed to assess the effectiveness of in situ simulation education. We searched databases including MEDLINE and Embase for studies comparing in situ simulation with other educational approaches. Two reviewers screened articles and extracted information. Sixty-two articles met inclusion criteria, of which 24 were synthesized quantitatively using random effects meta-analysis. When compared with current educational practices alone, the addition of in situ simulation to these practices was associated with small improvements in clinical outcomes, including mortality [odds ratio, 0.66; 95% confidence interval (CI), 0.55 to 0.78], care metrics (standardized mean difference, -0.34; 95% CI, -0.45 to -0.21), and nontechnical skills (standardized mean difference, -0.52; 95% CI, -0.99 to -0.05). Comparisons between in situ and traditional simulation showed mixed learner preference and knowledge improvement between groups, while technical skills showed improvement attributable to in situ simulation. In summary, available evidence suggests that adding in situ simulation to current educational practices may improve patient mortality and morbidity.

Personalisation and embodiment in e-Learning for health professionals: A randomised controlled trial.

PURPOSE: Mayer's theory of multimedia learning proposes that personalisation and embodiment (P/E) can improve outcomes in e-Learning. The authors hypothesised that an e-Learning module enhanced by P/E principles would lead to higher knowledge, perceived P/E and motivation among health care professionals, compared with an unenhanced module. METHODS: The authors conducted a randomised trial comparing two versions of a 30-minute multimedia e-Learning module addressing the antibiotic management of pneumonia. The unenhanced format used slides with voiceover (human voice but no visible speaker), formal language and no specific P/E strategies. The enhanced format additionally implemented P/E strategies including conversational style, polite language, visible author, social congruence, human-like presence and professional presence by subtly changing the script and substituting several short videos of subject matter experts. Participants included pharmacists, physicians and advanced practice providers from three academic and several community hospitals. Outcomes included knowledge, perceived P/E (assessed by the Congruence Personalisation Questionnaire, CPQ), motivation (assessed via the Instructional Materials Motivation Survey [IMMS] and Motivated Strategies for Learning Questionnaire [MSLQ]) and course satisfaction. RESULTS: There were 406 participants including 225 pharmacists, 109 physicians and 72 advanced practice providers. Post-module knowledge was slightly higher for the enhanced versus the unenhanced format, but the difference did not reach statistical significance (adjusted mean difference, 0.04 of 10 possible, [95% CI -0.26, 0.34], p = 0.78; Cohen d 0.02). Participant perceptions of P/E (measured via CPQ) were significantly greater for the enhanced format (difference 0.46 of 5 possible [0.35, 0.56], p < 0.001; Cohen d 0.85), as were motivational features of the e-Learning course (measured via IMMS) (difference 0.14 of 5 possible [0.02, 0.26], p = 0.02; Cohen d 0.24). Participants' overall motivational orientation (measured via MSLQ) and course satisfaction were not significantly different between the two formats (p > 0.05). CONCLUSION: Application of P/E principles to an e-Learning module led to greater perceived P/E and motivational features but did not influence knowledge.

Targeted Training Reduces Search Errors but Not Classification Errors for Hepatic Metastasis Detection at Contrast-Enhanced CT.

RATIONALE AND OBJECTIVES: Methods are needed to improve the detection of hepatic metastases. Errors occur in both lesion detection (search) and decisions of benign versus malignant (classification). Our purpose was to evaluate a training program to reduce search errors and classification errors in the detection of hepatic metastases in contrast-enhanced abdominal computed tomography (CT). MATERIALS AND METHODS: After Institutional Review Board approval, we conducted a single-group prospective pretest-posttest study. Pretest and posttest were identical and consisted of interpreting 40 contrast-enhanced abdominal CT exams containing 91 liver metastases under eye tracking. Between pretest and posttest, readers completed search training with eye-tracker feedback and coaching to increase interpretation time, use liver windows, and use coronal reformations. They also completed classification training with part-task practice, rating lesions as benign or malignant. The primary outcome was metastases missed due to search errors (<2 seconds gaze under eye tracker) and classification errors (>2 seconds). Jackknife free-response receiver operator characteristic (JAFROC) analysis was also conducted. RESULTS: A total of 31 radiologist readers (8 abdominal subspecialists, 8 nonabdominal subspecialists, 15 senior residents/fellows) participated. Search errors were reduced (pretest 11%, posttest 8%, difference 3% [95% confidence interval, 0.3%-5.1%], P = .01), but there was no difference in classification errors (difference 0%, P = .97) or in JAFROC figure of merit (difference -0.01, P = .36). In subgroup analysis, abdominal subspecialists demonstrated no evidence of change. CONCLUSION: Targeted training reduced search errors but not classification errors for the detection of hepatic metastases at contrast-enhanced abdominal CT. Improvements were not seen in all subgroups.

Scrubbing With Aristotle: Cultivating Surgical Wisdom through Inflection Points.

Surgical educators and researchers have well-considered a breadth of topics related to surgery. However, there is one concept that is notably absent in this corpus: surgical wisdom. In this perspective, we draw on work from Aristotle and psychology research to introduce the concept of phronesis, which we believe is useful for understanding surgical wisdom. We further illustrate how this concept can be a useful tool for surgical educators through the discussion of four distinct functions of phronesis, and illustrating the ability of these functions to help learners cultivate knowledge at important decision points, or inflection points, in surgical training and a surgical career.

A training program to reduce reader search errors for liver metastasis detection in CT.

Detection of low contrast liver metastases varies between radiologists. Training may improve performance for lower-performing readers and reduce inter-radiologist variability. We recruited 31 radiologists (15 trainees, 8 non-abdominal staff, and 8 abdominal staff) to participate in four separate reading sessions: pre-test, search training, classification training, and post-test. In the pre-test, each radiologist interpreted 40 liver CT exams containing 91 metastases, circumscribed suspected hepatic metastases while under eye tracker observation, and rated confidence. In search training, radiologists interpreted a separate set of 30 liver CT exams while receiving eye tracker feedback and after coaching to increase use of coronal reformations, interpretation time, and use of liver windows. In classification training, radiologists interpreted up to 100 liver CT image patches, most with benign or malignant lesions, and compared their annotations to ground truth. Post-test was identical to pre-test. Between pre- and post-test, sensitivity increased by 2.8% (p = 0.01) but AUC did not change significantly. Missed metastases were classified as search errors (<2 seconds gaze time) or classification errors (>2 seconds gaze time) using the eye tracker. Out of 2775 possible detections, search errors decreased (10.8% to 8.1%; p < 0.01) but classification errors were unchanged (5.7% vs 5.7%). When stratified by difficulty, easier metastases showed larger reductions in search errors: for metastases with average sensitivity of 0-50%, 50-90%, and 90-100%, reductions in search errors were 16%, 35%, and 58%, respectively. The training program studied here may be able to improve radiologist performance by reducing errors but not classification errors.

Technology-enhanced simulation in emergency medicine: Updated systematic review and meta-analysis 1991-2021.

Background: Over the past decade, the use of technology-enhanced simulation in emergency medicine (EM) education has grown, yet we still lack a clear understanding of its effectiveness. This systematic review aims to identify and synthesize studies evaluating the comparative effectiveness of technology-enhanced simulation in EM. Methods: We searched MEDLINE, EMBASE, PsycINFO, CINAHL, ERIC, Web of Science, and Scopus to identify EM simulation research that compares technology-enhanced simulation with other instructional modalities. Two reviewers screened articles for inclusion and abstracted information on learners, clinical topics, instructional design features, outcomes, cost, and study quality. Standardized mean difference (SMD) effect sizes were pooled using random effects. Results: We identified 60 studies, enrolling at least 5279 learners. Of these, 23 compared technology-enhanced simulation with another instructional modality (e.g., living humans, lecture, small group), and 37 compared two forms of technology-enhanced simulation. Compared to lecture or small groups, we found simulation to have nonsignificant differences for time skills (SMD 0.33, 95% confidence interval [CI] -0.23 to 0.89, n = 3), but a large, significant effect for non-time skills (SMD 0.82, 95% CI 0.18 to 1.46, n = 8). Comparison of alternative types of technology-enhanced simulation found favorable associations with skills acquisition, of moderate magnitude, for computer-assisted guidance (compared to no computer-assisted guidance), for time skills (SMD 0.50, 95% CI -1.66 to 2.65, n = 2) and non-time skills (SMD 0.57, 95% CI 0.33 to 0.80, n = 6), and for more task repetitions (time skills SMD 1.01, 95% CI 0.16 to 1.86, n = 2) and active participation (compared to observation) for time skills (SMD 0.85, 95% CI 0.25 to 1.45, n = 2) and non-time skills (SMD 0.33 95% CI 0.08 to 0.58, n = 3). Conclusions: Technology-enhanced simulation is effective for EM learners for skills acquisition. Features such as computer-assisted guidance, repetition, and active learning are associated with greater effectiveness.

Video-Based (Livestream) and In-Person Continuing Professional Development: Participant and Course Features Associated With Choice.

PURPOSE: Information on factors influencing learner choice between in-person and video-based continuing professional development (CPD) would help course leaders with planning and implementation. This study aimed to investigate how registration patterns differ between in-person and video-based offerings for the same CPD course. METHOD: The authors obtained data from 55 CPD courses offered in-person (at various U.S. locations) and via livestreamed video, January 2020 to April 2022. Participants included physicians, advanced practice providers, allied health professionals, nurses, and pharmacists. Registration rates were compared by participant and course features including professional role, age, country of residence, distance to and perceived desirability of the in-person destination, and timing of registration. RESULTS: Analyses included 11,072 registrations, of which 4,336 (39.2%) were for video-based learning. There was significant heterogeneity in video-based registrations across courses, ranging 14.3% to 71.4%. Multivariable analysis showed higher video-based registrations for advanced practice providers (vs physicians; adjusted odds ratio [AOR] 1.80 [99% confidence interval, 1.55-2.10]), non-U.S. residents (AOR 3.26 [1.18-9.01]), longer distance (AOR 1.19 [1.16-1.23] for each doubling of distance), and courses in July-September 2021 (vs January-April 2022; AOR 1.59 [1.24-2.02]); video-based registrations were lower for current or former employees or trainees of our institution (AOR 0.53 [0.45-0.61]), destinations of moderate or high desirability (vs low; AOR 0.42 [0.34-0.51] and 0.44 [0.33-0.58], respectively), and early registration (AOR 0.67 [0.64-0.69] for each doubling of days between registration and course start). There was no significant difference by age (AOR 0.92 [0.82-1.05] for participants > 46 years old vs younger participants). The multivariable model correctly predicted actual registration 78.5% of the time. CONCLUSIONS: Video-based livestreamed CPD is popular (chosen by nearly 40% of participants), although choices varied widely by course. Professional role, institutional affiliation, distance traveled, location desirability, and registration timing have small but statistically significant associations with choice of video-based vs in-person CPD.

Management reasoning and patient-clinician interactions: Insights from shared decision-making and simulated outpatient encounters.

PURPOSE: To expand understanding of patient-clinician interactions in management reasoning. METHODS: We reviewed 10 videos of simulated patient-clinician encounters to identify instances of problematic and successful communication, then reviewed the videos again through the lens of two models of shared decision-making (SDM): an 'involvement-focused' model and a 'problem-focused' model. Using constant comparative qualitative analysis we explored the connections between these patient-clinician interactions and management reasoning. RESULTS: Problems in patient-clinician interactions included failures to: encourage patient autonomy; invite the patient's involvement in decision-making; convey the health impact of the problem; explore and address concerns and questions; explore the context of decision-making (including patient preferences); meet the patient where they are; integrate situational preferences and priorities; offer >1 viable option; work with the patient to solve a problem of mutual concern; explicitly agree to a final care plan; and build the patient-clinician relationship. Clinicians' 'management scripts' varied along a continuum of prioritizing clinician vs patient needs. Patients also have their own cognitive scripts that guide their interactions with clinicians. The involvement-focused and problem-focused SDM models illuminated distinct, complementary issues. CONCLUSIONS: Management reasoning is a deliberative interaction occurring in the space between individuals. Juxtaposing management reasoning alongside SDM generated numerous insights.

Management Reasoning: Empirical Determination of Key Features and a Conceptual Model.

PURPOSE: Management reasoning is a critical yet understudied phenomenon in clinical practice and medical education. The authors sought to empirically identify key features of management reasoning and construct a model describing the management reasoning process. METHOD: In November 2020, 4 investigators each reviewed 10 video clips of simulated outpatient physician-patient encounters and used a coding form to document key features and insights related to management reasoning. The team used a constant comparative approach to distill 120 pages of raw observations into an 18-page list of management tasks, processes, and insights. The team then had a series of discussions to iteratively refine these findings into a parsimonious model of management reasoning. RESULTS: The investigators empirically identified 12 distinct features of management reasoning: contrasting and selection among multiple solutions; prioritization of patient, clinician, and system preferences and constraints; communication and shared decision making; ongoing monitoring and adjustment of the management plan; dynamic interplay among people, systems, and competing priorities; illness-specific knowledge; process knowledge; management scripts; clinician roles as patient teacher and salesperson; clinician-patient relationship; prognostication; and organization of the clinical encounter (sequencing and time management). Management scripts seemed to play a prominent and critical role. The model of management reasoning comprised 4 steps: instantiation of a management script, identifying (multiple) options and beginning to teach the patient, shared decision making, and ongoing monitoring and adjustment. This model also conceives 2 overarching features: that management reasoning is personalized to the patient and that it occurs between individuals rather than exclusively within the clinician's mind. CONCLUSIONS: Management scripts constitute a key feature of management reasoning, along with teaching patients about viable options, shared decision making, ongoing monitoring and adjustment, and personalization. Management reasoning seems to be constructed and negotiated between individuals rather than exclusively within the clinician.

Learner engagement and teaching effectiveness in livestreamed versus in-person CME.

INTRODUCTION: Engaging learners in continuing medical education (CME) is challenging. Recently, CME courses have transitioned to livestreamed CME, with learners viewing live, in-person courses online. The authors aimed to (1) compare learner engagement and teaching effectiveness in livestreamed with in-person CME and (2) determine how livestream engagement and teaching effectiveness is associated with (A) interactivity metrics, (B) presentation characteristics and (C) medical knowledge. METHODS: A 3-year, non-randomised study of in-person and livestream CME was performed. The course was in-person for 2018 but transitioned to livestream for 2020 and 2021. Learners completed the Learner Engagement Inventory and Teaching Effectiveness Instrument after each presentation. Both instruments were supported by content, internal structure and relations to other variables' validity evidence. Interactivity metrics included learner use of audience response, questions asked by learners and presentation views. Presentation characteristics included presentations using audience response, using pre/post-test format, time of day and words per slide. Medical knowledge was assessed by audience response. A repeated measures analysis of variance (anova) was used for comparisons and a mixed model approach for correlations. RESULTS: A total of 159 learners (response rate 27%) completed questionnaires. Engagement did not significantly differ between in-person or livestream CME. (4.56 versus 4.53, p = 0.64, maximum 5 = highly engaged). However, teacher effectiveness scores were higher for in-person compared with livestream (4.77 versus 4.71 p = 0.01, maximum 5 = highly effective). For livestreamed courses, learner engagement was associated with presentation characteristics, including presentation using of audience response (yes = 4.57, no = 4.45, p < .0001), use of a pre/post-test (yes = 4.62, no = 4.54, p < .0001) and time of presentation (morning = 4.58, afternoon = 4.53, p = .0002). Significant associations were not seen for interactivity metrics or medical knowledge. DISCUSSION: Livestreaming may be as engaging as in-person CME. Although teaching effectiveness in livestreaming was lower, this difference was small. CME course planners should consider offering livestream CME while exploring strategies to enhance teaching effectiveness in livestreamed settings.

Measuring Personalization, Embodiment, and Congruence in Online Learning: A Validation Study.

PURPOSE: Instructional designs promoting personalization (conversational style and visible author), embodiment (human-like mannerisms and presence), and social and cognitive congruence have been shown to enhance learning. However, measurements of these instructional design features are lacking. The authors sought to develop and validate an instrument to measure personalization, embodiment, and congruence in an online course, collecting evidence of content, internal structure, and relationships with other variables. METHOD: Through literature review, expert review, and pilot testing, the authors developed a 15-item, 5-domain instrument. Participants completed this instrument along with assessments of motivation (Instructional Materials Motivation Survey and Motivated Strategies for Learning Questionnaire), knowledge, and course satisfaction immediately after a 30-minute online multimedia learning module comprising slides with voiceover and (for some participants) short video clips, from January to April 2021. Participants included pharmacists, physicians, and advanced practice providers at academic and community hospitals. Confirmatory factor analysis (CFA) was used to examine model fit, followed by structural revisions guided by exploratory factor analysis (EFA) and tested in a new CFA using a split-sample approach. The authors evaluated associations between the revised instrument scores and motivation, knowledge, and satisfaction. RESULTS: There were 406 participants. CFA of the original 15-item instrument revealed poor model fit. EFA using 214 randomly selected participants suggested a 2-domain structure. CFA of the revised structure using the other 192 participants revealed good fit for a 9-item, 2-domain instrument: the Congruence-Personalization Questionnaire (CPQ). Linear regression confirmed positive, statistically significant associations for most hypothesized relationships (R 2 ≥ 0.12 and P < .001 for 14 of 18 associations with motivation and satisfaction). Associations with knowledge were weak and nonsignificant (R 2 ≤ 0.004, P > .23). CONCLUSIONS: Evidence of content, internal structure, and relationships with other variables support the validity of CPQ scores for measuring personalization and congruence in an online module.

Modeling Diagnostic Expertise in Cases of Irreducible Uncertainty: The Decision-Aligned Response Model.

PURPOSE: Assessing expertise using psychometric models usually yields a measure of ability that is difficult to generalize to the complexity of diagnoses in clinical practice. However, using an item response modeling framework, it is possible to create a decision-aligned response model that captures a clinician's decision-making behavior on a continuous scale that fully represents competing diagnostic possibilities. In this proof-of-concept study, the authors demonstrate the necessary statistical conceptualization of this model using a specific electrocardiogram (ECG) example. METHOD: The authors collected a range of ECGs with elevated ST segments due to either ST-elevation myocardial infarction (STEMI) or pericarditis. Based on pilot data, 20 ECGs were chosen to represent a continuum from "definitely STEMI" to "definitely pericarditis," including intermediate cases in which the diagnosis was intentionally unclear. Emergency medicine and cardiology physicians rated these ECGs on a 5-point scale ("definitely STEMI" to "definitely pericarditis"). The authors analyzed these ratings using a graded response model showing the degree to which each participant could separate the ECGs along the diagnostic continuum. The authors compared these metrics with the discharge diagnoses noted on chart review. RESULTS: Thirty-seven participants rated the ECGs. As desired, the ECGs represented a range of phenotypes, including cases where participants were uncertain in their diagnosis. The response model showed that participants varied both in their propensity to diagnose one condition over another and in where they placed the thresholds between the 5 diagnostic categories. The most capable participants were able to meaningfully use all categories, with precise thresholds between categories. CONCLUSIONS: The authors present a decision-aligned response model that demonstrates the confusability of a particular ECG and the skill with which a clinician can distinguish 2 diagnoses along a continuum of confusability. These results have broad implications for testing and for learning to manage uncertainty in diagnosis.

Understanding Reader Variability: A 25-Radiologist Study on Liver Metastasis Detection at CT.

Background Substantial interreader variability exists for common tasks in CT imaging, such as detection of hepatic metastases. This variability can undermine patient care by leading to misdiagnosis. Purpose To determine the impact of interreader variability associated with (a) reader experience, (b) image navigation patterns (eg, eye movements, workstation interactions), and (c) eye gaze time at missed liver metastases on contrast-enhanced abdominal CT images. Materials and Methods In a single-center prospective observational trial at an academic institution between December 2020 and February 2021, readers were recruited to examine 40 contrast-enhanced abdominal CT studies (eight normal, 32 containing 91 liver metastases). Readers circumscribed hepatic metastases and reported confidence. The workstation tracked image navigation and eye movements. Performance was quantified by using the area under the jackknife alternative free-response receiver operator characteristic (JAFROC-1) curve and per-metastasis sensitivity and was associated with reader experience and image navigation variables. Differences in area under JAFROC curve were assessed with the Kruskal-Wallis test followed by the Dunn test, and effects of image navigation were assessed by using the Wilcoxon signed-rank test. Results Twenty-five readers (median age, 38 years; IQR, 31-45 years; 19 men) were recruited and included nine subspecialized abdominal radiologists, five nonabdominal staff radiologists, and 11 senior residents or fellows. Reader experience explained differences in area under the JAFROC curve, with abdominal radiologists demonstrating greater area under the JAFROC curve (mean, 0.77; 95% CI: 0.75, 0.79) than trainees (mean, 0.71; 95% CI: 0.69, 0.73) (P = .02) or nonabdominal subspecialists (mean, 0.69; 95% CI: 0.60, 0.78) (P = .03). Sensitivity was similar within the reader experience groups (P = .96). Image navigation variables that were associated with higher sensitivity included longer interpretation time (P = .003) and greater use of coronal images (P < .001). The eye gaze time was at least 0.5 and 2.0 seconds for 71% (266 of 377) and 40% (149 of 377) of missed metastases, respectively. Conclusion Abdominal radiologists demonstrated better discrimination for the detection of liver metastases on abdominal contrast-enhanced CT images. Missed metastases frequently received at least a brief eye gaze. Higher sensitivity was associated with longer interpretation time and greater use of liver display windows and coronal images. © RSNA, 2022 Online supplemental material is available for this article.