Reliability, validity and acceptability of an online clinical reasoning simulator for medical students: An international pilot.

INTRODUCTION: Clinical reasoning skills are essential for decision-making. Current assessment methods are limited when testing clinical reasoning and management of uncertainty. This study evaluates the reliability, validity and acceptability of Practicum Script, an online simulation-based programme, for developing medical students' clinical reasoning skills using real-life cases. METHODS: In 2020, we conducted an international, multicentre pilot study using 20 clinical cases with 2457 final-year medical students from 21 schools worldwide. Psychometric analysis was performed (n = 1502 students completing at least 80% of cases). Classical estimates of reliability for three test domains (hypothesis generation, hypothesis argumentation and knowledge application) were calculated using Cronbach's alpha and McDonald's omega coefficients. Validity evidence was obtained by confirmatory factor analysis (CFA) and measurement alignment (MA). Items from the knowledge application domain were analysed using cognitive diagnostic modelling (CDM). Acceptability was evaluated by an anonymous student survey. RESULTS: Reliability estimates were high with narrow confidence intervals. CFA revealed acceptable goodness-of-fit indices for the proposed three-factor model. CDM analysis demonstrated good absolute test fit and high classification accuracy estimates. Student survey responses showed high levels of acceptability. CONCLUSION: Our findings suggest that Practicum Script is a useful resource for strengthening students' clinical reasoning skills and ability to manage uncertainty.

Experts' responses in script concordance tests: a response process validity investigation.

CONTEXT: The script concordance test (SCT), designed to measure clinical reasoning in complex cases, has recently been the subject of several critical research studies. Amongst other issues, response process validity evidence remains lacking. We explored the response processes of experts on an SCT scoring panel to better understand their seemingly divergent beliefs about how new clinical data alter the suitability of proposed actions within simulated patient cases. METHODS: A total of 10 Argentine gastroenterologists who served as the expert panel on an existing SCT re-answered 15 cases 9 months after their original panel participation. They then answered questions probing their reasoning and reactions to other experts' perspectives. RESULTS: The experts sometimes noted they would not ordinarily consider the actions proposed for the cases at all (30/150 instances [20%]) or would collect additional data first (54/150 instances [36%]). Even when groups of experts agreed about how new clinical data in a case affected the suitability of a proposed action, there was often disagreement (118/133 instances [89%]) about the suitability of the proposed action before the new clinical data had been introduced. Experts reported confidence in their responses, but showed limited consistency with the responses they had given 9 months earlier (linear weighted kappa = 0.33). Qualitative analyses showed nuanced and complex reasons behind experts' responses, revealing, for example, that experts often considered the unique affordances and constraints of their varying local practice environments when responding. Experts generally found other experts' alternative responses moderately compelling (mean ± standard deviation 2.93 ± 0.80 on a 5-point scale, where 3 = moderately compelling). Experts switched their own preferred responses after seeing others' reasoning in 30 of 150 (20%) instances. CONCLUSIONS: Expert response processes were not consistent with the classical interpretation and use of SCT scores. However, several fruitful and justifiable alternatives for the use of SCT-like methods are proposed, such as to guide assessments for learning.

The practicum script concordance test: an online continuing professional development format to foster reflection on clinical practice.

INTRODUCTION: Judgment in the face of uncertainty is an important dimension of expertise and clinical competence. However, it is challenging to conceive continuing professional development (CPD) initiatives aimed at helping physicians enhance their clinical judgment skills in ill-defined situations. We present an online script concordance-based CPD program (the Practicum Script Concordance Test, copyright ©2006 by Practicum Foundation), a tool that can be used to support health professionals in the development of their reflective clinical reasoning ability. We describe the rationale and principles and report on the implementation of 2 online programs based on this new CPD initiative. METHOD: The Practicum Script Concordance Test program consists of daily testing and feedback over the course of a year using SCT items. Feedback is both global (eg, health professionals are told their cumulative mean score) and specific (eg, they can view the expert panel's responses together with their justifications for their answers). Participants have the option of contacting a personal tutor, to whom they can send questions. Data regarding feasibility, participation, and acceptability were collected. RESULTS: Initial implementation took place in Mexico where 1901 physicians (1349 paediatricians, 552 cardiologists) were enrolled in Practicum programs. Around 70% of those enrolled pursued the program and were very satisfied with its format and content. The online format was an important factor in the development and maintenance of the programs. Dropouts had issues with the SCT concept and the time required to participate. DISCUSSION: The on-line Practicum Script Concordance Test program was designed to foster expertise development based on practice, reflection and feedback. Although further research is needed to examine its impact physicians' practice and ultimately on patient outcomes, it is an original and promising development in CPD.