Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions.

BACKGROUND: It remains unclear which item format would best suit the assessment of clinical reasoning: context-rich single best answer questions (crSBAs) or key-feature problems (KFPs). This study compared KFPs and crSBAs with respect to students' acceptance, their educational impact, and psychometric characteristics when used in a summative end-of-clinical-clerkship pediatric exam. METHODS: Fifth-year medical students (n = 377) took a computer-based exam that included 6-9 KFPs and 9-20 crSBAs which assessed their clinical reasoning skills, in addition to an objective structured clinical exam (OSCE) that assessed their clinical skills. Each KFP consisted of a case vignette and three key features using a "long-menu" question format. We explored students' perceptions of the KFPs and crSBAs in eight focus groups and analyzed statistical data of 11 exams. RESULTS: Compared to crSBAs, KFPs were perceived as more realistic and difficult, providing a greater stimulus for the intense study of clinical reasoning, and were generally well accepted. The statistical analysis revealed no difference in difficulty, but KFPs resulted more reliable and efficient than crSBAs. The correlation between the two formats was high, while KFPs correlated more closely with the OSCE score. CONCLUSIONS: KFPs with long-menu exams seem to bring about a positive educational effect without psychometric drawbacks.

Exploring the validity and reliability of a questionnaire for evaluating virtual patient design with a special emphasis on fostering clinical reasoning.

BACKGROUND: Virtual patients (VPs) are increasingly used to train clinical reasoning. So far, no validated evaluation instruments for VP design are available. AIMS: We examined the validity of an instrument for assessing the perception of VP design by learners. METHODS: Three sources of validity evidence were examined: (i) Content was examined based on theory of clinical reasoning and an international VP expert team. (ii) The response process was explored in think-aloud pilot studies with medical students and in content analyses of free text questions accompanying each item of the instrument. (iii) Internal structure was assessed by exploratory factor analysis (EFA) and inter-rater reliability by generalizability analysis. RESULTS: Content analysis was reasonably supported by the theoretical foundation and the VP expert team. The think-aloud studies and analysis of free text comments supported the validity of the instrument. In the EFA, using 2547 student evaluations of a total of 78 VPs, a three-factor model showed a reasonable fit with the data. At least 200 student responses are needed to obtain a reliable evaluation of a VP on all three factors. CONCLUSION: The instrument has the potential to provide valid information about VP design, provided that many responses per VP are available.

Learner preferences regarding integrating, sequencing and aligning virtual patients with other activities in the undergraduate medical curriculum: A focus group study.

CONTEXT: E-learning resources, such as virtual patients (VPs), can be more effective when they are integrated in the curriculum. To gain insights that can inform guidelines for the curricular integration of VPs, we explored students' perceptions of scenarios with integrated and non-integrated VPs aimed at promoting clinical reasoning skills. METHODS: During their paediatric clerkship, 116 fifth-year medical students were given at least ten VPs embedded in eight integrated scenarios and as non-integrated add-ons. The scenarios differed in the sequencing and alignment of VPs and related educational activities, tutor involvement, number of VPs, relevance to assessment and involvement of real patients. We sought students' perceptions on the VP scenarios in focus group interviews with eight groups of 4-7 randomly selected students (n = 39). The interviews were recorded, transcribed and analysed qualitatively. RESULTS: The analysis resulted in six themes reflecting students' perceptions of important features for effective curricular integration of VPs: (i) continuous and stable online access, (ii) increasing complexity, adapted to students' knowledge, (iii) VP-related workload offset by elimination of other activities, (iv) optimal sequencing (e.g.: lecture--1 to 2 VP(s)--tutor-led small group discussion--real patient) and (V) optimal alignment of VPs and educational activities, (vi) inclusion of VP topics in assessment. CONCLUSIONS: The themes appear to offer starting points for the development of a framework to guide the curricular integration of VPs. Their impact needs to be confirmed by studies using quantitative controlled designs.

Computer supported collaborative learning in a clerkship: an exploratory study on the relation of discussion activity and revision of critical appraisal papers.

BACKGROUND: Medical students in clerkship are continuously confronted with real and relevant patient problems. To support clinical problem solving skills, students perform a Critical Appraisal of a Topic (CAT) task, often resulting in a paper. Because such a paper may contain errors, students could profit from discussion with peers, leading to paper revision. Active peer discussion by a Computer Supported Collaborative Learning (CSCL) environment show positive medical students perceptions on subjective knowledge improvement. High students' activity during discussions in a CSCL environment demonstrated higher task-focussed discussion reflecting higher levels of knowledge construction. However, it remains unclear whether high discussion activity influences students' decisions revise their CAT paper. The aim of this research is to examine whether students who revise their critical appraisal papers after discussion in a CSCL environment show more task-focussed activity and discuss more intensively on critical appraisal topics than students who do not revise their papers. METHODS: Forty-seven medical students, stratified in subgroups, participated in a structured asynchronous online discussion of individual written CAT papers on self-selected clinical problems. The discussion was structured by three critical appraisal topics. After the discussion, the students could revise their paper. For analysis purposes, all students' postings were blinded and analysed by the investigator, unaware of students characteristics and whether or not the paper was revised. Postings were counted and analysed by an independent rater, Postings were assigned into outside activity, non-task-focussed activity or task-focussed activity. Additionally, postings were assigned to one of the three critical appraisal topics. Analysis results were compared by revised and unrevised papers. RESULTS: Twenty-four papers (51.6%) were revised after the online discussion. The discussions of the revised papers showed significantly higher numbers of postings, more task-focussed activities, and more postings about the two critical appraisal topics: "appraisal of the selected article(s)", and "relevant conclusion regarding the clinical problem". CONCLUSION: A CSCL environment can support medical students in the execution and critical appraisal of authentic tasks in the clinical workplace. Revision of CAT papers appears to be related to discussions activity, more specifically reflecting high task-focussed activity of critical appraisal topics.

Towards a typology of virtual patients.

INTRODUCTION: Although on-screen "virtual patients (VPs)" have been around for decades it is only now that they are entering the mainstream, and as such they are new to most of the medical education community. There is significant variety in the form, function, and efficacy of different VPs and there is, therefore, a growing need to clarify and distinguish between them. This article seeks to clarify VP concepts and approaches using a typology of VP designs. METHODS: The authors developed a VP design typology based on the literature, a review of existing VP systems, and their personal experience with VPs. This draft framework was refined using a Delphi study involving experts in the field, and was then validated by applying it in the description of different VP designs. RESULTS: Nineteen factors were synthesized around four categories: general (title, description, language, identifier, provenance, and typical study time); educational (educational level, educational modes, coverage, and objectives); instructional design (path type, user modality, media use, narrative use, interactivity use, and feedback use); technical (originating system, format, integration, and dependence). CONCLUSION: This empirically derived VP design typology provides a common reference point for all those wishing to report on or study VPs.

Design principles for virtual patients: a focus group study among students.

OBJECTIVES: This study aimed to examine what students perceive as the ideal features of virtual patient (VP) design in order to foster learning with a special focus on clinical reasoning. METHODS: A total of 104 Year 5 medical students worked through at least eight VPs representing four different designs during their paediatric clerkship. The VPs were presented in two modes and differed in terms of the authenticity of the user interface (with or without graphics support), predominant question type (long- versus short-menu questions) and freedom of navigation (relatively free versus predetermined). Each mode was presented in a rich and a poor version with regard to the use of different media and questions and explanations explicitly directed at clinical reasoning. Five groups of between four and nine randomly selected students (n = 27) participated in focus group interviews facilitated by a moderator using a questioning route. The interviews were videotaped, transcribed and analysed. Summary reports were approved by the students. RESULTS: Ten principles of VP design emerged from the analysis. A VP should be relevant, of an appropriate level of difficulty, highly interactive, offer specific feedback, make optimal use of media, help students focus on relevant learning points, offer recapitulation of key learning points, provide an authentic web-based interface and student tasks, and contain questions and explanations tailored to the clinical reasoning process. CONCLUSIONS: Students perceived the design principles identified as being conducive to their learning. Many of these principles are supported by the results of other published studies. Future studies should address the effects of these principles using quantitative controlled designs.

The effect of face-to-face collaborative learning on the elaboration of computer-based simulated cases.

INTRODUCTION: This study investigates the effects of working face to face in small groups on the processes that occur when students elaborate on computer-based simulated cases. METHODS: We performed a randomized controlled experiment that was designed to measure the effect of "social context" (triads versus individuals) on students' perceptions of the elaboration process and on the time spent on the different parts of the computer case. We sought students' perceptions using a questionnaire that was administered to all participating students (N = 47) and we examined the actions of the students working in triads (N = 12) and individually (N = 11) by analyzing the log files of the computer case. RESULTS: The results demonstrated no significant effect of social context on the degree of elaboration of the computer case. CONCLUSIONS: Working with computer-based simulated cases in small groups as opposed to individually in itself is not enough to increase the scope and depth of the elaboration of computer cases.

Student perceptions of a virtual learning environment for a problem-based learning undergraduate medical curriculum.

OBJECTIVE: To investigate the effects of a virtual learning environment (VLE) on group interaction and consultation of information resources during the preliminary phase, self-study phase and reporting phase of the problem-based learning process in an undergraduate medical curriculum. METHODS: A questionnaire was administered to 355 medical students in Years 1 and 2 to ask them about the perceived usefulness of a virtual learning environment that was created with Blackboard for group interaction and the use of learning resources. RESULTS: The students indicated that the VLE supported face-to-face interaction in the preliminary discussion and in the reporting phase but did not stimulate computer-mediated distance interaction during the self-study phase. They perceived that the use of multimedia in case presentations led to a better quality of group discussion than if case presentations were exclusively text-based. They also indicated that the information resources that were hyperlinked in the VLE stimulated the consultation of these resources during self-study, but not during the reporting phase. CONCLUSIONS: Students indicated that the use of a VLE in the tutorial room and the inclusion of multimedia in case presentations supported processes of active learning in the tutorial groups. However, if we want to exploit the full potential of asynchronous computer-mediated communication to initiate in-depth discussion during the self-study phase, its application will have to be selective and deliberate. Students indicated that the links in the VLE to selected information in library repositories supported their learning.

Analysis of verbal interactions in tutorial groups: a process study.

INTRODUCTION: Collaborative learning, including problem-based learning (PBL), is a powerful learning method. Group interaction plays a crucial role in stimulating student learning. However, few studies on learning processes in medical education have examined group interactions. Most studies on collaboration within PBL used self-reported data rather than observational data. We investigated the following types of interactions in PBL tutorial groups: learning-oriented interactions (exploratory questioning, cumulative reasoning and handling conflicts about knowledge); procedural interactions, and irrelevant/off-task interactions. AIM: The central question concerned how much time is spent on the different types of interaction during group sessions and how the types of interaction are distributed over the meeting. METHOD: Four tutorial group sessions in Year 2 of the PBL undergraduate curriculum of Maastricht Medical School were videotaped and analysed. The sessions concerned the reporting phase of the PBL process. We analysed the interactions using a coding scheme distinguishing several verbal interaction types, such as questions, arguments and evaluations. RESULTS: Learning-orientated interactions accounted for 80% of the interactions, with cumulative reasoning, exploratory questioning and handling conflicts about knowledge accounting for about 63%, 10% and 7% of the interactions, respectively. Exploratory questioning often preceded cumulative reasoning. Both types occurred throughout the meeting. Handling conflicts mainly occurred after the first 20 minutes. CONCLUSIONS: Task involvement in the tutorial groups was high. All types of learning-orientated interactions were observed. Relatively little time was spent on exploratory questions and handling conflicts about knowledge. Problem-based learning training should pay special attention to stimulating discussion about contradictory information.