Inter-school variations in the standard of examiners' graduation-level OSCE judgements.

INTRODUCTION: Ensuring equivalence in high-stakes performance exams is important for patient safety and candidate fairness. We compared inter-school examiner differences within a shared OSCE and resulting impact on students' pass/fail categorisation. METHODS: The same 6 station formative OSCE ran asynchronously in 4 medical schools, with 2 parallel circuits/school. We compared examiners' judgements using Video-based Examiner Score Comparison and Adjustment (VESCA): examiners scored station-specific comparator videos in addition to 'live' student performances, enabling 1/controlled score comparisons by a/examiner-cohorts and b/schools and 2/data linkage to adjust for the influence of examiner-cohorts. We calculated score impact and change in pass/fail categorisation by school. RESULTS: On controlled video-based comparisons, inter-school variations in examiners' scoring (16.3%) were nearly double within-school variations (8.8%). Students' scores received a median adjustment of 5.26% (IQR 2.87-7.17%). The impact of adjusting for examiner differences on students' pass/fail categorisation varied by school, with adjustment reducing failure rate from 39.13% to 8.70% (school 2) whilst increasing failure from 0.00% to 21.74% (school 4). DISCUSSION: Whilst the formative context may partly account for differences, these findings query whether variations may exist between medical schools in examiners' judgements. This may benefit from systematic appraisal to safeguard equivalence. VESCA provided a viable method for comparisons.

A realist evaluation of how, why and when objective structured clinical exams (OSCEs) are experienced as an authentic assessment of clinical preparedness.

INTRODUCTION: Whilst rarely researched, the authenticity with which Objective Structured Clinical Exams (OSCEs) simulate practice is arguably critical to making valid judgements about candidates' preparedness to progress in their training. We studied how and why an OSCE gave rise to different experiences of authenticity for different participants under different circumstances. METHODS: We used Realist evaluation, collecting data through interviews/focus groups from participants across four UK medical schools who participated in an OSCE which aimed to enhance authenticity. RESULTS: Several features of OSCE stations (realistic, complex, complete cases, sufficient time, autonomy, props, guidelines, limited examiner interaction etc) combined to enable students to project into their future roles, judge and integrate information, consider their actions and act naturally. When this occurred, their performances felt like an authentic representation of their clinical practice. This didn't work all the time: focusing on unavoidable differences with practice, incongruous features, anxiety and preoccupation with examiners' expectations sometimes disrupted immersion, producing inauthenticity. CONCLUSIONS: The perception of authenticity in OSCEs appears to originate from an interaction of station design with individual preferences and contextual expectations. Whilst tentatively suggesting ways to promote authenticity, more understanding is needed of candidates' interaction with simulation and scenario immersion in summative assessment.

Training healthcare professionals to be ready for practice in an era of social distancing: a realist evaluation.

BACKGROUND: Programme changes due to the COVID-19 pandemic have impacted variably on preparation for practice of healthcare professional students. Explanations for such variability need exploration. The aim of our study was to understand what clinical learning, whilst under socially distanced restrictions, worked and why (or why not). METHODS: We conducted a realist evaluation of the undergraduate healthcare programmes at one UK university in 2020-21. Initial programme theories to be tested in this study were derived from discussions with programme leads about the changes they implemented due to the pandemic. Study participants were students and teaching faculty. Online interview transcripts were coded, identifying why interventions had worked or not. This resulted in a set of 'context-mechanism-outcome' (CMO) statements about each intervention. The initial programme theories were refined as a result. RESULTS AND DISCUSSION: 29 students and 22 faculty members participated. 18 CMO configurations were identified relating to clinical skills learning and 25 relating to clinical placements. Clinical skills learning was successful whether in person, remote or hybrid if it followed the steps of: demonstration-explanation-mental rehearsal-attempt with feedback. Where it didn't work there was usually a lack of observation and corrective feedback. Placements were generally highly valued despite some deficiencies in student experience. Being useful on placements was felt to be good preparation for practice. If student numbers are to expand, findings about what works in distance learning of clinical skills and the value of various modes of induction to clinical workplace activity may also be relevant post-pandemic.

Using video-based examiner score comparison and adjustment (VESCA) to compare the influence of examiners at different sites in a distributed objective structured clinical exam (OSCE).

PURPOSE: Ensuring equivalence of examiners' judgements within distributed objective structured clinical exams (OSCEs) is key to both fairness and validity but is hampered by lack of cross-over in the performances which different groups of examiners observe. This study develops a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA) using it to compare examiners scoring from different OSCE sites for the first time. MATERIALS/ METHODS: Within a summative 16 station OSCE, volunteer students were videoed on each station and all examiners invited to score station-specific comparator videos in addition to usual student scoring. Linkage provided through the video-scores enabled use of Many Facet Rasch Modelling (MFRM) to compare 1/ examiner-cohort and 2/ site effects on students' scores. RESULTS: Examiner-cohorts varied by 6.9% in the overall score allocated to students of the same ability. Whilst only a tiny difference was apparent between sites, examiner-cohort variability was greater in one site than the other. Adjusting student scores produced a median change in rank position of 6 places (0.48 deciles), however 26.9% of students changed their rank position by at least 1 decile. By contrast, only 1 student's pass/fail classification was altered by score adjustment. CONCLUSIONS: Whilst comparatively limited examiner participation rates may limit interpretation of score adjustment in this instance, this study demonstrates the feasibility of using VESCA for quality assurance purposes in large scale distributed OSCEs.

Determining the influence of different linking patterns on the stability of students' score adjustments produced using Video-based Examiner Score Comparison and Adjustment (VESCA).

BACKGROUND: Ensuring equivalence of examiners' judgements across different groups of examiners is a priority for large scale performance assessments in clinical education, both to enhance fairness and reassure the public. This study extends insight into an innovation called Video-based Examiner Score Comparison and Adjustment (VESCA) which uses video scoring to link otherwise unlinked groups of examiners. This linkage enables comparison of the influence of different examiner-groups within a common frame of reference and provision of adjusted "fair" scores to students. Whilst this innovation promises substantial benefit to quality assurance of distributed Objective Structured Clinical Exams (OSCEs), questions remain about how the resulting score adjustments might be influenced by the specific parameters used to operationalise VESCA. Research questions, How similar are estimates of students' score adjustments when the model is run with either: fewer comparison videos per participating examiner?; reduced numbers of participating examiners? METHODS: Using secondary analysis of recent research which used VESCA to compare scoring tendencies of different examiner groups, we made numerous copies of the original data then selectively deleted video scores to reduce the number of 1/ linking videos per examiner (4 versus several permutations of 3,2,or 1 videos) or 2/examiner participation rates (all participating examiners (76%) versus several permutations of 70%, 60% or 50% participation). After analysing all resulting datasets with Many Facet Rasch Modelling (MFRM) we calculated students' score adjustments for each dataset and compared these with score adjustments in the original data using Spearman's correlations. RESULTS: Students' score adjustments derived form 3 videos per examiner correlated highly with score adjustments derived from 4 linking videos (median Rho = 0.93,IQR0.90-0.95,p < 0.001), with 2 (median Rho 0.85,IQR0.81-0.87,p < 0.001) and 1 linking videos (median Rho = 0.52(IQR0.46-0.64,p < 0.001) producing progressively smaller correlations. Score adjustments were similar for 76% participating examiners and 70% (median Rho = 0.97,IQR0.95-0.98,p < 0.001), and 60% (median Rho = 0.95,IQR0.94-0.98,p < 0.001) participation, but were lower and more variable for 50% examiner participation (median Rho = 0.78,IQR0.65-0.83, some ns). CONCLUSIONS: Whilst VESCA showed some sensitivity to the examined parameters, modest reductions in examiner participation rates or video numbers produced highly similar results. Employing VESCA in distributed or national exams could enhance quality assurance or exam fairness.

Determining influence, interaction and causality of contrast and sequence effects in objective structured clinical exams.

INTRODUCTION: Differential rater function over time (DRIFT) and contrast effects (examiners' scores biased away from the standard of preceding performances) both challenge the fairness of scoring in objective structured clinical exams (OSCEs). This is important as, under some circumstances, these effects could alter whether some candidates pass or fail assessments. Benefitting from experimental control, this study investigated the causality, operation and interaction of both effects simultaneously for the first time in an OSCE setting. METHODS: We used secondary analysis of data from an OSCE in which examiners scored embedded videos of student performances interspersed between live students. Embedded video position varied between examiners (early vs. late) whilst the standard of preceding performances naturally varied (previous high or low). We examined linear relationships suggestive of DRIFT and contrast effects in all within-OSCE data before comparing the influence and interaction of 'early' versus 'late' and 'previous high' versus 'previous low' conditions on embedded video scores. RESULTS: Linear relationships data did not support the presence of DRIFT or contrast effects. Embedded videos were scored higher early (19.9 [19.4-20.5]) versus late (18.6 [18.1-19.1], p < 0.001), but scores did not differ between previous high and previous low conditions. The interaction term was non-significant. CONCLUSIONS: In this instance, the small DRIFT effect we observed on embedded videos can be causally attributed to examiner behaviour. Contrast effects appear less ubiquitous than some prior research suggests. Possible mediators of these finding include the following: OSCE context, detail of task specification, examiners' cognitive load and the distribution of learners' ability. As the operation of these effects appears to vary across contexts, further research is needed to determine the prevalence and mechanisms of contrast and DRIFT effects, so that assessments may be designed in ways that are likely to avoid their occurrence. Quality assurance should monitor for these contextually variable effects in order to ensure OSCE equivalence.

Enhancing authenticity, diagnosticity and equivalence (AD-Equiv) in multicentre OSCE exams in health professionals education: protocol for a complex intervention study.

INTRODUCTION: Objective structured clinical exams (OSCEs) are a cornerstone of assessing the competence of trainee healthcare professionals, but have been criticised for (1) lacking authenticity, (2) variability in examiners' judgements which can challenge assessment equivalence and (3) for limited diagnosticity of trainees' focal strengths and weaknesses. In response, this study aims to investigate whether (1) sharing integrated-task OSCE stations across institutions can increase perceived authenticity, while (2) enhancing assessment equivalence by enabling comparison of the standard of examiners' judgements between institutions using a novel methodology (video-based score comparison and adjustment (VESCA)) and (3) exploring the potential to develop more diagnostic signals from data on students' performances. METHODS AND ANALYSIS: The study will use a complex intervention design, developing, implementing and sharing an integrated-task (research) OSCE across four UK medical schools. It will use VESCA to compare examiner scoring differences between groups of examiners and different sites, while studying how, why and for whom the shared OSCE and VESCA operate across participating schools. Quantitative analysis will use Many Facet Rasch Modelling to compare the influence of different examiners groups and sites on students' scores, while the operation of the two interventions (shared integrated task OSCEs; VESCA) will be studied through the theory-driven method of Realist evaluation. Further exploratory analyses will examine diagnostic performance signals within data. ETHICS AND DISSEMINATION: The study will be extra to usual course requirements and all participation will be voluntary. We will uphold principles of informed consent, the right to withdraw, confidentiality with pseudonymity and strict data security. The study has received ethical approval from Keele University Research Ethics Committee. Findings will be academically published and will contribute to good practice guidance on (1) the use of VESCA and (2) sharing and use of integrated-task OSCE stations.

Measuring the Effect of Examiner Variability in a Multiple-Circuit Objective Structured Clinical Examination (OSCE).

PURPOSE: Ensuring that examiners in different parallel circuits of objective structured clinical examinations (OSCEs) judge to the same standard is critical to the chain of validity. Recent work suggests examiner-cohort (i.e., the particular group of examiners) could significantly alter outcomes for some candidates. Despite this, examiner-cohort effects are rarely examined since fully nested data (i.e., no crossover between the students judged by different examiner groups) limit comparisons. In this study, the authors aim to replicate and further develop a novel method called Video-based Examiner Score Comparison and Adjustment (VESCA), so it can be used to enhance quality assurance of distributed or national OSCEs. METHOD: In 2019, 6 volunteer students were filmed on 12 stations in a summative OSCE. In addition to examining live student performances, examiners from 8 separate examiner-cohorts scored the pool of video performances. Examiners scored videos specific to their station. Video scores linked otherwise fully nested data, enabling comparisons by Many Facet Rasch Modeling. Authors compared and adjusted for examiner-cohort effects. They also compared examiners' scores when videos were embedded (interspersed between live students during the OSCE) or judged later via the Internet. RESULTS: Having accounted for differences in students' ability, different examiner-cohort scores for the same ability of student ranged from 18.57 of 27 (68.8%) to 20.49 (75.9%), Cohen's d = 1.3. Score adjustment changed the pass/fail classification for up to 16% of students depending on the modeled cut score. Internet and embedded video scoring showed no difference in mean scores or variability. Examiners' accuracy did not deteriorate over the 3-week Internet scoring period. CONCLUSIONS: Examiner-cohorts produced a replicable, significant influence on OSCE scores that was unaccounted for by typical assessment psychometrics. VESCA offers a promising means to enhance validity and fairness in distributed OSCEs or national exams. Internet-based scoring may enhance VESCA's feasibility.

Evaluation of stress-control layout at the Subtropolis Mine, Petersburg, Ohio.

The Subtropolis room-and-pillar mine extracts the Vanport Limestone (Allegheny Formation, Pennsylvanian System) near Petersburg, Ohio. Strata instability problems associated with excessive concentrations of lateral stress caused the mine operator to implement a change in layout design. This mining method has been identified as a stress control layout and has been used by other underground stone mines in the past with varying degrees of success. Practical experience has shown that entry headings advance in the direction of the principal lateral stress, producing lower stress concentrations with better mining conditions. It is important to minimize stress concentrations along the mining front, so an arrow-shaped advance is recommended. This technique advances more developments (headings) in a "good" direction and reduces developments (crosscuts) in the "bad direction." As is expected, the stress control layout enhances the potential for shear failures in crosscuts. It is, therefore, important to focus crosscut engineering interventions that either: (a) lower stress concentrations (for example, an arched roof) or (b) enhance strength of the strata containing the shears (for example, rock reinforcement). This study focuses on observing strata conditions on a regular basis and monitoring the response of these strata to changing geologic and mining conditions through 3D Dynamic LiDAR scans.

Exploring differences in individual and group judgements in standard setting.

CONTEXT: Standard setting is critically important to assessment decisions in medical education. Recent research has demonstrated variations between medical schools in the standards set for shared items. Despite the centrality of judgement to criterion-referenced standard setting methods, little is known about the individual or group processes that underpin them. This study aimed to explore the operation and interaction of these processes in order to illuminate potential sources of variability. METHODS: Using qualitative research, we purposively sampled across UK medical schools that set a low, medium or high standard on nationally shared items, collecting data by observation of graduation-level standard-setting meetings and semi-structured interviews with standard-setting judges. Data were analysed using thematic analysis based on the principles of grounded theory. RESULTS: Standard setting occurred through the complex interaction of institutional context, judges' individual perspectives and group interactions. Schools' procedures, panel members and atmosphere produced unique contexts. Individual judges formed varied understandings of the clinical and technical features of each question, relating these to their differing (sometimes contradictory) conceptions of minimally competent students, by balancing information and making suppositions. Conceptions of minimal competence variously comprised: limited attendance; limited knowledge; poor knowledge application; emotional responses to questions; 'test-savviness', or a strategic focus on safety. Judges experienced tensions trying to situate these abstract conceptions in reality, revealing uncertainty. Groups constructively revised scores through debate, sharing information and often constructing detailed clinical representations of cases. Groups frequently displayed conformity, illustrating a belief that outlying judges were likely to be incorrect. Less frequently, judges resisted change, using emphatic language, bargaining or, rarely, 'polarisation' to influence colleagues. CONCLUSIONS: Despite careful conduct through well-established procedures, standard setting is judgementally complex and involves uncertainty. Understanding whether or how these varied processes produce the previously observed variations in outcomes may offer routes to enhance equivalence of criterion-referenced standards.

Developing a video-based method to compare and adjust examiner effects in fully nested OSCEs.

BACKGROUND: Although averaging across multiple examiners' judgements reduces unwanted overall score variability in objective structured clinical examinations (OSCE), designs involving several parallel circuits of the OSCE require that different examiner cohorts collectively judge performances to the same standard in order to avoid bias. Prior research suggests the potential for important examiner-cohort effects in distributed or national examinations that could compromise fairness or patient safety, but despite their importance, these effects are rarely investigated because fully nested assessment designs make them very difficult to study. We describe initial use of a new method to measure and adjust for examiner-cohort effects on students' scores. METHODS: We developed video-based examiner score comparison and adjustment (VESCA): volunteer students were filmed 'live' on 10 out of 12 OSCE stations. Following the examination, examiners additionally scored station-specific common-comparator videos, producing partial crossing between examiner cohorts. Many-facet Rasch modelling and linear mixed modelling were used to estimate and adjust for examiner-cohort effects on students' scores. RESULTS: After accounting for students' ability, examiner cohorts differed substantially in their stringency or leniency (maximal global score difference of 0.47 out of 7.0 [Cohen's d = 0.96]; maximal total percentage score difference of 5.7% [Cohen's d = 1.06] for the same student ability by different examiner cohorts). Corresponding adjustment of students' global and total percentage scores altered the theoretical classification of 6.0% of students for both measures (either pass to fail or fail to pass), whereas 8.6-9.5% students' scores were altered by at least 0.5 standard deviations of student ability. CONCLUSIONS: Despite typical reliability, the examiner cohort that students encountered had a potentially important influence on their score, emphasising the need for adequate sampling and examiner training. Development and validation of VESCA may offer a means to measure and adjust for potential systematic differences in scoring patterns that could exist between locations in distributed or national OSCE examinations, thereby ensuring equivalence and fairness.

Variants in the DYX2 locus are associated with altered brain activation in reading-related brain regions in subjects with reading disability.

Reading disability (RD) is a complex genetic disorder with unknown etiology. Genes on chromosome 6p22, including DCDC2, KIAA0319, and TTRAP, have been identified as RD associated genes. Imaging studies have shown both functional and structural differences between brains of individuals with and without RD. There are limited association studies performed between RD genes, specifically genes on 6p22, and regional brain activation during reading tasks. Using fourteen variants in DCDC2, KIAA0319, and TTRAP and exhaustive reading measures, we first tested for association with reading performance in 82 parent-offspring families (326 individuals). Next, we determined the association of these variants with activation of sixteen brain regions of interest during four functional magnetic resonance imaging-reading tasks. We nominally replicated associations between reading performance and variants of DCDC2 and KIAA0319. Furthermore, we observed a number of associations with brain activation patterns during imaging-reading tasks with all three genes. The strongest association occurred between activation of the left anterior inferior parietal lobe and complex tandem repeat BV677278 in DCDC2 (uncorrected p=0.00003, q=0.0442). Our results show that activation patterns across regions of interest in the brain are influenced by variants in the DYX2 locus. The combination of genetic and functional imaging data show a link between genes and brain functioning during reading tasks in subjects with RD. This study highlights the many advantages of imaging data as an endophenotype for discerning genetic risk factors for RD and other communication disorders and underscores the importance of integrating neurocognitive, imaging, and genetic data in future investigations.

DCDC2 genetic variants and susceptibility to developmental dyslexia.

OBJECTIVE(S): Developmental dyslexia is a heritable condition, with genetic factors accounting for 44-75% of the variance in performance tests of reading component subphenotypes. Compelling genetic linkage and association evidence supports a quantitative trait locus in the 6p21.3 region that encodes a gene called DCDC2. In this study, we explored the contribution of two DCDC2 markers to dyslexia, related reading and memory phenotypes in nuclear families of Italian origin. METHODS: The 303 nuclear families recruited on the basis of having a proband with developmental dyslexia have been studied with 6p21.3 markers, BV677278 and rs793862. Marker-trait association was investigated by the quantitative transmission disequilibrium test (version 2.5.1) that allows for the analyses of quantitative traits. Seven phenotypes were used in association analyses, that is, word and nonword reading, word and nonword spelling, orthographic choice, memory, and the affected status based on inclusion criteria. RESULTS: Quantitative transmission disequilibrium test analyses yielded evidence for association between reading skills and the BV677278 deletion (empirical P-values=0.025-0.029) and between memory and BV677278 allele 10 (empirical P-value=0.0001). CONCLUSION: Our result adds further evidence in support of DCDC2 contributing to the deficits in developmental dyslexia. More specifically, our data support the view that DCDC2 influences both reading and memory impairments thus shedding further light into the etiologic basis and the phenotypic complexity of developmental dyslexia.

A dyslexia-associated variant in DCDC2 changes gene expression.

Reading disability (RD) or dyslexia is a common neurogenetic disorder. Two genes, KIAA0319 and DCDC2, have been identified by association studies of the DYX2 locus on 6p21.3. We previously identified a 2445 bp deletion, and a compound STR within the deleted region (BV677278), in intron 2 of DCDC2. The deletion and several alleles of the STR are strongly associated with RD (P = 0.00002). In this study we investigated whether BV677278 is a regulatory region for DCDC2 by electrophoretic mobility shift and luciferase reporter assays. We show that oligonucleotide probes from the STR bind nuclear protein from human brain, and that alleles of the STR have a range of DCDC2-specific enhancer activities. Five alleles displayed strong enhancer activity and increased gene expression, while allele 1 showed no enhancer activity. These studies suggest that the association of BV677278 with RD reflects a role as a modifier of DCDC2 expression.

Polymorphism of DCDC2 Reveals Differences in Cortical Morphology of Healthy Individuals-A Preliminary Voxel Based Morphometry Study.

OBJECTIVE: The purpose of this investigation was to determine whether there is an association between the putative reading disability (RD) susceptibility gene Doublecortin Domain Containing 2 (DCDC2), and gray matter (GM) distribution in the brain, in a sample of healthy control individuals. METHOD: Fifty-six control subjects were genotyped for an RD-associated deletion in intron 2 of DCDC2. Voxel based morphometry (VBM) was used to examine structural magnetic resonance imaging (MRI) scans to assess GM differences between the two groups. RESULTS: Individuals heterozygous for the deletion exhibited significantly higher GM volumes in reading/language and symbol-decoding related brain regions including superior, medial and inferior temporal, fusiform, hippocampal/para-hippocampal, inferior occipito-parietal, inferior and middle frontal gyri, especially in the left hemisphere. GM values correlated with published data on regional DCDC2 expression in a lateralized manner. CONCLUSIONS: These data suggest a role for DCDC2 in GM distribution in language-related brain regions in healthy individuals.

Strong evidence that KIAA0319 on chromosome 6p is a susceptibility gene for developmental dyslexia.

Linkage between developmental dyslexia (DD) and chromosome 6p has been replicated in a number of independent samples. Recent attempts to identify the gene responsible for the linkage have produced inconsistent evidence for association of DD with a number of genes in a 575-kb region of chromosome 6p22.2, including VMP, DCDC2, KIAA0319, TTRAP, and THEM2. We aimed to identify the specific gene or genes involved by performing a systematic, high-density (approximately 2-3-kb intervals) linkage disequilibrium screen of these genes in an independent sample, incorporating family-based and case-control designs in which dyslexia was defined as an extreme representation of reading disability. Using DNA pooling, we first observed evidence for association with 17 single-nucleotide polymorphisms (SNPs), 13 of which were located in the KIAA0319 gene (P<.01-.003). After redundant SNPs were excluded, 10 SNPs were individually genotyped in 223 subjects with DD and 273 controls. Those SNPs that were significant at P