Automated Item Generation: impact of item variants on performance and standard setting.

BACKGROUND: Automated Item Generation (AIG) uses computer software to create multiple items from a single question model. There is currently a lack of data looking at whether item variants to a single question result in differences in student performance or human-derived standard setting. The purpose of this study was to use 50 Multiple Choice Questions (MCQs) as models to create four distinct tests which would be standard set and given to final year UK medical students, and then to compare the performance and standard setting data for each. METHODS: Pre-existing questions from the UK Medical Schools Council (MSC) Assessment Alliance item bank, created using traditional item writing techniques, were used to generate four 'isomorphic' 50-item MCQ tests using AIG software. Isomorphic questions use the same question template with minor alterations to test the same learning outcome. All UK medical schools were invited to deliver one of the four papers as an online formative assessment for their final year students. Each test was standard set using a modified Angoff method. Thematic analysis was conducted for item variants with high and low levels of variance in facility (for student performance) and average scores (for standard setting). RESULTS: Two thousand two hundred eighteen students from 12 UK medical schools participated, with each school using one of the four papers. The average facility of the four papers ranged from 0.55-0.61, and the cut score ranged from 0.58-0.61. Twenty item models had a facility difference > 0.15 and 10 item models had a difference in standard setting of > 0.1. Variation in parameters that could alter clinical reasoning strategies had the greatest impact on item facility. CONCLUSIONS: Item facility varied to a greater extent than the standard set. This difference may relate to variants causing greater disruption of clinical reasoning strategies in novice learners compared to experts, but is confounded by the possibility that the performance differences may be explained at school level and therefore warrants further study.

The influence of candidates' physical attributes on patient ratings in simulated assessments of clinical practice.

BACKGROUND: We have previously shown that clinical examiners' scoring is not negatively impacted when a candidate has a tattoo, unnatural hair colour, or a regional accent. We investigated whether these physical attributes in exam candidates impact patient scoring. METHODS: Simulated/real patients were randomly assigned to watch five videos of simulated candidate performances of a cranial nerve examination: clear fail, borderline, good, 'clear pass' without an attribute, and 'clear pass' with one of the attributes (tattoo, purple hair, accent). Participants scored domains of communication and professionalism. We compared scores for the clear pass candidates with and without attributes. RESULTS: One hundred and eighty three patients participated. The total scores for the candidates with tattoos and purple hair were higher than the candidate with no physical attribute (p < 0.001). For the candidate with a Liverpool English accent no difference was identified (p = 0.120). CONCLUSIONS: The presence of certain physical attributes (tattoos or purple hair) was associated with higher scores given by patients to candidates in a simulated physical examination station.

Assessment of factual recall and higher-order cognitive domains in an open-book medical school examination.

Open-book examinations (OBEs) will likely become increasingly important assessment tools. We investigated how access to open-book resources affected questions testing factual recall, which might be easy to look-up, versus questions testing higher-order cognitive domains. Few studies have investigated OBEs using modern Internet resources or as summative assessments. We compared performance on an examination conducted as a traditional closed-book exam (CBE) in 2019 (N = 320) and a remote OBE with free access to Internet resources in 2020 (N = 337) due to COVID-19. This summative, end-of-year assessment focused on basic science for second-year medical students. We categorized questions by Bloom's taxonomy ('Remember', versus 'Understand/Apply'). We predicted higher performance on the OBE, driven by higher performance on 'Remember' questions. We used an item-centric analysis by using performance per item over all examinees as the outcome variable in logistic regression, with terms 'Open-Book, 'Bloom Category' and their interaction. Performance was higher on OBE questions than CBE questions (OR 2.2, 95% CI: 2.14-2.39), and higher on 'Remember' than 'Understand/Apply' questions (OR 1.13, 95% CI: 1.09-1.19). The difference in performance between 'Remember' and 'Understand/Apply' questions was greater in the OBE than the CBE ('Open-Book' * 'Bloom Category' interaction: OR 1.2, 95% CI: 1.19-1.37). Access to open-book resources had a greater effect on performance on factual recall questions than higher-order questions, though performance was higher in the OBE overall. OBE design must consider how searching for information affects performance, particularly on questions measuring different domains of knowledge.

The influence of candidates' physical attributes on assessors' ratings in clinical practice.

BACKGROUND: Assessments of physician competence in the work-place are common and often contribute to high-stakes assessments. Previous research suggests that assessors' judgements can be influenced by candidates' physical attributes. We investigated whether simulated candidates' scores were influenced by assessor bias based on tattoos, hair colour, and a regional accent. METHODS: We used an experimental, video-based, single-blinded, randomised, internet-based design. We created videos of simulated medical intern performances of a clinical examination at four different standards of competence. Four videos were also created of simulated candidates performing at a 'clear pass' standard, with either no stereotypical attribute (CPX), purple hair (CPH), tattoos (CPT) or a Liverpool English accent (CPA). Assessors were randomly assigned to watch five videos including the "clear pass" candidate without an attribute and one of the "clear pass" candidates with an attribute and asked to give an overall global grade for each candidate. We compared the global grades for the clear pass candidates with and without attributes. RESULTS: Ninety-eight assessors were included in the analysis. The total scores for the candidates with stereotyped attributes were not significantly lower than the candidate with no attribute. Assessors showed moderate levels of agreement between the global grades awarded for all the candidates. The global grades awarded to candidate with a stereotypical attribute were not significantly lower than for those without. CONCLUSIONS: The presence of tattoos, purple hair, or a regional accent did not systematically negatively influence the grade or score awarded by assessors to candidates in observed clinical examination scenarios.

Which patients with gynaecomastia require more detailed investigation?

Gynaecomastia may be due to medication, chronic liver or kidney disease, hypogonadism (primary or secondary to pituitary disease) or hyperthyroidism. Having excluded these aetiologies, it is imperative to be vigilant for underlying malignancy causing gynaecomastia. These include human chorionic gonadotrophin-secreting testicular and extratesticular tumours and oestrogen-secreting testicular tumours and feminising adrenal tumours.

Relative Importance of the Arcuate and Anteroventral Periventricular Kisspeptin Neurons in Control of Puberty and Reproductive Function in Female Rats.

Kisspeptin plays a critical role in pubertal timing and reproductive function. In rodents, kisspeptin perikarya within the hypothalamic arcuate (ARC) and anteroventral periventricular (AVPV) nuclei are thought to be involved in LH pulse and surge generation, respectively. Using bilateral microinjections of recombinant adeno-associated virus encoding kisspeptin antisense into the ARC or AVPV of female rats at postnatal day 10, we investigated the relative importance of these two kisspeptin populations in the control of pubertal timing, estrous cyclicity, and LH surge and pulse generation. A 37% knockdown of kisspeptin in the AVPV resulted in a significant delay in vaginal opening and first vaginal estrous, abnormal estrous cyclicity, and reduction in the occurrence of spontaneous LH surges, although these retained normal amplitude. This AVPV knockdown had no effect on LH pulse frequency, measured after ovariectomy. A 32% reduction of kisspeptin in the ARC had no effect on the onset of puberty but resulted in abnormal estrous cyclicity and decreased LH pulse frequency. Additionally, the knockdown of kisspeptin in the ARC decreased the amplitude but not the incidence of LH surges. These results might suggest that the role of AVPV kisspeptin in the control of pubertal timing is particularly sensitive to perturbation. In accordance with our previous studies, ARC kisspeptin signaling was critical for normal pulsatile LH secretion in female rats. Despite the widely reported role of AVPV kisspeptin neurons in LH surge generation, this study suggests that both AVPV and ARC populations are essential for normal LH surges and estrous cyclicity.

The physiological role of arcuate kisspeptin neurons in the control of reproductive function in female rats.

Kisspeptin plays a pivotal role in pubertal onset and reproductive function. In rodents, kisspeptin perikarya are located in 2 major populations: the anteroventral periventricular nucleus and the hypothalamic arcuate nucleus (ARC). These nuclei are believed to play functionally distinct roles in the control of reproduction. The anteroventral periventricular nucleus population is thought to be critical in the generation of the LH surge. However, the physiological role played by the ARC kisspeptin neurons remains to be fully elucidated. We used bilateral stereotactic injection of recombinant adeno-associated virus encoding kisspeptin antisense into the ARC of adult female rats to investigate the physiological role of kisspeptin neurons in this nucleus. Female rats with kisspeptin knockdown in the ARC displayed a significantly reduced number of both regular and complete oestrous cycles and significantly longer cycles over the 100-day period of the study. Further, kisspeptin knockdown in the ARC resulted in a decrease in LH pulse frequency. These data suggest that maintenance of ARC-kisspeptin levels is essential for normal pulsatile LH release and oestrous cyclicity.

Hepcidin levels in diabetes mellitus and polycystic ovary syndrome.

AIM: Increased body iron is associated with insulin resistance. Hepcidin is the key hormone that negatively regulates iron homeostasis. We hypothesized that individuals with insulin resistance have inadequate hepcidin levels for their iron load. METHODS: Serum concentrations of the active form of hepcidin (hepcidin-25) and hepcidin:ferritin ratio were evaluated in participants with Type 2 diabetes (n = 33, control subjects matched for age, gender and BMI, n = 33) and participants with polycystic ovary syndrome (n = 27, control subjects matched for age and BMI, n = 16). To investigate whether any changes observed were associated with insulin resistance rather than insulin deficiency or hyperglycaemia per se, the same measurements were made in participants with Type 1 diabetes (n = 28, control subjects matched for age, gender and BMI, n = 30). Finally, the relationship between homeostasis model assessment of insulin resistance and serum hepcidin:ferritin ratio was explored in overweight or obese participants without diabetes (n = 16). RESULTS: Participants with Type 2 diabetes had significantly lower hepcidin and hepcidin:ferritin ratio than control subjects (P < 0.05 and P < 0.01, respectively). Participants with polycystic ovary syndrome had a significantly lower hepcidin:ferritin ratio than control subjects (P < 0.05). There was no significant difference in hepcidin or hepcidin:ferritin ratio between participants with Type 1 diabetes and control subjects (P = 0.88 and P = 0.94). Serum hepcidin:ferritin ratio inversely correlated with homeostasis model assessment of insulin resistance (r = -0.59, P < 0.05). CONCLUSION: Insulin resistance, but not insulin deficiency or hyperglycaemia per se, is associated with inadequate hepcidin levels. Reduced hepcidin concentrations may cause increased body iron stores in insulin-resistant states.