What Stops Fairness from Emerging in Assessment? The Forces on a Complex Adaptive System.

Introduction: Workplace-based assessment occurs in authentic, dynamic clinical environments where reproducible, measurement-based assessments can often not be implemented. In these environments, research approaches that respect these multiple dynamic interactions, such as complexity perspectives, are encouraged. Previous research has shown that fairness in assessment is a nonlinear phenomenon that emerges from interactions between its components and behaves like a complex adaptative system. The aim of this study was to understand the external forces on the complex adaptive system which may disrupt fairness from emerging. Methods: We conducted online focus groups with a purposeful sample of nineteen academic leaders in the Netherlands. We used an iterative approach to collection, analysis and coding of the data and interpreted the results using a lens of complexity, focusing on how individual elements of fairness work in concert to create systems with complex behaviour. Results: We identified three themes of forces which can disrupt fairness: forces impairing interactivity, forces impairing adaption and forces impairing embeddedness. Within each of these themes, we identified subthemes: assessor and student forces, tool forces and system forces. Discussion: Consistent with complexity theory, this study suggests there are multiple forces which can hamper the emergence of fairness. Whilst complexity thinking does not reduce the scale of the challenge, viewing forces through this lens provides insight into why and how these forces are disrupting fairness. This allows for more purposeful, meaningful changes to support the use of fair judgement in assessment in dynamic authentic clinical workplaces.

Fairness in Assessment: Identifying a Complex Adaptive System.

Introduction: Assessment design in health professions education is continuously evolving. There is an increasing desire to better embrace human judgement in assessment. Thus, it is essential to understand what makes this judgement fair. This study builds upon existing literature by studying how assessment leaders conceptualise the characteristics of fair judgement. Methods: Sixteen assessment leaders from 15 medical schools in Australia and New Zealand participated in online focus groups. Data collection and analysis occurred concurrently and iteratively. We used the constant comparison method to identify themes and build on an existing conceptual model of fair judgement in assessment. Results: Fairness is a multi-dimensional construct with components at environment, system and individual levels. Components influencing fairness include articulated and agreed learning outcomes relating to the needs of society, a culture which allows for learner support, stakeholder agency and learning (environmental level), collection, interpretation and combination of evidence, procedural strategies (system level) and appropriate individual assessments and assessor expertise and agility (individual level). Discussion: We observed that within the data at fractal, that is an infinite pattern repeating at different scales, could be seen suggesting fair judgement should be considered a complex adaptive system. Within complex adaptive systems, it is primarily the interaction between the entities which influences the outcome it produces, not simply the components themselves. Viewing fairness in assessment through a lens of complexity rather than as a linear, causal model has significant implications for how we design assessment programs and seek to utilise human judgement in assessment.

Clinical phenotype and complications of large vessel giant cell arteritis: A systematic review and meta-analysis.

BACKGROUND: Giant Cell Arteritis (GCA) is a heterogenous systemic granulomatous vasculitis involving the aorta and any of its major tributaries. Despite increased awareness of large vessel (LV) involvement, studies reporting incidence, clinical characteristics and complications of large-vessel GCA (LV-GCA) show conflicting results due to inconsistent disease definitions, differences in study methodologies and the broad spectrum of clinical presentations. The aim of this systematic literature review was to better define LV-GCA based on the available literature and identify distinguishing characteristics that may differentiate LV-GCA patients from those with limited cranial disease. METHODS: Published studies indexed in MEDLINE and EMBASE were searched from database inception to 7th May 2021. Studies were included if they presented cohort or cross-sectional data on a minimum of 25 patients with LV-GCA. Control groups were included if data was available on patients with limited cranial GCA (C-GCA). Data was quantitatively synthesised with application of a random effects meta-regression model, using Stata. RESULTS: The search yielded 3488 studies, of which 46 were included. Diagnostic criteria for LV-GCA differed between papers, but was typically dependent on imaging or histopathology. Patients with LV-GCA were generally younger at diagnosis compared to C-GCA patients (mean age difference -4.53 years), had longer delay to diagnosis (mean difference 3.03 months) and lower rates of positive temporal artery biopsy (OR: 0.52 [95% CI: 0.3, 0.91]). Fewer LV-GCA patients presented with cranial manifestations and only 53% met the 1990 ACR Classification Criteria for GCA. Vasculitis was detected most commonly in the thoracic aorta, followed by the subclavian, brachiocephalic trunk and axillary arteries. The mean cumulative prednisolone dose at 12-months was 6056.5mg for LV-GCA patients, relapse rates were similar between LV- and C-GCA patients, and 12% of deaths in LV-GCA patients could be directly attributed to an LV complication. CONCLUSION: Patients with LV-GCA have distinct disease features when compared to C-GCA, and this has implications on diagnosis, treatment strategies and surveillance of long-term sequalae.

Giant cell arteritis: A population-based retrospective cohort study exploring incidence and clinical presentation in Canterbury, Aotearoa New Zealand.

Background/aim: To determine the epidemiology and clinical features of giant cell arteritis (GCA) in Canterbury, Aotearoa New Zealand, with a particular focus on extra-cranial large vessel disease. Methods: Patients with GCA were identified from radiology and pathology reports, outpatient letters and inpatient hospital admissions in the Canterbury New Zealand from 1 June 2011 to 31 May 2016. Data was collected retrospectively based on review of electronic medical records. Results: There were 142 cases of GCA identified. 65.5% of cases were female with a mean age of 74.2 years. The estimated population incidence for biopsy-proven GCA was 10.5 per 100,000 people over the age of 50 and incidence peaked between 80 and 84 years of age. 10/142 (7%) people were diagnosed with large vessel GCA, often presenting with non-specific symptoms and evidence of vascular insufficiency including limb claudication, vascular bruits, blood pressure and pulse discrepancy, or cerebrovascular accident. Those with limited cranial GCA were more likely to present with the cardinal clinical features of headache and jaw claudication. Patients across the two groups were treated similarly, but those with large vessel disease had greater long-term steroid burden. Rates of aortic complication were low across both groups, although available follow-up data was limited. Conclusion: This study is the first of its kind to describe the clinical characteristics of large vessel GCA in a New Zealand cohort. Despite small case numbers, two distinct subsets of disease were recognized, differentiating patients with cranial and large vessel disease. Our results suggest that utilization of an alternative diagnostic and therapeutic approach may be needed to manage patients with large vessel disease.

Living with a frozen shoulder - a phenomenological inquiry.

BACKGROUND: Frozen shoulder (adhesive capsulitis) is an inflammatory condition affecting the capsule of the glenohumeral joint. It is characterised by a painful restricted range of passive and active movement in all planes of motion. The impact of frozen shoulder on affected individuals remains poorly characterised. In this study we sought to better understand the lived experience of people suffering from frozen shoulder to characterise the physical, psychological and socioeconomic impact of the condition. METHODS: A qualitative study using a phenomenological approach was undertaken. Purposeful sampling was used to identify individuals for interview. Semi-structured interviews were performed and continued until saturation was achieved. A biopsychosocial framework was used during the analysis in order to generate themes which best described the phenomenon and reflected the lived experience of individuals' suffering from this condition. RESULTS: Ten interviews were conducted, and five main themes emerged including; the severity of the pain experience, a loss of independence, an altered sense of self, the significant psychological impact, and the variable experience with healthcare providers. CONCLUSIONS: These findings offer an insight into the lived experience of individuals with frozen shoulder, both on a personal and sociocultural level. The pain endured has profound impacts on physical and mental health, with loss of function resulting in a narrative reconstruction and altered sense of self. Our findings illustrate that frozen shoulder is much more than a benign self-limiting musculoskeletal condition and should be managed accordingly. TRIAL REGISTRATION: ANZCTR 12620000677909 Registered 28/04/2020 https://www.anzctr.org.au/Trial/Registration/TrialReview.aspx?id=379719&isReview=true.

The pursuit of fairness in assessment: Looking beyond the objective.

Health professions education has undergone significant changes over the last few decades, including the rise of competency-based medical education, a shift to authentic workplace-based assessments, and increased emphasis on programmes of assessment. Despite these changes, there is still a commonly held assumption that objectivity always leads to and is the only way to achieve fairness in assessment. However, there are well-documented limitations to using objectivity as the 'gold standard' to which assessments are judged. Fairness, on the other hand, is a fundamental quality of assessment and a principle that almost no one contests. Taking a step back and changing perspectives to focus on fairness in assessment may help re-set a traditional objective approach and identify an equal role for subjective human judgement in assessment alongside objective methods. This paper explores fairness as a fundamental quality of assessments. This approach legitimises human judgement and shared subjectivity in assessment decisions alongside objective methods. Widening the answer to the question: 'What is fair assessment' to include not only objectivity but also expert human judgement and shared subjectivity can add significant value in ensuring learners are better equipped to be the health professionals required of the 21st century.

Making it fair: Learners' and assessors' perspectives of the attributes of fair judgement.

INTRODUCTION: Optimising the use of subjective human judgement in assessment requires understanding what makes judgement fair. Whilst fairness cannot be simplistically defined, the underpinnings of fair judgement within the literature have been previously combined to create a theoretically-constructed conceptual model. However understanding assessors' and learners' perceptions of what is fair human judgement is also necessary. The aim of this study is to explore assessors' and learners' perceptions of fair human judgement, and to compare these to the conceptual model. METHODS: A thematic analysis approach was used. A purposive sample of twelve assessors and eight post-graduate trainees undertook semi-structured interviews using vignettes. Themes were identified using the process of constant comparison. Collection, analysis and coding of the data occurred simultaneously in an iterative manner until saturation was reached. RESULTS: This study supported the literature-derived conceptual model suggesting fairness is a multi-dimensional construct with components at individual, system and environmental levels. At an individual level, contextual, longitudinally-collected evidence, which is supported by narrative, and falls within ill-defined boundaries is essential for fair judgement. Assessor agility and expertise are needed to interpret and interrogate evidence, identify boundaries and provide narrative feedback to allow for improvement. At a system level, factors such as multiple opportunities to demonstrate competence and improvement, multiple assessors to allow for different perspectives to be triangulated, and documentation are needed for fair judgement. These system features can be optimized through procedural fairness. Finally, appropriate learning and working environments which considers patient needs and learners personal circumstances are needed for fair judgments. DISCUSSION: This study builds on the theory-derived conceptual model demonstrating the components of fair judgement can be explicitly articulated whilst embracing the complexity and contextual nature of health-professions assessment. Thus it provides a narrative to support dialogue between learner, assessor and institutions about ensuring fair judgements in assessment.

Fairness in human judgement in assessment: a hermeneutic literature review and conceptual framework.

Human judgement is widely used in workplace-based assessment despite criticism that it does not meet standards of objectivity. There is an ongoing push within the literature to better embrace subjective human judgement in assessment not as a 'problem' to be corrected psychometrically but as legitimate perceptions of performance. Taking a step back and changing perspectives to focus on the fundamental underlying value of fairness in assessment may help re-set the traditional objective approach and provide a more relevant way to determine the appropriateness of subjective human judgements. Changing focus to look at what is 'fair' human judgement in assessment, rather than what is 'objective' human judgement in assessment allows for the embracing of many different perspectives, and the legitimising of human judgement in assessment. However, this requires addressing the question: what makes human judgements fair in health professions assessment? This is not a straightforward question with a single unambiguously 'correct' answer. In this hermeneutic literature review we aimed to produce a scholarly knowledge synthesis and understanding of the factors, definitions and key questions associated with fairness in human judgement in assessment and a resulting conceptual framework, with a view to informing ongoing further research. The complex construct of fair human judgement could be conceptualised through values (credibility, fitness for purpose, transparency and defensibility) which are upheld at an individual level by characteristics of fair human judgement (narrative, boundaries, expertise, agility and evidence) and at a systems level by procedures (procedural fairness, documentation, multiple opportunities, multiple assessors, validity evidence) which help translate fairness in human judgement from concepts into practical components.