A Comparison of Checklist and Domain-Based Ratings in the Assessment of Objective Structured Clinical Examination (OSCE) Performance.

INTRODUCTION: The Objective Structured Clinical Examination (OSCE) is the mainstay of clinical assessment in the final-year undergraduate Family Medicine clerkship at King Faisal Specialist Hospital and Research Centre (KFSHRC). The gold standard for OSCE assessment is the checklist rating, completed by physician examiners. Numerous studies have suggested that global or domain-based OSCE ratings may be a better indicator of competence than checklist ratings. The aim of this study was to examine the utility of domain-based OSCE ratings in the context of final-year, undergraduate, Family Medicine OSCE examinations in Riyadh, Saudi Arabia. This is akin to an exercise in quality improvement, as we continuously look for ways to improve our OSCE assessment processes. METHODS: This study utilised a quantitative methodology. Three final year OSCE exams were chosen. Physicians rated each student using a checklist score and using a more holistic domain-based score. Physician checklist scores and physician domain-based scores were then compared, and correlation was assessed. We also looked at the internal consistency of the scoring methods. RESULTS: A significant correlation was found between checklist and domain-based scores by physicians for all exams (r=0.858, p<0.01), with a good internal consistency for these methodologies for all exams. CONCLUSION: The results demonstrate that both checklist and domain-based scores offer some benefit to the assessment, with a similar internal consistency and strong correlation. Domain-based ratings should be utilised for softer skills that are not easily assessed by checklists. There is clearly a need to rethink our OSCE assessment. The assessment should combine checklist and domain-based physician scores. As trainees become more experienced, checklist OSCE may penalise directness and efficiency, while domain-based ratings would offer a better appraisal of competence, and have been shown to be more sensitive to the level of training and expertise. Changing the assessment methods will lead to necessary changes in the student approach to the OSCE and improve authenticity and validity.

Análisis de la aceptación y validez de los métodos utilizados para la implementación de un programa de formación basado en competencias en un servicio de Medicina Intensiva de un hospital universitario de referencia / Acceptance and validity of the methods used to implement a competency based medical education programme in an Intensive Care Department of a teaching referral center

Objetivos: 1) Determinar la satisfacción de tutores y residentes con la metodología utilizada para la implementación de CoBaTrICE, y 2) determinar la validez y la fiabilidad de las escalas de valoración global diseñados ad hoc para analizar el desempeño de los residentes con fines formativos. Diseño: Estudio prospectivo de cohortes. Participantes: Todos los residentes y tutores del Servicio de Medicina Intensiva del Hospital Universitario y Politécnico La Fe de Valencia. Intervención: En marzo del 2016 se inició la implementación de CoBaTrICE sustentada en: 1) formación de los tutores en técnicas de retroalimentación; 2) realización por los residentes de múltiples ejercicios reales de evaluación objetiva y estructurada para adquirir las competencias del programa, y 3) uso de un portafolio electrónico para registrar las evidencias del progreso y estimular la reflexión. Métodos: La satisfacción con CoBaTrICE se evaluó mediante una encuesta realizada tras 9 meses de implementación a los 15 residentes y 5 tutores del servicio. Se preguntó sobre la metodología de las evaluaciones, calidad de la retroalimentación, autorregulación del aprendizaje y utilidad del portafolio. Se determinaron la consistencia interna (alfa de Cronbach), índices de generalizabilidad y fiabilidad interjueces (índice de correlación intraclase) de las escalas de valoración global. Resultados: La aplicación de CoBaTrICE fue satisfactoria en todas las dimensiones estudiadas. Se constataron la validez y la fiabilidad de las escalas de valoración utilizadas. Conclusiones: La metodología utilizada para implementar CoBaTrICE fue valorada positivamente por tutores y residentes. Las escalas de valoración global utilizadas en la evaluación formativa demostraron ser válidas, fiables y reproducibles. (AU)

Objectives; 1. To determine the satisfaction of tutors and residents with a specific methodology used to implement CoBaTrICE; 2. To determine the reliability and validity of the global rating scales designed ad hoc to assess the performance of the residents for training purposes. Design Prospective cohort study Participants All the residents and tutors of the ICU Department of the Hospital Universitario y Politécnico la Fe de Valencia. Intervention CoBaTrICE implementation started in March 2016, it was based on: 1) Training the tutors in feedback techniques; 2) Performing multiple objective and structured work based assessments to achieve the competences of the program; and 3) The use of an electronic portfolio to promote learning reflection and to collect the evidence that learning was taking place. Methods The acceptance of CoBaTrICE was explored through a satisfaction survey conducted after 9 months of implementation of the training program. The 15 residents and 5 tutors of the ICU Department were asked about the methodology of the formative assessments, the quality of the feedback, self-learning regulation and the electronic portfolio usefulness. The validity of the global rating scales was assessed through the tests alfa de Cronbach, reliability and generalizability indexes, and intraclass correlation coefficient. Results The implementation of CoBaTrICE was satisfactory in all the dimensions studied. The global rating scales used for formative purposes showed reliability and validity. Conclusions The methodology used to implement CoBaTrICE was highly valued by tutors and residents. The global rating scales used for formative purposes showed reliability and validity. (AU)

Acceptance and validity of the methods used to implement a competency based medical education programme in an Intensive Care Department of a teaching referral center.

OBJECTIVES: 1. To determine the satisfaction of tutors and residents with a specific methodology used to implement CoBaTrICE. 2. To determine the reliability and validity of the global rating scales designed ad hoc to assess the performance of the residents for training purposes. DESIGN: Prospective cohort study. PARTICIPANTS: All the residents and tutors of the ICU Department of the Hospital Universitario y Politécnico la Fe de Valencia. INTERVENTION: CoBaTrICE implementation started in March 2016, it was based on: (1) Training the tutors in feedback techniques; (2) Performing multiple objective and structured work based assessments to achieve the competences of the program; and (3) The use of an electronic portfolio to promote learning reflection and to collect the evidence that learning was taking place. METHODS: The acceptance of CoBaTrICE was explored through a satisfaction survey conducted after 9 months of implementation of the training program. The 15 residents and 5 tutors of the ICU Department were asked about the methodology of the formative assessments, the quality of the feedback, self-learning regulation and the electronic portfolio usefulness. The validity of the global rating scales was assessed through the tests alfa de Cronbach, reliability and generalizability indexes, and intraclass correlation coefficient. RESULTS: The implementation of CoBaTrICE was satisfactory in all the dimensions studied. The global rating scales used for formative purposes showed reliability and validity. CONCLUSIONS: The methodology used to implement CoBaTrICE was highly valued by tutors and residents. The global rating scales used for formative purposes showed reliability and validity.

The assessment of researchers' competence in experimental procedures with laboratory animals: A three-step methodology to develop a global rating scale.

To conduct animal experiments, researchers must be competent to handle and perform interventions on living animals in compliance with regulations. Laboratory animal science training programmes and licensing bodies therefore need to be able to reliably ensure and certify the professional competence of researchers and technicians. This requires access to assessment strategies which can verify knowledge as well as capturing performative and behavioural dimensions of assessment. In this paper, we describe the process of developing different global rating scales measuring candidates' competence in a performative assessment. We set out the following sequence, with three crucial phases, in the process of scale development: (a) Item Development, (b) Scale Development and (c) Piloting of the Scale. We note each phase's different sub-steps. Despite the emergent need to ensure the competence of researchers using animals in scientific procedures, to our best knowledge there are very few species and procedure/skill specific assessment tools for this purpose, and the assessment methodology literature in the field is very limited. This paper provides guidance for those who need to develop and assess proficiency in laboratory animal procedures by setting out a method that can be used to create the required tools and illustrating how competence assessment strategies can be implemented.

Assessment of procedural skills in residents working in a research and training institute: An effort to ensure patient safety and quality control.

BACKGROUND: To ensure patient safety, it is important to regularly assess the knowledge and practical skills of anesthesia trainees. This study was conducted to evaluate the competency of the residents and the impact of various corrective measures in the form of didactic lectures and clinical skill demonstrations on the conduct of various procedural skills by the residents. MATERIALS AND METHODS: Ninety-five junior residents were enrolled in this study. Assessment of competency of 1st, 2nd, and 3rd year residents in performing various procedure skills of anesthesia was done in two stages using procedure specific checklist (PSC) and Global Rating Scales (GRSs). Preliminary results of the first assessment (Score 1) were discussed with the residents; deficiencies were identified and corrective measures suggested by didactic lectures and clinical skill demonstrations which were followed by a subsequent assessment after 3 months (Score 2). RESULTS: There was a statistically significant improvement in the PSC and GRS scores after corrective measures for all the procedural interventions studied. Percentage increase in scores was maximum in 1st year (42.98 ± 6.62) followed by 2nd year (34.62 ± 5.49) and minimum in 3rd year residents (18.06 ± 3.69). The percentage increase of scores was almost similar for all subset of procedural skills; low, intermediate, and high skill anesthetic procedures. CONCLUSION: For assessment of procedural skills of residents, use of PSC and GRS scores should be incorporated and the same should be used to monitor the impact of various corrective measures (didactic lectures and clinical skill demonstrations) on the conduct of various procedural skills by the resident.

The value of patient global assessment in lumbar spine surgery: an evaluation based on more than 90,000 patients.

PURPOSE: There are two, principally different ways to obtain patient opinions regarding the outcome of spine surgery: using prospective multi-item questionnaires preoperatively and at follow-up, and using a retrospective single-item question at follow-up-both methods have distinct advantages and limitations. The purpose of the study was to explore the utility of using the simple transition question global assessment, GA, ("How is your back/leg pain today as compared to before the surgery?") as an overall patient-reported outcome measure (PROM) based on the large real-life database in the Swedish spine registry (Swespine). METHODS: The correlation between GA and the score-changes and the final scores at 1 year of follow-up for the PROMs VAS, ODI, and EQ-5D was examined. The correlations between GA and item-specific domains within the ODI, EQ-5D and SF-36 as well as the discriminative ability of PROMs with GA as reference criterion were also analysed. The cohort consisted of 94,132 patients registered in Swespine who were surgically treated for disc herniation, spinal stenosis or degenerative disc disease. RESULTS: The correlation coefficients for GA vs. the score-changes were lower than for GA vs final scores. For VAS they ranged for the different diagnosis groups from 0.33 to 0.61 and from 0.50 to 0.79, respectively. For ODI, the corresponding values ranged from 0.43 to 0.65 and 0.63 to 0.76; for the EQ-5D from 0.32 to 0.45 and 0.54 to 0.71. Further, GA showed a somewhat stronger correlation to pain-specific PROMs than to quality-of-life PROMs. CONCLUSIONS: The single-item outcome measure global assessment (GA) appears to be a feasible overall patient-reported outcome measure (PROM) and a useful reference for interpreting the scores of patient-reported outcome measures.

Function and clinical meaningfulness of treatments for mild Alzheimer's disease.

INTRODUCTION: Effectiveness of Alzheimer's disease (AD) treatments is commonly evaluated with coprimary outcomes; cognition with function to ensure clinical meaningfulness of a cognitive effect. METHODS: We reviewed the literature for functional outcomes in mild AD or mild cognitive impairment (MCI) patients (distinct from combined mild-moderate/severe AD) treated with approved AD drugs. Cognitive and functional treatment differences in mild AD patients in solanezumab EXPEDITION/EXPEDITION2 studies were compared across time. RESULTS: Seven publications provided MCI/mild AD functional outcomes, one of which reported a significant functional treatment effect. Secondary analyses of EXPEDITION studies suggested a smaller functional effect of solanezumab relative to cognition. An increasing effect of solanezumab over 18 months was shown for cognition and function. DISCUSSION: Function as the sole measure to demonstrate clinical meaningfulness of cognitive effects in mild AD may have limitations. For disease-modifying treatments, point differences on cognitive and functional scales should be qualified with duration of treatment.

Improving the content and face validity of OSCE assessment marking criteria on an undergraduate midwifery programme: a quality initiative.

UNLABELLED: Objective Structured Clinical Examinations (OSCE's) have been adopted as a means of assessing midwifery students' clinical skills. The purpose of the OSCE is to provide a standardised method for the evaluation of clinical skill performance in a simulated environment. This paper describes how a quality improvement initiative using both internal and external expert review was utilised to improve OSCE assessment marking criteria. The purpose of the quality initiative was to review the content and face validity of the marking criteria for assessing performance. The design and choice of tools used to score students' performance is central to reliability and validity. 20 videos of students from year one of a midwifery preregistration programme undertaking an OSCE assessment on abdominal examination and 18 videos of students response to obstetric emergencies e.g. PPH, and shoulder dystocia were available for review. The quality initiative aimed to strengthen the reliability and validity of the OSCE in assessing student performance. CONCLUSION: the use of global rating scales allows for the capturing of elements of professional competency that do not appear on specific criteria for skills performance checklists.