ABSTRACT
The coronavirus disease 2019 (COVID-19) pandemic has required educators to adapt the in-person objective structured clinical examination (OSCE) to online settings in order for it to remain a critical component of the multifaceted assessment of a student’s competency. This systematic scoping review aimed to summarize the assessment methods and validity and reliability of the measurement tools used in current online OSCE (hereafter, referred to as teleOSCE) approaches. A comprehensive literature review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Articles were eligible if they reported any form of performance assessment, in any field of healthcare, delivered in an online format. Two reviewers independently screened the results and analyzed relevant studies. Eleven articles were included in the analysis. Pre-recorded videos were used in 3 studies, while observations by remote examiners through an online platform were used in 7 studies. Acceptability as perceived by students was reported in 2 studies. This systematic scoping review identified several insights garnered from implementing teleOSCEs, the components transferable from telemedicine, and the need for systemic research to establish the ideal teleOSCE framework. TeleOSCEs may be able to improve the accessibility and reproducibility of clinical assessments and equip students with the requisite skills to effectively practice telemedicine in the future.
ABSTRACT
Purpose@#It aimed to compare the use of the tele-objective structured clinical examination (teleOSCE) with in-person assessment in high-stakes clinical examination so as to determine the impact of the teleOSCE on the assessment undertaken. Discussion follows regarding what skills and domains can effectively be assessed in a teleOSCE. @*Methods@#This study is a retrospective observational analysis. It compares the results achieved by final year medical students in their clinical examination, assessed using the teleOSCE in 2020 (n=285), with those who were examined using the traditional in-person format in 2019 (n=280). The study was undertaken at the University of New South Wales, Australia. @*Results@#In the domain of physical examination, students in 2020 scored 0.277 points higher than those in 2019 (mean difference=–0.277, P<0.001, effect size=0.332). Across all other domains, there was no significant difference in mean scores between 2019 and 2020. @*Conclusion@#The teleOSCE does not negatively impact assessment in clinical examination in all domains except physical examination. If the teleOSCE is the future of clinical skills examination, assessment of physical examination will require concomitant workplace-based assessment.
ABSTRACT
The coronavirus disease 2019 (COVID-19) pandemic has required educators to adapt the in-person objective structured clinical examination (OSCE) to online settings in order for it to remain a critical component of the multifaceted assessment of a student’s competency. This systematic scoping review aimed to summarize the assessment methods and validity and reliability of the measurement tools used in current online OSCE (hereafter, referred to as teleOSCE) approaches. A comprehensive literature review was undertaken following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews guidelines. Articles were eligible if they reported any form of performance assessment, in any field of healthcare, delivered in an online format. Two reviewers independently screened the results and analyzed relevant studies. Eleven articles were included in the analysis. Pre-recorded videos were used in 3 studies, while observations by remote examiners through an online platform were used in 7 studies. Acceptability as perceived by students was reported in 2 studies. This systematic scoping review identified several insights garnered from implementing teleOSCEs, the components transferable from telemedicine, and the need for systemic research to establish the ideal teleOSCE framework. TeleOSCEs may be able to improve the accessibility and reproducibility of clinical assessments and equip students with the requisite skills to effectively practice telemedicine in the future.
ABSTRACT
Purpose@#It aimed to compare the use of the tele-objective structured clinical examination (teleOSCE) with in-person assessment in high-stakes clinical examination so as to determine the impact of the teleOSCE on the assessment undertaken. Discussion follows regarding what skills and domains can effectively be assessed in a teleOSCE. @*Methods@#This study is a retrospective observational analysis. It compares the results achieved by final year medical students in their clinical examination, assessed using the teleOSCE in 2020 (n=285), with those who were examined using the traditional in-person format in 2019 (n=280). The study was undertaken at the University of New South Wales, Australia. @*Results@#In the domain of physical examination, students in 2020 scored 0.277 points higher than those in 2019 (mean difference=–0.277, P<0.001, effect size=0.332). Across all other domains, there was no significant difference in mean scores between 2019 and 2020. @*Conclusion@#The teleOSCE does not negatively impact assessment in clinical examination in all domains except physical examination. If the teleOSCE is the future of clinical skills examination, assessment of physical examination will require concomitant workplace-based assessment.
ABSTRACT
Purpose@#Senior nursing and medical interns’ lack of familiarity and confidence with respect to practicing universal precaution for the prevention of occupational needlestick or sharp injuries may harm themselves. Trainees’ self-reported needlestick or sharp injury rate was known to be especially high during the first 2 months of internship in Taiwan. This prospective cohort study aimed to assess the effect of newly developed virtual reality (VR) game, which uses Gagne’s learning model to improve universal precaution for needlestick or sharp injury prevention and decrease the rates of needle stick or sharp injuries in new-coming medical and nursing interns in Taiwan. @*Methods@#From 2017 to 2019, the VR system was developed and applied in training of 59 new-coming nursing and 50 medical interns. Occupational needlestick or sharp injury prevention was sought to be achieved through a game of right and wrong choices for safe or unsafe universal precaution behaviors. @*Results@#In comparison with medical interns, a higher proportion of nursing interns had past experiences of deep occupational needlestick or sharp injury. Before VR training, the familiarity and confidence for needlestick or sharp injury prevention were higher among nursing interns than medical interns. Trainees with past experiences of deep needlestick or sharp injury exhibited better performance on the accuracy rate and time needed to complete 20 decisions than those without past experiences in VR practice. All trainees showed an improved performance after VR training. A high proportion of trainees reported that the VR-based training significantly decreased their anxiety about needlestick or sharp injury prevention. @*Conclusion@#This self-developed VR game system using Gagne’s flow improved universal precaution for needlestick or sharp injury prevention and reduced the needlestick or sharp injury rates in the first 2 months of nursing and medical internship.
ABSTRACT
PURPOSE: Optimal methods for communication skills training (CST) are an active research area, but the effects of CST on communication performance in objective structured clinical examinations (OSCEs) has not been closely studied. Student roleplay (RP) for CST is common, although volunteer simulated patient (SP) CST is cost-effective and provides authentic interactions. We assessed whether our volunteer SP CST program improved OSCE performance compared to our previous RP strategy. METHODS: We performed a retrospective, quasi-experimental study of 2 second-year medical student cohorts' OSCE data in Australia. The 2014 cohort received RP-only CST (N=182) while the 2016 cohort received SP-only CST (N=148). The t-test and analysis of variance were used to compare the total scores in 3 assessment domains: generic communication, clinical communication, and physical examination/procedural skills. RESULTS: The baseline characteristics of groups (scores on the Australian Tertiary Admission Rank, Undergraduate Medicine and Health Sciences Admission Test, and medicine program interviews) showed no significant differences between groups. For each domain, the SP-only CST group demonstrated superior OSCE outcomes, and the difference between cohorts was significant (P<0.01). The superiority of volunteer SP CST over student RP CST in terms of OSCE performance outcomes was found for generic communication, clinical communication, and physical examination/procedural skills. CONCLUSION: The better performance of the SP cohort in physical examination/procedural skills might be explained by the requirement for patient compliance and cooperation, facilitated by good generic communication skills. We recommend a volunteer SP program as an effective and efficient way to improve CST among junior medical students.
Subject(s)
Humans , Australia , Clinical Competence , Cohort Studies , Non-Randomized Controlled Trials as Topic , Patient Compliance , Patient Simulation , Retrospective Studies , Students, Medical , VolunteersABSTRACT
PURPOSE: Lack of confidence in suturing/ligature skills due to insufficient practice and assessments is common among novice Chinese medical interns. This study aimed to improve the skill acquisition of medical interns through a new intervention program. METHODS: In addition to regular clinical training, expert-led or expert-led plus artificial intelligence (AI) system tutoring courses were implemented during the first 2 weeks of the surgical block. Interns could voluntarily join the regular (no additional tutoring), expert-led tutoring, or expert-led+AI tutoring groups freely. In the regular group, interns (n=25) did not receive additional tutoring. The expert-led group received 3-hour expert-led tutoring and in-training formative assessments after 2 practice sessions. After a similar expert-led course, the expert-led+AI group (n=23) practiced and assessed their skills on an AI system. Through a comparison with the internal standard, the system automatically recorded and evaluated every intern's suturing/ligature skills. In the expert-led+AI group, performance and confidence were compared between interns who participated in 1, 2, or 3 AI practice sessions. RESULTS: The end-of-surgical block objective structured clinical examination (OSCE) performance and self-assessed confidence in suturing/ligature skills were highest in the expert-led+AI group. In comparison with the expert-led group, the expert-led+AI group showed similar performance in the in-training assessment and greater improvement in the end-of-surgical block OSCE. In the expert-led+AI group, the best performance and highest post-OSCE confidence were noted in those who engaged in 3 AI practice sessions. CONCLUSION: This pilot study demonstrated the potential value of incorporating an additional expert-led+AI system–assisted tutoring course into the regular surgical curriculum.
Subject(s)
Humans , Artificial Intelligence , Asian People , Curriculum , Ligation , Pilot Projects , Prospective Studies , TaiwanABSTRACT
PURPOSE@#Optimal methods for communication skills training (CST) are an active research area, but the effects of CST on communication performance in objective structured clinical examinations (OSCEs) has not been closely studied. Student roleplay (RP) for CST is common, although volunteer simulated patient (SP) CST is cost-effective and provides authentic interactions. We assessed whether our volunteer SP CST program improved OSCE performance compared to our previous RP strategy.@*METHODS@#We performed a retrospective, quasi-experimental study of 2 second-year medical student cohorts' OSCE data in Australia. The 2014 cohort received RP-only CST (N=182) while the 2016 cohort received SP-only CST (N=148). The t-test and analysis of variance were used to compare the total scores in 3 assessment domains: generic communication, clinical communication, and physical examination/procedural skills.@*RESULTS@#The baseline characteristics of groups (scores on the Australian Tertiary Admission Rank, Undergraduate Medicine and Health Sciences Admission Test, and medicine program interviews) showed no significant differences between groups. For each domain, the SP-only CST group demonstrated superior OSCE outcomes, and the difference between cohorts was significant (P<0.01). The superiority of volunteer SP CST over student RP CST in terms of OSCE performance outcomes was found for generic communication, clinical communication, and physical examination/procedural skills.@*CONCLUSION@#The better performance of the SP cohort in physical examination/procedural skills might be explained by the requirement for patient compliance and cooperation, facilitated by good generic communication skills. We recommend a volunteer SP program as an effective and efficient way to improve CST among junior medical students.
ABSTRACT
PURPOSE@#Lack of confidence in suturing/ligature skills due to insufficient practice and assessments is common among novice Chinese medical interns. This study aimed to improve the skill acquisition of medical interns through a new intervention program.@*METHODS@#In addition to regular clinical training, expert-led or expert-led plus artificial intelligence (AI) system tutoring courses were implemented during the first 2 weeks of the surgical block. Interns could voluntarily join the regular (no additional tutoring), expert-led tutoring, or expert-led+AI tutoring groups freely. In the regular group, interns (n=25) did not receive additional tutoring. The expert-led group received 3-hour expert-led tutoring and in-training formative assessments after 2 practice sessions. After a similar expert-led course, the expert-led+AI group (n=23) practiced and assessed their skills on an AI system. Through a comparison with the internal standard, the system automatically recorded and evaluated every intern's suturing/ligature skills. In the expert-led+AI group, performance and confidence were compared between interns who participated in 1, 2, or 3 AI practice sessions.@*RESULTS@#The end-of-surgical block objective structured clinical examination (OSCE) performance and self-assessed confidence in suturing/ligature skills were highest in the expert-led+AI group. In comparison with the expert-led group, the expert-led+AI group showed similar performance in the in-training assessment and greater improvement in the end-of-surgical block OSCE. In the expert-led+AI group, the best performance and highest post-OSCE confidence were noted in those who engaged in 3 AI practice sessions.@*CONCLUSION@#This pilot study demonstrated the potential value of incorporating an additional expert-led+AI system–assisted tutoring course into the regular surgical curriculum.
ABSTRACT
Purpose@#In contemporary pharmacy, the role of pharmacists has become more multifaceted, as they now handle a wider range of tasks and take more responsibility for providing patient care than 20 years ago. This evolution in pharmacists’ responsibilities has been accompanied by the need for pharmacists to display high-quality patient-centred care and counselling, and to demonstrate professionalism, which now needs to be taught and assessed as part of pharmacy education and practice. This study aimed at identifying definitions of professionalism in pharmacy practice and critically evaluating published instruments for assessing professionalism in pharmacy practice. @*Methods@#We searched the medical literature listed in Scopus, MEDLINE, and PsycINFO databases from 1 January 2000 to 31 December 2018. All papers meeting our selection criteria were reviewed and summarised into a clear review of professionalism requirements in pharmacy practice. Details of the instruments measuring professionalism were reviewed in detail. @*Results@#There is no accepted simple definition of professionalism, although we identified several theoretical and policy frameworks required for professional pharmaceutical practice. We identified 4 instruments (the Behavioural Professionalism Assessment Instrument, Lerkiatbundit’s instrument, the Pharmacy Professionalism Instrument, and the Professionalism Assessment Tool that build on these frameworks and measure professional practice in pharmacy students. These were found to be reliable and valid, but had only been used and tested in student populations. @*Conclusion@#Given the increasingly broad role of community pharmacists, there is a need for assessments of professionalism in practice. Professionalism is a complex concept that is challenging to measure because it has no standardised definition and the existing literature related to the topic is limited. Currently available instruments focus on measuring the development of the elements of professionalism among pharmacy students, rather than pharmacists.
ABSTRACT
Purpose@#In contrast to the core part of the clinical interviewing and physical examination (PE) skills course, corresponding to the basic, head-to-toe, and thoracic systems, learners need structured feedback in the cluster part of the course, which includes the abdominal, neuromuscular, and musculoskeletal systems. This study evaluated the effects of using Dreyfus scale-based feedback, which has elements of continuous professional development, instead of Likert scale-based feedback in the cluster part of training in Taiwan. @*Methods@#Instructors and final-year medical students in the 2015–2016 classes of National Yang-Ming University, Taiwan comprised the regular cohort, whereas those in the 2017–2018 classes formed the intervention cohort. In the intervention cohort, Dreyfus scale-based feedback, rather than Likert scale-based feedback, was used in the cluster part of the course. @*Results@#In the cluster part of the course in the regular cohort, pre-trained standardized patients rated the class climate as poor, and students expressed low satisfaction with the instructors and course and low self-assessed readiness. In comparison with the regular cohort, improved end-of-course group objective structured clinical examination scores after the cluster part were noted in the intervention cohort. In other words, the implementation of Dreyfus scale-based feedback in the intervention cohort for the cluster part improved the deficit in this section of the course. @*Conclusion@#The implementation of Dreyfus scale-based feedback helped instructors to create a good class climate in the cluster part of the clinical interviewing and PE skills course. Simultaneously, this new intervention achieved the goal of promoting medical students’ readiness for interviewing, PE, and self-directed learning.
ABSTRACT
PURPOSE: The biases that may influence objective structured clinical examination (OSCE) scoring are well understood, and recent research has attempted to establish the magnitude of their impact. However, the influence of examiner experience, clinical seniority, and occupation on communication and physical examination scores in OSCEs has not yet been clearly established. METHODS: We compared the mean scores awarded for generic and clinical communication and physical examination skills in 2 undergraduate medicine OSCEs in relation to examiner characteristics (gender, examining experience, occupation, seniority, and speciality). The statistical significance of the differences was calculated using the 2-tailed independent t-test and analysis of variance. RESULTS: Five hundred and seventeen students were examined by 237 examiners at the University of New South Wales in 2014 and 2016. Examiner gender, occupation (academic, clinician, or clinical tutor), and job type (specialist or generalist) did not significantly impact scores. Junior doctors gave consistently higher scores than senior doctors in all domains, and this difference was statistically significant for generic and clinical communication scores. Examiner experience was significantly inversely correlated with generic communication scores. CONCLUSION: We suggest that the assessment of examination skills may be less susceptible to bias because this process is fairly prescriptive, affording greater scoring objectivity. We recommend training to define the marking criteria, teaching curriculum, and expected level of performance in communication skills to reduce bias in OSCE assessment.
Subject(s)
Humans , Australia , Awards and Prizes , Bias , Curriculum , New South Wales , Occupations , Physical ExaminationABSTRACT
PURPOSE@#The biases that may influence objective structured clinical examination (OSCE) scoring are well understood, and recent research has attempted to establish the magnitude of their impact. However, the influence of examiner experience, clinical seniority, and occupation on communication and physical examination scores in OSCEs has not yet been clearly established.@*METHODS@#We compared the mean scores awarded for generic and clinical communication and physical examination skills in 2 undergraduate medicine OSCEs in relation to examiner characteristics (gender, examining experience, occupation, seniority, and speciality). The statistical significance of the differences was calculated using the 2-tailed independent t-test and analysis of variance.@*RESULTS@#Five hundred and seventeen students were examined by 237 examiners at the University of New South Wales in 2014 and 2016. Examiner gender, occupation (academic, clinician, or clinical tutor), and job type (specialist or generalist) did not significantly impact scores. Junior doctors gave consistently higher scores than senior doctors in all domains, and this difference was statistically significant for generic and clinical communication scores. Examiner experience was significantly inversely correlated with generic communication scores.@*CONCLUSION@#We suggest that the assessment of examination skills may be less susceptible to bias because this process is fairly prescriptive, affording greater scoring objectivity. We recommend training to define the marking criteria, teaching curriculum, and expected level of performance in communication skills to reduce bias in OSCE assessment.
ABSTRACT
PURPOSE: The objective structured clinical examination (OSCE) is considered to be one of the most robust methods of clinical assessment. One of its strengths lies in its ability to minimise the effects of examiner bias due to the standardisation of items and tasks for each candidate. However, OSCE examiners' assessment scores are influenced by several factors that may jeopardise the assumed objectivity of OSCEs. To better understand this phenomenon, the current review aims to determine and describe important sources of examiner bias and the factors affecting examiners' assessments. METHODS: We performed a narrative review of the medical literature using Medline. All articles meeting the selection criteria were reviewed, with salient points extracted and synthesised into a clear and comprehensive summary of the knowledge in this area. RESULTS: OSCE examiners' assessment scores are influenced by factors belonging to 4 different domains: examination context, examinee characteristics, examinee-examiner interactions, and examiner characteristics. These domains are composed of several factors including halo, hawk/dove and OSCE contrast effects; the examiner's gender and ethnicity; training; lifetime experience in assessing; leadership and familiarity with students; station type; and site effects. CONCLUSION: Several factors may influence the presumed objectivity of examiners' assessments, and these factors need to be addressed to ensure the objectivity of OSCEs. We offer insights into directions for future research to better understand and address the phenomenon of examiner bias.
Subject(s)
Humans , Bias , Leadership , Patient Selection , Problem Solving , Recognition, PsychologyABSTRACT
PURPOSE: Peer physical examination (PPE), by which junior medical students learn physical examination skills before practicing on patients, is a widely implemented and accepted part of medical curricula. However, the ethical implications of PPE have been debated, since issues including student gender impact on its acceptability. Research has previously demonstrated the phenomenon of ‘attitude-behavior inconsistency' showing that students' predictions about their participation in PPE differ from what they actually do in practice. This study asks whether gender and student self-ratings of outlook affect engagement in PPE. METHODS: This study gathered data from students who had completed PPE with the objective of determining what factors have the greatest impact on the actual practice of PPE by students. Data were used to derive the number of opportunities students had to examine a peer, for various body parts. Respondent gender and self-ratings of outlook were recorded. RESULTS: Responses from 130 students were analysed: 74 female (57%) and 56 male (43%). Students have fewer opportunities to examine peers of the opposite gender; this is statistically significant for all body parts when male students examine female peers. CONCLUSION: Gender is the factor of overriding importance on whether these peer interactions actually occur, such that students have fewer opportunities to examine peers of the opposite gender, particularly male students examining female peers. Student outlook has little impact. We speculate that the more acceptable PPE is to participants, paradoxically, the more complicated these interactions become, possibly with implications for future practice.
Subject(s)
Female , Humans , Male , Australia , Curriculum , Human Body , Peer Group , Physical Examination , Students, Medical , Surveys and QuestionnairesABSTRACT
This study aimed to identify those features within secondary school curricula and assessment, particularly science subjects that best predict academic achievement in the first year of three different three-year undergraduate health professional programmes (nursing, pharmacy, and health sciences) at a large New Zealand university. In particular, this study compared the contribution of breadth of knowledge (number of credits acquired) versus grade level (grade point average) and explored the impact of demographic variables on achievement. The findings indicated that grades are the most important factor predicting student success in the first year of university. Although taking biology and physics at secondary school has some impact on university first year achievement, the effect is relatively minor.