Assessing Student Performance Using a Novel Rubric Based on the Dreyfus Model of Skill Acquisition.

OBJECTIVE: Pharmacy student performance on practicum was previously assessed using a Likert scale from 0 to 9, resulting in challenges with clarity and assessor subjectivity. To address these issues, an assessment rubric based on the Dreyfus model of skill acquisition was developed and implemented. This study sought to evaluate student, practice educator (PE), and faculty perceptions related to the rubric's effectiveness in assessing student performance within the direct patient care practicum setting. METHODS: An exploratory sequential mixed methods approach was used. A qualitative component using focus groups and semistructured interviews was followed by a quantitative component using a survey questionnaire. Data gathered from the qualitative component were collectively analyzed and used to inform questionnaire development intended to confirm identified themes and collect further data on stakeholder perceptions. RESULTS: A total of 7 students, 7 PEs, and 4 faculty participated in the focus groups/interviews and 70 of 645 (10.9%) students and 103 of 756 (13.6%) PEs participated in the survey questionnaire. The majority of the participants felt that the rubric clearly communicated the expectations for student performance, is relevant and consistent with pharmacy practice, and is useful in accurately assessing performance. For PEs with experience, the novel rubric was an improvement over the previous assessment processes and perceived as more thorough and clearer in describing performance expectations. The identified challenges included the rubric's visual organization, length, and redundancy of some of the assessment elements. CONCLUSION: Our findings suggest that a novel rubric based on the Dreyfus model is effective in assessing student performance on practicum and may address some of the challenges commonly observed with performance assessment.

Pharmacist trainees narrow scope of interprofessional collaboration and communication in hospital practice.

Early curricular exposure to interprofessional education (IPE) is intended to acclimatize health professional trainees to shared-care in the practice settings they will ultimately join. However, IPE activities typically reside outside actual organizational and social systems in which interprofessional care is delivered. We aimed to explore how pharmacist trainees experience collaborator and communicator competency roles during team-based workplace-based learning. Participants maintained written diaries reflecting on interprofessional collaboration and communication during an eight-week hospital clerkship. Diary entries and transcripts from semi-structured follow-up interviews were analyzed from the social constructivist perspective using reflective thematic analysis. Participant accounts of on-ward activities represented most collaborator and communicator roles outlined in pharmacy and interprofessional competency frameworks, but were predominantly between the pharmacist trainee and physicians. Pharmacist trainees did not routinely engage with other health professions on a daily basis. Additionally, reported encounters with other team members were typically information exchanges and not episodes of authentic interdependent or shared care. Interactions were almost completely devoid of perceived interpersonal or role conflict. These findings offer insight into how pharmacist trainees perceive and develop competencies for team-based care. Further work is required to understand how such limited scope of interprofessional communication and collaboration might ultimately impair quality patient care.

Status of simulation in health care education: an international survey.

Simulation is rapidly penetrating the terrain of health care education and has gained growing acceptance as an educational method and patient safety tool. Despite this, the state of simulation in health care education has not yet been evaluated on a global scale. In this project, we studied the global status of simulation in health care education by determining the degree of financial support, infrastructure, manpower, information technology capabilities, engagement of groups of learners, and research and scholarly activities, as well as the barriers, strengths, opportunities for growth, and other aspects of simulation in health care education. We utilized a two-stage process, including an online survey and a site visit that included interviews and debriefings. Forty-two simulation centers worldwide participated in this study, the results of which show that despite enormous interest and enthusiasm in the health care community, use of simulation in health care education is limited to specific areas and is not a budgeted item in many institutions. Absence of a sustainable business model, as well as sufficient financial support in terms of budget, infrastructure, manpower, research, and scholarly activities, slows down the movement of simulation. Specific recommendations are made based on current findings to support simulation in the next developmental stages.

Objective assessment of laparoscopic skills: dual-task approach.

BACKGROUND: Assessment of surgical performance is often accomplished with traditional methods that often provide only subjective data. Trainees who perform well on a simulator in a controlled environment may not perform well in a real operating room environment with distractions. This project uses the ideas of dual-task methodology and applies them to the assessment of performance of laparoscopic surgical skills. The level of performance on distracting secondary tasks while trying to perform a primary task becomes an indirect but objective measure of the surgical skill of the trainee. METHODS: Nine surgery residents and 6 experienced laparoscopic surgeons performed 3 primary tasks on a laparoscopic virtual reality simulator (camera position, grasping, and cholecystectomy) while being distracted by 3 secondary tasks (counting beeps, selective responses, and mental arithmetic). Completion time and error rates were recorded for each combination of tasks. RESULTS: When performed separately, time to completion and error rates for primary and secondary tasks were similar for learners and experts. When performing the tasks simultaneously, learners had more errors than experts. Error rates increased for learners when distracting tasks became more difficult or required more attention. Expert surgeons maintained consistent error rates despite the increasing difficulty of task combinations. CONCLUSIONS: The use of dual-task methodology may help trainers to identify which surgical trainees require more preparation before entering the real operating room environment. Expert surgeons are capable of maintaining performance levels on a primary task in the face of distractions that may occur in the operating room.

Comparing the use of global rating scale with checklists for the assessment of central venous catheterization skills using simulation.

The use of checklists is recommended for the assessment of competency in central venous catheterization (CVC) insertion. To explore the use of a global rating scale in the assessment of CVC skills, this study seeks to compare its use with two checklists, within the context of a formative examination using simulation. Video-recorded performances of CVC insertion by 34 first-year medical residents were reviewed by two independent, trained evaluators. Each evaluator used three assessment tools: a ten-item checklist, a 21-item checklist, and a nine-item global rating scale. Exploratory principal component analysis of the global rating scale revealed two factors, accounting for 84.1% of the variance: technical ability and safety. The two checklist scores correlated positively with the weighted factor score on technical ability (0.49 [95% CI 0.17-0.71] for the 10-item checklist; 0.43 [95% CI 0.10-0.67] for the 21-item checklist) and negatively with the weighted factor score on safety (-0.17 [95% CI -0.48-0.18] for the 10-item checklist; -0.13 [95% CI -0.45-0.22] for the 21-item checklist). A checklist score of <80% was strong indication of incompetence. However, a high checklist score did not preclude incompetence. Ratings using the global rating scale identified an additional 11 candidates (32%) who were deemed incompetent despite scoring >80% on both checklists. All these candidates committed serious errors. In conclusion, the practice of universal adoption of checklists as the preferred method of assessment of procedural skills should be questioned. The inclusion of global rating scales should be considered.

Reliability and acceptance of the mini-clinical evaluation exercise as a performance assessment of practicing physicians.

BACKGROUND: The purpose of this study was to evaluate the reliability and acceptance of the mini-Clinical Evaluation Exercise (mini-CEX) as an assessment of practicing primary care physicians. METHOD: Six raters were recruited to conduct the assessments. After a training session, their ability to discriminate between levels of performance was evaluated using videotaped clinical scenarios. Fifteen physicians were assessed in an office setting by the raters who scored multiple clinical encounters using a validated mini-CEX form for each encounter. Participants were given a postassessment survey regarding the process. RESULTS: Raters distinguished between performance levels on the videotaped scenarios (P < .001). A total of 188 physician-patient interactions were assessed. The generalizability coefficient for 10 encounters was 0.92. In the postassessment survey, the raters (94%) and physicians assessed (75%) both felt that the mini-CEX is an acceptable assessment. CONCLUSIONS: The mini-CEX seems to be a reliable and acceptable instrument for the assessment of practicing physicians.

The mini clinical evaluation exercise (mini-CEX) for assessing clinical performance of international medical graduates.

OBJECTIVE: To evaluate the feasibility, reliability and acceptability of the mini clinical evaluation exercise (mini-CEX) for performance assessment among international medical graduates (IMGs). DESIGN, SETTING AND PARTICIPANTS: Observational study of 209 patient encounters involving 28 IMGs and 35 examiners at three metropolitan teaching hospitals in New South Wales, Victoria and Queensland, September-December 2006. MAIN OUTCOME MEASURES: The reliability of the mini-CEX was estimated using generalisability (G) analysis, and its acceptability was evaluated by a written survey of the examiners and IMGs. RESULTS: The G coefficient for eight encounters was 0.88, suggesting that the reliability of the mini-CEX was 0.90 for 10 encounters. Almost half of the IMGs (7/16) and most examiners (14/18) were satisfied with the mini-CEX as a learning tool. Most of the IMGs and examiners enjoyed the immediate feedback, which is a strong component of the tool. CONCLUSION: The mini-CEX is a reliable tool for performance assessment of IMGs, and is acceptable to and well received by both learners and supervisors.

Otolaryngology training during paediatric residency: A survey of paediatricians in Canada.

INTRODUCTION: There is a significant overlap between paediatrics and otolaryngology relating to clinical practice of the two specialties. A lack of cross-training has been identified in previous studies, but the specifics have not been established. The present study was directed at paediatricians in Canada, and examined the need for mandatory otolaryngology training during paediatric residency. METHODS: Surveys were mailed out to paediatricians in Canada who had completed residency within the past 20 years. Guidelines for the mailing procedure were regulated by the Royal College of Physicians and Surgeons of Canada. A cover letter, survey form and return envelope were included in the package. Data were tabulated and described using descriptive statistics. RESULTS: Six hundred sixty-six surveys were mailed; the response rate was 48%. Seventy-three per cent of paediatricians indicated that otolaryngology training should be mandatory during paediatric residency. Seventy-nine per cent of general paediatricians and 68% of subspecialists also believed that it should be mandatory training. Seventy per cent of paediatricians indicated that clinical experience was the best format for otolaryngology training, the other options being lectures or rotations. Postgraduate year 2 was the most preferred year for this training. For paediatricians who indicated mandatory training, 45% indicated that it could not replace something else, 38% said that it could replace another experience and the remainder were undecided. The respondents provided helpful commentary. INTERPRETATION: The majority of surveyed paediatricians in Canada believe that otolaryngology training should be mandatory during paediatric residency. There was also a general consensus relating to the format (clinical experience) and the duration (two to four weeks) of the training.

Embedding medical student computer tutorials into a busy emergency department.

OBJECTIVES: To explore medical students' use of computer tutorials embedded in a busy clinical setting; to demonstrate that such tutorials can increase knowledge gain over and above that attributable to the clinical rotation itself. METHODS: Six tutorials were installed on a computer placed in a central area in an emergency department. Each tutorial was made up of between 33 and 85 screens of information that include text, graphics, animations, and questions. They were designed to be brief (10 minutes), focused, interactive, and immediately relevant. The authors evaluated the intervention using quantitative research methods, including usage tracking, surveys of faculty and students, and a randomized pretest-posttest study. RESULTS: Over 46 weeks, 95 medical students used the tutorials 544 times, for an overall average of 1.7 times a day. The median time spent on completed tutorials was 11 minutes (average [SD], 14 [+/-12] minutes). Seventy-four students completed the randomized study. They completed 65% of the assigned tutorials, resulting in improved examination scores compared with the control (effect size, 0.39; 95% confidence interval = 0.15 to 0.62). Students were positively disposed to the tutorials, ranking them as "valuable." Fifty-four percent preferred the tutorials to small group teaching sessions with a preceptor. The faculty was also positive about the tutorials, although they did not appear to integrate the tutorials directly into their teaching. CONCLUSIONS: Medical students on rotation in a busy clinical setting can and will use appropriately presented computer tutorials. The tutorials are effective in raising examination scores.

Workplace injury management: using new technology to deliver and evaluate physician continuing medical education.

OBJECTIVE: Physicians typically receive little continuing medical education (CME) about their role in workplace injury management as well as on workplace injuries and disease. Although new technologies may help educate physicians in these areas, careful evaluation is required, given the understudied nature of these interventions. The objective of this study is to evaluate two promising new technologies to deliver CME (online learning and videoconferencing) and to compare the effectiveness of these delivery methods to traditional CME interventions (large urban traditional conference lectures and small group local face-to-face outreach) in their impact on physician knowledge related to workplace injury management. METHODS: This study utilized a prospective, controlled evaluation of two educational programs for BC physicians: 1) The Diagnosis and Management of Lateral Epicondylitis; and 2) Is Return-to-Work Good Medicine? Each educational module was delivered in each of four ways (Outreach Visit, Videoconference Session, Conference Lecture, Online) and physicians self-selected their participation--both in terms of topic and delivery method. Questionnaires related to knowledge as well as learner attitude and satisfaction were administered prior (pre-test) and following (post-test) all educational sessions. RESULTS: 581 physician encounters occurred as a result of the educational interventions and a significant percentage of the physicians participated in the research per se (i.e. there were 358 completed sets of pre-test and post-test 'Knowledge' questionnaires). Overall the results showed that the developed training programs increased physicians' knowledge of both Lateral Epicondylitis and the physician's role in Return-To-Work planning as reflected in improved post-test performance when compared to pre-test scores. Furthermore, videoconferencing and online training were at least as effective as conference lectures and instructor-led small group outreach sessions in their impact on physician knowledge. CONCLUSIONS: Use of effective videoconferencing and online learning activities will increase physician access to quality CME related to workplace injury management and will overcome access barriers intrinsic to types of CME interventions based on instructor-student face-to-face interactions.

Assessing cognitive & motor performance in minimally invasive surgery (MIS) for training & tool design.

The objective of this paper is to present the development of a new modelling diagram (MCMD) to represent MIS procedures in terms of both motor and cognitive actions. Through observation and analysis of several laparoscopic cholecystectomy procedures and based on task analysis techniques, we created a diagram language composed of six primary symbols: processes, decisions, interrupt service routines (ISRs), options points and AND and OR gates. We then tested and refined them during 10 new cases until no further changes seemed necessary, we have since applied this approach to 6 laparoscopic colorectal surgeries and have found that no further symbols were necessary though the procedural representation was naturally different. This modelling diagram allowed us to represent both cognitive and motor performance aspects of surgical procedures in a unified framework and will in future allow us to assess motor performance on particular surgical tasks at particular points in the procedure (i.e., the surgical context).

Medical student selection: choice of a semi-structured panel interview or an unstructured one-on-one interview.

Reliability has been shown to be higher in structured medical admissions interviews as compared to unstructured interviews. This study reports the comparison of a proposed semi-structured panel interview with a current individual unstructured medical admissions interview. Inter-rater reliability coefficients were calculated, and correlations were estimated between panel, individual and academic scores. Admission status in 2003 was related to these scores by means of logistic regression. Both individual and panel interviews were significantly correlated with admissions status. The inter rater reliability coefficient (from individual interviews) was 0.12 whereas the interpanel reliability coefficient was 0.52. Panel interview: good across panel and within panel consistency of scoring. No effect of who asked the questions, question order, or interview duration on scoring. No correlation between panel interview scores and academic variables (MCAT, GPA). We found good inter-panel reliability, a high consistency within and between interview panels, and uniformly positive questionnaire responses. The panel interview measures something different from academic variables. These data, in conjunction with a strong sense from the medical and psychological literature supporting the reliability and validity of a semi-structured panel interview, support our decision to replace our individual interview with the panel interview.

Differences in cardiovascular risks in the aboriginal and non-aboriginal people living in Bella Coola, British Columbia.

BACKGROUND: There is a lack of information regarding cardiovascular disease (CVD) in rural areas in British Columbia, Canada. To establish the relative prevalence of CVD risk factors in a rural area weanalyzed biochemical and clinical data collected on aboriginal and non-aboriginal descent people living in the community of Bella Coola. MATERIAL/METHODS: A retrospective review of the 2378 charts located in the Bella Coola Medical Clinic was done in the Fall of 2002. Anthropometric, biochemical and clinical information was obtained from 1,120 people of Nuxalk descent and 1258 people of mainly European descent. Children and adults of either sex, pre- and post-menopausal women, and patients with hyperlipidaemia, type 2 diabetes, and hypertension were included in the review. RESULTS: After adjusting for age, gender and BMI, non-aboriginal origin was associated with higher total and LDL-cholesterol (p<0.01) and aboriginal origin was associated with higher glucose levels (p < 0.0Ol). A significantly higher value for Log (TG/HDL), a plasma parameter defined as "Atherogenic index plasma (AIP)" was found in the aboriginal group. In addition, both aboriginal women and men had higher prevalence of metabolic syndrome than the non-aboriginal men and women. CONCLUSIONS: This study found different patterns of CVD risk factors i n non-aboriginal and aboriginalgroups living in the same geographic area. Based on the prevalence of metabolic syndrome, higher triglycerides and AIP, aboriginal participants, especially the women, are at a higher risk for CVD.

Randomized trial of problem-based versus didactic seminars for disseminating evidence-based guidelines on asthma management to primary care physicians.

INTRODUCTION: This randomized controlled trial (RCT) investigated the effectiveness of and satisfaction with small-group problem-based learning (PBL) versus a didactic lecture approach to guideline dissemination in asthma management controlling for confounders common in comparative educational interventions. METHODS: Sites were selected as either lecture or PBL using simple randomization. All participants were exposed to similar educational resources to ensure treatment equivalency. Instruments included standardized program/speaker evaluation forms and a validated case-based questionnaire with a visual analogue scale measuring the level of confidence of responses. The latter was presented immediately pre- and post-intervention and 3 months later. The statistician was blinded to intervention groups. RESULTS: Overall, 52 family physicians agreed to participate, 23 in the PBL sessions (mean 4.6 per group) and 29 in the didactic lecture sessions (mean 7.25). There was no significant difference between the groups with respect to the knowledge gained at each test administration. Participants rated the lecturer or facilitator equally well as having established a positive learning environment. PBL participants rated the perceived educational value of the program higher than did lecture participants (4.36 vs. 3.93; p = .04). Both groups experienced a significant increase in asthma-related knowledge post-intervention. Attrition rates for the 3-month post-test were 14% for PBL participants versus 32% for lecture-based participants. DISCUSSION: PBL was as effective in knowledge uptake and retention as lecture-based continuing medical education (CME) programs. Further study is warranted to investigate whether the assessment of higher educational value or an increase in response rate to delayed testing is replicable in other RCTs addressing common confounders and if these factors influence future CME participation, changes in physician clinical behavior, or patient health outcomes.