The limited use of instructional design guidelines in healthcare simulation scenarios: an expert appraisal.

BACKGROUND: Systematic reviews on simulation training effectiveness have pointed to the need to adhere to evidence-based instructional design (ID) guidelines. ID guidelines derive from sound cognitive theories and aim to optimize complex learning (integration of knowledge, skills, and attitudes) and learning transfer (application of acquired knowledge and skills in the workplace). The purpose of this study was to explore adherence to ID guidelines in simulation training programs for dealing with postpartum hemorrhage (PPH), a high-risk situation and the leading cause of maternal mortality worldwide. METHODS: A total of 40 raters analyzed simulation training programs as described in 32 articles. The articles were divided into four subsets of seven articles and one subset of four articles. Each subset was judged by seven to ten raters on adherence to ID guidelines. The 5-point Likert score rating scale was based on Merrill's First Principles of Instruction and included items relating to key ID features categorized into five subscales: authenticity, activation of prior knowledge, demonstration, application, and integration/transfer. The authors searched for articles published in English between January 2007 and March 2017 in PubMed, Eric, and Google Scholar and calculated the mean Likert-scale score, per subscale, and interrater reliability (IRR). RESULTS: The mean Likert-scale scores calculated for all subscales were < 3.00. For the number of raters used to judge the papers in this study (varying between 7 and 10), the IRR was found to be excellent for the authenticity and integration/transfer subscales, good-to-excellent for the activation of prior knowledge and application subscales, and fair-to-good for the demonstration subscale. CONCLUSION: The results demonstrate a paucity of the description of adherence to evidence-based ID guidelines in current simulation trainings for a high-risk situation such as PPH.

Effects of an in situ instructional design based postpartum hemorrhage simulation training on patient outcomes: an uncontrolled before-and-after study.

Objective: To compare postpartum hemorrhage (PPH) patient outcomes before and after an in situ instructional design based PPH simulation attended by obstetrics and gynecology (OBGYN) residents.Methods: This uncontrolled before-and-after study was conducted in Recife, Brazil including all 1388 women delivering from June to August 2012 and all 1357 delivering from June to August 2013. The 36 OBGYN residents were divided into13 teams of two or three participants and were trained through ID based PPH simulation training with the following eight steps: (1) prior knowledge activation, (2) video demonstration, (3) dual-coding PPH protocol discussion-an image association during the training, (4) training scenario # 1, (5) debriefing, (6) training scenario # 2 with immediate feedback, (7) training scenario # 3, and (8) debriefing with self-assessment. The training scenarios had an increasing level of complexity. The main goal of the training was the adequate management of PPH and situational awareness improvement-the ability to anticipate, recognize, and intercept unfolding error chains. The primary patient outcomes rates used for the before and after comparison were therapeutic uterotonics use within 24 h of birth and blood transfusion. Secondary outcomes were therapeutic oxytocin mean dosage IU within 24 h of birth, postpartum Hb < 6 g/dL, among others. Chi-square test was used for categorical variables comparison and independent t-test for continuous variables.Results: PPH rates were 100 (7.2% of 2012 deliveries) and 80 cases (5.9% of 2013 deliveries), respectively. Comparison of primary post- and pre-simulation outcomes revealed no significant differences. However, in the comparison for therapeutic oxytocin mean dosage IU within 24 h of birth, there was an increase found after the simulation (15.98 ± 7.4 versus 25.1 ± 12.3; p < .001). For all other outcome measures, there were no statistical differences.Conclusions: In situ ID based PPH simulation leads to an increase in the mean dosage of oxytocin after training, in selected cases. This may indicate better situational awareness when managing women with PPH.

High enthusiasm about long lasting mentoring relationships and older mentors.

BACKGROUND: Mentoring plays a pivotal role in workplace-based learning, especially in the medical realm. Organising a formal mentoring programme can be labor and time intensive and generally impractical in resource constrained medical schools with limited numbers of mentors. Hence, informal mentoring offers a valuable alternative, but will be more likely to be effective when mentors and protégés share similar views. It is therefore important to gain more insight into factors influencing perceptions of informal mentoring. This study aims to explore mentors and protégés' perceptions of informal mentoring and how these vary (or not) with gender, age and the duration of the relationship. METHOD: We administered an Informal Mentor Role Instrument (IMRI) to medical practitioners and academics from Egypt, Pakistan and Saudi Arabia. The questionnaire was developed for the study from other validated instruments. It contained 39 items grouped into 7 domains: acceptance, counselling, friendship, parenting, psychological support, role modelling and sociability. RESULTS: A total of 103 mentors and 91 protégés completed the IMRI. Mentors had a better appreciation for the interpersonal aspects of informal mentoring than protégés, especially regarding acceptance, counselling and friendship. Moreover, being older and engaged in a longer mentoring relationship contributed to more positive perceptions of interpersonal aspects of mentoring, regardless of one's role (mentor or protégé). CONCLUSION: It can be concluded that the expectations of mentors and protégés differed regarding the content and aim of the interpersonal characteristics of their mentoring relationship. We recommend mentors and protégés to more explicitly exchange their expectations of the informal mentoring relationship, as typically practiced in formal mentoring. Additionally, in our study, seniority and lasting relationships seem crucial for good informal mentoring. It appears beneficial to foster lasting informal mentoring relationships and to give more guidance to younger mentors.

Exploring medical residents' perceived need for negotiation skills training.

OBJECTIVES: This study explores the optimal focus for negotiation skills development training by investigating how often medical residents negotiate in practice, and how they perceive the effectiveness of their negotiation capabilities. METHODS: An exploratory study was performed using a questionnaire regarding the medical residents' working environment, negotiation frequency, knowledge and skills using a 5-point Likert scale, multiple choice questions and open questions. Exploratory factor analysis with principal component analysis, varimax rotation, reliability analysis, and content analysis were used to reduce the number of variables. Descriptive and interferential statistics and multiple regression analysis were used to analyze the data. RESULTS: We analyzed the responses of 60 medical residents. The findings showed that the perceived development of their negotiation knowledge (M=3.06, SD=0.83) was less than their negotiation skills (M=3.69, SD=0.47). Their attitude during negotiations, especially females, differed substantially in the interactions with nurses than with their supervisors. Medical residents with more working experience, better negotiation skills or who worked in hierarchical environments negotiated more frequently with their supervisors. Medical residents with better collaboration skills and negotiation knowledge demonstrated better negotiation skills. CONCLUSIONS: This study underlines medical residents' need for negotiation training. In addition to the basic negotiation knowledge and skills, training programs in negotiation should focus on the medical residents' awareness of their attitudes during negotiations, combining the assertiveness shown in interactions with supervisors with the empathy and emotional engagement present in interactions with nurses.  Furthermore, attention should be paid to the influence of the environmental hierarchy on negotiation skill development.

Does selection pay off? A cost-benefit comparison of medical school selection and lottery systems.

CONTEXT: Resources for medical education are becoming more constrained, whereas accountability in medical education is increasing. In this constrictive environment, medical schools need to consider and justify their selection procedures in terms of costs and benefits. To date, there have been no studies focusing on this aspect of selection. OBJECTIVES: We aimed to examine and compare the costs and benefits of two different approaches to admission into medical school: a tailored, multimethod selection process versus a lottery procedure. Our goal was to assess the relative effectiveness of each approach and to compare these in terms of benefits and costs from the perspective of the medical school. METHODS: The study was conducted at Maastricht University Medical School, at which the selection process and a weighted lottery procedure ran in parallel for 3 years (2011-2013). The costs and benefits of the selection process were compared with those of the lottery procedure over three student cohorts throughout the Bachelor's programme. The extra costs of selection represented the monetary investment of the medical school in conducting the selection procedure; the benefits were derived from the increase in income generated by the prevention of dropout and the reductions in extra costs facilitated by decreases in the repetition of blocks and objective structured clinical examinations. RESULTS: The tailor-made selection procedure cost about €139 000 when extrapolated to a full cohort of students (n = 286). The lottery procedure came with negligible costs for the medical school. However, the average benefits of selection compared with the lottery system added up to almost €207 000. CONCLUSIONS: This study not only shows that conducting a cost-benefit comparison is feasible in the context of selection for medical school, but also that an 'expensive' selection process can be cost-beneficial in comparison with an 'inexpensive' lottery system. We encourage other medical schools to examine the cost-effectiveness of their own selection processes in relation to student outcomes in order to extend knowledge on this important topic.

Selection into medicine: the predictive validity of an outcome-based procedure.

BACKGROUND: Medical schools must select students from a large pool of well-qualified applicants. A challenging issue set forward in the broader literature is that of which cognitive and (inter)personal qualities should be measured to predict diverse later performance. To address this gap, we designed a 'backward chaining' approach to selection, based on the competences of a 'good doctor'. Our aim was to examine if this outcome-based selection procedure was predictive of study success in a medical bachelor program. METHODS: We designed a multi-tool selection procedure, blueprinted to the CanMEDS competency framework. The relationship between performance at selection and later study success across a three-year bachelor program was examined in three cohorts. Study results were compared between selection-positive and selection-negative (i.e. primarily rejected) students. RESULTS: Selection-positive students outperformed their selection-negative counterparts throughout the entire bachelor program on assessments measuring cognitive (e.g. written exams), (inter)personal and combined outcomes (i.e. OSCEs). Of the 30 outcome variables, selection-positive students scored significantly higher in 11 cases. Fifteen other, non-significant between-group differences were also in favor of the selection-positives. An overall comparison using a sign test indicated a significant difference between both groups (p < 0.001), despite equal pre-university GPAs. CONCLUSIONS: The use of an outcome-based selection approach seems to address some of the predictive validity limitations of commonly-used selection tools. Selection-positive students significantly outperformed their selection-negative counterparts across a range of cognitive, (inter)personal, and mixed outcomes throughout the entire three-year bachelor in medicine.

Development and validation of the TOCO-TURBT tool: a summative assessment tool that measures surgical competency in transurethral resection of bladder tumour.

BACKGROUND: The current shift towards competency-based residency training has increased the need for objective assessment of skills. In this study, we developed and validated an assessment tool that measures technical and non-technical competency in transurethral resection of bladder tumour (TURBT). METHODS: The 'Test Objective Competency' (TOCO)-TURBT tool was designed by means of cognitive task analysis (CTA), which included expert consensus. The tool consists of 51 items, divided into 3 phases: preparatory (n = 15), procedural (n = 21), and completion (n = 15). For validation of the TOCO-TURBT tool, 2 TURBT procedures were performed and videotaped by 25 urologists and 51 residents in a simulated setting. The participants' degree of competence was assessed by a panel of eight independent expert urologists using the TOCO-TURBT tool. Each procedure was assessed by two raters. Feasibility, acceptability and content validity were evaluated by means of a quantitative cross-sectional survey. Regression analyses were performed to assess the strength of the relation between experience and test scores (construct validity). Reliability was analysed by generalizability theory. RESULTS: The majority of assessors and urologists indicated the TOCO-TURBT tool to be a valid assessment of competency and would support the implementation of the TOCO-TURBT assessment as a certification method for residents. Construct validity was clearly established for all outcome measures of the procedural phase (all r > 0.5, p < 0.01). Generalizability-theory analysis showed high reliability (coefficient Phi ≥ 0.8) when using the format of two assessors and two cases. CONCLUSIONS: This study provides first evidence that the TOCO-TURBT tool is a feasible, valid and reliable assessment tool for measuring competency in TURBT. The tool has the potential to be used for future certification of competencies for residents and urologists. The methodology of CTA might be valuable in the development of assessment tools in other areas of clinical practice.

Connected, attracted, and concerned: A Q study on medical crossborder curriculum partnerships.

INTRODUCTION: A new form of internationalization has been trending upward in the medical education realm: crossborder medical curriculum partnerships established to deliver the same, or adapted, curriculum to groups of geographically separated students. This study aims to investigate crossborder medical curriculum partnerships by exploring the experiences of teachers at the recipient institution who have a key role in delivering the program. METHODS: From four pioneering recipient medical schools, 24 teachers participated in a Q-sort study. Each participant rank-ordered 42 statements about teaching in a crossborder medical curriculum on a scale from -5 (indicating strong disagreement) to +5 (indicating strong agreement). The authors conducted a "by-person" factor analysis to uncover distinct patterns in the ranking of statements, using the statistical results and participants' comments about their Q sorts to interpret these patterns and translate them into distinct viewpoints. RESULTS: Three viewpoints emerged, reflecting: (1) a feeling of connectedness with the partner institution, trust in the quality of the curriculum, and appreciation of interinstitutional relationships; (2) the partnership's attractiveness because of the career opportunities it offers; and (3) concerns over the quality of graduates because of doubts about the appropriateness of the didactic model and insufficient attention to local healthcare needs, and over the practical feasibility of such partnerships. CONCLUSIONS: The three viewpoints identified revealed a pallet of views on how host teachers might experience their work. It shows the heterogeneous features of this group and seems to counterbalance reports that they are feeling "deprived" from their role as teacher. Two viewpoints featured an appreciation of interinstitutional relationships and of the partnership, especially when perceiving a degree of autonomy. Partners can capitalize on all different viewpoints by deploying procedure and policies to raise the quality of education delivery.

Cross-validation of a learning climate instrument in a non-western postgraduate clinical environment.

BACKGROUND: In postgraduate training, there is a need to continuously assess the learning and working conditions to optimize learning. Students or trainees respond to the learning climate as they perceive it. The Dutch Residency Educational Climate Test (D-RECT) is a learning climate measurement tool with well-substantiated validity. However, it was originally designed for Dutch postgraduate trainees and it remains to be shown whether extrapolation to non-Western settings is viable. The dual objective of this study was to revalidate D-RECT outside of a Western setting and to evaluate the factor structure of a recently revised version of the D-RECT containing 35 items. METHODS: We invited Filipino internal medicine residents from 96 hospitals to complete the revised 35-item D-RECT. Subsequently, we performed a confirmatory factor analysis to check the fit of the 9 scale model of the revised 35-item D-RECT. Inter-rater reliability was assessed using generalizability theory. RESULTS: Confirmatory factor analysis unveiled that the factor structure of the revised 35-item D-RECT provided a reasonable fit to the Filipino data, after removal of 7 items. Five to seven evaluations of individual residents were needed per scale to obtain a reliable result. CONCLUSION: Even in a non-Western setting, the D-RECT exhibited psychometric validity. This study validated the factor structure of the revised 35-item D-RECT after some modifications. We recommend that its application be extended to other Asian countries and specialties.

Selection and ranking of patient video cases in paediatric neurology in relation to learner levels.

BACKGROUND: Teaching and learning with patient video cases may add authenticity, enhance diagnostic accuracy and improve chances of early diagnosis. The aim of this study is firstly to identify selection criteria for key Patient video cases (PVCs), secondly to identify trends in relevance of PVCs for learner levels and thirdly, to rank PVCs for learner levels. METHODS: Based on a literature review, we identified criteria for key PVCs for use in paediatric neurology. We then performed a multi-round Delphi analysis to obtain agreement between 28 expert clinician teachers concerning key PVCs for four learner levels. RESULTS: We identified two major criteria: key PVCs should demonstrate key movements, and these movements should be subtle and/or difficult to note. The expert clinician teachers subsequently assessed a list of 14 topics for key PVCs. We found a clear, increasing trend in relevance scores, from medical students to young residents to experienced residents and specialists. For medical students and residents, epileptic spasms, Down syndrome, developmental delay, cerebral palsy and absence epilepsy were highly ranked. For specialists, conditions like chorea, focal seizures or eye movement disorders topped the ranking list, although ranking was less clear for this group of advanced learners. DISCUSSION AND CONCLUSION: Key PVCs should demonstrate movements that are difficult to note for learners. Ranked lists of key PVCs for teaching and learning at different learner levels are now available and may help institutions build validated local libraries of PVCs.

Stakeholders' perceptions on competency and assessment program of entry-level pharmacists in developing countries.

OBJECTIVES: To assess the stakeholders' perceptions on the competency of entry-level pharmacists and the use of written licensure examination as the primary assessment for licensure decisions on entry-level pharmacists who have completed the Pharmacy Internship Program1 (PIP) in developing countries. METHOD: A cross-sectional survey was conducted among stakeholders in which they completed a web-based 21-item pre-tested questionnaire to determine their views regarding the competency outcomes and assessment program for entry-level pharmacist. RESULTS: The stakeholders rated the entry-level pharmacists to possess all competencies except research skills. Stakeholders suggested improvement of the program by defining the competency framework and training preceptors. However, stakeholders disagree on using written examination as the primary assessment for licensure decision and suggested the incorporation of other performance-based assessments like preceptor's assessment reports. CONCLUSION: Stakeholders are uncertain on entry-level pharmacists in developing countries possessing adequate research competencies and think their assessment program for licensure need more than written examination to assess all required competencies.

Toward a better judgment of item relevance in progress testing.

BACKGROUND: Items must be relevant to ensure item quality and test validity. Since "item relevance" has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement. METHODS: Members of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group. RESULTS: The proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p < 0.001). Agreement between the RC and STA was high, with an ICC of >0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p < 0.001), and did not differ between RC and STA. To achieve an acceptable inter-rater reliability for relevance and inclusion, 6 members must serve on the RC. CONCLUSIONS: Use of the rubric results in a stricter evaluation of items' appropriateness for inclusion in the PT and facilitates agreement between the RC and other stakeholders. Hence, it may help increase the acceptability and validity of the PT.

Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions.

BACKGROUND: It remains unclear which item format would best suit the assessment of clinical reasoning: context-rich single best answer questions (crSBAs) or key-feature problems (KFPs). This study compared KFPs and crSBAs with respect to students' acceptance, their educational impact, and psychometric characteristics when used in a summative end-of-clinical-clerkship pediatric exam. METHODS: Fifth-year medical students (n = 377) took a computer-based exam that included 6-9 KFPs and 9-20 crSBAs which assessed their clinical reasoning skills, in addition to an objective structured clinical exam (OSCE) that assessed their clinical skills. Each KFP consisted of a case vignette and three key features using a "long-menu" question format. We explored students' perceptions of the KFPs and crSBAs in eight focus groups and analyzed statistical data of 11 exams. RESULTS: Compared to crSBAs, KFPs were perceived as more realistic and difficult, providing a greater stimulus for the intense study of clinical reasoning, and were generally well accepted. The statistical analysis revealed no difference in difficulty, but KFPs resulted more reliable and efficient than crSBAs. The correlation between the two formats was high, while KFPs correlated more closely with the OSCE score. CONCLUSIONS: KFPs with long-menu exams seem to bring about a positive educational effect without psychometric drawbacks.

Stakeholder perspectives on workplace-based performance assessment: towards a better understanding of assessor behaviour.

Workplace-Based Assessment (WBA) plays a pivotal role in present-day competency-based medical curricula. Validity in WBA mainly depends on how stakeholders (e.g. clinical supervisors and learners) use the assessments-rather than on the intrinsic qualities of instruments and methods. Current research on assessment in clinical contexts seems to imply that variable behaviours during performance assessment of both assessors and learners may well reflect their respective beliefs and perspectives towards WBA. We therefore performed a Q methodological study to explore perspectives underlying stakeholders' behaviours in WBA in a postgraduate medical training program. Five different perspectives on performance assessment were extracted: Agency, Mutuality, Objectivity, Adaptivity and Accountability. These perspectives reflect both differences and similarities in stakeholder perceptions and preferences regarding the utility of WBA. In comparing and contrasting the various perspectives, we identified two key areas of disagreement, specifically 'the locus of regulation of learning' (i.e., self-regulated versus externally regulated learning) and 'the extent to which assessment should be standardised' (i.e., tailored versus standardised assessment). Differing perspectives may variously affect stakeholders' acceptance, use-and, consequently, the effectiveness-of assessment programmes. Continuous interaction between all stakeholders is essential to monitor, adapt and improve assessment practices and to stimulate the development of a shared mental model. Better understanding of underlying stakeholder perspectives could be an important step in bridging the gap between psychometric and socio-constructivist approaches in WBA.

The use of instructional design guidelines to increase effectiveness of postpartum hemorrhage simulation training.

OBJECTIVE: To compare learning outcomes of postpartum hemorrhage simulation training based on either instructional design guidelines or best practice. METHODS: A pretest-post-test non-equivalent groups study was conducted among obstetrics and gynecology residents in Recife, Brazil, from June 8 to August 30, 2013. The instructional design group included 13 teams, whereas the best practice group included seven teams. A standardized task checklist was used for scenario analysis and the proportion of correctly executed tasks compared (post-test minus pretest). RESULTS: The instructional design group scored higher than the best practice group for total number of tasks completed (median difference 0.46 vs 0.17; P<0.001; effect size [r]=0.72). Similar results were observed for communication (median difference 0.56 vs 0.22; P=0.004; r=0.58), laboratory evaluation (median difference 0.83 vs 0.00; P<0.001; r=0.76), and mechanical management (median difference 0.25 vs -0.15; P=0.048; r=0.39). Speed of learning was also increased. The median differences were 0.20 for the instructional design group compared with 0.05 for the best practice group at 60 seconds (P=0.015; r=0.49), and 0.49 versus 0.26 (P=0.001; r=0.65) at 360 seconds. CONCLUSION: The use of simulation training for postpartum hemorrhage that was based on instructional design guidelines yielded better learning outcomes than did training based on best practice.

High educational impact of a national simulation-based urological curriculum including technical and non-technical skills.

BACKGROUND: Although simulation training is increasingly used to meet modern technology and patient safety demands, its successful integration within surgical curricula is still rare. The Dutch Urological Practical Skills (D-UPS) curriculum provides modular simulation-based training of technical and non-technical basic urological skills in the local hospital setting. This study aims to assess the educational impact of implementing the D-UPS curriculum in the Netherlands and to provide focus points for improvement of the D-UPS curriculum according to the participants. METHODS: Educational impact was assessed by means of qualitative individual module-specific feedback and a quantitative cross-sectional survey among residents and supervisors. Twenty out of 26 Dutch teaching hospitals participated. The survey focussed on practical aspects, the D-UPS curriculum in general, and the impact of the D-UPS curriculum on the development of technical and non-technical skills. RESULTS: A considerable survey response of 95 % for residents and 76 % for supervisors was obtained. Modules were attended by junior and senior residents, supervised by a urologist, and peer teaching was used. Ninety percent of supervisors versus 67 % of residents judged the D-UPS curriculum as an important addition to current residency training (p = 0.007). Participants' aggregated general judgement of the modules showed a substantial percentage favorable score (M ± SE: 57 ± 4 %). The impact of training on, e.g., knowledge of materials/equipment and ability to anticipate on complications was high, especially for junior residents (77 ± 5 and 71 ± 7 %, respectively). Focus points for improvement of the D-UPS curriculum according to the participants include adaptation of the training level to residents' level of experience and focus on logistics. CONCLUSION: The simulation-based D-UPS curriculum has a high educational impact. Residents and supervisors consider the curriculum to be an important addition to current residency training. Focus points for improvement of the D-UPS curriculum according to the participants include increased attention to logistics and integration of a spiral learning approach.

Embedding of the progress test in an assessment program designed according to the principles of programmatic assessment.

BACKGROUND: Progress tests (PT) are used to assess students on topics from all medical disciplines. Progress testing is usually one of the assessment methods of the cognitive domain. There is limited knowledge on how positioning of the PT in a program of assessment (PoA) influences students' PT scores, use of PT feedback and perceived learning value. METHODS: We compared PT total scores and use of a PT test feedback (ProF) system in two medical courses, where the PT is either used as a summative assessment or embedded in a comprehensive PoA and used formatively. In addition, an interview study was used to explore the students' perception on use of PT feedback and perceived learning value. RESULTS: PT total scores were higher, with considerable effect sizes (ESs) and students made more use of ProF when the PT was embedded in a comprehensive PoA. Analysis of feedback in the portfolio stimulated students to look for patterns in PT results, link the PT to other assessment results, follow-up on learning objectives, and integrate the PT in their learning for the entire PoA. CONCLUSIONS: Embedding the PT in an assessment program designed according to the principles of programmatic assessment positively affects PT total scores, use of PT feedback, and perceived learning value.

A first report of East Asian students' perception of progress testing: a focus group study.

BACKGROUND: Progress testing (PT) is used in Western countries to evaluate students' level of functional knowledge, and to enhance meaning-oriented and self-directed learning. However, the use of PT has not been investigated in East Asia, where reproduction-oriented and teacher-centered learning styles prevail. Here, we explored the applicability of PT by focusing on student perceptions. METHODS: Twenty-four students from Years 2, 3, and 5 at Jichi Medical University in Japan attended a pilot PT session preceded by a brief introduction of its concept and procedures. Variations in obtained test scores were analyzed by year, and student perceptions of PT were explored using focus groups. RESULTS: Formula scores (mean ± standard deviation) in Years 2, 3, and 5 were 12.63 ± 3.53, 35.88 ± 14.53, and 71.00 ± 18.31, respectively. Qualitative descriptive analysis of focus group data showed that students disfavored testing of medical knowledge without tangible goals, but instead favored repetitive assessment of knowledge that had been learned and was tested on a unit basis in the past in order to achieve deep learning. Further, students of all school years considered that post-test explanatory lectures by teachers were necessary. CONCLUSIONS: East Asian students' perceptions indicated that, in addition to their intensive memorization within narrow test domains compartmentalized by end-of-unit tests, the concept of PT was suitable for repetitive memorization, as it helped them to integrate their knowledge and to increase their understanding. Post-test explanatory lectures might lessen their dislike of the intangible goals of PT, but at the expense of delaying the development of self-directed learning. Key issues for the optimization of PT in East Asia may include administration of PT after completed end-of-unit tests and a gradual change in feedback methodology over school years from test-oriented post-test lectures to the provision of literature references only, as a means of enhancing test self-review and self-directed learning.

Content validity of workplace-based portfolios: A multi-centre study.

BACKGROUND: Portfolios are used as tools to coach and assess students in the workplace. This study sought to evaluate the content validity of portfolios as reflected in their capacity to adequately assess achieved competences of medical students during clerkships. METHODS: We reviewed 120 workplace portfolios at three medical universities (Belgium and the Netherlands). To validate their content, we developed a Validity Inventory for Portfolio Assessment (VIPA) based on the CanMEDS roles. Two raters evaluated each portfolio and indicated for each VIPA item whether the portfolio provided sufficient information to enable satisfactory assessment of the item. We ran a descriptive analysis on the validation data and computed Cohen's Kappa to investigate interrater agreement. RESULTS: The portfolios adequately covered the items pertaining to the communicator (90%) and professional (87%) roles. Coverage of the medical expert, collaborator, scholar and manager roles ranged between 75% and 85%. The health advocate role, covering 59%, was clearly less well represented. This role also exhibited little interrater agreement (Kappa < 0.4). CONCLUSIONS: This study lends further credence to the evidence that portfolios can indeed adequately assess the different CanMEDS roles during clerkships, the health advocate role, which was less well represented in the portfolio content, excepted.

The Simbla TURBT Simulator in Urological Residency Training: From Needs Analysis to Validation.

OBJECTIVE: To investigate the value of the physical Simbla Transurethral Resection of a Bladder Tumor (TURBT) simulator as an educational tool within urological residency training, by means of a training needs analysis (TNA) and assessment of its feasibility, acceptability, and face, content, and construct validity. METHODS: To analyze the training needs for TURBT, procedural steps and pitfalls were identified and the TNA was completed during an expert consensus meeting. Participants (n = 76) were divided into three groups based on their experience in TURBT: novices, intermediates, and experts. Participants performed two standardized TURBT procedures on the simulator. Face validity and content validity, as well as feasibility and acceptability, were assessed with a quantitative survey. Construct validity was assessed by comparing the performance of novices, intermediates, and experts on resection time, quality of tumor resection, and overall performance. RESULTS: Of the 21 procedural steps and 17 pitfalls defined in TNA, 13 steps and 8 pitfalls were covered by the Simbla. Participants rated the Simbla's overall realism (face validity) with a score of 8 of 10 (range 6-9). The simulator was judged to be most useful (content validity) for learning eye-hand coordination: score 8 (6-10). All aspects regarding realism and usefulness were rated above the acceptability threshold of 6/10. Intermediates (100%) and experts (96%) considered the Simbla to be a useful educational tool within the urological curriculum. Resection time was longer for novices than for experts (p < 0.05; construct validity). In addition, the overall performance of novices was rated lower compared with intermediates and experts, and novices showed more irradical resections and bladder perforations (all p < 0.05). CONCLUSIONS: The Simbla TURBT simulator is a valid, feasible, and acceptable educational tool for training procedural skills and may be implemented in the urological curriculum to complement learning in clinical practice. TNA is valuable in defining training objectives and evaluating the educational value of a simulator.