A Grant-Based Experiment to Train Clinical Investigators: The AACR/ASCO Methods in Clinical Cancer Research Workshop.

To address the need for clinical investigators in oncology, American Association for Cancer Research (AACR) and American Society for Clinical Oncology (ASCO) established the Methods in Clinical Cancer Research Workshop (MCCRW). The workshop's objectives were to: (i) provide training in the methods, design, and conduct of clinical trials; (ii) ensure that clinical trials met federal and international ethical guidelines; (iii) evaluate the effectiveness of the workshop; and (iv) create networking opportunities for young investigators with mentoring senior faculty. Educational methods included: (i) didactic lectures, (ii) Small Group Discussion Sessions, (iii) Protocol Development Groups, and (iv) one-on-one mentoring. Learning focused on the development of an Institutional Review Board (IRB)-ready protocol, which was submitted on the last day of the workshop. Evaluation methods included: (i) pre- and postworkshop tests, (ii) students' workshop evaluations, (iii) faculty's ratings of protocol development, (iv) students' productivity in clinical research after the workshop, and (v) an independent assessment of the workshop. From 1996 to 2014, 1,932 students from diverse backgrounds attended the workshop. There was a significant improvement in the students' level of knowledge from the pre- to the postworkshop exams (P < 0.001). Across the classes, student evaluations were very favorable. At the end of the workshop, faculty rated 92% to 100% of the students' protocols as ready for IRB submission. Intermediate and long-term follow-ups indicated that more than 92% of students were actively involved in patient-related research, and 66% had implemented five or more protocols. This NCI-sponsored MCCRW has had a major impact on the training of clinicians in their ability to design and implement clinical trials in cancer research.

Effects of a patient's name and image on medical knowledge acquisition.

PURPOSE: To assess whether there are differences in medical students' (MS) knowledge acquisition after being provided a virtual patient (VP) case summary with a patient's name and facial picture included compared to no patient's name or image. METHOD: 76 MS from four clerkship blocks participated. Blocks one and three (Treatment group) were provided case materials containing the patient's name and facial picture while blocks two and four (Control group) were provided similar materials without the patient's name or image. Knowledge acquisition was evaluated with a multiple-choice-question examination (CQA_K). RESULTS: Treatment group CQA_K scores were 64.6% (block one, n = 18) and 76.0% (block three, n = 22). Control group scores were 71.7%, (block two, n = 17) and 68.4% (block four, n = 19). ANOVA F-test among the four block mean scores was not significant; F (3, 72) = 1.68, p = 0.18, η2=0.07. Only 22.2% and 27.3% of the MS from blocks one and three respectively correctly recalled the patient's name while 16.7% and 40.9% recalled the correct final diagnosis of the patient. CONCLUSIONS: These results suggest that including a patient's name and facial picture on reading materials may not improve MS knowledge acquisition. Corroborating studies should be performed before applying these results to the design of instructional materials.

Reporting school data on the dental licensure examination.

Beginning in 2012, candidate performance on the National Board Dental Examination (NBDE) has been reported as pass/fail, and only the failing candidates receive numerical scores for remediation purposes. The Joint Commission on National Dental Examinations recognizes that the numerical scores have been important information to dental schools for curriculum evaluation and that the pass/fail reports do not provide meaningful information to the schools. This article describes the process of evaluating and validating a new model for reporting standardized school-level performance data on the NBDE. Under this new model, schools are able to monitor the overall performance of their students compared to a national cohort.

Constructing licensure exams: a reliability study of case-based questions on the National Board Dental Hygiene Examination.

Patient cases with associated questions are a method for increasing the clinical relevance of licensure exams. This study used generalizability theory to assess changes in score reliability when the number of questions per case varied in the National Board Dental Hygiene Examination (NBDHE). The experimental design maintained the same total number of case-based items, while varying the number of cases and items within cases to assess changes in score reliability. Using generalizability theory, the amounts of error variance within cases and between cases on the NBDHE were assessed. Impact on score reliability (generalizability) was computed. The data were from the responses of 4,528 candidates who took the paper-pencil version of the NBDHE in spring 2009. Results showed that the minimum value of generalizability occurred when fourteen cases with ten items each were used in the examination. The maximum value of generalizability occurred when fifty cases with three items each were used in the examination. The research findings support the development of more cases with fewer items per case on the NBDHE in order to enhance test score reliability and validity. Practical constraints should be considered if more cases with fewer items per case are developed for future examinations.

Career transition and dental school faculty development program.

Academic dentistry, as a career track, is not attracting sufficient numbers of new recruits to maintain a corps of skilled dental educators. The Faculty Development Program (FDP) at the University of Texas Health Science Center at San Antonio Dental School received federal funds to institute a 7-component program to enhance faculty recruitment and retention and provide training in skills associated with success in academics including:(1) a Teaching Excellence and Academic Skills (TExAS)Fellowship, (2) training in research methodology,evidence-based practice research, and information management, (3) an annual dental hygiene faculty development workshop for dental hygiene faculty, (4) a Teaching Honors Program and Academic Dental Careers Fellowship to cultivate students' interest in educational careers, (5) an Interprofessional Primary Care Rotation,(6) advanced education support toward a master's degree in public health, and (7) a key focus of the entire FDP, an annual Career Transition Workshop to facilitate movement from the practice arena to the educational arm of the profession.The Career Transition Workshop is a cap stone for the FDP; its goal is to build a bridge from practice to academic environment. It will provide guidance for private practice, public health, and military dentists and hygienists considering a career transition into academic dentistry. Topics will be addressed including: academic culture, preparation for the academic environment,academic responsibilities, terms of employment,compensation and benefits, career planning, and job search / interviewing. Instructors for the workshop will include dental school faculty who have transitioned from the practice, military, and public health sectors into dental education.Objectives of the Overall Faculty Development Program:• Provide training in teaching and research skills,career planning, and leadership in order to address faculty shortages in dental schools and under representation of minority faculty.• Provide resident and faculty training in cultural and linguistic competency.• Develop and conduct a collaborative inter professional education project with a Pediatric Medicine department, a nursing school, and other health professions' education programs.• Provide faculty and residents with financial support to pursue a master's degree in public health; and • Provide support and assistance for dental practitioners desiring to explore a transition into the educational environment.

The design and utility of institutional teaching awards: a literature review.

BACKGROUND: Institutional teaching awards have been used widely in higher education since the 1970s. Nevertheless, a comprehensive review of the literature on such awards has not been published since 1997. AIM: We conducted a literature review to learn as much as possible about the design (e.g., formats, selection processes) and utility (e.g., impact on individuals and institutions) of teaching awards in order to provide information for use in designing, implementing, or evaluating award programs. METHODS: We searched electronic databases for English-language publications on awards for exemplary teaching. Targeted publications included descriptions and/or investigations of award programs, their impact, and theoretical or conceptual models for awards programs. Screening was conducted by dual review; a third reviewer was assigned for disagreements. Data were analyzed qualitatively. Results were summarized descriptively. RESULTS: We identified 1302 publications for initial relevancy screening by title and abstract. We identified an additional 23 publications in a follow-up search. The full text of 126 publications was reviewed for further relevance. A total of 62 publications were identified as relevant, and of these 43 met our criteria for inclusion. Of the 43, 19 described the design features of 24 awards; 20 reports discussed award utility. Nomination and selection processes and benefits (e.g., plaques) varied as did perceived impact on individuals and institutions. CONCLUSION: Limited evidence exists regarding design and utility of teaching awards. Awards are perceived as having potential for positive impact, including promotions, but may also have unintended negative consequences. Future research should investigate the impact of awards on personal and professional development, and how promotion and tenure committees perceive awards.

Validating the standard for the National Board Dental Examination Part II.

As part of the overall exam validation process, the Joint Commission on National Dental Examinations periodically reviews and validates the pass/fail standard for the National Board Dental Examination (NBDE), Parts I and II. The most recent standard-setting activities for NBDE Part II used the Objective Standard Setting method. This report describes the process used to set the pass/fail standard for the 2009 exam. The failure rate on the NBDE Part II increased from 5.3 percent in 2008 to 13.7 percent in 2009 and then decreased to 10 percent in 2010. This article describes the Objective Standard Setting method and presents the estimated probabilities of classification errors based on the beta binomial mathematical model. The results show that the probability of correct classifications of candidate performance is very high (0.97) and that probabilities of false negative and false positive errors are very small (.03 and <0.001, respectively). The low probability of classification errors supports the conclusion that the pass/fail score on the NBDE Part II is a valid guide for making decisions about candidates for dental licensure.

Short-answer questions and formula scoring separately enhance dental student academic performance.

In this study, numerical course scores of second-year dental students in four successive classes in an oral and maxillofacial pathology course were compared. While the course content and teaching methods were essentially unchanged throughout the four years, two modest departures from the sole use of multiple-choice format questions were made in the assessment of student achievements. The modifications consisted of creating a more challenging examination procedure through the inclusion of un-cued short-answer format questions and the institution of correction-for-guessing scoring on multiple-choice examinations. Academically, the students in the four classes were comparable, as indicated by their respective numerical course score distributions in a prerequisite general pathology course in which the course content was unchanged, and all multiple-choice format questions were used to assess student academic achievements. This four-year study demonstrated that two qualitative changes in the educational environment-utilization of un-cued short-answer questions and correction for guessing scoring of multiple-choice questions-separately resulted in significant improvements in student course scores. Our results support the notion that, without any changes in curricular content or emphasis, combinations of qualitative changes in the assessment procedures alter student behavior and, as a consequence, appreciably improve their academic achievements.

Short-answer examinations improve student performance in an oral and maxillofacial pathology course.

The effect of examination question format on student performance was assessed by investigating three academically comparable second-year dental school classes in an oral and maxillofacial pathology course. One class was given examinations with all multiple-choice questions, one class was given examinations with all short-answer questions, and one class was given examinations with half multiple-choice questions and half short-answer questions. The class given examinations with half short-answer questions along with half multiple-choice questions had a significantly higher average score and grade category distribution (80-100 percent, 70-79 percent, <70 percent) than the class given examinations with all multiple-choice questions. When students in these two classes were divided into three academic ability groups based on the student's score in a prerequisite general pathology course, the class given examinations with half short-answer questions and half multiple-choice questions in the oral and maxillofacial pathology course had significantly higher scores and grade category distributions in all three ability groups. The average score and grade category distribution in the class given examinations with all short-answer questions in the oral and maxillofacial pathology course were not significantly different from the average score and grade category distribution in the class given examinations with half short-answer and half multiple-choice questions. Our interpretation of these results is that the utilization of examinations containing short-answer questions created a more challenging learning environment that motivated students to adopt more effective study regimens.

Prospective implementation of correction for guessing in oral and maxillofacial pathology multiple-choice examinations: did student performance improve?

A standard correction for random guessing on multiple-choice examinations was implemented prospectively in an Oral and Maxillofacial Pathology course for second-year dental students. The correction was a weighted scoring formula for points awarded for correct answers, incorrect answers, and unanswered questions such that the expected gain in the multiple-choice examination score due to random guessing was zero. An equally weighted combination of four examinations using equal numbers of short-answer questions and multiple-choice questions was used for student evaluation. Scores on both types of examinations, after implementation of the correction for guessing on the multiple-choice component (academic year 2005-06), were compared with the previous year (academic year 2004-05) when correction for guessing was not used for student evaluation but was investigated retrospectively. Academically, the two classes were comparable as indicated by the grade distributions in a General Pathology course taken immediately prior to the Oral and Maxillofacial Pathology course. Agreement between scores on short-answer examinations and multiple-choice examinations was improved in the 2005-06 class compared with the 2004-05 class. Importantly, the test score means were higher on both the short-answer and multiple-choice examinations in the Oral and Maxillofacial Pathology course, and the standard deviations were significantly smaller in 2005-06 compared to 2004-05; these differences reflected an upward shift in the lower part of the grade distributions to higher grades in 2005-06. Furthermore, when students were classified by their grade in the General Pathology course, students receiving a C (numerical grade of 70-79 percent) in General Pathology had significantly improved performance in the Oral and Maxillofacial Pathology course in 2005-06, relative to 2004-05, on both short-answer and multiple-choice examinations representing an aptitude-treatment interaction. We interpret this improved performance as a response to a higher expectation imposed on the 2005-06 students by the prospective implementation of correction for guessing.

Improving resident performance assessment data: numeric precision and narrative specificity.

PURPOSE: To evaluate the use of a systems approach for diagnosing performance assessment problems in surgery residencies, and intervene to improve the numeric precision of global rating scores and the behavioral specificity of narrative comments. METHOD: Faculty and residents at two surgery programs participated in parallel before- and-after trials. During the baseline year, quality assurance data were gathered and problems were identified. During two subsequent intervention years, an educational specialist at each program intervened with an organizational change strategy to improve information feedback loops. Three quality-assurance measures were analyzed: (1) percentage return rate of forms, (2) generalizability coefficients and 95% confidence intervals of scores, and (3) percentage of forms with behaviorally specific narrative comments. RESULTS: Median return rates of forms increased significantly from baseline to intervention Year 1 at Site A (71% to 100%) and Site B (75% to 100%), and then remained stable during Year 2. Generalizability coefficients increased between baseline and intervention Year 1 at Site A (0.65 to 0.85) and Site B (0.58 to 0.79), and then remained stable. The 95% confidence interval around resident mean scores improved at Site A from baseline to intervention Year 1 (0.78 to 0.58) and then remained stable; at Site B, it remained constant throughout (0.55 to 0.56). The median percentage of forms with behaviorally specific narrative comments at Site A increased significantly from baseline to intervention Years 1 and 2 (50%, 57%, 82%); at Site B, the percentage increased significantly in intervention Year 1, and then remained constant (50%, 60%, 67%). CONCLUSIONS: Diagnosing performance assessment system problems and improving information feedback loops improved the quality of resident performance assessment data at both programs.

Educating treatment professionals about addiction science research: demographics of knowledge and belief changes.

Communication of accurate, objective, and timely scientific information to treatment professionals is important--especially in the "drug abuse" and addiction field where misinformation and a lack of exposure to new information are common. The purpose of this study was to assess knowledge and belief changes that accompanied educational workshops (3 or 6 hr-long) on addiction science targeted to treatment professionals (N=1403) given in the United States and Puerto Rico between July 2000 and August 2001. Each workshop covered three main concepts: (1) terms and definitions; (2) basic neurochemistry of addiction; and (3) how new neurobiological knowledge will affect the treatment of addictions in the future. Analysis of variance was used to compare mean pretest to posttest change scores among levels of four independent variables: gender, age, occupation/position, and race/ethnicity. Workshop participants achieved a significant improvement in knowledge about addiction with younger groups achieving greater gains. Participants' beliefs shifted in the desired direction. Significant differences in belief shifts occurred among occupational and gender groups, but not among race/ethnicity or age groups. There was also a consistent change in the policy belief subscale that related to how strongly the audience members believed research on addiction was important. We conclude that addiction science education provided to treatment professionals can increase their knowledge and change their beliefs about the causes of addictions. In addition, the workshop participants form a base of constituents who are likely to support greater addiction research funding.

Do peer chart audits improve residents' performance in providing preventive care?

PURPOSE: One recommended method to evaluate residents' competence in practice-based learning and improvement is chart audit. This study determined whether residents improved in providing preventive care after a peer chart audit program was initiated. METHOD: Residents audited 1,005 charts and scored their peers on 12 clinical preventive services. The mean total chart audit scores were compared across five time blocks of the 45-month study. RESULTS: Residents' performance in providing preventive care initially improved significantly but declined in the last ten months. However, their performance remained significantly higher than at the beginning. CONCLUSIONS: By auditing their peers' charts, residents improved their own performance in providing preventive care. The diffusion of innovations theory may explain the prolonged implementation phase and problems in maintaining a chart audit program.

Effectiveness of addiction science presentations to treatment professionals, using a modified Solomon study design.

OBJECTIVES: Knowledge of addiction research findings is critical for healthcare professionals who treat addicted patients. However, there is little information available about the instructional effectiveness of lecture-slide presentations in changing knowledge vs. beliefs of such professionals. DESIGN: A modified Solomon four-group experimental design was used to assess the instructional effectiveness (knowledge gain vs. belief changes) of three-hour addiction science workshops presented to health-care professionals by neurobiologically-trained academic researchers. Effectiveness of the workshops was assessed by a 28-item questionnaire on participant versus control group knowledge/beliefs on addiction. Six-month follow-up questionnaires measured "retention" of knowledge and belief changes. RESULTS: The workshop participants showed significant knowledge gain and belief changes, whereas the two control groups showed no change in knowledge or beliefs. After six months, knowledge gains decreased, but were still higher than pre-test scores. In contrast, belief changes on three subscales persisted over six months in 40 to 52 percent of the subjects. CONCLUSIONS: These results illustrate a successful continuing education model by which academic researchers who are skilled teachers present a three-hour lecture-slide workshop with extensive question-and-answer sessions on addictions. We conclude that motivated health-care professionals can experience important knowledge gains and belief changes by participating in such workshops. In contrast to the transient retention of knowledge, belief changes persisted surprisingly well for at least six months in about half the subjects. These results suggest that long-term changes in the professional orientation of these health-care workers are possible.

A multimedia patient simulation for teaching and assessing endodontic diagnosis.

Teaching and assessing diagnostic skills are difficult due to relatively small numbers of total clinical experiences and a shortage of clinical faculty. Patient simulations could help teach and assess diagnosis by displaying a well-defined diagnostic task, then providing informative feedback and opportunities for repetition and correction of errors. This report describes the development and initial evaluation of SimEndo I, a multimedia patient simulation program that could be used for teaching or assessing endodontic diagnosis. Students interact with a graphical interface that has four pull-down menus and related submenus. In response to student requests, the program presents patient information. Scoring is based on diagnosis of each case by endodontists. Pilot testing with seventy-four junior dental students identified numerous needed improvements to the user interface program. A multi-school field test of the interface program using three patient cases addressed three research questions: 1) How did the field test students evaluate SimEndo I? Overall mean evaluation was 8.1 on a 0 to 10 scale; 2) How many cases are needed to generate a reproducible diagnostic proficiency score for an individual student using the Rimoldi scoring procedure? Mean diagnostic proficiency scores by case ranged from .27 to .40 on a 0 to 1 scale; five cases would produce a score with a 0.80 reliability coefficient; and 3) Did students accurately diagnose each case? Mean correct diagnosis scores by case ranged from .54 to .78 on a 0 to 1 scale. We conclude that multimedia patient simulations offer a promising alternative for teaching and assessing student diagnostic skills.

The effectiveness of a Mock Board experience in coaching students for the Dental Hygiene National Board Examination.

A Mock Board is available through the American Dental Association to any student or dental hygiene program as a study aid for the Dental Hygiene National Board Examination (DHNBE). The purpose of this study was to evaluate the usefulness of this Mock Board as a learning activity to help students gain familiarity with the question formats and the overall board exam experience. A sensitivity-specificity model was applied to validate the Mock Board's accuracy in identifying students who would pass or fail the DHNBE. A survey was conducted to assess students' opinions of the Mock Board experience. The Mock Board accurately identified success or failure on the DHNBE for 75 percent of the participants. However, the Mock Board's sensitivity was much better than its specificity. A majority of students reported the Mock Board motivated them to study and prepared them well for the national board experience. Comments showed, however, that students did not feel the Mock Board experience was as intense as taking the DHNBE. Findings indicate the Mock Board can be a valid and effective addition to board preparation activities. Dental hygiene faculty members are urged to consider incorporating the Mock Board experience with more traditional methods of coaching in preparing students for the DHNBE.

Evaluation of a Role-Play Learning Exercise in an Ambulatory Clinic Setting.

Learning in ambulatory settings can be aided by teaching activities which do not slow the pace of the clinic. In this study, simulated-parent role-play scripts were developed for use with students prior to seeing actual patients. During the learning exercise, a faculty member role-played various parents, a medical student role-played the physician while another student observed. Students were randomly assigned to either the Treatment Group (participated in role-play exercise) or Control Group (did not participate). Sixteen Treatment Group students completed approximately 20 role-play cases on the first day of the Pediatrics clerkship. Both Treatment and Control Group students were then individually assessed one to two days later using two role-play cases by an examiner blinded to the students' experimental status. Student performance was scored on three criteria: history taking, differential diagnosis, and correct diagnosis. The Treatment Group achieved higher scores than the Control Group for history taking, but not for differential diagnosis or correct diagnosis. Role-play learning exercises can be used to improve student history taking in an ambulatory clinic prior to seeing actual patients. Potential use of role-play cases as a screening exam to exempt some students from the role-play learning exercise is discussed.