Small programs, big challenges: Reimagining the evaluation of clinical teaching in genetic counseling.

Within the health professions education system, a significant proportion of teaching and learning occurs in the clinical setting. As such, the need to measure effective teaching for accreditation standards, faculty development, merit pay, academic promotion, and for monitoring the safety of the learning environment has led to numerous universities developing instruments to evaluate teaching effectiveness in this context. To date; however, these instruments typically focus on the student perspective, despite evidence demonstrating that student evaluations of teaching (SETs) lack correlation with learning outcomes and are not a true measure of teaching effectiveness. This issue is further exacerbated in small health professional training programs, such as genetic counseling, where clinical teachers may only supervise 1-3 students per year. As a result, not only are SETs more confounded due to small sample sizes, but a direct conflict exists between respecting learner anonymity and providing timely and relevant feedback to faculty. In such contexts, even using SETs to evaluate the nature of the learning environment may be unreliable due to student concerns about identifiability and fear of retaliation for unfavorable evaluation. This paper will review the literature regarding SETs, barriers to this process within the clinical setting, and the unintended downstream consequences. Options for addressing issues related to the use of SETs will be considered, with particular focus on the process of reflection and the use of teaching consultations or peer support groups as a means to improve teaching effectiveness in this learning environment.

Approaches to Item-Level Data with Cross-Classified Structure: An Illustration with Student Evaluation of Teaching.

Student evaluation of teaching (SET) questionnaires are ubiquitously applied in higher education institutions in North America for both formative and summative purposes. Data collected from SET questionnaires are usually item-level data with cross-classified structure, which are characterized by multivariate categorical outcomes (i.e., multiple Likert-type items in the questionnaires) and cross-classified structure (i.e., non-nested students and instructors). Recently, a new approach, namely the cross-classified IRT model, was proposed for appropriately handling SET data. To inform researchers in higher education, in this article, the cross-classified IRT model, along with three existing approaches applied in SET studies, including the cross-classified random effects model (CCREM), the multilevel item response theory (MLIRT) model, and a two-step integrated strategy, was reviewed. The strengths and weaknesses of each of the four approaches were also discussed. Additionally, the new and existing approaches were compared through an empirical data analysis and a preliminary simulation study. This article concluded by providing general suggestions to researchers for analyzing SET data and discussing limitations and future research directions.

Faculty Process for Reviewing and Utilizing a School's Course Evaluation Comments.

OBJECTIVE: This study aimed to explore faculty engagement with qualitative comments from course evaluations. METHODS: Course faculty from the University of North Carolina Eshelman School of Pharmacy were recruited via email to participate in a 30-minute interview session. Previous course evaluation comments were adapted to create a de-identified mock evaluation. Six interviews were conducted via Zoom, consisting of a think-aloud protocol based on the mock course evaluation followed by a cognitive interview focused on goals and current utilization of comments, and common patterns and issues sought by faculty. Interview transcripts were manually cleaned and de-identified. Transcripts were inductively coded by 1 researcher using MAXQDA. RESULTS: Three overarching themes were identified: general faculty process for reviewing comments (ie, how faculty perceive and analyze comments), comments utilization for course change (ie, how faculty utilize comments in making course changes), and faculty analysis strategy (ie, faculty approach to locating common patterns in evaluation comments). The most common subthemes included usefulness of comments, methods for tracking comment patterns, and challenges with the large number of comments each semester. CONCLUSION: Faculty provided useful insight and feedback regarding the current state of the course evaluation process that can be used to improve the structure, organization, and utilization of course evaluations by faculty. These findings could inform the creation of the course evaluation comment automated analysis program in the next stage of an ongoing project.

Online Parameter Estimation for Student Evaluation of Teaching.

Student evaluation of teaching (SET) assesses students' experiences in a class to evaluate teachers' performance in class. SET essentially comprises three facets: teaching proficiency, student rating harshness, and item properties. The computerized adaptive testing form of SET with an established item pool has been used in educational environments. However, conventional scoring methods ignore the harshness of students toward teachers and, therefore, are unable to provide a valid assessment. In addition, simultaneously estimating teachers' teaching proficiency and students' harshness remains an unaddressed issue in the context of online SET. In the current study, we develop and compare three novel methods-marginal, iterative once, and hybrid approaches-to improve the precision of parameter estimations. A simulation study is conducted to demonstrate that the hybrid method is a promising technique that can substantially outperform traditional methods.

Perceptions and attitudes of medical students towards student evaluation of teaching: A cross-sectional study.

BACKGROUND: Faculty evaluation surveys in the frame of student evaluation of teaching (SETs) are a widely utilized tool to assess faculty teaching. Although SETs are used regularly to evaluate teaching effectiveness, their sole use for making administrative decisions and as an indicator of teaching quality has been controversial. METHODS: A survey containing 22 items assessing demographics, perception, and factors for evaluating faculty was distributed to medical students at our institute. Statistical analyses were conducted using Microsoft Excel and R Software utilizing regression analysis and ANOVA test. RESULTS: The survey received 374 responses consisting of 191 (51.1%) male students and 183 (48.9%) female students. In all, 178 (47.5%) students considered the optimal time for providing faculty evaluation to be after the release of the exam results, compared to 127 (33.9%) students, who chose the after the exam but before the release of exam results option. When asked what happens whenever the tutor is aware about the SETs data, 273 (72.9%) and 254 (67.9%) students believed that it would influence the difficulty of the exam and grading/curving of the exam results, respectively. Better teaching skills (93%, 348), being responsive and open to student feedback and suggestions (84.7%, 317), being committed to class time and schedule (80.1%, 300), and an easier exam (68.6%, 257) were considered important factors to acquire a positive evaluation by a considerable proportion of students. Fewer lectures (P < 0.05), decreased number of slides per lecture (P < 0.01), easier exam (P < 0.05), and giving clues to students about the exam (P < 0.05) were found to be very important to obtain a positive tutor evaluation by students. CONCLUSIONS: Institutions ought to continue exploring areas of improvement in the faculty evaluation process while raising awareness among students about the importance and administrative implications of their feedback.

Critique of Approaches for Evaluating Teaching and Proposal for a New Institutional Policy.

A review of current literature provides ample evidence that Student Evaluation of Teaching, as currently practiced, can be a biased and flawed evaluation of educator performance and student learning. Due to this, many institutions have developed their own internal multimodal approach for faculty evaluation and professional advancement review that removes the underlying bias associated with survey results. Ross University School of Veterinary Medicine created a taskforce to evaluate the current teaching assessment practices and provide an alternative method. Based on an extensive literature review, an institutional policy was created using a multimodal approach, combining student evaluations of teaching scores with peer review of teaching, review of teaching materials, mentorship, self-ratings, and self-reflection. The implementation plan for this policy is described to allow for a teaching evaluation process built on a philosophy of mentorship, guidance, and self-reflection; as well as the use of external professional exam metrics to review the institutional academic quality at large.

In the eye of the beholder - how course delivery affects anatomy education.

BACKGROUND: The COVID-19 pandemic caused major shifts in students' learning strategies as well as teaching environments that profoundly affected the delivery of anatomy courses in medical schools. The Department of Anatomy at the University of Zagreb School of Medicine had a unique experience where the anatomy course in 2019/2020 was first taught in-person before transferring to an online course delivery, while the inverse happened in 2020/2021. The core curriculum, course material and examination criteria were the same in both academic years. The aim of the study was to determine whether course delivery affected students' perceptions of the course and whether it impacted students' engagement and success. METHODS: The students' perceptions of the course were assessed via an anonymous course survey (student evaluation of teaching, SET). The questions in the SET assessed the usefulness of teaching modalities rather than students' satisfaction. Most questions were in the form of statements to which students responded with their level of agreement on a five-point Likert scale. Differences between responses in 2019/2020 and 2020/2021 were analyzed using the Mann-Whitney test. Effect size was estimated using Cliff's delta and association between responses was assessed using Spearman's r coefficient. RESULTS: Students' perceptions were significantly affected by changes in course delivery. Students' success and engagement were higher in 2019/2020 when in-person teaching preceded online teaching. Furthermore, students' views on course organization and the usefulness of continuous assessment were more positive in 2019/2020. Finally, students' perceptions of the usefulness of online materials and activities were more positive in 2019/2020. All comparisons between the two academic years were statistically significant (P ≤ 0.0001 for all comparisons, Mann-Whitney test). CONCLUSIONS: Students' perceptions of the anatomy course were dependent on the teaching environment they were exposed to at the beginning of the course. A transfer from in-person to online course delivery was more successful than vice-versa. This has important implications for structuring hybrid courses in medical education in the future.

A WeChat-based competency and performance checklist in basic surgical skills course for military medical academy undergraduates.

BACKGROUND: Basic surgical skills (BSS) is a key bridging course for medical students to acquire basic surgical maneuvers and practice animal surgery before clinical rotation, but the complexity of operational procedures and high demands on asepsis may lead to poor performance and frequent error during practice. The current study intended to improve BSS teaching outcomes by implementing smartphone app-based competency and performance checklists for medical academy undergraduates. METHODS: WeChat-based checklists containing competency and performance modules were designed, distributed and collected via smartphone. One hundred seventy-six third-grade undergraduate cadets majoring in clinical medicine or anesthesiology were prospectively enrolled, with 92 set as study group and 84 as control group. Checklists were distributed for self-evaluation before and after each class throughout the semester of autumn 2021-2022. Student age, previous Grade Point Average (GPA), average grades of BSS (including grade-A rate and pass rate), operative time, error rate, and perioperative complications of intestinal anastomosis performed on Beagle dogs between the groups were compared to evaluate the efficacy of the checklists. RESULTS: The students aged 20.2 ± 0.63 in Group A and 20.3 ± 0.92 in Group B (P = 0.15), with a previous GPA of 2.9 ± 0.61 vs. 2.87 ± 0.58 (P = 0.61). The average operative time on their final lesson of intestinal anastomosis was 192.3 ± 27.18 min vs. 213.8 ± 29.48 min (P < 0.001). All students passed in BSS course, with a final grade of 89.45 ± 4.360 in Group A and 86.64 ± 4.026 in Group B (P < 0.001), in which grade-A rate was 46.7% vs. 26.2% (P = 0.005). For perioperative comorbidities, 4/23 (17.4%) animals in Group A and 5/21 (23.8%) in Group B recorded wound dehiscence or other incision-related complications; no animals died in Group A, and 2 died in Group B due to hemorrhagic shock or sepsis. CONCLUSIONS: The implementation of WeChat-based checklist is a reflection of improved quality of teaching in BSS course that may promote the students' competency and performance.

A literature review of the research on students' evaluation of teaching in higher education.

Students' evaluation of teaching is a teaching quality evaluation method and teacher performance evaluation tool commonly used in Chinese and foreign universities, and it is also a controversial hot issue in the field of teaching evaluation. At present, the research results of students' evaluation of teaching in higher education are relatively rich, mainly focusing on reliability, validity and its influencing factors, construction of index system, problems in practical application and improvement strategies. The purpose of this article is to study the relevant research results of the current Chinese and foreign academic circles, in order to provide useful inspiration for the construction of the index system and practical application of the ideological and political theory course evaluation and teaching of Chinese college students.

Why teaching innovation matters: Evidence from a pre- versus peri-COVID-19 pandemic comparison of student evaluation data.

The COVID-19 pandemic has robustly affected global education environments, so higher education institutions need to emphasize innovation and creativity in educational methods for teachers to improve their teaching performance as well as enhance the engagement and motivation of students in this changing environment. Accordingly, it is essential to discuss the role of teaching innovation in the setting of the COVID-19 pandemic compared to the pre-COVID-19 period. The aim of this study is to empirically validate the importance of teaching innovation in student evaluation of teaching before and during the COVID-19 pandemic. Data were collected from the medical college of a comprehensive university in Taiwan and were subjected to t-tests and multiple linear regression analysis. Findings from a quantitative study with 44 teachers revealed that teaching innovation was positively correlated to student evaluation of teaching. We also found that teachers who implemented teaching innovation strategies performed better than those teachers who used conventional teaching strategies on student evaluation of teaching. In particular, teaching innovative teachers had improvement in student evaluation of teaching during the time of the COVID-19 pandemic, but not non-teaching innovative teachers. The evidence from this study suggests that teaching innovation can not only enhance teachers' teaching development and performance but also boost students' motivation for learning, especially in the time of the COVID-19 pandemic. Our findings have important implications for future research on teaching innovation and for higher education institutions and faculty wishing to provide high-quality learning environments to their students.

Spain is not different: teaching quantitative courses can also be hazardous to one's career (at least in undergraduate courses).

Student evaluations of teaching (SETs) have become a widely used tool for assessing teaching in higher education. However, numerous investigations have shown that SETs are subject to multiple biases, one of which is particularly relevant, namely, the area of knowledge to which the subject belongs. This article aims to replicate the article by Uttl & Smibert (2017, https://doi.org/10.7717/peerj.3299) in a different educational context to verify whether the negative bias toward instructors who teach quantitative courses found by the authors in the US also appears in the Spanish university system. The study was conducted at the Business and Law School of the Universidad Pontificia Comillas, a private Spanish university, using two different samples. First, we analyzed undergraduate courses using a sample of 80,667 SETs in which 2,885 classes (defined as a single semester-long course taught by an individual instructor to a specific group of students), 488 instructors, and 322 different courses were evaluated over a time period of four academic years (2016/2017-2019/2020). Second, in the same period, 16,083 SETs corresponding to master's degree courses were analyzed, which involved the study of 871 classes, 275 instructors, and 155 different courses. All the data included in the analysis were obtained from official university surveys developed by a team of professionals specialized in teaching quality responsible for ensuring the reliability of the information. At the degree level, the results show that despite the considerable cultural and temporal difference between the samples, the results are very similar to those obtained by Uttl & Smibert (2017, https://doi.org/10.7717/peerj.3299); i.e., professors teaching quantitative courses are far more likely to obtain worse SETs than instructors in other areas. There are hardly any differences at the master's degree level, regardless of whether nearly 75% of master's degree instructors also teach at the undergraduate level. This leads us to three different conclusions. (1) Evidence suggests that the reason for these differences is not due to faculty teaching quantitative courses being less effective than faculty teaching in some other fields. Our results indicate that the same instructor is evaluated very differently depending on whether he or she teaches at the undergraduate or master's level. (2) It is essential to avoid comparisons of SETs between different areas of knowledge, at least at the undergraduate level. (3) A significant change in the use and interpretation of SETs is imperative, or its replacement by other evaluation mechanisms should be considered. If this does not occur, it is possible that in the future, there will be an adverse selection effect among professors of quantitative methods; i.e., only the worst professionals in quantitative methods will opt for teaching since the good professionals will prefer other jobs.

Psychometric properties of the Persian version of System for Evaluation of Teaching Qualities by students: A tool for assessing clinical tutors from students' viewpoint.

BACKGROUND: Effective clinical teaching is crucially important for patient care in future. Therefore, proper clinical training is essential to make physicians capable of delivering high-quality health care. MATERIALS AND METHODS: The present study was a cross-sectional research. After translating the questionnaire into Persian, it was distributed among medical students in the clinical years of medical education in teaching hospitals affiliated to Ahvaz Jundishapur University of Medical Sciences in 2018. The System for Evaluation of Teaching Qualities (SETQ) has 25 questions in a 6-scale Likert scale that evaluates clinical tutors in five dimensions of teaching and learning environment, professional attitude toward students, transferring of goals, evaluation of students, feedback, and promoting self-directed learning. Instrument reliability was assessed by calculating the Cronbach's alpha coefficient, whereas questionnaire content validity was evaluated by relative content validity ratio (CVR) and content validity index (CVI). To evaluate the structural validity, an exploratory factor analysis was conducted. RESULTS: The SETQ was completed by 127 medical students. Cronbach's alpha coefficient of the total questionnaire was estimated as 0.908. The factor analysis showed that the questionnaire was composed of six factors, explaining 66.14% of the total variance. The CVI and CVR indices of the individual items were also acceptable. CONCLUSION: The findings of our study showed that the Persian version of SETQ questionnaire had the acceptable reliability and validity to be used in assessing clinical tutors in different hospitals in Iran.

[Exploration and practice on the teaching reform of "Bioremediation of Environmental Pollution"].

The course Bioremediation of Environmental Pollution, which plays a vital role in the professional training system, is a professional elective course for college students majored in environmental science, environmental engineering and agricultural resources and environment. In view of the problems identified in previous teaching experiences, the teachers carried out teaching reform to meet the demand for high-quality personnel training. The teaching reform included optimization of course objectives, reconstruction of course content and knowledge integration, reform and innovation of teaching methods. The practices indicate that a reformed curriculum teaching significantly improves the achievement of the teaching objectives. Moreover, it effectively enhances the students' independent learning, thinking and comprehensive knowledge application ability, achieving sound teaching effects.

The effect of teaching style and academic motivation on student evaluation of teaching: Insights from social cognition.

Student evaluation of teaching (SET) is ubiquitous in higher education as a metric for assessing teachers, gaining student feedback, and informing faculty personnel decisions. It is thus imperative to examine the dimensions along which a teacher is judged. This study tested the application of the universal dimensions of social judgment (i.e., warmth and competence) in SET. A total of 108 psychology undergraduates (M age = 23.63, SD age = 3.14) in Singapore rated a fictitious teacher (i.e., either relationship-oriented or task-oriented) based on their interactions over a programmed online chat. Participants responded to the social judgment measures of warmth and competence and rated their academic motivation. Results indicated a higher SET rating for a relationship-oriented than a task-oriented teacher. Further, student academic motivation mediated the link between teaching style and judgment of competence. The findings extend the supremacy of warmth in the context of SET, thus supporting the application of social cognition literature to educational research. In addition, the findings suggest that fostering a match in task goals between a teacher and student improves ratings of teacher competence.

Exploration and practice on the teaching reform of "Bioremediation of Environmental Pollution" / 生物工程学报

The course Bioremediation of Environmental Pollution, which plays a vital role in the professional training system, is a professional elective course for college students majored in environmental science, environmental engineering and agricultural resources and environment. In view of the problems identified in previous teaching experiences, the teachers carried out teaching reform to meet the demand for high-quality personnel training. The teaching reform included optimization of course objectives, reconstruction of course content and knowledge integration, reform and innovation of teaching methods. The practices indicate that a reformed curriculum teaching significantly improves the achievement of the teaching objectives. Moreover, it effectively enhances the students' independent learning, thinking and comprehensive knowledge application ability, achieving sound teaching effects.

Application of task-oriented method combined with two-way evaluation in hypertension nursing teaching / 中华医学教育探索杂志

Objective:To explore the application of task-orientated method combined with two-way evaluation in hypertension nursing teaching.Methods:A total of 90 nursing students who practiced in the Department of Cardiovascular Medicine, Haikou Affiliated Hospital of Central South University Xiangya School of Medicine from September 2019 to December 2020 were selected as the research objects. According to the order of their admission, the nursing students were divided into control group and research group, with 45 students in each group. The control group used traditional teaching, while the research group used task-orientated method combined with two-way evaluation. At the end of the practice, the two groups of practice nursing students were compared to evaluate the teaching quality of teaching teachers, meanwhile the two groups of teaching teachers evaluated the quality of practice nursing students. SPSS 22.0 was used for t test. Results:The four aspects (teaching attitude, teaching method, teaching content and teaching effect) of the nursing students' evaluation on the teachers in the research group were higher than those in the control group, and the difference was statistically significant ( P<0.05). The three aspects (theoretical knowledge, professional nursing skills and comprehensive quality) of the teachers' evaluation on the students in the study group were higher than those in the control group, and the difference was statistically significant ( P<0.05). Conclusion:The task-orientated method combined with two-way evaluation can improve the teaching quality of clinical nursing teaching and the recognition of clinical nursing students to the teaching quality of teaching teachers, so as to promote teaching and learning.

Validation of a Short Scale for Student Evaluation of Teaching Ratings in a Polytechnic Higher Education Institution.

The general purpose of this work is 2-fold, to validate scales and to present the methodological procedure to reduce these scales to validate a rating scale for the student evaluation of teaching in the context of a Polytechnic Higher Education Institution. We explored the relationship between the long and short versions of the scale; examine their invariance in relation to relevant variables such as gender. Data were obtained from a sample of 6,110 students enrolled in a polytechnic higher education institution, most of whom were male. Data analysis included descriptive analysis, intraclass correlation, exploratory structural equation modeling (ESEM), confirmatory factorial analysis, correlations between the short and long form corrected for the shared error variance, gender measurement invariance, reliability using congeneric correlated factors, and correlations with academic achievement for the class as unit with an analysis following a multisection design. Results showed four highly correlated factors that do not exclude a general factor, with an excellent fit to data; configural, metric, and scalar gender measurement invariance; high reliability for both the long and short scale and subscales; high short and long-form scale correlations; and moderate but significant correlations between the long and short versions of the scales with academic performance, with individual and aggregate data collected from classes or sections. To conclude, this work shows the possibility of developing student evaluation of teaching scales with a short form scale, which maintains the same high reliability and validity indexes as the longer scale.

Does professors' gender impact how students evaluate their teaching and the recommendations for the best professor?

This study examined the impact of the professors' gender according to a student evaluation of teaching (SET) in a private university. The study took place in a private university (n = 103,833) on six different campuses in the north region of Mexico. The distribution of the professors' gender was analyzed according to semesters, campuses, and schools. Our findings suggested that when undergraduates evaluated their professors on specific criteria concerning teaching performance, they expressed their opinion regardless of the professors' gender. However, when being asked for a single overall evaluation, as whether they would recommend the professor as one of their best professors, the students tended to favor male professors over their female peers by a slight margin. While such perceptions might not be representative of the actual teaching quality, it would be interesting in the future to delve deeper into the causes of possible biases.

Barriers to obtaining reliable results from evaluations of teaching quality in undergraduate medical education.

BACKGROUND: Medical education is characterized by numerous features that are different from other higher education programmes, and evaluations of teaching quality are an integral part of medical education. Although scholars have made extensive efforts to enhance the quality of teaching, various issues unrelated to teaching that interfere with the accuracy of evaluation results remain. The purpose of this study is to identify the barriers that prevent objective and reliable results from being obtained during the evaluation process. METHODS: This study used mixed methods (3 data sources) to collect opinions from different stakeholders. Based on purposive sampling, 16 experts familiar with teaching management and 12 s- and third-year students were invited to participate in interviews and discussions, respectively. Additionally, based on systematic random sampling, 74 teachers were invited to complete a questionnaire survey. All qualitative data were imported into NVivo software and analysed using thematic analysis in chronological order and based on grounded theory. Statistical analyses of the questionnaire results were conducted using SPSS software. RESULTS: Sixty-nine valid questionnaires (93.24%) were recovered. A total of 29 open codes were extracted, and 14 axial codes were summarized and divided into four selective codes: evaluation preparation, the index system, the operation process, and the consequences of evaluation. The main barriers to obtaining reliable evaluation results included inadequate attention, unreasonable weighting, poor teaching facilities, an index without pertinence and appropriate descriptions, bad time-points, incomplete information on the system, lagged feedback, and disappointing result application. Almost all participants suggested lowering the weight of students as subjects, with a weight of 50-60% being appropriate. Students showed dissatisfaction with evaluation software, and the participants disagreed over the definition of good teaching and the management of student attendance. CONCLUSIONS: This study reveals the difficulties and problems in current evaluations of teaching in medical education. Collecting data from multiple stakeholders helps in better understanding the evaluation process. Educators need to be aware of various issues that may affect the final results when designing the evaluation system and interpreting the results. More research on solutions to these problems and the development of a reasonable evaluation system is warranted.

Student evaluations of teaching (SET): implications for medical education in psychiatry and an approach to evaluating SET and student performance.

OBJECTIVE: We present reflections on student evaluation of teaching (SET) in the context of recent higher educational research that assesses SET, as well as concurrent and/or subsequent student performance. CONCLUSIONS: In a sense, there is in-built cynicism in SET, with more favourable SET for easier assessment. There is emerging evidence that SET is inversely proportional to the performance of students in subsequent courses, i.e. the higher the ratings, the poorer the students perform in subsequent studies. It is proposed that SET should be combined with contemporaneous formative and summative assessments of student performance in medical school settings, especially in psychiatry education.