Work-in-Progress: Using Latent Dirichlet Allocation to uncover themes in student comments from peer evaluations of teamwork

ABSTRACT

This work-in-process research paper investigates common themes in peer-to-peer comments of teamwork behavior effectiveness collected with peer evaluations in engineering student teams in three time horizons - prior to COVID-19 pandemic, early phase of pandemic, and mature phase of pandemic. Constructive feedback is imperative to maintaining healthy team climate and dynamic, which facilitates positive individual and team learning outcomes. Asking engineering students to provide self- and peer-evaluation feedback in comments accomplishes multiple objectives. Students reflect on teammates' behavior and performance rather than relying on (potentially biased) general perceptions to provide evidence-based comments for the assessment period. Repeated practice giving feedback also tends to improve students' ability to provide constructive and insightful evaluations. To better understand what and how engineering students provide feedback in teamwork, the Comprehensive Assessment of Team-Member Effectiveness (CATME) peer evaluation tool suite was used to provide a framework to teach students about effective team behaviors using a behavioral-anchored rating scale. Using CATME also provided a mechanism for collecting self- and peer- evaluation survey data in both structured (the behavioral scale) and open-ended (comments) ways. Latent Dirichlet Allocation (LDA) was used as the classic method for topic modeling to analyze first-year engineering students' self- and peer- comments in the introductory engineering foundation courses in a large Midwestern R1 university. Topic Coherence measure (c_v) for topic quality was used to determine the optimal number of topics to represent the comment data. The themes of each of the topics identified were interpreted by thematic analysis of the most commonly used words and responses associated with each topic identified by the LDA model. The preliminary results showed that pre-pandemic themes closely matched the five behavioral dimensions of the CATME instrument. Data collected in Spring 2020 required more themes to capture the complexity of the transition to online learning. Comments from Spring 2021 required an even larger number of themes to describe the experience of teamwork during a fully virtual class implementation. © American Society for Engineering Education, 2022.