A Qualitative Textual Analysis of Feedback Comments in ePortfolios: Quality and Alignment with the CanMEDS Roles.

Introduction: Competency-based education requires high-quality feedback to guide students' acquisition of competencies. Sound assessment and feedback systems, such as ePortfolios, are needed to facilitate seeking and giving feedback during clinical placements. However, it is unclear whether the written feedback comments in ePortfolios are of high quality and aligned with the current competency focus. Therefore, this study investigates the quality of written feedback comments in ePortfolios of healthcare students, as well as how these feedback comments align with the CanMEDS roles. Methods: A qualitative textual analysis was conducted. 2,349 written feedback comments retrieved from the ePortfolios of 149 healthcare students (specialist medicine, general practice, occupational therapy, speech therapy and midwifery) were analysed retrospectively using deductive content analysis. Two structured categorisation matrices, one based on four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and another one on the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional), guided the analysis. Results: The minority of the feedback comments (n = 352; 14.9%) could be considered of high quality because they met all four quality criteria. Most feedback comments were of moderate quality and met only two to three quality criteria. Regarding the CanMEDS roles, the Medical Expert role was most frequently represented in the feedback comments, as opposed to the roles Leader and Health Advocate. Discussion: The results highlighted that providing high-quality feedback is challenging. To respond to these challenges, it is recommended to set up individual and continuous feedback training.

Automating the Identification of Feedback Quality Criteria and the CanMEDS Roles in Written Feedback Comments Using Natural Language Processing.

Introduction: Manually analysing the quality of large amounts of written feedback comments is time-consuming and demands extensive resources and human effort. Therefore, this study aimed to explore whether a state-of-the-art large language model (LLM) could be fine-tuned to identify the presence of four literature-derived feedback quality criteria (performance, judgment, elaboration and improvement) and the seven CanMEDS roles (Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional) in written feedback comments. Methods: A set of 2,349 labelled feedback comments of five healthcare educational programs in Flanders (Belgium) (specialistic medicine, general practice, midwifery, speech therapy and occupational therapy) was split into 12,452 sentences to create two datasets for the machine learning analysis. The Dutch BERT models BERTje and RobBERT were used to train four multiclass-multilabel classification models: two to identify the four feedback quality criteria and two to identify the seven CanMEDS roles. Results: The classification models trained with BERTje and RobBERT to predict the presence of the four feedback quality criteria attained macro average F1-scores of 0.73 and 0.76, respectively. The F1-score of the model predicting the presence of the CanMEDS roles trained with BERTje was 0.71 and 0.72 with RobBERT. Discussion: The results showed that a state-of-the-art LLM is able to identify the presence of the four feedback quality criteria and the CanMEDS roles in written feedback comments. This implies that the quality analysis of written feedback comments can be automated using an LLM, leading to savings of time and resources.

Training to Support ePortfolio Users During Clinical Placements: a Scoping Review.

ePortfolios are frequently used to support students' competency development, and teachers' and clinical mentors' supervision during clinical placements. User training is considered a critical success factor for the implementation of these ePortfolios. However, there is ambiguity about the design and outcomes of ePortfolio user training. A scoping review was conducted to consolidate evidence from studies describing the design of ePortfolio user training initiatives and their outcomes. The search yielded 1180 articles of which 16 were included in this review. Based on the results, an individual, ongoing training approach which grounds in a fitting theoretical framework is recommended. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-022-01583-0.

Decreasing Risky Behavior on Social Network Sites: The Impact of Parental Involvement in Secondary Education Interventions.

Teenagers face significant risks when using increasingly popular social network sites. Prevention and intervention efforts to raise awareness about these risks and to change risky behavior (so-called "e-safety" interventions) are essential for the wellbeing of these minors. However, several studies have revealed that while school interventions often affect awareness, they have only a limited impact on pupils' unsafe behavior. Utilizing the Theory of Planned Behavior and theories about parental involvement, we hypothesized that involving parents in an e-safety intervention would positively influence pupils' intentions and behavior. In a quasi-experimental study with pre- and post-test measures involving 207 pupils in secondary education, we compared the impact of an intervention without parental involvement with one that included active parental involvement by means of a homework task. We found that whereas parental involvement was not necessary to improve the intervention's impact on risk awareness, it did change intentions to engage in certain unsafe behavior, such as posting personal and sexual information on the profile page of a social network site, and in reducing existing problematic behavior. This beneficial impact was particularly evident for boys. These findings suggest that developing prevention campaigns with active parental involvement is well worth the effort. Researchers and developers should therefore focus on other efficient strategies to involve parents.

Teaching and Learning with Mobile Technology: A Qualitative Explorative Study about the Introduction of Tablet Devices in Secondary Education.

This paper investigates teachers' and students' perceptions concerning the impact of using tablet devices for teaching and learning purposes. An explorative focus group study was conducted with teachers (n = 18) and students (n = 39) in a secondary school that has implemented tablet devices since 2012. The general finding of this study shows that the use of tablet devices in the classroom setting has an impact on both teaching and learning practices. The results suggest that teachers can be divided into two categories: the innovative teachers and the instrumental teachers. Innovative teachers attempt to shift from a teacher-centered to a learning-centered approach. They have changed their teaching style by transforming lessons in accordance with the advantages tablet computers can offer. Instrumental teachers seem to use the device as a 'book behind glass'. The distinction between the two groups has consequences for both the way courses are given and how students experience them. In general, the introduction of tablet devices entails a shift in the way students learn, as the devices provide interactive, media-rich, and exciting new environments. The results of this study indicate that policy makers should consider introducing technical and pedagogical support in order to facilitate both teachers' and students' understanding of the full potential of this kind of technology in education.

How safe do teenagers behave on Facebook? An observational study.

The substantial use of social network sites by teenagers has raised concerns about privacy and security. Previous research about behavior on social network sites was mostly based on surveys and interviews. Observational research overcomes problems inherent to this research method, for example social desirability. However, existing observational research mostly focuses on public profiles of young adults. Therefore, the current observation-study includes 1050 public and non-public Facebook-profiles of teenagers (13-18) to investigate (1) what kind of information teenagers post on their profile, (2) to what extent they protect this information using privacy-settings and (3) how much risky information they have on their profile. It was found that young people mostly post pictures, interests and some basic personal information on their profile. Some of them manage their privacy-settings as such that this information is reserved for friends' eyes only, but a lot of information is accessible on the friends-of-friends' pages. Although general risk scores are rather low, more detailed analyses show that teenagers nevertheless post a significant amount of risky information. Moreover, older teenagers and girls post more (risky) information while there are no differences in applying privacy settings. We found no differences in the Facebook behavior of teenagers enrolled in different education forms. Implications of these results are discussed.

The effects of two computer-supported collaborative learning (CSCL) scripts on university students' critical thinking

The present study focuses on the use of two different types of scripts as possible ways to structure university students' discourse in asynchronous discussion groups and consequently promote their learning. More specifically, the aim of the study is to determine how requiring students to label their contributions by means of De Bono's Thinking Hats (script 1) and Weinberger's script for the construction of argumentation sequences (script 2) affects the ongoing critical thinking processes reflected in the discussion. The results suggest that both scripts successfully facilitated critical thinking. The results showed that the labeling condition (script 1) surpasses the argumentation script (script 2) with regard to the overall depth of critical thinking in the discussion, and the critical thinking processes during the stages of problem identification and problem integration in particular. Further, it can be argued that students in the labeling condition are engaged in more focused, more critical, and more practically-oriented discussions.(AU)

O objetivo desse estudo é enfatizar o uso de dois diferentes roteiros como possibilidades para estruturar o discurso de estudantes universitários em grupos de discussão assíncrona e, conseqüentemente, promover o aprendizados dos mesmos. Mais especificadamente, esse estudo têm o objetivo de ensinar alunos a dar suas contribuições por meio do roteiro de De Bono's Thinking Hats (roteiro 1) e pelo roteiro Weinberger para a construção de seqüências argumentativas (roteiro 2) que afetam o curso do processo de pensamento presente numa discussão. Os resultados revelaram que rotulação (roteiro 1) supera o roteiro argumentativo (roteiro 2) em relação a profundidade do pensamento crítico global e do pensamento crítico durante o estágio de identificação de problemas e, em particular, na etapa relativa à integração de problemas. Além disso, pode ser considerado que alunos competentes em rotulação são mais engajados, críticos e orientados para a prática nas discussões.(AU)

The effects of two computer-supported collaborative learning (CSCL) scripts on university students' critical thinking

The present study focuses on the use of two different types of scripts as possible ways to structure university students' discourse in asynchronous discussion groups and consequently promote their learning. More specifically, the aim of the study is to determine how requiring students to label their contributions by means of De Bono's Thinking Hats (script 1) and Weinberger's script for the construction of argumentation sequences (script 2) affects the ongoing critical thinking processes reflected in the discussion. The results suggest that both scripts successfully facilitated critical thinking. The results showed that the labeling condition (script 1) surpasses the argumentation script (script 2) with regard to the overall depth of critical thinking in the discussion, and the critical thinking processes during the stages of problem identification and problem integration in particular. Further, it can be argued that students in the labeling condition are engaged in more focused, more critical, and more practically-oriented discussions.

O objetivo desse estudo é enfatizar o uso de dois diferentes roteiros como possibilidades para estruturar o discurso de estudantes universitários em grupos de discussão assíncrona e, conseqüentemente, promover o aprendizados dos mesmos. Mais especificadamente, esse estudo têm o objetivo de ensinar alunos a dar suas contribuições por meio do roteiro de De Bono's Thinking Hats (roteiro 1) e pelo roteiro Weinberger para a construção de seqüências argumentativas (roteiro 2) que afetam o curso do processo de pensamento presente numa discussão. Os resultados revelaram que rotulação (roteiro 1) supera o roteiro argumentativo (roteiro 2) em relação a profundidade do pensamento crítico global e do pensamento crítico durante o estágio de identificação de problemas e, em particular, na etapa relativa à integração de problemas. Além disso, pode ser considerado que alunos competentes em rotulação são mais engajados, críticos e orientados para a prática nas discussões.