Peer overmarking and insufficient diagnosticity: the impact of the rating method for peer assessment.

The present study explores two rating methods for peer assessment (analytical rating using criteria and comparative judgement) in light of concurrent validity, reliability and insufficient diagnosticity (i.e. the degree to which substandard work is recognised by the peer raters). During a second-year undergraduate course, students wrote a one-page essay on an air pollutant. A first cohort (N = 260) relied on analytical rating using criteria to assess their peers' essays. A total of 1297 evaluations were made, and each essay received at least four peer ratings. Results indicate a small correlation between peer and teacher marks, and three essays of substandard quality were not recognised by the group of peer raters. A second cohort (N = 230) used comparative judgement. They completed 1289 comparisons, from which a rank order was calculated. Results suggest a large correlation between the university teacher marks and the peer scores and acceptable reliability of the rank order. In addition, the three essays of substandard quality were discerned as such by the group of peer raters. Although replication research is warranted, the results provide the first evidence that, when peer raters overmark and fail to identify substandard work using analytical rating with criteria, university teachers may consider changing the rating method of the peer assessment to comparative judgement.

Entering Higher Professional Education: Unveiling First-Year Students' Key Academic Experiences and Their Occurrence Over Time.

To date, little understanding exists of how first-year students in professionally oriented higher-education (HE) programs (i.e., those that provide vocational education to prepare students for a particular occupation) experience their academic transition process. In the present study, we first argued how the constructs of academic adjustment and academic integration can provide complementary perspectives on the academic transition of first-year students in (professional) HE. Next, we examined what first-year students in professional HE contexts perceive to be the most important experiences associated with their academic transition process in the first semester of their first year of higher education (FYHE). To this end, we adopted the fundamentals of the critical incident technique and asked 104 students in a Flemish (Dutch-speaking part of Belgium) university college (which offers professional HE programs, such as nursing) to complete "reflective logs" with open questions at the start of the second semester of their FYHE, wherein they reflected on three critical academic experiences during their first semester. An inductive, cross-case content analysis of the collected narratives showed that students reported on nine themes of academic experiences, which relate to five adjustment themes (dealing with the organization of the study program, organizing study work, committing to the study, following class and taking notes, and processing learning content outside class) and four integration themes (feeling competent, feeling stressed, feeling prepared, and feeling supported). Further analyses showed that although some of the nine themes of academic experiences appear to be more important at different times in the first semester, they all seem to be meaningful throughout the whole semester.

Improving Self-Reflection Assessment Practices: Comparative Judgment as an Alternative to Rubrics.

CONSTRUCT: The authors aimed to investigate the utility of the comparative judgment method for assessing students' written self-reflections. BACKGROUND: Medical practitioners' reflective skills are increasingly considered important and therefore included in the medical education curriculum. However, assessing students' reflective skills using rubrics does not appear to guarantee adequate inter-rater reliabilities. Recently, comparative judgment was introduced as a new method to evaluate performance assessments. This study investigates the merits and limitations of the comparative judgment method for assessing students' written self-reflections. More specifically, it examines the reliability in relation to the time spent assessing, the correlation between the scores obtained using the two methods (rubrics and comparative judgment), and, raters' perceptions of the comparative judgment method. APPROACH: Twenty-two self-reflections, that had previously been scored using a rubric, were assessed by a group of eight raters using comparative judgment. Two hundred comparisons were completed and a rank order was calculated. Raters' impressions were investigated using a focus group. FINDINGS: Using comparative judgment, each self-reflection needed to be compared seven times with another self-reflection to reach a scale separation reliability of .55. The inter-rater reliability of rating (ICC, (1, k)) using rubrics was .56. The time investment required for these reliability levels in both methods was around 24 minutes. The Kendall's tau rank correlation indicated a strong correlation between the scores obtained via both methods. Raters reported that making comparisons made them evaluate the quality of self-reflections in a more nuanced way. Time investment was, however, considered heavy, especially for the first comparisons. Although raters appreciated that they did not have to assign a grade to each self-reflection, the fact that the method does not automatically lead to a grade or feedback was considered a downside. CONCLUSIONS: First evidence was provided for the comparative judgment method as an alternative to using rubrics for assessing students' written self-reflections. Before comparative judgment can be implemented for summative assessment, more research is needed on the time investment required to ensure no contradictory feedback is given back to students. Moreover, as the comparative judgment method requires an additional standard setting exercise to obtain grades, more research is warranted on the merits and limitations of this method when a pass/fail approach is used.

Work-related learning in the transition from higher education to work: The role of the development of self-efficacy and achievement goals.

BACKGROUND: Work-related learning is particularly important at the start of graduates' careers. Preparing students for work-related learning is a chief aim of higher education, demonstrating its relevance when investigating the transition to work. AIM: This study aimed to investigate the role of personal factors for work-related learning during the transition from higher education to work. This study took a longitudinal and person-centred approach by examining differences in work-related learning between different motivational profiles based on the conjoint development of self-efficacy and achievement goals. SAMPLE: Questionnaires were administered four times, starting in the beginning of students' final year of higher education until 4 months after graduation (N = 814). METHODS: In order to assess the development of the personal factors, different multi-indicator latent growth curve models were estimated. Afterwards, a multidimensional latent class growth analysis was used to identify latent profiles of participants with similar growth trajectories. Differences in work-related learning were investigated by a multivariate analysis of variance. RESULTS AND CONCLUSIONS: The results revealed an average increase in self-efficacy, learning-goal, and performance-approach orientation, indicating that the transition to work triggers change in these personal factors. Performance-avoidance orientation decreased on average. Four distinct motivational profiles were discerned: strongly efficacious and approach-oriented, moderate efficacious and approach-oriented, moderate efficacious and learning-oriented, and undecided profile. Furthermore, differences were found concerning work-related learning, in which the strongly efficacious and approach-oriented profile reported more participation in formal and informal learning activities using personal sources, while the undecided profile showed least participation in the latter.

The relation between cognitive and metacognitive processing: Building bridges between the SRL, MDL, and SAL domains.

AIM: The main aim of this commentary was to connect the insights from the contributions of the special issue on the intersection between depth and the regulation of strategy use. The seven contributions in this special issue stem from three perspectives: self-regulated learning (SRL), model of domain learning (MDL), or the student approaches to learning (SAL). PROCEDURE: Prior to combining insights from different studies, the definition and operationalization of cognitive and metacognitive processing in the seven contributions is described. Subsequently, the grain size and statistical methods used in these contributions are discussed. This information allows us to - albeit cautiously - combine the results from the different studies regarding the relation between cognitive and metacognitive processing. CONCLUSION: Deep processing and self-regulation/monitoring showed a strong correlation, regardless of the theoretical framework or data collection method chosen. The strength of the correlation between surface processing and metacognitive processing differed, however, between the studies. Pathways for future research on students' cognitive and metacognitive processing are suggested, at the methodological level as well as regarding the conceptualization of unregulated learning and surface processing.

Scale Separation Reliability: What Does It Mean in the Context of Comparative Judgment?

Comparative judgment (CJ) is an alternative method for assessing competences based on Thurstone's law of comparative judgment. Assessors are asked to compare pairs of students work (representations) and judge which one is better on a certain competence. These judgments are analyzed using the Bradly-Terry-Luce model resulting in logit estimates for the representations. In this context, the Scale Separation Reliability (SSR), coming from Rasch modeling, is typically used as reliability measure. But, to the knowledge of the authors, it has never been systematically investigated if the meaning of the SSR can be transferred from Rasch to CJ. As the meaning of the reliability is an important question for both assessment theory and practice, the current study looks into this. A meta-analysis is performed on 26 CJ assessments. For every assessment, split-halves are performed based on assessor. The rank orders of the whole assessment and the halves are correlated and compared with SSR values using Bland-Altman plots. The correlation between the halves of an assessment was compared with the SSR of the whole assessment showing that the SSR is a good measure for split-half reliability. Comparing the SSR of one of the halves with the correlation between the two respective halves showed that the SSR can also be interpreted as an interrater correlation. Regarding SSR as expressing a correlation with the truth, the results are mixed.

To what degree does the missing-data technique influence the estimated growth in learning strategies over time? A tutorial example of sensitivity analysis for longitudinal data.

Longitudinal data is almost always burdened with missing data. However, in educational and psychological research, there is a large discrepancy between methodological suggestions and research practice. The former suggests applying sensitivity analysis in order to the robustness of the results in terms of varying assumptions regarding the mechanism generating the missing data. However, in research practice, participants with missing data are usually discarded by relying on listwise deletion. To help bridge the gap between methodological recommendations and applied research in the educational and psychological domain, this study provides a tutorial example of sensitivity analysis for latent growth analysis. The example data concern students' changes in learning strategies during higher education. One cohort of students in a Belgian university college was asked to complete the Inventory of Learning Styles-Short Version, in three measurement waves. A substantial number of students did not participate on each occasion. Change over time in student learning strategies was assessed using eight missing data techniques, which assume different mechanisms for missingness. The results indicated that, for some learning strategy subscales, growth estimates differed between the models. Guidelines in terms of reporting the results from sensitivity analysis are synthesised and applied to the results from the tutorial example.

Modeling change in learning strategies throughout higher education: a multi-indicator latent growth perspective.

The change in learning strategies during higher education is an important topic of research in the Student Approaches to Learning field. Although the studies on this topic are increasingly longitudinal, analyses have continued to rely primarily on traditional statistical methods. The present research is innovative in the way it uses a multi-indicator latent growth analysis in order to more accurately estimate the general and differential development in learning strategy scales. Moreover, the predictive strength of the latent growth models are estimated. The sample consists of one cohort of Flemish University College students, 245 of whom participated in the three measurement waves by filling out the processing and regulation strategies scales of the Inventory of Learning Styles--Short Versions. Independent-samples t-tests revealed that the longitudinal group is a non-random subset of students starting University College. For each scale, a multi-indicator latent growth model is estimated using Mplus 6.1. Results suggest that, on average, during higher education, students persisting in their studies in a non-delayed manner seem to shift towards high-quality learning and away from undirected and surface-oriented learning. Moreover, students from the longitudinal group are found to vary in their initial levels, while, unexpectedly, not in their change over time. Although the growth models fit the data well, significant residual variances in the latent factors remain.

Differential use of learning strategies in first-year higher education: the impact of personality, academic motivation, and teaching strategies.

BACKGROUND: Although the evidence in support of the variability of students' learning strategies has expanded in recent years, less is known about the explanatory base of these individual differences in terms of the joint influences of personal and contextual characteristics. AIMS: Previous studies have often investigated how student learning is associated with either personal or contextual factors. This study takes an integrative research perspective into account and examines the joint effects of personality, academic motivation, and teaching strategies on students' learning strategies in a same educational context in first-year higher education. SAMPLE: In this study, 1,126 undergraduate students and 90 lecturers from eight professional bachelor programmes in a university college participated. METHODS: Self-report measures were used to measure students' personality, academic motivation, and learning strategies. Students' processing and regulation strategies are mapped using the Inventory of Learning Styles. Key characteristics of more content-focused versus learning-focused teaching strategies were measured. Multivariate multi-level analysis was used to take the nested data structure and interrelatedness of learning strategies into account. RESULTS: Different personality traits (openness, conscientiousness, and neuroticism) and academic motivation (amotivation, autonomous, and controlled motivation) were found to be independently associated with student learning strategies. Besides these student characteristics, also teaching strategies were found to be directly associated with learning strategies. CONCLUSIONS: The study makes clear that the impact of teaching strategies on learning strategies in first-year higher education cannot be overlooked nor overinterpreted, due to the importance of students' personality and academic motivation which also partly explain why students learn the way they do.