EFAR-MMLA: An Evaluation Framework to Assess and Report Generalizability of Machine Learning Models in MMLA.

Multimodal Learning Analytics (MMLA) researchers are progressively employing machine learning (ML) techniques to develop predictive models to improve learning and teaching practices. These predictive models are often evaluated for their generalizability using methods from the ML domain, which do not take into account MMLA's educational nature. Furthermore, there is a lack of systematization in model evaluation in MMLA, which is also reflected in the heterogeneous reporting of the evaluation results. To overcome these issues, this paper proposes an evaluation framework to assess and report the generalizability of ML models in MMLA (EFAR-MMLA). To illustrate the usefulness of EFAR-MMLA, we present a case study with two datasets, each with audio and log data collected from a classroom during a collaborative learning session. In this case study, regression models are developed for collaboration quality and its sub-dimensions, and their generalizability is evaluated and reported. The framework helped us to systematically detect and report that the models achieved better performance when evaluated using hold-out or cross-validation but quickly degraded when evaluated across different student groups and learning contexts. The framework helps to open up a "wicked problem" in MMLA research that remains fuzzy (i.e., the generalizability of ML models), which is critical to both accumulating knowledge in the research community and demonstrating the practical relevance of these techniques.

Understanding teacher design practices for digital inquiry-based science learning: the case of Go-Lab.

Designing and implementing online or digital learning material is a demanding task for teachers. This is even more the case when this material is used for more engaged forms of learning, such as inquiry learning. In this article, we give an informed account of Go-Lab, an ecosystem that supports teachers in creating Inquiry Learning Spaces (ILSs). These ILSs are built around STEM-related online laboratories. Within the Go-Lab ecosystem, teachers can combine these online laboratories with multimedia material and learning apps, which are small applications that support learners in their inquiry learning process. The Go-Lab ecosystem offers teachers ready-made structures, such as a standard inquiry cycle, alternative scenarios or complete ILSs that can be used as they are, but it also allows teachers to configure these structures to create personalized ILSs. For this article, we analyzed data on the design process and structure of 2414 ILSs that were (co)created by teachers and that our usage data suggest have been used in classrooms. Our data show that teachers prefer to start their design from empty templates instead of more domain-related elements, that the makeup of the design team (a single teacher, a group of collaborating teachers, or a mix of teachers and project members) influences key design process characteristics such as time spent designing the ILS and number of actions involved, that the characteristics of the resulting ILSs also depend on the type of design team and that ILSs that are openly shared (i.e., published in a public repository) have different characteristics than those that are kept private.

A Scalable Architecture for the Dynamic Deployment of Multimodal Learning Analytics Applications in Smart Classrooms.

The smart classrooms of the future will use different software, devices and wearables as an integral part of the learning process. These educational applications generate a large amount of data from different sources. The area of Multimodal Learning Analytics (MMLA) explores the affordances of processing these heterogeneous data to understand and improve both learning and the context where it occurs. However, a review of different MMLA studies highlighted that ad-hoc and rigid architectures cannot be scaled up to real contexts. In this work, we propose a novel MMLA architecture that builds on software-defined networks and network function virtualization principles. We exemplify how this architecture can solve some of the detected challenges to deploy, dismantle and reconfigure the MMLA applications in a scalable way. Additionally, through some experiments, we demonstrate the feasibility and performance of our architecture when different classroom devices are reconfigured with diverse learning tools. These findings and the proposed architecture can be useful for other researchers in the area of MMLA and educational technologies envisioning the future of smart classrooms. Future work should aim to deploy this architecture in real educational scenarios with MMLA applications.

Multimodal Teaching Analytics: Automated Extraction of Orchestration Graphs from Wearable Sensor Data.

The pedagogical modelling of everyday classroom practice is an interesting kind of evidence, both for educational research and teachers' own professional development. This paper explores the usage of wearable sensors and machine learning techniques to automatically extract orchestration graphs (teaching activities and their social plane over time), on a dataset of 12 classroom sessions enacted by two different teachers in different classroom settings. The dataset included mobile eye-tracking as well as audiovisual and accelerometry data from sensors worn by the teacher. We evaluated both time-independent and time-aware models, achieving median F1 scores of about 0.7-0.8 on leave-one-session-out k-fold cross-validation. Although these results show the feasibility of this approach, they also highlight the need for larger datasets, recorded in a wider variety of classroom settings, to provide automated tagging of classroom practice that can be used in everyday practice across multiple teachers.