Granular estimation of user cognitive workload using multi-modal physiological sensors.

Mental workload (MWL) is a crucial area of study due to its significant influence on task performance and potential for significant operator error. However, measuring MWL presents challenges, as it is a multi-dimensional construct. Previous research on MWL models has focused on differentiating between two to three levels. Nonetheless, tasks can vary widely in their complexity, and little is known about how subtle variations in task difficulty influence workload indicators. To address this, we conducted an experiment inducing MWL in up to 5 levels, hypothesizing that our multi-modal metrics would be able to distinguish between each MWL stage. We measured the induced workload using task performance, subjective assessment, and physiological metrics. Our simulated task was designed to induce diverse MWL degrees, including five different math and three different verbal tiers. Our findings indicate that all investigated metrics successfully differentiated between various MWL levels induced by different tiers of math problems. Notably, performance metrics emerged as the most effective assessment, being the only metric capable of distinguishing all the levels. Some limitations were observed in the granularity of subjective and physiological metrics. Specifically, the subjective overall mental workload couldn't distinguish lower levels of workload, while all physiological metrics could detect a shift from lower to higher levels, but did not distinguish between workload tiers at the higher or lower ends of the scale (e.g., between the easy and the easy-medium tiers). Despite these limitations, each pair of levels was effectively differentiated by one or more metrics. This suggests a promising avenue for future research, exploring the integration or combination of multiple metrics. The findings suggest that subtle differences in workload levels may be distinguishable using combinations of subjective and physiological metrics.

A methodology for projecting the return on investment of training technologies.

Each year significant tax dollars are spent on the development of new technologies to increase efficiency and/or reduce costs of military training. However, there are currently no validated methods or measures to quantify the return on investment for adopting these new technologies for military training. Estimating the return on investment (ROI) for training technology adoption involves 1) developing a methodology or framework, 2) validating measures and methods, and 3) assessing predictive validity. The current paper describes a projective methodology using the Kirkpatrick framework to compare projected tangible and intangible benefits against tangible and intangible costs to estimate future ROI. The use-case involved an advanced technology demonstration in which sixty aircrew participated in a series of live, virtual, and constructive (LVC) exercises over a five-week period. Participants evaluated the technology's potential costs and benefits according to the Kirkpatrick framework of training program evaluation, and analyses resulted in a nominal projection of $488 million dollars saved, significant enhancements in large-force proficiency, and 1.4 lives saved over ten years at an implementation rate of 0.5% of budgeted flight hours. A discussion of theoretical implications, data-based limitations, and recommendations for future research are provided.

Teammates Instead of Tools: The Impacts of Level of Autonomy on Mission Performance and Human-Agent Teaming Dynamics in Multi-Agent Distributed Teams.

Human-agent teaming (HAT) is becoming more commonplace across industry, military, and consumer settings. Agents are becoming more advanced, more integrated, and more responsible for tasks previously assigned to humans. In addition, the dyadic human-agent teaming nature is evolving from a one-one pair to one-many, in which the human is working with numerous agents to accomplish a task. As capabilities become more advanced and humanlike, the best method for humans and agents to effectively coordinate is still unknown. Therefore, current research must start diverting focus from how many agents can a human manage to how can agents and humans work together effectively. Levels of autonomy (LOAs), or varying levels of responsibility given to the agents, implemented specifically in the decision-making process could potentially address some of the issues related to workload, stress, performance, and trust. This study sought to explore the effects of different LOAs on human-machine team coordination, performance, trust, and decision making in hand with assessments of operator workload and stress in a simulated multi-unmanned aircraft vehicle (UAV) intelligence surveillance and reconnaissance (ISR) task. The results of the study can be used to identify human factor roadblocks to effective HAT and provide guidance for future designs of HAT. Additionally, the unique impacts of LOA and autonomous decision making by agents on trust are explored.

Team task analysis: differentiating between tasks using team relatedness and team workflow as metrics of team task interdependence.

OBJECTIVE: As a constructive replication and extension of Arthur, Edwards, Bell, Villado, and Bennett (2005), the objective of the current study was to further investigate the efficacy of team relatedness and team workflow ratings (along with their composite) as metrics of interdependence. BACKGROUND: Although an analysis of task and job interdependence has important implications and uses in domains such as job design, selection, and training, the job analysis literature has been slow to develop an effective method to identify team-based tasks and jobs. METHOD: To achieve the study's objectives, 140 F-16 fighter pilots (35 four-person teams) rated 34 task and activity statements in terms of their team relatedness and team workflow. RESULTS: The results indicated that team relatedness and team workflow effectively differentiated between tasks with varying levels of interdependency (as identified by instructor pilots who served as subject matter experts) within the same job. In addition, teams that accurately perceived the level of interdependency performed better on a four-ship F-16 flight-training program than those that did not. CONCLUSION: Team relatedness and team workflow ratings can effectively differentiate between tasks with varying levels of interdependency. APPLICATION: Like traditional individual task or job analysis, this information can serve as the basis for specified human resource functions and interventions, and as diagnostic indicators as well.

Assessing measurement equivalence across rating sources: a multitrait-multirater approach.

Research examining the structure of multisource performance ratings has demonstrated that ratings are a direct function of both who is doing the rating (rating source) as well as what is being rated (performance dimension). A separate line of research has focused on the extent to which performance ratings are equivalent across sources. To date no research has examined the measurement equivalence of multisource ratings within the context of both dimension and rating source direct effects on ratings. We examine the impact of both performance dimension and rating source as well as the degree of measurement equivalence across sources. Results indicate that (a) the impact of the underlying performance dimension is the same across rating sources, (b) the impact of rating source is substantial and only slightly smaller than the impact of the underlying performance dimension, and (c) the impact of rating source differs substantially depending on the source.

Team task analysis: identifying tasks and jobs that are team based.

This paper presents initial information on the development and validation of three team task analysis scales. These scales were designed to quantitatively assess the extent to which a group of tasks or a job is team based. During a 2-week period, 52 male students working in 4-person teams were trained to perform a complex highly interdependent computer-simulated combat mission consisting of both individual- and team-based tasks. Our results indicated that the scales demonstrated high levels of interrater agreement. In addition, the scales differentiated between tasks that were predetermined to be individual versus team based. Finally, the results indicated that job-level ratings of team workflow were more strongly related to team performance than were aggregated task-level ratings of team-relatedness or team workflow. These results suggest that the scales presented here are an effective means of quantifying the extent to which tasks or jobs are team based. A research and practical implication of our findings is that the team task analysis scales could serve as criterion measures in the evaluation of team training interventions or predictors of team performance.

Effectiveness of training in organizations: a meta-analysis of design and evaluation features.

The authors used meta-analytic procedures to examine the relationship between specified training design and evaluation features and the effectiveness of training in organizations. Results of the meta-analysis revealed training effectiveness sample-weighted mean ds of 0.60 (k = 15, N = 936) for reaction criteria, 0.63 (k = 234, N = 15,014) for learning criteria, 0.62 (k = 122, N = 15,627) for behavioral criteria, and 0.62 (k = 26, N = 1,748) for results criteria. These results suggest a medium to large effect size for organizational training. In addition, the training method used, the skill or task characteristic trained, and the choice of evaluation criteria were related to the effectiveness of training programs. Limitations of the study along with suggestions for future research are discussed.