Introduction to the Emerging Cognitive Science of Distributed Human-Autonomy Teams.

Teams are a fundamental aspect of life-from sports to business, to defense, to science, to education. While the cognitive sciences tend to focus on information processing within individuals, others have argued that teams are also capable of demonstrating cognitive capacities similar to humans, such as skill acquisition and forgetting (cf., Cooke, Gorman, Myers, & Duran, 2013; Fiore et al., 2010). As artificially intelligent and autonomous systems improve in their ability to learn, reason, interact, and coordinate with human teammates combined with the observation that teams can express cognitive capacities typically seen in individuals, a cognitive science of teams is emerging. Consequently, new questions are being asked about teams regarding teamness, trust, the introduction and effects of autonomous systems on teams, and how best to measure team behavior and phenomena. In this topic, four facets of human-autonomy team cognition are introduced with leaders in the field providing in-depth articles associated with one or more of the facets: (1) defining teams; (2) how trust is established, maintained, and repaired when broken; (3) autonomous systems operating as teammates; and (4) metrics for evaluating team cognition across communication, coordination, and performance.

An automated content-based measure of closed loop communication among critical care air transport teams.

Successful teamwork is essential to ensure critical care air transport (CCAT) patients receive effective care. Despite the importance of team performance, current training methods rely on subjective performance assessments and do not evaluate performance at the team level. Researchers have developed the Team Dynamics Measurement System (TDMS) to provide real-time, objective measures of team coordination to assist trainers in providing CCAT aircrew with feedback to improve performance. The first iteration of TDMS relied exclusively on communication flow patterns (i.e., who was speaking and when) to identify instances of various communication types such as closed loop communication (CLC). The research presented in this paper significantly advances the TDMS project by incorporating natural language processing (NLP) to identify CLC. The addition of NLP to the existing TDMS resulted in greater accuracy and fewer false alarms in identifying instances of CLC compared to the previous flow-based implementation. We discuss ways in which these improvements will facilitate instructor feedback and support further refinement of the TDMS.

Communication Strategies in Human-Autonomy Teams During Technological Failures.

OBJECTIVE: This study examines low-, medium-, and high-performing Human-Autonomy Teams' (HATs') communication strategies during various technological failures that impact routine communication strategies to adapt to the task environment. BACKGROUND: Teams must adapt their communication strategies during dynamic tasks, where more successful teams make more substantial adaptations. Adaptations in communication strategies may explain how successful HATs overcome technological failures. Further, technological failures of variable severity may alter communication strategies of HATs at different performance levels in their attempts to overcome each failure. METHOD: HATs in a Remotely Piloted Aircraft System-Synthetic Task Environment (RPAS-STE), involving three team members, were tasked with photographing targets. Each triad had two randomly assigned participants in navigator and photographer roles, teaming with an experimenter who simulated an AI pilot in a Wizard of Oz paradigm. Teams encountered two different technological failures, automation and autonomy, where autonomy failures were more challenging to overcome. RESULTS: High-performing HATs calibrated their communication strategy to the complexity of the different failures better than medium- and low-performing teams. Further, HATs adjusted their communication strategies over time. Finally, only the most severe failures required teams to increase the efficiency of their communication. CONCLUSION: HAT effectiveness under degraded conditions depends on the type of communication strategies enacted by the team. Previous findings from studies of all-human teams apply here; however, novel results suggest information requests are particularly important to HAT success during failures. APPLICATION: Understanding the communication strategies of HATs under degraded conditions can inform training protocols to help HATs overcome failures.

The Impact of Training on Human-Autonomy Team Communications and Trust Calibration.

OBJECTIVE: This work examines two human-autonomy team (HAT) training approaches that target communication and trust calibration to improve team effectiveness under degraded conditions. BACKGROUND: Human-autonomy teaming presents challenges to teamwork, some of which may be addressed through training. Factors vital to HAT performance include communication and calibrated trust. METHOD: Thirty teams of three, including one confederate acting as an autonomous agent, received either entrainment-based coordination training, trust calibration training, or control training before executing a series of missions operating a simulated remotely piloted aircraft. Automation and autonomy failures simulating degraded conditions were injected during missions, and measures of team communication, trust, and task efficiency were collected. RESULTS: Teams receiving coordination training had higher communication anticipation ratios, took photos of targets faster, and overcame more autonomy failures. Although autonomy failures were introduced in all conditions, teams receiving the calibration training reported that their overall trust in the agent was more robust over time. However, they did not perform better than the control condition. CONCLUSIONS: Training based on entrainment of communications, wherein introduction of timely information exchange through one team member has lasting effects throughout the team, was positively associated with improvements in HAT communications and performance under degraded conditions. Training that emphasized the shortcomings of the autonomous agent appeared to calibrate expectations and maintain trust. APPLICATIONS: Team training that includes an autonomous agent that models effective information exchange may positively impact team communication and coordination. Training that emphasizes the limitations of an autonomous agent may help calibrate trust.

The Role of Visual and Auditory Communication in the Performance of a Joint Team Task.

OBJECTIVE: This study examines visual, auditory, and the combination of both (bimodal) coupling modes in the performance of a two-person perceptual-motor task, in which one person provides the perceptual inputs and the other the motor inputs. BACKGROUND: Parking a plane or landing a helicopter on a mountain top requires one person to provide motor inputs while another person provides perceptual inputs. Perceptual inputs are communicated either visually, auditorily, or through both cues. METHODS: One participant drove a remote-controlled car around an obstacle and through a target, while another participant provided auditory, visual, or bimodal cues for steering and acceleration. Difficulty was manipulated using target size. Performance (trial time, path variability), cue rate, and spatial ability were measured. RESULTS: Visual coupling outperformed auditory coupling. Bimodal performance was best in the most difficult task condition but also high in the easiest condition. Cue rate predicted performance in all coupling modes. Drivers with lower spatial ability required a faster auditory cue rate, whereas drivers with higher ability performed best with a lower rate. CONCLUSION: Visual cues result in better performance when only one coupling mode is available. As predicted by multiple resource theory, when both cues are available, performance depends more on auditory cueing. In particular, drivers must be able to transform auditory cues into spatial actions. APPLICATION: Spotters should be trained to provide an appropriate cue rate to match the spatial ability of the driver or pilot. Auditory cues can enhance visual communication when the interpersonal task is visual with spatial outputs.

A Typology for the Application of Team Coordination Dynamics Across Increasing Levels of Dynamic Complexity.

OBJECTIVE: This review and synthesis examines approaches for measuring and assessing team coordination dynamics (TCD). The authors advance a system typology for classifying TCD approaches and their applications for increasing levels of dynamic complexity. BACKGROUND: There is an increasing focus on how teams adapt their coordination in response to changing and uncertain operational conditions. Understanding coordination is significant because poor coordination is associated with maladaptive responses, whereas adaptive coordination is associated with effective responses. This issue has been met with TCD approaches that handle increasing complexity in the types of TCD teams exhibit. METHOD: A three-level system typology of TCD approaches for increasing dynamic complexity is provided, with examples of research at each level. For System I TCD, team states converge toward a stable, fixed-point attractor. For System II TCD, team states are periodic, which can appear complex, yet are regular and relatively stable. In System III TCD, teams can exhibit periodic patterns, but those patterns change continuously to maintain effectiveness. RESULTS: System I and System II are applicable to TCD with known or discoverable behavioral attractors that are stationary across mid-to long-range timescales. System III TCD is the most generalizable to dynamic environments with high requirements for adaptive coordination across a range of timescales. CONCLUSION: We outline current challenges for TCD and next steps in this burgeoning field of research. APPLICATION: System III approaches are becoming widespread, as they are generalizable to time- and/or scale-varying TCD and multimodal analyses. Recommendations for deploying TCD in team settings are provided.

Using Communication to Modulate Neural Synchronization in Teams.

Throughout training and team performance, teams may be assessed based on their communication patterns to identify which behaviors contributed to the team's performance; however, this process of establishing meaning in communication is burdensome and time consuming despite the low monetary cost. A current topic in team research is developing covert measures, which are easier to analyze in real-time, to identify team processes as they occur during team performance; however, little is known about how overt and covert measures of team process relate to one another. In this study, we investigated the relationship between overt (communication) and covert (neural) measures of team process by manipulating the interaction partner (participant or experimenter) team members worked with and the type of task (decision-making or action-based) teams performed to assess their effects on team neural synchronization (measured as neurodynamic entropy) and communication (measured as both flow and content). The results indicated that the type of task affected how the teams structured their communication but had unpredictable effects on the neural synchronization of the team when averaged across the task session. The interaction partner did not affect team neural synchronization when averaged. However, there were significant relationships when communication and neural processes were examined over time between the neurodynamic entropy and the communication flow time series due to both the type of task and the interaction partner. Specifically, significant relationships across time were observed when participants were interacting with the other participant, during the second task trial, and across different regions of the cortex depending on the type of task being performed. The findings from the time series analyses suggest that factors that are previously known to affect communication (interaction partner and task type) also structure the relationship between team communication and neural synchronization-cross-level effects-but only when examined across time. Future research should consider these factors when developing new conceptualizations of team process measurement for measuring team performance over time.

Measuring Real-Time Team Cognition During Team Training.

OBJECTIVE: A method for detecting real-time changes in team cognition in the form of significant communication reorganizations is described. We demonstrate the method in the context of scenario-based simulation training. BACKGROUND: We present the dynamical view that individual- and team-level aspects of team cognition are temporally intertwined in a team's real-time response to challenging events. We suggest that this real-time response represents a fundamental team cognitive skill regarding the rapidity and appropriateness of the response, and methods and metrics are needed to track this skill. METHOD: Communication data from medical teams (Study 1) and submarine crews (Study 2) were analyzed for significant communication reorganization in response to training events. Mutual information between team members informed post hoc filtering to identify which team members contributed to reorganization. RESULTS: Significant communication reorganizations corresponding to challenging training events were detected for all teams. Less experienced teams tended to show delayed and sometimes ineffective responses that more experienced teams did not. Mutual information and post hoc filtering identified the individual-level inputs driving reorganization and potential mechanisms (e.g., leadership emergence, role restructuring) underlying reorganization. CONCLUSION: The ability of teams to rapidly and effectively reorganize coordination patterns as the situation demands is a team cognitive skill that can be measured and tracked. APPLICATION: Potential applications include team monitoring and assessment that would allow for visualization of a team's real-time response and provide individualized feedback based on team member's contributions to the team response.

Evaluating sociotechnical dynamics in a simulated remotely-piloted aircraft system: a layered dynamics approach.

As coordination mechanisms change and technology failures occur, a sociotechnical system must reorganise itself across human and technological layers to maintain effectiveness. We present a study examining reorganisation across communication, controls and vehicle layers of a remotely-piloted aircraft system (RPAS) using a layered dynamics approach. Team members (pilot; navigator; photographer) performed 5 simulated RPAS missions using different operator configurations, including all-human and human-autonomy teams. Reorganization (operationally defined using entropy) time series measured the changing system reorganisation profiles under different operator configurations and following autonomy failures. Correlations between these reorganisation profiles and team effectiveness scores describe the manner in which the system had to be coordinated to maintain effectiveness under these changing conditions. Four unplanned autonomy failures were analysed to visualise system reorganisation following a technology failure. With its objective and real-time modelling and measurement capabilities, layered dynamics complements existing systems thinking tools for understanding sociotechnical complexity and enhancing system effectiveness. Practitioner summary: A layered dynamics approach for understanding how a sociotechnical system dynamically reorganises itself is presented. The layered dynamics of RPAS were analysed under different operator configurations and following autonomy failures. Layered dynamics complements existing system-thinking tools for modelling sociotechnical system complexity and effectiveness. Abbreviation: RPAS: remotely-piloted aircraft system; HIS: human-systems integration; EAST: event analysis of systemic teamwork; H1: hypothesis 1; H2: hypothesis 2; H3: hypothesis 3; CERTT-STE: cognitive engineering research on team tasks--synthetic task environment; AVO: air vehicle operator; PLO: payload operator; DEMPC: data exploitation, mission planning, and communications; ACT-R: adaptive control of thought-rational; sec: seconds; ANOVA: analysis of variance.

Understanding and Modeling Teams As Dynamical Systems.

By its very nature, much of teamwork is distributed across, and not stored within, interdependent people working toward a common goal. In this light, we advocate a systems perspective on teamwork that is based on general coordination principles that are not limited to cognitive, motor, and physiological levels of explanation within the individual. In this article, we present a framework for understanding and modeling teams as dynamical systems and review our empirical findings on teams as dynamical systems. We proceed by (a) considering the question of why study teams as dynamical systems, (b) considering the meaning of dynamical systems concepts (attractors; perturbation; synchronization; fractals) in the context of teams, (c) describe empirical studies of team coordination dynamics at the perceptual-motor, cognitive-behavioral, and cognitive-neurophysiological levels of analysis, and (d) consider the theoretical and practical implications of this approach, including new kinds of explanations of human performance and real-time analysis and performance modeling. Throughout our discussion of the topics we consider how to describe teamwork using equations and/or modeling techniques that describe the dynamics. Finally, we consider what dynamical equations and models do and do not tell us about human performance in teams and suggest future research directions in this area.

Deviations from mirroring in interpersonal multifrequency coordination when visual information is occluded.

In activities such as dancing and sports, people synchronize behaviors in many different ways. Synchronization between people has traditionally been characterized as either perfect mirroring (1:1 in-phase synchronization, spontaneous synchrony, and mimicry) or reflectional mirroring (1:1 antiphase synchronization), but most activities require partners to synchronize more complicated patterns. We asked visually coupled dyads to coordinate finger movements to perform multifrequency ratios (1:1, 2:1, 3:1, 4:1, and 5:1). Because these patterns are coordinated across and not just within individual physiological and motor systems, we based our predictions on frequency-locking dynamics, which is a general coordination principle that is not limited to physiological explanations. Twenty dyads performed five multifrequency ratios under three levels of visual coupling, with half using a subcritical visual information update rate. The dynamical principle was supported, such that multifrequency performance tends to abide by the strictures of frequency locking. However, these constraints are relaxed if the visual information rate is beyond the critical information update rate. An analysis of turning points in the oscillatory finger movements suggested that dyads did not rely on this visual information to stabilize coordination. How the laboratory findings align with naturalistic observations of multifrequency performance in actual sports teams (Double Dutch) is discussed. Frequency-locking accounts not only for the human propensity for perfect mirroring but also for variations in performance when dyads deviate from mirroring.

Top-down (Prior Knowledge) and Bottom-up (Perceptual Modality) Influences on Spontaneous Interpersonal Synchronization.

Coordination with others is such a fundamental part of human activity that it can happen unintentionally. This unintentional coordination can manifest as synchronization and is observed in physical and human systems alike. We investigated the role of top-down influences (prior knowledge of the perceptual modality their partner is using) and bottom-up factors (perceptual modality combination) on spontaneous interpersonal synchronization. We examine this phenomena with respect to two different theoretical perspectives that differently emphasize top-down and bottom-up factors in interpersonal synchronization: joint-action/shared cognition theories and ecological-interactive theories. In an empirical study twelve dyads performed a finger oscillation task while attending to each other's movements through either visual, auditory, or visual and auditory perceptual modalities. Half of the participants were given prior knowledge of their partner's perceptual capabilities for coordinating across these different perceptual modality combinations. We found that the effect of top-down influence depends on the perceptual modality combination between two individuals. When people used the same perceptual modalities, top-down influence resulted in less synchronization and when people used different perceptual modalities, top-down influence resulted in more synchronization. Furthermore, persistence in the change in behavior as a result of having perceptual information about each other ('social memory') was stronger when this top-down influence was present.

Cross-Level Effects Between Neurophysiology and Communication During Team Training.

OBJECTIVE: We investigated cross-level effects, which are concurrent changes across neural and cognitive-behavioral levels of analysis as teams interact, between neurophysiology and team communication variables under variations in team training. BACKGROUND: When people work together as a team, they develop neural, cognitive, and behavioral patterns that they would not develop individually. It is currently unknown whether these patterns are associated with each other in the form of cross-level effects. METHOD: Team-level neurophysiology and latent semantic analysis communication data were collected from submarine teams in a training simulation. We analyzed whether (a) both neural and communication variables change together in response to changes in training segments (briefing, scenario, or debriefing), (b) neural and communication variables mutually discriminate teams of different experience levels, and (c) peak cross-correlations between neural and communication variables identify how the levels are linked. RESULTS: Changes in training segment led to changes in both neural and communication variables, neural and communication variables mutually discriminated between teams of different experience levels, and peak cross-correlations indicated that changes in communication precede changes in neural patterns in more experienced teams. CONCLUSION: Cross-level effects suggest that teamwork is not reducible to a fundamental level of analysis and that training effects are spread out across neural and cognitive-behavioral levels of analysis. Cross-level effects are important to consider for theories of team performance and practical aspects of team training. APPLICATION: Cross-level effects suggest that measurements could be taken at one level (e.g., neural) to assess team experience (or skill) on another level (e.g., cognitive-behavioral).

Learning to tie well with others: bimanual versus intermanual performance of a highly practised skill.

Studies indicate that novices are faster in manual tasks when performing with a partner ('intermanual') than with their own two hands ('bimanual'). The generality of this 'mode effect' was examined using a highly practised bimanual task, shoe tying, at which participants were experts. Speed-variability correlations confirmed participants were bimanually skilled but not intermanually skilled. Contrary to results using novices, intermanual was slower, such that prior skill reverses the effect. Analyses incorporating the similarity of shoe-tying strategies across dyads implicated a perceptual rather than shared knowledge/representation basis for intermanual performance. Practice effects indicated that intermanual performance built upon prior bimanual skill, such that novel relative timings between dyads' hands must be acquired. Motor transfer effects provided support for this conclusion. During shoe tying, hands were tightly coupled in the intermanual mode due to the perceptual coupling constraints of intermanual performance. Increased coupling was correlated with slower performance. Implications for real-world tasks (e.g. surgical knot tying) are described.

Neural signatures of team coordination are revealed by multifractal analysis.

The quality of a team depends on its ability to deliver information through a hierarchy of team members and negotiate processes spanning different time scales. That structure and the behavior that results from it pose problems for researchers because multiply-nested interactions are not easily separated. We explored the behavior of a six-person team engaged in a Submarine Piloting and Navigation (SPAN) task using the tools of dynamical systems. The data were a single entropy time series that showed the distribution of activity across six team members, as recorded by nine-channel electroencephalography (EEG). A single team's data were analyzed for the purposes of illustrating the utility of multifractal analysis and allowing for in-depth exploratory analysis of temporal characteristics. Could the meaningful events experienced by one of these teams be captured using multifractal analysis, a dynamical systems tool that is specifically designed to extract patterns across levels of analysis? Results indicate that nested patterns of team activity can be identified from neural data streams, including both routine and novel events. The novelty of this tool is the ability to identify social patterns from the brain activity of individuals in the social interaction. Implications for application and future directions of this research are discussed.

Are two hands (from different people) better than one? Mode effects and differential transfer between manual coordination modes.

OBJECTIVE: We report an experiment in which we investigated differential transfer between unimanual (one-handed), bimanual (two-handed), and intermanual (different peoples' hands) coordination modes. BACKGROUND: People perform some manual tasks faster than others ("mode effects"). However, little is known about transfer between coordination modes. To investigate differential transfer, we draw hypotheses from two perspectives--information based and constraint based--of bimanual and interpersonal coordination and skill acquisition. METHOD: Participants drove a teleoperated rover around a circular path in sets of two 2-min trials using two of the different coordination modes. Speed and variability of the rover's path were measured. Order of coordination modes was manipulated to examine differential transfer and mode effects. RESULTS: Differential transfer analyses revealed patterns of positive transfer from simpler (localized spatiotemporal constraints) to more complex (distributed spatiotemporal constraints) coordination modes paired with negative transfer in the opposite direction. Mode effects indicated that intermanual performance was significantly faster than unimanual performance, and bimanual performance was intermediate. Importantly, all of these effects disappeared with practice. CONCLUSION: The observed patterns of differential transfer between coordination modes may be better accounted for by a constraint-based explanation of differential transfer than by an information-based one. Mode effects may be attributable to anticipatory movements based on dyads' access to mutual visual information. APPLICATION: Although people may be faster using more-complex coordination modes, when operators transition between modes, they may be more effective transitioning from simpler (e.g., bimanual) to more complex (e.g., intermanual) modes than vice versa. However, this difference may be critical only for novel or rarely practiced tasks.

Interactive team cognition.

Cognition in work teams has been predominantly understood and explained in terms of shared cognition with a focus on the similarity of static knowledge structures across individual team members. Inspired by the current zeitgeist in cognitive science, as well as by empirical data and pragmatic concerns, we offer an alternative theory of team cognition. Interactive Team Cognition (ITC) theory posits that (1) team cognition is an activity, not a property or a product; (2) team cognition should be measured and studied at the team level; and (3) team cognition is inextricably tied to context. There are implications of ITC for theory building, modeling, measurement, and applications that make teams more effective performers.

The organizational neurodynamics of teams.

Our objective was to apply ideas from complexity theory to derive expanded neurodynamic models of Submarine Piloting and Navigation showing how teams cognitively organize around task changes. The cognitive metric highlighted was an electroencephalography-derived measure of engagement (termed neurophysiologic synchronies of engagement) that was modeled into collective team variables showing the engagement of each of six team members as well as that of the team as a whole. We modeled the cognitive organization of teams using the information content of the neurophysiologic data streams derived from calculations of their Shannon entropy. We show that the periods of team cognitive reorganization (a) occurred as a natural product of teamwork particularly around periods of stress, (b) appeared structured around episodes of communication, (c) occurred following deliberate external perturbation to team function, and (d) were less frequent in experienced navigation teams. These periods of reorganization were lengthy, lasting up to 10 minutes. As the overall entropy levels of the neurophysiologic data stream are significantly higher for expert teams, this measure may be a useful candidate for modeling teamwork and its development over prolonged periods of training.

Measuring patterns in team interaction sequences using a discrete recurrence approach.

OBJECTIVE: Recurrence-based measures of communication determinism and pattern information are described and validated using previously collected team interaction data. BACKGROUND: Team coordination dynamics has revealed that"mixing" team membership can lead to flexible interaction processes, but keeping a team "intact" can lead to rigid interaction processes. We hypothesized that communication of intact teams would have greater determinism and higher pattern information compared to that of mixed teams. METHOD: Determinism and pattern information were measured from three-person Uninhabited Air Vehicle team communication sequences over a series of 40-minute missions. Because team members communicated using push-to-talk buttons, communication sequences were automatically generated during each mission. RESULTS: The Composition x Mission determinism effect was significant. Intact teams' determinism increased over missions, whereas mixed teams' determinism did not change. Intact teams had significantly higher maximum pattern information than mixed teams. CONCLUSION: Results from these new communication analysis methods converge with content-based methods and support our hypotheses. APPLICATION: Because they are not content based, and because they are automatic and fast, these new methods may be amenable to real-time communication pattern analysis.