An Exploratory Multi-Session Study of Learning High-Dimensional Body-Machine Interfacing for Assistive Robot Control.

Individuals who suffer from severe paralysis often lose the capacity to perform fundamental body movements and everyday activities. Empowering these individuals with the ability to operate robotic arms, in high degrees-of-freedom (DoFs), can help to maximize both functional utility and independence. However, robot teleoperation in high DoFs currently lacks accessibility due to the challenge in capturing high-dimensional control signals from the human, especially in the face of motor impairments. Body-machine interfacing is a viable option that offers the necessary high-dimensional motion capture, and it moreover is noninvasive, affordable, and promotes movement and motor recovery. Nevertheless, to what extent body-machine interfacing is able to scale to high-DoF robot control, and whether it is feasible for humans to learn, remains an open question. In this exploratory multi-session study, we demonstrate the feasibility of human learning to operate a body-machine interface to control a complex, assistive robotic arm. We use a sensor net of four inertial measurement unit sensors, bilaterally placed on the scapulae and humeri. Ten uninjured participants are familiarized, trained, and evaluated in reaching and Activities of Daily Living tasks, using the body- machine interface. Our results suggest the manner of control space mapping (joint-space control versus task-space control), from interface to robot, plays a critical role in the evolution of human learning. Though joint-space control shows to be more intuitive initially, task-space control is found to have a greater capacity for longer-term improvement and learning.

Characterizing Eye Gaze for Assistive Device Control.

Eye gaze tracking is increasingly popular due to improved technology and availability. However, in assistive device control, eye gaze tracking is often limited to discrete control inputs. In this paper, we present a method for collecting both reactionary and control eye gaze signals to build an individualized characterization for eye gaze interface use. Results from a study conducted with motor-impaired participants are presented, offering insights into maximizing the potential of eye gaze for assistive device control. These findings can inform the development of continuous control paradigms using eye gaze.

The Impact of Control Interface on Features of Heart Rate Variability.

Shared human-robot control for assistive machines can improve the independence of individuals with motor impairments. Monitoring elevated levels of workload can enable the assistive autonomy to adjust the control sharing in an assist-as-needed way, to achieve a balance between user fatigue, stress, and independent control. In this work, we aim to investigate how heart rate variability features can be utilized to monitor elevated levels of mental workload while operating a powered wheelchair, and how that utilization might vary under different control interfaces. To that end, we conduct a 22 person study with three commercial interfaces. Our results show that the validity and reliability of using the ultra-short-term heart-rate variability features as predictors of workload indeed are affected by the type of interface in use.

Active Intent Disambiguation for Shared Control Robots.

Assistive shared-control robots have the potential to transform the lives of millions of people afflicted with severe motor impairments. The usefulness of shared-control robots typically relies on the underlying autonomy's ability to infer the user's needs and intentions, and the ability to do so unambiguously is often a limiting factor for providing appropriate assistance confidently and accurately. The contributions of this paper are four-fold. First, we introduce the idea of intent disambiguation via control mode selection, and present a mathematical formalism for the same. Second, we develop a control mode selection algorithm which selects the control mode in which the user-initiated motion helps the autonomy to maximally disambiguate user intent. Third, we present a pilot study with eight subjects to evaluate the efficacy of the disambiguation algorithm. Our results suggest that the disambiguation system (a) helps to significantly reduce task effort, as measured by number of button presses, and (b) is of greater utility for more limited control interfaces and more complex tasks. We also observe that (c) subjects demonstrated a wide range of disambiguation request behaviors, with the common thread of concentrating requests early in the execution. As our last contribution, we introduce a novel field-theoretic approach to intent inference inspired by dynamic field theory that works in tandem with the disambiguation scheme.

Interface Operation and Implications for Shared-Control Assistive Robots.

Shared-control for assistive devices can increase the independence of individuals with motor impairments. However, each person is unique in their level of injury and physical constraints. Consequently, a plethora of interfaces are used to control the assistive device, depending on the individual. In order to be effective, the shared-autonomy assistance should be aware of the usage characteristics of the interface and adjust to varying performance characteristics of the person. To that end, we conduct a 23 person (9 spinal cord injured and 14 uninjured) study using three commercial interfaces used to operate powered wheelchairs, and establish performance measures to characterize interface usage. The analyses of our performance measures unveil key aspects of the interface operation that can inform features of a customizable and interface-aware control sharing framework.

Autonomy in Rehabilitation Robotics: An Intersection.

Within the field of human rehabilitation, robotic machines are used both to rehabilitate the body and to perform functional tasks. Robotics autonomy able to perceive the external world and reason about high-level control decisions, however, seldom is present in these machines. For functional tasks in particular, autonomy could help to decrease the operational burden on the human and perhaps even to increase access-and this potential only grows as human motor impairments become more severe. There are however serious, and often subtle, considerations to introducing clinically-feasible robotics autonomy to rehabilitation robots and machines. Today the fields of robotics autonomy and rehabilitation robotics are largely separate. The topic of this article is at the intersection of these fields: the introduction of clinically-feasible autonomy solutions to rehabilitation robots, and opportunities for autonomy within the rehabilitation domain.

Prediction of user preference over shared-control paradigms for a robotic wheelchair.

The design of intelligent powered wheelchairs has traditionally focused heavily on providing effective and efficient navigation assistance. Significantly less attention has been given to the end-user's preference between different assistance paradigms. It is possible to include these subjective evaluations in the design process, for example by soliciting feedback in post-experiment questionnaires. However, constantly querying the user for feedback during real-world operation is not practical. In this paper, we present a model that correlates objective performance metrics and subjective evaluations of autonomous wheelchair control paradigms. Using off-the-shelf machine learning techniques, we show that it is possible to build a model that can predict the most preferred shared-control method from task execution metrics such as effort, safety, performance and utilization. We further characterize the relative contributions of each of these metrics to the individual choice of most preferred assistance paradigm. Our evaluation includes Spinal Cord Injured (SCI) and uninjured subject groups. The results show that our proposed correlation model enables the continuous tracking of user preference and offers the possibility of autonomy that is customized to each user.

A Human-Robot Interaction Perspective on Assistive and Rehabilitation Robotics.

Assistive and rehabilitation devices are a promising and challenging field of recent robotics research. Motivated by societal needs such as aging populations, such devices can support motor functionality and subject training. The design, control, sensing, and assessment of the devices become more sophisticated due to a human in the loop. This paper gives a human-robot interaction perspective on current issues and opportunities in the field. On the topic of control and machine learning, approaches that support but do not distract subjects are reviewed. Options to provide sensory user feedback that are currently missing from robotic devices are outlined. Parallels between device acceptance and affective computing are made. Furthermore, requirements for functional assessment protocols that relate to real-world tasks are discussed. In all topic areas, the design of human-oriented frameworks and methods is dominated by challenges related to the close interaction between the human and robotic device. This paper discusses the aforementioned aspects in order to open up new perspectives for future robotic solutions.

Human-in-the-Loop Optimization of Shared Autonomy in Assistive Robotics.

In this paper, we propose a mathematical framework which formalizes user-driven customization of shared autonomy in assistive robotics as a nonlinear optimization problem. Our insight is to allow the end-user, rather than relying on standard optimization techniques, to perform the optimization procedure, thereby allowing us to leave the exact nature of the cost function indeterminate. We ground our formalism with an interactive optimization procedure that customizes control sharing using an assistive robotic arm. We also present a pilot study that explores interactive optimization with end-users. This study was performed with 17 subjects (4 with spinal cord injury, 13 without injury). Results show all subjects were able to converge to an assistance paradigm, suggesting the existence of optimal solutions. Notably, the amount of assistance was not always optimized for task performance. Instead, some subjects favored retaining more control during the execution over better task performance. The study supports the case for user-driven customization and provides guidance for its continued development and study.

Using machine learning to blend human and robot controls for assisted wheelchair navigation.

This work presents an algorithm for collaborative control of an assistive semi-autonomous wheelchair. Our approach is based on a statistical machine learning technique to learn task variability from demonstration examples. The algorithm has been developed in the context of shared-control powered wheelchairs that provide assistance to individuals with impairments that affect their control in challenging driving scenarios, like doorway navigation. We validate our algorithm within a simulation environment, and find that with relatively few demonstrations, our approach allows for safe traversal of the doorway while maintaining a high level of user control.

Simplified intersubject averaging on the cortical surface using SUMA.

Task and group comparisons in functional magnetic resonance imaging (fMRI) studies are often accomplished through the creation of intersubject average activation maps. Compared with traditional volume-based intersubject averages, averages made using computational models of the cortical surface have the potential to increase statistical power because they reduce intersubject variability in cortical folding patterns. We describe a two-step method for creating intersubject surface averages. In the first step cortical surface models are created for each subject and the locations of the anterior and posterior commissures (AC and PC) are aligned. In the second step each surface is standardized to contain the same number of nodes with identical indexing. An anatomical average from 28 subjects created using the AC-PC technique showed greater sulcal and gyral definition than the corresponding volume-based average. When applied to an fMRI dataset, the AC-PC method produced greater maximum, median, and mean t-statistics in the average activation map than did the volume average and gave a better approximation to the theoretical-ideal average calculated from individual subjects. The AC-PC method produced average activation maps equivalent to those produced with surface-averaging methods that use high-dimensional morphing. In comparison with morphing methods, the AC-PC technique does not require selection of a template brain and does not introduce deformations of sulcal and gyral patterns, allowing for group analysis within the original folded topology of each individual subject. The tools for performing AC-PC surface averaging are implemented and freely available in the SUMA software package.

Unraveling multisensory integration: patchy organization within human STS multisensory cortex.

Although early sensory cortex is organized along dimensions encoded by receptor organs, little is known about the organization of higher areas in which different modalities are integrated. We investigated multisensory integration in human superior temporal sulcus using recent advances in parallel imaging to perform functional magnetic resonance imaging (fMRI) at very high resolution. These studies suggest a functional architecture in which information from different modalities is brought into close proximity via a patchy distribution of inputs, followed by integration in the intervening cortex.

Integration of auditory and visual information about objects in superior temporal sulcus.

Two categories of objects in the environment-animals and man-made manipulable objects (tools)-are easily recognized by either their auditory or visual features. Although these features differ across modalities, the brain integrates them into a coherent percept. In three separate fMRI experiments, posterior superior temporal sulcus and middle temporal gyrus (pSTS/MTG) fulfilled objective criteria for an integration site. pSTS/MTG showed signal increases in response to either auditory or visual stimuli and responded more to auditory or visual objects than to meaningless (but complex) control stimuli. pSTS/MTG showed an enhanced response when auditory and visual object features were presented together, relative to presentation in a single modality. Finally, pSTS/MTG responded more to object identification than to other components of the behavioral task. We suggest that pSTS/MTG is specialized for integrating different types of information both within modalities (e.g., visual form, visual motion) and across modalities (auditory and visual).