Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation.

Modeling human motor control and predicting how humans will move in novel environments is a grand scientific challenge. Researchers in the fields of biomechanics and motor control have proposed and evaluated motor control models via neuromechanical simulations, which produce physically correct motions of a musculoskeletal model. Typically, researchers have developed control models that encode physiologically plausible motor control hypotheses and compared the resulting simulation behaviors to measurable human motion data. While such plausible control models were able to simulate and explain many basic locomotion behaviors (e.g. walking, running, and climbing stairs), modeling higher layer controls (e.g. processing environment cues, planning long-term motion strategies, and coordinating basic motor skills to navigate in dynamic and complex environments) remains a challenge. Recent advances in deep reinforcement learning lay a foundation for modeling these complex control processes and controlling a diverse repertoire of human movement; however, reinforcement learning has been rarely applied in neuromechanical simulation to model human control. In this paper, we review the current state of neuromechanical simulations, along with the fundamentals of reinforcement learning, as it applies to human locomotion. We also present a scientific competition and accompanying software platform, which we have organized to accelerate the use of reinforcement learning in neuromechanical simulations. This "Learn to Move" competition was an official competition at the NeurIPS conference from 2017 to 2019 and attracted over 1300 teams from around the world. Top teams adapted state-of-the-art deep reinforcement learning techniques and produced motions, such as quick turning and walk-to-stand transitions, that have not been demonstrated before in neuromechanical simulations without utilizing reference motion data. We close with a discussion of future opportunities at the intersection of human movement simulation and reinforcement learning and our plans to extend the Learn to Move competition to further facilitate interdisciplinary collaboration in modeling human motor control for biomechanics and rehabilitation research.

Design of a Lightweight Soft Robotic Arm Using Pneumatic Artificial Muscles and Inflatable Sleeves.

As robots begin to interact with humans and operate in human environments, safety becomes a major concern. Conventional robots, although reliable and consistent, can cause injury to anyone within its range of motion. Soft robotics, wherein systems are made to be soft and mechanically compliant, are thus a promising alternative due to their lightweight nature and ability to cushion impacts, but current designs often sacrifice accuracy and usefulness for safety. We, therefore, have developed a bioinspired robotic arm combining elements of rigid and soft robotics such that it exhibits the positive qualities of both, namely compliance and accuracy, while maintaining a low weight. This article describes the design of a robotic arm-wrist-hand system with seven degrees of freedom (DOFs). The shoulder and elbow each has two DOFs for two perpendicular rotational motions on each joint, and the hand has two DOFs for wrist rotations and one DOF for a grasp motion. The arm is pneumatically powered using custom-built McKibben type pneumatic artificial muscles, which are inflated and deflated using binary and proportional valves. The wrist and hand motions are actuated through servomotors. In addition to the actuators, the arm is equipped with a potentiometer in each joint for detecting joint angle changes. Simulation and experimental results for closed-loop position control are also presented in the article.

Human-in-the-loop optimization of exoskeleton assistance during walking.

Exoskeletons and active prostheses promise to enhance human mobility, but few have succeeded. Optimizing device characteristics on the basis of measured human performance could lead to improved designs. We have developed a method for identifying the exoskeleton assistance that minimizes human energy cost during walking. Optimized torque patterns from an exoskeleton worn on one ankle reduced metabolic energy consumption by 24.2 ± 7.4% compared to no torque. The approach was effective with exoskeletons worn on one or both ankles, during a variety of walking conditions, during running, and when optimizing muscle activity. Finding a good generic assistance pattern, customizing it to individual needs, and helping users learn to take advantage of the device all contributed to improved economy. Optimization methods with these features can substantially improve performance.

Perturbation-dependent selection of postural feedback gain and its scaling.

In this study we examined whether the selection of postural feedback gain and its scaling is dependent on perturbation type. We compare forward pushes applied to the back of a standing subject to previous work on responses to support translation. As was done in the previous work, we quantified the subject's response in terms of perturbation-dependent feedback gains. Seven healthy young subjects (25±3 yr) experienced five different magnitudes of forward push applied by a 1.25 m-long pendulum falling from the height of 1.4m toward the center of mass of the subject's torso. The loads on the pendulum ranged from 2 to 10 kg. Impulsive force, ground reaction forces and joint kinematics were measured, and joint torques were calculated from inverse dynamics. A full-state feedback control model was used to quantify the empirical data, and the feedback gains that minimized the fitting error between the data and model simulation were identified. As in previously published feedback gains for support translation trials, gradual gain scaling with push perturbation magnitude was consistently observed, but a different feedback gain set was obtained. The results imply that the nervous system may be aware of body dynamics being subjected to various perturbation types and may select perturbation-dependent postural feedback gains that satisfy postural stability and feasible joint torque constraints.

Physical human interaction for an inflatable manipulator.

There is a growing need for robots that can function in close proximity to human beings and also physically interact with them safely. We believe inherent safety is extremely important for robots in human environments. Towards this end, we are exploring the use of inflatable structures for manipulators instead of traditional rigid structures, to improve safety in physical human robot interaction (pHRI). This paper develops a contact detection and reaction scheme for an inflatable manipulator prototype. The resulting scheme is used for physical interaction tasks with humans. Experiments verifying the efficacy of the contact detection scheme are shown using two interaction scenarios.

Random sampling of states in dynamic programming.

We combine three threads of research on approximate dynamic programming: sparse random sampling of states, value function and policy approximation using local models, and using local trajectory optimizers to globally optimize a policy and associated value function. Our focus is on finding steady-state policies for deterministic time-invariant discrete time control problems with continuous states and actions often found in robotics. In this paper, we describe our approach and provide initial results on several simulated robotics problems.

Sensory adaptation in human balance control: lessons for biomimetic robotic bipeds.

This paper describes mechanisms used by humans to stand on moving platforms, such as a bus or ship, and to combine body orientation and motion information from multiple sensors including vision, vestibular, and proprioception. A simple mechanism, sensory re-weighting, has been proposed to explain how human subjects learn to reduce the effects of inconsistent sensors on balance. Our goal is to replicate this robust balance behavior in bipedal robots. We review results exploring sensory re-weighting in humans and describe implementations of sensory re-weighting in simulation and on a robot.