Making Bipedal Robot Experiments Reproducible and Comparable: The Eurobench Software Approach.

This study describes the software methodology designed for systematic benchmarking of bipedal systems through the computation of performance indicators from data collected during an experimentation stage. Under the umbrella of the European project Eurobench, we collected approximately 30 protocols with related testbeds and scoring algorithms, aiming at characterizing the performances of humanoids, exoskeletons, and/or prosthesis under different conditions. The main challenge addressed in this study concerns the standardization of the scoring process to permit a systematic benchmark of the experiments. The complexity of this process is mainly due to the lack of consistency in how to store and organize experimental data, how to define the input and output of benchmarking algorithms, and how to implement these algorithms. We propose a simple but efficient methodology for preparing scoring algorithms, to ensure reproducibility and replicability of results. This methodology mainly constrains the interface of the software and enables the engineer to develop his/her metric in his/her favorite language. Continuous integration and deployment tools are then used to verify the replicability of the software and to generate an executable instance independent of the language through dockerization. This article presents this methodology and points at all the metrics and documentation repositories designed with this policy in Eurobench. Applying this approach to other protocols and metrics would ease the reproduction, replication, and comparison of experiments.

Optimization of Dynamic Sit-to-Stand Trajectories to Assess Whole-Body Motion Performance of the Humanoid Robot REEM-C.

To enable the application of humanoid robots outside of laboratory environments, the biped must meet certain requirements. These include, in particular, coping with dynamic motions such as climbing stairs or ramps or walking over irregular terrain. Sit-to-stand transitions also belong to this category. In addition to their actual application such as getting out of vehicles or standing up after sitting, for example, at a table, these motions also provide benefits in terms of performance assessment. Therefore, they have long been used as a sports medical and geriatric assessment for humans. Here, we develop optimized sit-to-stand trajectories using optimal control, which are characterized by their dynamic and humanlike nature. We implement these motions on the humanoid robot REEM-C. Based on the obtained sensor data, we present a unified benchmarking procedure based on two different experimental protocols. These protocols are characterized by their increasing level of difficulty for quantifying different aspects of lower limb performance. We report performance results obtained by REEM-C using two categories of indicators: primary, scenario-specific indicators that assess overall performance (chair height and ankle-to-chair distance) and subsidiary, general indicators that further describe performance. The latter provide a more detailed analysis of the applied motion and are based on metrics such as the angular momentum, zero moment point, capture point, or foot placement estimator. In the process, we identify performance deficiencies of the robot based on the collected data. Thus, this work is an important step toward a unified quantification of bipedal performance in the execution of humanlike and dynamically demanding motions.

Performance Evaluation of Lower Limb Exoskeletons: A Systematic Review.

Benchmarks have long been used to verify and compare the readiness level of different technologies in many application domains. In the field of wearable robots, the lack of a recognized benchmarking methodology is one important impediment that may hamper the efficient translation of research prototypes into actual products. At the same time, an exponentially growing number of research studies are addressing the problem of quantifying the performance of robotic exoskeletons, resulting in a rich and highly heterogeneous picture of methods, variables and protocols. This review aims to organize this information, and identify the most promising performance indicators that can be converted into practical benchmarks. We focus our analysis on lower limb functions, including a wide spectrum of motor skills and performance indicators. We found that, in general, the evaluation of lower limb exoskeletons is still largely focused on straight walking, with poor coverage of most of the basic motor skills that make up the activities of daily life. Our analysis also reveals a clear bias towards generic kinematics and kinetic indicators, in spite of the metrics of human-robot interaction. Based on these results, we identify and discuss a number of promising research directions that may help the community to attain a comprehensive benchmarking methodology for robot-assisted locomotion more efficiently.