Toward robust and scalable deep spiking reinforcement learning.

Deep reinforcement learning (DRL) combines reinforcement learning algorithms with deep neural networks (DNNs). Spiking neural networks (SNNs) have been shown to be a biologically plausible and energy efficient alternative to DNNs. Since the introduction of surrogate gradient approaches that allowed to overcome the discontinuity in the spike function, SNNs can now be trained with the backpropagation through time (BPTT) algorithm. While largely explored on supervised learning problems, little work has been done on investigating the use of SNNs as function approximators in DRL. Here we show how SNNs can be applied to different DRL algorithms like Deep Q-Network (DQN) and Twin-Delayed Deep Deteministic Policy Gradient (TD3) for discrete and continuous action space environments, respectively. We found that SNNs are sensitive to the additional hyperparameters introduced by spiking neuron models like current and voltage decay factors, firing thresholds, and that extensive hyperparameter tuning is inevitable. However, we show that increasing the simulation time of SNNs, as well as applying a two-neuron encoding to the input observations helps reduce the sensitivity to the membrane parameters. Furthermore, we show that randomizing the membrane parameters, instead of selecting uniform values for all neurons, has stabilizing effects on the training. We conclude that SNNs can be utilized for learning complex continuous control problems with state-of-the-art DRL algorithms. While the training complexity increases, the resulting SNNs can be directly executed on neuromorphic processors and potentially benefit from their high energy efficiency.

The orthopedic diseases of ancient Egypt.

BACKGROUND: CT scanning of ancient human remains has the potential to provide insights into health and diseases. While Egyptian mummies have undergone CT scans in prior studies, a systematic survey of the orthopedic conditions afflicting a group of these ancient individuals has never been carried out. METHODS: We performed whole body CT scanning on 52 ancient Egyptian mummies using technique comparable to that of medical imaging. All of the large joints and the spine were systematically examined and osteoarthritic (OA) changes were scored 0-4 using Kellgren and Lawrence classification. RESULTS: The cruciate ligaments and menisci could be identified frequently. There were much more frequent OA changes in the spine (25 mummies) than in the large joints (15 cases of acromioclavicular and/or glenohumeral joint OA changes, five involvement of the ankle, one in the elbow, four in the knee, and one in the hip). There were six cases of scoliosis. Individual mummies had the following conditions: juvenile aseptic necrosis of the hip (Perthes disease), stage 4 osteochondritis dissecans of the knee, vertebral compression fracture, lateral patella-femoral joint hyper-compression syndrome, severe rotator cuff arthropathy, rotator cuff impingement, hip pincer impingement, and combined fracture of the greater trochantor and vertebral bodies indicating obvious traumatic injury. This report includes the most ancient discovery of several of these syndromes. CONCLUSIONS: Ancient Egyptians often suffered painful orthopedic conditions. The high frequency of scoliosis merits further study. The pattern of degenerative changes in the spine and joints may offer insights into activity levels of these people.