Drosophila flying in augmented reality reveals the vision-based control autonomy of the optomotor response.

For walking, swimming, and flying animals, the optomotor response is essential to stabilize gaze. How flexible is the optomotor response? Classic work in Drosophila has argued that flies adapt flight control under augmented visual feedback conditions during goal-directed bar fixation. However, whether the lower-level, reflexive optomotor response can similarly adapt to augmented visual feedback (partially autonomous) or not (autonomous) over long timescales is poorly understood. To address this question, we developed an augmented reality paradigm to study the vision-based control autonomy of the yaw optomotor response of flying fruit flies (Drosophila). Flies were placed in a flight simulator, which permitted free body rotation about the yaw axis. By feeding back body movements in real time to a visual display, we augmented and inverted visual feedback. Thus, this experimental paradigm caused a constant visual error between expected and actual visual feedback to study potential adaptive visuomotor control. By combining experiments with control theory, we demonstrate that the optomotor response is autonomous during augmented reality flight bouts of up to 30 min, which exceeds the reported learning epoch during bar fixation. Agreement between predictions from linear systems theory and experimental data supports the notion that the optomotor response is approximately linear and time invariant within our experimental assay. Even under positive visual feedback, which revealed the stability limit of flies in augmented reality, the optomotor response was autonomous. Our results support a hierarchical motor control architecture in flies with fast and autonomous reflexes at the bottom and more flexible behavior at higher levels.

The influence of saccades on yaw gaze stabilization in fly flight.

In a way analogous to human vision, the fruit fly D. melanogaster and many other flying insects generate smooth and saccadic movements to stabilize and shift their gaze in flight, respectively. It has been hypothesized that this combination of continuous and discrete movements benefits both flight stability and performance, particularly at high frequencies or speeds. Here we develop a hybrid control system model to explore the effects of saccades on the yaw stabilization reflex of D. melanogaster. Inspired from experimental data, the model includes a first order plant, a Proportional-Integral (PI) continuous controller, and a saccadic reset system that fires based on the integrated error of the continuous controller. We explore the gain, delay and switching threshold parameter space to quantify the optimum regions for yaw stability and performance. We show that the addition of saccades to a continuous controller provides benefits to both stability and performance across a range of frequencies. Our model suggests that Drosophila operates near its optimal switching threshold for its experimental gain set. We also show that based on experimental data, D. melanogaster operates in a region that trades off performance and stability. This trade-off increases flight robustness to compensate for internal perturbations such as wing damage.

Flies adaptively control flight to compensate for added inertia.

Animal locomotion is highly adaptive, displaying a large degree of flexibility, yet how this flexibility arises from the integration of mechanics and neural control remains elusive. For instance, animals require flexible strategies to maintain performance as changes in mass or inertia impact stability. Compensatory strategies to mechanical loading are especially critical for animals that rely on flight for survival. To shed light on the capacity and flexibility of flight neuromechanics to mechanical loading, we pushed the performance of fruit flies (Drosophila) near its limit and implemented a control theoretic framework. Flies with added inertia were placed inside a virtual reality arena which permitted free rotation about the vertical (yaw) axis. Adding inertia increased the fly's response time yet had little influence on overall gaze stabilization performance. Flies maintained stability following the addition of inertia by adaptively modulating both visuomotor gain and damping. By contrast, mathematical modelling predicted a significant decrease in gaze stabilization performance. Adding inertia altered saccades, however, flies compensated for the added inertia by increasing saccade torque. Taken together, in response to added inertia flies increase reaction time but maintain flight performance through adaptive neural control. Overall, adding inertia decreases closed-loop flight robustness. Our work highlights the flexibility and capacity of motor control in flight.

Hawkmoths regulate flight torques with their abdomen for yaw control.

Many animals use body parts such as tails to stabilize posture while moving at high speed. In flying insects, leg or abdominal inertia can influence flight posture. In the hawkmoth Manduca sexta, the abdomen contributes ∼50% of the total body weight and it can therefore serve to inertially redirect flight forces. How do torques generated by the wings and abdomen interact for flight control? We studied the yaw optomotor response of M. sexta by using a torque sensor attached to their thorax. In response to yaw visual motion, the abdomen moved antiphase with the stimulus, head and total torque. By studying moths with ablated wings and a fixed abdomen, we resolved abdomen and wing torques and revealed their individual contribution to total yaw torque production. Frequency-domain analysis revealed that the abdomen torque is overall smaller than wing torque, although the abdomen torque is ∼80% of the wing torque at higher visual stimulus temporal frequency. Experimental data and modeling revealed that the wing and abdomen torque are transmitted linearly to the thorax. By modeling the thorax and abdomen as a two-link system, we show that abdomen flexion can inertially redirect the thorax to add constructively to wing steering efforts. Our work argues for considering the role of the abdomen in tethered insect flight experiments that use force/torque sensors. Taken together, the hawkmoth abdomen can regulate wing torques in free flight, which could modulate flight trajectories and increase maneuverability.

Flies trade off stability and performance via adaptive compensation to wing damage.

Physical injury often impairs mobility, which can have dire consequences for survival in animals. Revealing mechanisms of robust biological intelligence to prevent system failure can provide critical insights into how complex brains generate adaptive movement and inspiration to design fault-tolerant robots. For flying animals, physical injury to a wing can have severe consequences, as flight is inherently unstable. Using a virtual reality flight arena, we studied how flying fruit flies compensate for damage to one wing. By combining experimental and mathematical methods, we show that flies compensate for wing damage by corrective wing movement modulated by closed-loop sensing and robust mechanics. Injured flies actively increase damping and, in doing so, modestly decrease flight performance but fly as stably as uninjured flies. Quantifying responses to injury can uncover the flexibility and robustness of biological systems while informing the development of bio-inspired fault-tolerant strategies.

Nested mechanosensory feedback actively damps visually guided head movements in Drosophila.

Executing agile locomotion requires animals to integrate sensory feedback, often from multiple sources. For example, human gaze is mediated by multiple feedback loops that integrate visual and vestibular information. A central challenge in studying biological feedback loops is that they are nested and dynamically coupled. Here, we develop a framework based on control theory for unraveling nested feedback systems and apply it to study gaze stabilization in the fruit fly (Drosophila). By combining experimental and mathematical methods to manipulate control topologies, we uncovered the role of body-generated mechanosensory feedback nested within visual feedback in the control of head movements. We discovered that visual feedback changed the tuning of head movements across visual motion frequencies whereas mechanosensory feedback damped head movements. Head saccades had slower dynamics when the body was free to move, further pointing to the role of damping via mechanosensory feedback. By comparing head responses between self-generated and externally generated body motion, we revealed a nonlinear gating of mechanosensory feedback that is motor-context dependent. Altogether, our findings reveal the role of nested feedback loops in flies and uncover mechanisms that reconcile differences in head kinematics between body-free and body-fixed flies. Our framework is generalizable to biological and robotic systems relying on nested feedback control for guiding locomotion.

Complementary feedback control enables effective gaze stabilization in animals.

Visually active animals coordinate vision and movement to achieve spectacular tasks. An essential prerequisite to guide agile locomotion is to keep gaze level and stable. Since the eyes, head and body can move independently to control gaze, how does the brain effectively coordinate these distinct motor outputs? Furthermore, since the eyes, head, and body have distinct mechanical constraints (e.g., inertia), how does the nervous system adapt its control to these constraints? To address these questions, we studied gaze control in flying fruit flies (Drosophila) using a paradigm which permitted direct measurement of head and body movements. By combining experiments with mathematical modeling, we show that body movements are sensitive to the speed of visual motion whereas head movements are sensitive to its acceleration. This complementary tuning of the head and body permitted flies to stabilize a broader range of visual motion frequencies. We discovered that flies implement proportional-derivative (PD) control, but unlike classical engineering control systems, relay the proportional and derivative signals in parallel to two distinct motor outputs. This scheme, although derived from flies, recapitulated classic primate vision responses thus suggesting convergent mechanisms across phyla. By applying scaling laws, we quantify that animals as diverse as flies, mice, and humans as well as bio-inspired robots can benefit energetically by having a high ratio between head, body, and eye inertias. Our results provide insights into the mechanical constraints that may have shaped the evolution of active vision and present testable neural control hypotheses for visually guided behavior across phyla.

Mechanisms of punctuated vision in fly flight.

To guide locomotion, animals control gaze via movements of their eyes, head, and/or body, but how the nervous system controls gaze during complex motor tasks remains elusive. In many animals, shifts in gaze consist of periods of smooth movement punctuated by rapid eye saccades. Notably, eye movements are constrained by anatomical limits, which requires resetting eye position. By studying tethered, flying fruit flies (Drosophila), we show that flies perform stereotyped head saccades to reset gaze, analogous to optokinetic nystagmus in primates. Head-reset saccades interrupted head smooth movement for as little as 50 ms-representing less than 5% of the total flight time-thereby enabling punctuated gaze stabilization. By revealing the passive mechanics of the neck joint, we show that head-reset saccades leverage the neck's natural elastic recoil, enabling mechanically assisted redirection of gaze. The consistent head orientation at saccade initiation, the influence of the head's angular position on saccade rate, the decrease in wing saccade frequency in head-fixed flies, and the decrease in head-reset saccade rate in flies with their head range of motion restricted together implicate proprioception as the primary trigger of head-reset saccades. Wing-reset saccades were influenced by head orientation, establishing a causal link between neck sensory signals and the execution of body saccades. Head-reset saccades were abolished when flies switched to a landing state, demonstrating that head movements are gated by behavioral state. We propose a control architecture for active vision systems with limits in sensor range of motion. VIDEO ABSTRACT.

Multimodal Integration Across Spatiotemporal Scales to Guide Invertebrate Locomotion.

Locomotion is a hallmark of organisms which has enabled adaptive radiation to an extraordinarily diverse class of ecological niches, and allows animals to move across vast distances. Sampling from multiple sensory modalities enables animals to acquire rich information to guide locomotion. Locomotion without sensory feedback is haphazard; therefore, sensory and motor systems have evolved complex interactions to generate adaptive behavior. Notably, sensory-guided locomotion acts over broad spatial and temporal scales to permit goal-seeking behavior, whether to localize food by tracking an attractive odor plume or to search for a potential mate. How does the brain integrate multimodal stimuli over different temporal and spatial scales to effectively control behavior? In this review, we classify locomotion into three ordinally ranked hierarchical layers that act over distinct spatiotemporal scales: stabilization, motor primitives, and higher-order tasks, respectively. We discuss how these layers present unique challenges and opportunities for sensorimotor integration. We focus on recent advances in invertebrate locomotion due to their accessible neural and mechanical signals from the whole brain, limbs, and sensors. Throughout, we emphasize neural-level description of computations for multimodal integration in genetic model systems, including the fruit fly, Drosophila melanogaster, and the yellow fever mosquito, Aedes aegypti. We identify that summation (e.g., gating) and weighting-which are inherent computations of spiking neurons-underlie multimodal integration across spatial and temporal scales, therefore suggesting collective strategies to guide locomotion.

Hybrid visual control in fly flight: insights into gaze shift via saccades.

Flies fly by alternating between periods of fixation and body saccades, analogous to how our own eyes move. Gaze fixation via smooth movement in fly flight has been studied extensively, but comparatively less is known about the mechanism by which flies trigger and control body saccades to shift their gaze. Why do flies implement a hybrid fixate-and-saccade locomotion strategy? Here we review recent developments that provide new insights into this question. We focus on the interplay between smooth movement and saccades, the trigger classes of saccades, and the timeline of saccade execution. We emphasize recent mechanistic advances in Drosophila, where genetic tools have enabled cellular circuit analysis at an unprecedented level in a flying insect. In addition, we review trade-offs in behavioral paradigms used to study saccades. Throughout we highlight exciting avenues for future research in the control of fly flight.

Active vision shapes and coordinates flight motor responses in flies.

Animals use active sensing to respond to sensory inputs and guide future motor decisions. In flight, flies generate a pattern of head and body movements to stabilize gaze. How the brain relays visual information to control head and body movements and how active head movements influence downstream motor control remains elusive. Using a control theoretic framework, we studied the optomotor gaze stabilization reflex in tethered flight and quantified how head movements stabilize visual motion and shape wing steering efforts in fruit flies (Drosophila). By shaping visual inputs, head movements increased the gain of wing steering responses and coordination between stimulus and wings, pointing to a tight coupling between head and wing movements. Head movements followed the visual stimulus in as little as 10 ms-a delay similar to the human vestibulo-ocular reflex-whereas wing steering responses lagged by more than 40 ms. This timing difference suggests a temporal order in the flow of visual information such that the head filters visual information eliciting downstream wing steering responses. Head fixation significantly decreased the mechanical power generated by the flight motor by reducing wingbeat frequency and overall thrust. By simulating an elementary motion detector array, we show that head movements shift the effective visual input dynamic range onto the sensitivity optimum of the motion vision pathway. Taken together, our results reveal a transformative influence of active vision on flight motor responses in flies. Our work provides a framework for understanding how to coordinate moving sensors on a moving body.

Non-canonical Receptive Field Properties and Neuromodulation of Feature-Detecting Neurons in Flies.

Several fundamental aspects of motion vision circuitry are prevalent across flies and mice. Both taxa segregate ON and OFF signals. For any given spatial pattern, motion detectors in both taxa are tuned to speed, selective for one of four cardinal directions, and modulated by catecholamine neurotransmitters. These similarities represent conserved, canonical properties of the functional circuits and computational algorithms for motion vision. Less is known about feature detectors, including how receptive field properties differ from the motion pathway or whether they are under neuromodulatory control to impart functional plasticity for the detection of salient objects from a moving background. Here, we investigated 19 types of putative feature selective lobula columnar (LC) neurons in the optic lobe of the fruit fly Drosophila melanogaster to characterize divergent properties of feature selection. We identified LC12 and LC15 as feature detectors. LC15 encodes moving bars, whereas LC12 is selective for the motion of discrete objects, mostly independent of size. Neither is selective for contrast polarity, speed, or direction, highlighting key differences in the underlying algorithms for feature detection and motion vision. We show that the onset of background motion suppresses object responses by LC12 and LC15. Surprisingly, the application of octopamine, which is released during flight, reverses the suppressive influence of background motion, rendering both LCs able to track moving objects superimposed against background motion. Our results provide a comparative framework for the function and modulation of feature detectors and new insights into the underlying neuronal mechanisms involved in visual feature detection.

Fly eyes are not still: a motion illusion in Drosophila flight supports parallel visual processing.

Most animals shift gaze by a 'fixate and saccade' strategy, where the fixation phase stabilizes background motion. A logical prerequisite for robust detection and tracking of moving foreground objects, therefore, is to suppress the perception of background motion. In a virtual reality magnetic tether system enabling free yaw movement, Drosophila implemented a fixate and saccade strategy in the presence of a static panorama. When the spatial wavelength of a vertical grating was below the Nyquist wavelength of the compound eyes, flies drifted continuously and gaze could not be maintained at a single location. Because the drift occurs from a motionless stimulus - thus any perceived motion stimuli are generated by the fly itself - it is illusory, driven by perceptual aliasing. Notably, the drift speed was significantly faster than under a uniform panorama, suggesting perceptual enhancement as a result of aliasing. Under the same visual conditions in a rigid-tether paradigm, wing steering responses to the unresolvable static panorama were not distinguishable from those to a resolvable static pattern, suggesting visual aliasing is induced by ego motion. We hypothesized that obstructing the control of gaze fixation also disrupts detection and tracking of objects. Using the illusory motion stimulus, we show that magnetically tethered Drosophila track objects robustly in flight even when gaze is not fixated as flies continuously drift. Taken together, our study provides further support for parallel visual motion processing and reveals the critical influence of body motion on visuomotor processing. Motion illusions can reveal important shared principles of information processing across taxa.

Flies land upside down on a ceiling using rapid visually mediated rotational maneuvers.

Flies and other insects routinely land upside down on a ceiling. These inverted landing maneuvers are among the most remarkable aerobatic feats, yet the full range of these behaviors and their underlying sensorimotor processes remain largely unknown. Here, we report that successful inverted landing in flies involves a serial sequence of well-coordinated behavioral modules, consisting of an initial upward acceleration followed by rapid body rotation and leg extension, before terminating with a leg-assisted body swing pivoted around legs firmly attached to the ceiling. Statistical analyses suggest that rotational maneuvers are triggered when flies' relative retinal expansion velocity reaches a threshold. Also, flies exhibit highly variable pitch and roll rates, which are strongly correlated to and likely mediated by multiple sensory cues. When flying with higher forward or lower upward velocities, flies decrease the pitch rate but increase the degree of leg-assisted swing, thereby leveraging the transfer of body linear momentum.

Object features and T4/T5 motion detectors modulate the dynamics of bar tracking by Drosophila.

Visual objects can be discriminated by static spatial features such as luminance or dynamic features such as relative movement. Flies track a solid dark vertical bar moving on a bright background, a behavioral reaction so strong that for a rigidly tethered fly, the steering trajectory is phase advanced relative to the moving bar, apparently in anticipation of its future position. By contrast, flickering bars that generate no coherent motion or have a surface texture that moves in the direction opposite to the bar generate steering responses that lag behind the stimulus. It remains unclear how the spatial properties of a bar influence behavioral response dynamics. Here,  we show that a dark bar defined by its luminance contrast to the uniform background drives a co-directional steering response that is phase advanced relative to the response to a textured bar defined only by its motion relative to a stationary textured background. The textured bar drives an initial contra-directional turn and phase-locked tracking. The qualitatively distinct response dynamics could indicate parallel visual processing of a luminance versus motion-defined object. Calcium imaging shows that T4/T5 motion-detecting neurons are more responsive to a solid dark bar than a motion-defined bar. Genetically blocking T4/T5 neurons eliminates the phase-advanced co-directional response to the luminance-defined bar, leaving the orientation response largely intact. We conclude that T4/T5 neurons mediate a co-directional optomotor response to a luminance-defined bar, thereby driving phase-advanced wing kinematics, whereas separate unknown visual pathways elicit the contra-directional orientation response.

Visuomotor strategies for object approach and aversion in Drosophila melanogaster.

Animals classify stimuli to generate appropriate motor actions. In flight, Drosophila melanogaster classify equidistant large and small objects with categorically different behaviors: a tall object evokes approach whereas a small object elicits avoidance. We studied visuomotor behavior in rigidly and magnetically tethered D. melanogaster to reveal strategies that generate aversion to a small object. We discovered that small-object aversion in tethered flight is enabled by aversive saccades and smooth movement, which vary with the stimulus type. Aversive saccades in response to a short bar had different dynamics from approach saccades in response to a tall bar and the distribution of pre-saccade error angles was more stochastic for a short bar. Taken together, we show that aversive responses in D. melanogaster are driven in part by processes that elicit signed saccades with distinct dynamics and trigger mechanisms. Our work generates new hypotheses to study brain circuits that underlie classification of objects in D. melanogaster.

Transition by head-on collision: mechanically mediated manoeuvres in cockroaches and small robots.

Exceptional performance is often considered to be elegant and free of 'errors' or missteps. During the most extreme escape behaviours, neural control can approach or exceed its operating limits in response time and bandwidth. Here we show that small, rapid running cockroaches with robust exoskeletons select head-on collisions with obstacles to maintain the fastest escape speeds possible to transition up a vertical wall. Instead of avoidance, animals use their passive body shape and compliance to negotiate challenging environments. Cockroaches running at over 1 m or 50 body lengths per second transition from the floor to a vertical wall within 75 ms by using their head like an automobile bumper, mechanically mediating the manoeuvre. Inspired by the animal's behaviour, we demonstrate a passive, high-speed, mechanically mediated vertical transitions with a small, palm-sized legged robot. By creating a collision model for animal and human materials, we suggest a size dependence favouring mechanical mediation below 1 kg that we term the 'Haldane limit'. Relying on the mechanical control offered by soft exoskeletons represents a paradigm shift for understanding the control of small animals and the next generation of running, climbing and flying robots where the use of the body can off-load the demand for rapid sensing and actuation.

Drosophila Spatiotemporally Integrates Visual Signals to Control Saccades.

Like many visually active animals, including humans, flies generate both smooth and rapid saccadic movements to stabilize their gaze. How rapid body saccades and smooth movement interact for simultaneous object pursuit and gaze stabilization is not understood. We directly observed these interactions in magnetically tethered Drosophila free to rotate about the yaw axis. A moving bar elicited sustained bouts of saccades following the bar, with surprisingly little smooth movement. By contrast, a moving panorama elicited robust smooth movement interspersed with occasional optomotor saccades. The amplitude, angular velocity, and torque transients of bar-fixation saccades were finely tuned to the speed of bar motion and were triggered by a threshold in the temporal integral of the bar error angle rather than its absolute retinal position error. Optomotor saccades were tuned to the dynamics of panoramic image motion and were triggered by a threshold in the integral of velocity over time. A hybrid control model based on integrated motion cues simulates saccade trigger and dynamics. We propose a novel algorithm for tuning fixation saccades in flies.

Sensory processing within cockroach antenna enables rapid implementation of feedback control for high-speed running maneuvers.

Animals are remarkably stable during high-speed maneuvers. As the speed of locomotion increases, neural bandwidth and processing delays can limit the ability to achieve and maintain stable control. Processing the information of sensory stimuli into a control signal within the sensor itself could enable rapid implementation of whole-body feedback control during high-speed locomotion. Here, we show that processing in antennal afferents is sufficient to act as the control signal for a fast sensorimotor loop. American cockroaches Periplaneta americana use their antennae to mediate escape running by tracking vertical surfaces such as walls. A control theoretic model of wall following predicts that stable control is possible if the animal can compute wall position (P) and velocity, its derivative (D). Previous whole-nerve recordings from the antenna during simulated turning experiments demonstrated a population response consistent with P and D encoding, and suggested that the response was synchronized with the timing of a turn executed while wall following. Here, we record extracellularly from individual mechanoreceptors distributed along the antenna and show that these receptors encode D and have distinct latencies and filtering properties. The summed output of these receptors can be used as a control signal for rapid steering maneuvers. The D encoding within the antenna in addition to the temporal filtering properties and P dependence of the population of afferents support a sensory-encoding notion from control theory. Our findings support the notion that peripheral sensory processing can enable rapid implementation of whole-body feedback control during rapid running maneuvers.

Method and software for using m-sequences to characterize parallel components of higher-order visual tracking behavior in Drosophila.

A moving visual figure may contain first-order signals defined by variation in mean luminance, as well as second-order signals defined by constant mean luminance and variation in luminance envelope, or higher-order signals that cannot be estimated by taking higher moments of the luminance distribution. Separating these properties of a moving figure to experimentally probe the visual subsystems that encode them is technically challenging and has resulted in debated mechanisms of visual object detection by flies. Our prior work took a white noise systems identification approach using a commercially available electronic display system to characterize the spatial variation in the temporal dynamics of two distinct subsystems for first- and higher-order components of visual figure tracking. The method relied on the use of single pixel displacements of two visual stimuli according to two binary maximum length shift register sequences (m-sequences) and cross-correlation of each m-sequence with time-varying flight steering measurements. The resultant spatio-temporal action fields represent temporal impulse responses parameterized by the azimuthal location of the visual figure, one STAF for first-order and another for higher-order components of compound stimuli. Here we review m-sequence and reverse correlation procedures, then describe our application in detail, provide Matlab code, validate the STAFs, and demonstrate the utility and robustness of STAFs by predicting the results of other published experimental procedures. This method has demonstrated how two relatively modest innovations on classical white noise analysis--the inclusion of space as a way to organize response kernels and the use of linear decoupling to measure the response to two channels of visual information simultaneously--could substantially improve our basic understanding of visual processing in the fly.