Understanding and mitigating noise in trained deep neural networks.

Deep neural networks unlocked a vast range of new applications by solving tasks of which many were previously deemed as reserved to higher human intelligence. One of the developments enabling this success was a boost in computing power provided by special purpose hardware, such as graphic or tensor processing units. However, these do not leverage fundamental features of neural networks like parallelism and analog state variables. Instead, they emulate neural networks relying on binary computing, which results in unsustainable energy consumption and comparatively low speed. Fully parallel and analogue hardware promises to overcome these challenges, yet the impact of analogue neuron noise and its propagation, i.e. accumulation, threatens rendering such approaches inept. Here, we determine for the first time the propagation of noise in deep neural networks comprising noisy nonlinear neurons in trained fully connected layers. We study additive and multiplicative as well as correlated and uncorrelated noise, and develop analytical methods that predict the noise level in any layer of symmetric deep neural networks or deep neural networks trained with back propagation. We find that noise accumulation is generally bound, and adding additional network layers does not worsen the signal to noise ratio beyond a limit. Most importantly, noise accumulation can be suppressed entirely when neuron activation functions have a slope smaller than unity. We therefore developed the framework for noise in fully connected deep neural networks implemented in analog systems, and identify criteria allowing engineers to design noise-resilient novel neural network hardware.

Two-color optically addressed spatial light modulator as a generic spatiotemporal system.

Nonlinear spatiotemporal systems are the basis for countless physical phenomena in such diverse fields as ecology, optics, electronics, and neuroscience. The canonical approach to unify models originating from different fields is the normal form description, which determines the generic dynamical aspects and different bifurcation scenarios. Realizing different types of dynamical systems via one experimental platform that enables continuous transition between normal forms through tuning accessible system parameters is, therefore, highly relevant. Here, we show that a transmissive, optically addressed spatial light modulator under coherent optical illumination and optical feedback coupling allows tuning between pitchfork, transcritical, and saddle-node bifurcations of steady states. We demonstrate this by analytically deriving the system's dynamical equations in correspondence to the normal forms of the associated bifurcations and confirm these results via extensive numerical simulations. Our model describes a nematic liquid crystal device using nano-dimensional dichalcogenide (a-As 2S 3) glassy thin films as photo sensors and alignment layers, and we use device parameters obtained from experimental characterization. Optical coupling, for example, using diffraction, holography, or integrated unitary maps allows implementing a variety of system topologies of technological relevance for neural networks and potentially Ising or XY-Hamiltonian models with ultralow energy consumption.

Advancing Fourier: space-time concepts in ultrafast optics, imaging, and photonic neural networks.

The concepts of Fourier optics were established in France in the 1940s by Pierre-Michel Duffieux, and laid the foundations of an extensive series of activities in the French research community that have touched on nearly every aspect of contemporary optics and photonics. In this paper, we review a selection of results where applications of the Fourier transform and transfer functions in optics have been applied to yield significant advances in unexpected areas of optics, including the spatial shaping of complex laser beams in amplitude and in phase, real-time ultrafast measurements, novel ghost imaging techniques, and the development of parallel processing methodologies for photonic artificial intelligence.

Coupled Nonlinear Delay Systems as Deep Convolutional Neural Networks.

Neural networks are transforming the field of computer algorithms, yet their emulation on current computing substrates is highly inefficient. Reservoir computing was successfully implemented on a large variety of substrates and gave new insight in overcoming this implementation bottleneck. Despite its success, the approach lags behind the state of the art in deep learning. We therefore extend time-delay reservoirs to deep networks and demonstrate that these conceptually correspond to deep convolutional neural networks. Convolution is intrinsically realized on a substrate level by generic drive-response properties of dynamical systems. The resulting novelty is avoiding vector matrix products between layers, which cause low efficiency in today's substrates. Compared to singleton time-delay reservoirs, our deep network achieves accuracy improvements by at least an order of magnitude in Mackey-Glass and Lorenz time series prediction.

Delayed dynamical systems: networks, chimeras and reservoir computing.

We present a systematic approach to reveal the correspondence between time delay dynamics and networks of coupled oscillators. After early demonstrations of the usefulness of spatio-temporal representations of time-delay system dynamics, extensive research on optoelectronic feedback loops has revealed their immense potential for realizing complex system dynamics such as chimeras in rings of coupled oscillators and applications to reservoir computing. Delayed dynamical systems have been enriched in recent years through the application of digital signal processing techniques. Very recently, we have showed that one can significantly extend the capabilities and implement networks with arbitrary topologies through the use of field programmable gate arrays. This architecture allows the design of appropriate filters and multiple time delays, and greatly extends the possibilities for exploring synchronization patterns in arbitrary network topologies. This has enabled us to explore complex dynamics on networks with nodes that can be perfectly identical, introduce parameter heterogeneities and multiple time delays, as well as change network topologies to control the formation and evolution of patterns of synchrony. This article is part of the theme issue 'Nonlinear dynamics of delay systems'.

Dependence of quality factor on surface roughness in crystalline whispering-gallery mode resonators.

We present an experimental study of the variation of quality factor (Q-factor) of WGM resonators as a function of surface roughness. We consider mm-size whispering-gallery mode resonators manufactured with fluoride crystals, featuring Q-factors of the order of 1 billion at 1550 nm. The experimental procedure consists of repeated polishing steps, after which the surface roughness is evaluated using profilometry by white-light phase-shifting interferometry, while the Q-factors are determined using the cavity-ring-down method. This protocol permits us to establish an explicit curve linking the Q-factor of the disk-resonator to the surface roughness of the rim. We have performed measurements with four different crystals, namely, magnesium, calcium, strontium, and lithium fluoride. We have thereby found that the variations of Q-factor as a function of surface roughness is universal, in the sense that it is globally independent of the bulk material under consideration. We also discuss our experimental results in the light of theoretical estimates of surface scattering Q-factors already published in the literature.

Dynamical complexity and computation in recurrent neural networks beyond their fixed point.

Spontaneous activity found in neural networks usually results in a reduction of computational performance. As a consequence, artificial neural networks are often operated at the edge of chaos, where the network is stable yet highly susceptible to input information. Surprisingly, regular spontaneous dynamics in Neural Networks beyond their resting state possess a high degree of spatio-temporal synchronization, a situation that can also be found in biological neural networks. Characterizing information preservation via complexity indices, we show how spatial synchronization allows rRNNs to reduce the negative impact of regular spontaneous dynamics on their computational performance.

Multi-Gbit/s optical phase chaos communications using a time-delayed optoelectronic oscillator with a three-wave interferometer nonlinearity.

We propose a chaos communication scheme based on a chaotic optical phase carrier generated with an optoelectronic oscillator with nonlinear time-delay feedback. The system includes a dedicated non-local nonlinearity, which is a customized three-wave imbalanced interferometer. This particular feature increases the complexity of the chaotic waveform and thus the security of the transmitted information, as these interferometers are characterized by four independent parameters which are part of the secret key for the chaos encryption scheme. We first analyze the route to chaos in the system, and evidence a sequence of period doubling bifurcations from the steady-state to fully developed chaos. Then, in the chaotic regime, we study the synchronization between the emitter and the receiver, and achieve chaotic carrier cancellation with a signal-to-noise ratio up to 20 dB. We finally demonstrate error-free chaos communications at a data rate of 3 Gbit/s.

Consistency in experiments on multistable driven delay systems.

We investigate the consistency properties in the responses of a nonlinear delay optoelectronic intensity oscillator subject to different drives, in particular, harmonic and self-generated waveforms. This system, an implementation of the Ikeda oscillator, is operating in a closed-loop configuration, exhibiting its autonomous dynamics while the drive signals are additionally introduced. Applying the same drive multiple times, we compare the dynamical responses of the optoelectronic oscillator and quantify the degree of consistency among them via their correlation. Our results show that consistency is not restricted to conditions close to the first Hopf bifurcation but can be found in a broad range of dynamical regimes, even in the presence of multistability. Finally, we discuss the dependence of consistency on the nature of the drive signal.

Nonlinear Memory Capacity of Parallel Time-Delay Reservoir Computers in the Processing of Multidimensional Signals.

This letter addresses the reservoir design problem in the context of delay-based reservoir computers for multidimensional input signals, parallel architectures, and real-time multitasking. First, an approximating reservoir model is presented in those frameworks that provides an explicit functional link between the reservoir architecture and its performance in the execution of a specific task. Second, the inference properties of the ridge regression estimator in the multivariate context are used to assess the impact of finite sample training on the decrease of the reservoir capacity. Finally, an empirical study is conducted that shows the adequacy of the theoretical results with the empirical performances exhibited by various reservoir architectures in the execution of several nonlinear tasks with multidimensional inputs. Our results confirm the robustness properties of the parallel reservoir architecture with respect to task misspecification and parameter choice already documented in the literature.

Interaction between Liénard and Ikeda dynamics in a nonlinear electro-optical oscillator with delayed bandpass feedback.

We report on experimental and theoretical analysis of the complex dynamics generated by a nonlinear time-delayed electro-optic bandpass oscillator. We investigate the interaction between the slow- and fast-scale dynamics of autonomous oscillations in the breather regime. We analyze in detail the coupling between the fast-scale behavior associated to a characteristic low-pass Ikeda behavior and the slow-scale dynamics associated to a Liénard limit-cycle. Finally, we show that when projected onto a two-dimensional phase space, the attractors corresponding to periodic and chaotic breathers display a spiral-like pattern, which strongly depends on the shape of the nonlinear function.

Optimal nonlinear information processing capacity in delay-based reservoir computers.

Reservoir computing is a recently introduced brain-inspired machine learning paradigm capable of excellent performances in the processing of empirical data. We focus in a particular kind of time-delay based reservoir computers that have been physically implemented using optical and electronic systems and have shown unprecedented data processing rates. Reservoir computing is well-known for the ease of the associated training scheme but also for the problematic sensitivity of its performance to architecture parameters. This article addresses the reservoir design problem, which remains the biggest challenge in the applicability of this information processing scheme. More specifically, we use the information available regarding the optimal reservoir working regimes to construct a functional link between the reservoir parameters and its performance. This function is used to explore various properties of the device and to choose the optimal reservoir architecture, thus replacing the tedious and time consuming parameter scannings used so far in the literature.

Laser chimeras as a paradigm for multistable patterns in complex systems.

A chimera state is a rich and fascinating class of self-organized solutions developed in high-dimensional networks. Necessary features of the network for the emergence of such complex but structured motions are non-local and symmetry breaking coupling. An accurate understanding of chimera states is expected to bring important insights on deterministic mechanism occurring in many structurally similar high-dimensional dynamics such as living systems, brain operation principles and even turbulence in hydrodynamics. Here we report on a powerful and highly controllable experiment based on an optoelectronic delayed feedback applied to a wavelength tuneable semiconductor laser, with which a wide variety of chimera patterns can be accurately investigated and interpreted. We uncover a cascade of higher-order chimeras as a pattern transition from N to N+1 clusters of chaoticity. Finally, we follow visually, as the gain increases, how chimera state is gradually destroyed on the way to apparent turbulence-like system behaviour.

Kerr optical frequency comb generation in strontium fluoride whispering-gallery mode resonators with billion quality factor.

We report the fabrication for the first time of a strontium fluoride (SrF(2)) whispering-gallery mode resonator with quality factor in excess of 1 billion. The millimeter-size disk-resonator is polished until the surface roughness decreases down to a root-mean square value of 1.2 nm, as measured with a vertical scanning profilometer. We also demonstrate that this ultrahigh Q resonator allows for the generation of a normal-dispersion Kerr optical frequency comb at 1550 nm.

Optimally coherent Kerr combs generated with crystalline whispering gallery mode resonators for ultrahigh capacity fiber communications.

Optical Kerr frequency combs are known to be effective coherent multiwavelength sources for ultrahigh capacity fiber communications. These combs are the frequency-domain counterparts of a wide variety of spatiotemporal dissipative structures, such as cavity solitons, chaos, or Turing patterns (rolls). In this Letter, we demonstrate that Turing patterns, which correspond to the so-called primary combs in the spectral domain, are optimally coherent in the sense that for the same pump power they provide the most robust carriers for coherent data transmission in fiber communications using advanced modulation formats. Our model is based on a stochastic Lugiato-Lefever equation which accounts for laser pump frequency jitter and amplified spontaneous emission noise induced by the erbium-doped fiber amplifier. Using crystalline whispering-gallery-mode resonators with quality factor Q∼10^{9} for the comb generation, we show that when the noise is accounted for, the coherence of a primary comb is significantly higher than the coherence of their solitonic or chaotic counterparts for the same pump power. In order to confirm this theoretical finding, we perform an optical fiber transmission experiment using advanced modulation formats, and we show that the coherence of the primary comb is high enough to enable data transmission of up to 144 Gbit/s per comb line, the highest value achieved with a Kerr comb so far. This performance evidences that compact crystalline photonic systems have the potential to play a key role in a new generation of coherent fiber communication networks, alongside fully integrated systems.

Mixed-mode oscillations in slow-fast delayed optoelectronic systems.

In this article, we investigate the dynamical behavior of breathers in optoelectronic oscillators from the standpoint of mixed-mode oscillations. In the phase space, these breathers are composite oscillations that are damped to the attractive branches of an invariant manifold. Our study shows that the emergence of breather dynamics is linked to the apparition of inflection points in the phase space, and we develop an analytical framework based on the Liénard reduction form in order to provide an analytical insight into this phenomenology. Our theoretical results are in excellent agreement with experimental measurements.

Stochastic nonlinear time series forecasting using time-delay reservoir computers: performance and universality.

Reservoir computing is a recently introduced machine learning paradigm that has already shown excellent performances in the processing of empirical data. We study a particular kind of reservoir computers called time-delay reservoirs that are constructed out of the sampling of the solution of a time-delay differential equation and show their good performance in the forecasting of the conditional covariances associated to multivariate discrete-time nonlinear stochastic processes of VEC-GARCH type as well as in the prediction of factual daily market realized volatilities computed with intraday quotes, using as training input daily log-return series of moderate size. We tackle some problems associated to the lack of task-universality for individually operating reservoirs and propose a solution based on the use of parallel arrays of time-delay reservoirs.

Incoherent resonant seeding of modulation instability in optical fiber.

We report control of the spectral and noise properties of spontaneous modulation instability (MI) in optical fiber using an incoherent seed with power at the 10(-6) level relative to the pump. We sweep the seed wavelength across the MI gain band, and observe significant enhancement of MI bandwidth and improvement in the signal-to-noise ratio as the seed coincides with the MI gain peak. We also vary the seed bandwidth and find a reduced effect on the MI spectrum as the seed coherence decreases. Stochastic nonlinear Schrödinger equation simulations of spectral and noise properties are in excellent agreement with experiment.

Microwave photonics systems based on whispering-gallery-mode resonators.

Microwave photonics systems rely fundamentally on the interaction between microwave and optical signals. These systems are extremely promising for various areas of technology and applied science, such as aerospace and communication engineering, sensing, metrology, nonlinear photonics, and quantum optics. In this article, we present the principal techniques used in our lab to build microwave photonics systems based on ultra-high Q whispering gallery mode resonators. First detailed in this article is the protocol for resonator polishing, which is based on a grind-and-polish technique close to the ones used to polish optical components such as lenses or telescope mirrors. Then, a white light interferometric profilometer measures surface roughness, which is a key parameter to characterize the quality of the polishing. In order to launch light in the resonator, a tapered silica fiber with diameter in the micrometer range is used. To reach such small diameters, we adopt the "flame-brushing" technique, using simultaneously computer-controlled motors to pull the fiber apart, and a blowtorch to heat the fiber area to be tapered. The resonator and the tapered fiber are later approached to one another to visualize the resonance signal of the whispering gallery modes using a wavelength-scanning laser. By increasing the optical power in the resonator, nonlinear phenomena are triggered until the formation of a Kerr optical frequency comb is observed with a spectrum made of equidistant spectral lines. These Kerr comb spectra have exceptional characteristics that are suitable for several applications in science and technology. We consider the application related to ultra-stable microwave frequency synthesis and demonstrate the generation of a Kerr comb with GHz intermodal frequency.