A deep learning method for reflective boundary estimation.

Environment estimation is a challenging task in reverberant settings such as the underwater and indoor acoustic domains. The locations of reflective boundaries, for example, can be estimated using acoustic echoes and leveraged for subsequent, more accurate localization and mapping. Current boundary estimation methods are constrained to high signal-to-noise ratios or are customized to specific environments. Existing methods also often require a correct assignment of echoes to boundaries, which is difficult if spurious echoes are detected. To evade these limitations, a convolutional neural network (NN) method is developed for robust two-dimensional boundary estimation, given known emitter and receiver locations. A Hough transform-inspired algorithm is leveraged to transform echo times of arrival into images, which are amenable to multi-resolution regression by NNs. The same architecture is trained on transform images of different resolutions to obtain diverse NNs, deployed sequentially for increasingly refined boundary estimation. A correct echo labeling solution is not required, and the method is robust to reverberation. The proposed method is tested in simulation and for real data from a water tank, where it outperforms state-of-the-art alternatives. These results are encouraging for the future development of data-driven three-dimensional environment estimation with high practical value in underwater acoustic detection and tracking.

Imaging privacy threats from an ambient light sensor.

Embedded sensors in smart devices pose privacy risks, often unintentionally leaking user information. We investigate how combining an ambient light sensor with a device display can capture an image of touch interaction without a camera. By displaying a known video sequence, we use the light sensor to capture reflected light intensity variations partially blocked by the touching hand, formulating an inverse problem similar to single-pixel imaging. Because of the sensors' heavy quantization and low sensitivity, we propose an inversion algorithm involving an ℓp-norm dequantizer and a deep denoiser as natural image priors, to reconstruct images from the screen's perspective. We demonstrate touch interactions and eavesdropping hand gestures on an off-the-shelf Android tablet. Despite limitations in resolution and speed, we aim to raise awareness of potential security/privacy threats induced by the combination of passive and active components in smart devices and promote the development of ways to mitigate them.

An architecture for passive joint localization and structure learning in reverberant environments.

Passive localization and tracking of a mobile emitter, and joint learning of its reverberant three-dimensional (3D) acoustic environment, where critical structural features are unknown, is a key open problem. Unaccounted-for occluders are potentially present, so that the emitter can lose line-of-sight to the receivers, and can only be observed through its reflected raypaths. The locations of reflective boundaries must therefore be jointly estimated with the emitter's position. A multistage global optimization and tracking architecture is developed to solve this problem with a relatively unconstrained model. Each stage of this architecture establishes domain knowledge such as synchronization and initial environment estimation, which are inputs for the following stages of more refined algorithms. This approach is generalizable to different physical scales and modalities and improves on methods that do not exploit the motion of the emitter. In one stage of this architecture, particle swarm optimization is used to simultaneously estimate the environment and the emitter location. In another stage, a Hough transform-inspired boundary localization algorithm is extended to 3D settings, to establish an initial estimate of the environment. The performance of this holistic approach is analyzed and its reliability is demonstrated in a reverberant watertank testbed, which models the shallow-water underwater acoustic setting.

A Maximal Correlation Framework for Fair Machine Learning.

As machine learning algorithms grow in popularity and diversify to many industries, ethical and legal concerns regarding their fairness have become increasingly relevant. We explore the problem of algorithmic fairness, taking an information-theoretic view. The maximal correlation framework is introduced for expressing fairness constraints and is shown to be capable of being used to derive regularizers that enforce independence and separation-based fairness criteria, which admit optimization algorithms for both discrete and continuous variables that are more computationally efficient than existing algorithms. We show that these algorithms provide smooth performance-fairness tradeoff curves and perform competitively with state-of-the-art methods on both discrete datasets (COMPAS, Adult) and continuous datasets (Communities and Crimes).

An Information Theoretic Interpretation to Deep Neural Networks.

With the unprecedented performance achieved by deep learning, it is commonly believed that deep neural networks (DNNs) attempt to extract informative features for learning tasks. To formalize this intuition, we apply the local information geometric analysis and establish an information-theoretic framework for feature selection, which demonstrates the information-theoretic optimality of DNN features. Moreover, we conduct a quantitative analysis to characterize the impact of network structure on the feature extraction process of DNNs. Our investigation naturally leads to a performance metric for evaluating the effectiveness of extracted features, called the H-score, which illustrates the connection between the practical training process of DNNs and the information-theoretic framework. Finally, we validate our theoretical results by experimental designs on synthesized data and the ImageNet dataset.

Revealing hidden scenes by photon-efficient occlusion-based opportunistic active imaging.

The ability to see around corners, i.e., recover details of a hidden scene from its reflections in the surrounding environment, is of considerable interest in a wide range of applications. However, the diffuse nature of light reflected from typical surfaces leads to mixing of spatial information in the collected light, precluding useful scene reconstruction. Here, we employ a computational imaging technique that opportunistically exploits the presence of occluding objects, which obstruct probe-light propagation in the hidden scene, to undo the mixing and greatly improve scene recovery. Importantly, our technique obviates the need for the ultrafast time-of-flight measurements employed by most previous approaches to hidden-scene imaging. Moreover, it does so in a photon-efficient manner (i.e., it only requires a small number of photon detections) based on an accurate forward model and a computational algorithm that, together, respect the physics of three-bounce light propagation and single-photon detection. Using our methodology, we demonstrate reconstruction of hidden-surface reflectivity patterns in a meter-scale environment from non-time-resolved measurements. Ultimately, our technique represents an instance of a rich and promising new imaging modality with important potential implications for imaging science.

A Modulo-Based Architecture for Analog-to-Digital Conversion.

Systems that capture and process analog signals must first acquire them through an analog-to-digital converter. While subsequent digital processing can remove statistical correlations present in the acquired data, the dynamic range of the converter is typically scaled to match that of the input analog signal. The present paper develops an approach for analog-to-digital conversion that aims at minimizing the number of bits per sample at the output of the converter. This is attained by reducing the dynamic range of the analog signal by performing a modulo operation on its amplitude, and then quantizing the result. While the converter itself is universal and agnostic of the statistics of the signal, the decoder operation on the output of the quantizer can exploit the statistical structure in order to unwrap the modulo folding. The performance of this method is shown to approach information theoretical limits, as captured by the rate-distortion function, in various settings. An architecture for modulo analog-to-digital conversion via ring oscillators is suggested, and its merits are numerically demonstrated.

A real-time brain-machine interface combining motor target and trajectory intent using an optimal feedback control design.

Real-time brain-machine interfaces (BMI) have focused on either estimating the continuous movement trajectory or target intent. However, natural movement often incorporates both. Additionally, BMIs can be modeled as a feedback control system in which the subject modulates the neural activity to move the prosthetic device towards a desired target while receiving real-time sensory feedback of the state of the movement. We develop a novel real-time BMI using an optimal feedback control design that jointly estimates the movement target and trajectory of monkeys in two stages. First, the target is decoded from neural spiking activity before movement initiation. Second, the trajectory is decoded by combining the decoded target with the peri-movement spiking activity using an optimal feedback control design. This design exploits a recursive Bayesian decoder that uses an optimal feedback control model of the sensorimotor system to take into account the intended target location and the sensory feedback in its trajectory estimation from spiking activity. The real-time BMI processes the spiking activity directly using point process modeling. We implement the BMI in experiments consisting of an instructed-delay center-out task in which monkeys are presented with a target location on the screen during a delay period and then have to move a cursor to it without touching the incorrect targets. We show that the two-stage BMI performs more accurately than either stage alone. Correct target prediction can compensate for inaccurate trajectory estimation and vice versa. The optimal feedback control design also results in trajectories that are smoother and have lower estimation error. The two-stage decoder also performs better than linear regression approaches in offline cross-validation analyses. Our results demonstrate the advantage of a BMI design that jointly estimates the target and trajectory of movement and more closely mimics the sensorimotor control system.

Feedback-controlled parallel point process filter for estimation of goal-directed movements from neural signals.

Real-time brain-machine interfaces have estimated either the target of a movement, or its kinematics. However, both are encoded in the brain. Moreover, movements are often goal-directed and made to reach a target. Hence, modeling the goal-directed nature of movements and incorporating the target information in the kinematic decoder can increase its accuracy. Using an optimal feedback control design, we develop a recursive Bayesian kinematic decoder that models goal-directed movements and combines the target information with the neural spiking activity during movement. To do so, we build a prior goal-directed state-space model for the movement using an optimal feedback control model of the sensorimotor system that aims to emulate the processes underlying actual motor control and takes into account the sensory feedback. Most goal-directed models, however, depend on the movement duration, not known a priori to the decoder. This has prevented their real-time implementation. To resolve this duration uncertainty, the decoder discretizes the duration and consists of a bank of parallel point process filters, each combining the prior model of a discretized duration with the neural activity. The kinematics are computed by optimally combining these filter estimates. Using the feedback-controlled model and even a coarse discretization, the decoder significantly reduces the root mean square error in estimation of reaching movements performed by a monkey.

Neural population partitioning and a concurrent brain-machine interface for sequential motor function.

Although brain-machine interfaces (BMIs) have focused largely on performing single-targeted movements, many natural tasks involve planning a complete sequence of such movements before execution. For these tasks, a BMI that can concurrently decode the full planned sequence before its execution may also consider the higher-level goal of the task to reformulate and perform it more effectively. Using population-wide modeling, we discovered two distinct subpopulations of neurons in the rhesus monkey premotor cortex that allow two planned targets of a sequential movement to be simultaneously held in working memory without degradation. Such marked stability occurred because each subpopulation encoded either only currently held or only newly added target information irrespective of the exact sequence. On the basis of these findings, we developed a BMI that concurrently decodes a full motor sequence in advance of movement and can then accurately execute it as desired.