Pix2HDR - A pixel-wise acquisition and deep learning-based synthesis approach for high-speed HDR videos.

Accurately capturing dynamic scenes with wideranging motion and light intensity is crucial for many vision applications. However, acquiring high-speed high dynamic range (HDR) video is challenging because the camera's frame rate restricts its dynamic range. Existing methods sacrifice speed to acquire multi-exposure frames. Yet, misaligned motion in these frames can still pose complications for HDR fusion algorithms, resulting in artifacts. Instead of frame-based exposures, we sample the videos using individual pixels at varying exposures and phase offsets. Implemented on a monochrome pixel-wise programmable image sensor, our sampling pattern captures fast motion at a high dynamic range. We then transform pixel-wise outputs into an HDR video using end-to-end learned weights from deep neural networks, achieving high spatiotemporal resolution with minimized motion blurring. We demonstrate aliasing-free HDR video acquisition at 1000 FPS, resolving fast motion under low-light conditions and against bright backgrounds - both challenging conditions for conventional cameras. By combining the versatility of pixel-wise sampling patterns with the strength of deep neural networks at decoding complex scenes, our method greatly enhances the vision system's adaptability and performance in dynamic conditions.

Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy.

High-speed wide-field fluorescence microscopy has the potential to capture biological processes with exceptional spatiotemporal resolution. However, conventional cameras suffer from low signal-to-noise ratio at high frame rates, limiting their ability to detect faint fluorescent events. Here, we introduce an image sensor where each pixel has individually programmable sampling speed and phase, so that pixels can be arranged to simultaneously sample at high speed with a high signal-to-noise ratio. In high-speed voltage imaging experiments, our image sensor significantly increases the output signal-to-noise ratio compared to a low-noise scientific CMOS camera (~2-3 folds). This signal-to-noise ratio gain enables the detection of weak neuronal action potentials and subthreshold activities missed by the standard scientific CMOS cameras. Our camera with flexible pixel exposure configurations offers versatile sampling strategies to improve signal quality in various experimental conditions.

Neuromorphic applications in medicine.

In recent years, there has been a growing demand for miniaturization, low power consumption, quick treatments, and non-invasive clinical strategies in the healthcare industry. To meet these demands, healthcare professionals are seeking new technological paradigms that can improve diagnostic accuracy while ensuring patient compliance. Neuromorphic engineering, which uses neural models in hardware and software to replicate brain-like behaviors, can help usher in a new era of medicine by delivering low power, low latency, small footprint, and high bandwidth solutions. This paper provides an overview of recent neuromorphic advancements in medicine, including medical imaging and cancer diagnosis, processing of biosignals for diagnosis, and biomedical interfaces, such as motor, cognitive, and perception prostheses. For each section, we provide examples of how brain-inspired models can successfully compete with conventional artificial intelligence algorithms, demonstrating the potential of neuromorphic engineering to meet demands and improve patient outcomes. Lastly, we discuss current struggles in fitting neuromorphic hardware with non-neuromorphic technologies and propose potential solutions for future bottlenecks in hardware compatibility.

The influence of stereopsis on visual saliency in a proto-object based model of selective attention.

Some animals including humans use stereoscopic vision which reconstructs spatial information about the environment from the disparity between images captured by eyes in two separate adjacent locations. Like other sensory information, such stereoscopic information is expected to influence attentional selection. We develop a biologically plausible model of binocular vision to study its effect on bottom-up visual attention, i.e., visual saliency. In our model, the scene is organized in terms of proto-objects on which attention acts, rather than on unbound sets of elementary features. We show that taking into account the stereoscopic information improves the performance of the model in the prediction of human eye movements with statistically significant differences.

Pixel-wise programmability enables dynamic high-SNR cameras for high-speed microscopy.

High-speed widefield fluorescence microscopy has the potential to capture biological processes with exceptional spatiotemporal resolution. However, conventional cameras suffer from low signal-to-noise ratio (SNR) at high frame rates, limiting their ability to detect faint fluorescent events. Here we introduce an image sensor where each pixel has individually programmable sampling speed and phase, so that pixels can be arranged to simultaneously sample at high speed with a high SNR. In high-speed voltage imaging experiments, our image sensor significantly increases the output SNR compared to a low-noise scientific CMOS camera (~2-3 folds). This SNR gain enables the detection of weak neuronal action potentials and subthreshold activities missed by the standard scientific CMOS cameras. Our proposed camera with flexible pixel exposure configurations offers versatile sampling strategies to improve signal quality in various experimental conditions.

First Organoid Intelligence (OI) workshop to form an OI community.

The brain is arguably the most powerful computation system known. It is extremely efficient in processing large amounts of information and can discern signals from noise, adapt, and filter faulty information all while running on only 20 watts of power. The human brain's processing efficiency, progressive learning, and plasticity are unmatched by any computer system. Recent advances in stem cell technology have elevated the field of cell culture to higher levels of complexity, such as the development of three-dimensional (3D) brain organoids that recapitulate human brain functionality better than traditional monolayer cell systems. Organoid Intelligence (OI) aims to harness the innate biological capabilities of brain organoids for biocomputing and synthetic intelligence by interfacing them with computer technology. With the latest strides in stem cell technology, bioengineering, and machine learning, we can explore the ability of brain organoids to compute, and store given information (input), execute a task (output), and study how this affects the structural and functional connections in the organoids themselves. Furthermore, understanding how learning generates and changes patterns of connectivity in organoids can shed light on the early stages of cognition in the human brain. Investigating and understanding these concepts is an enormous, multidisciplinary endeavor that necessitates the engagement of both the scientific community and the public. Thus, on Feb 22-24 of 2022, the Johns Hopkins University held the first Organoid Intelligence Workshop to form an OI Community and to lay out the groundwork for the establishment of OI as a new scientific discipline. The potential of OI to revolutionize computing, neurological research, and drug development was discussed, along with a vision and roadmap for its development over the coming decade.

Analysis and Design Methodology of RF Energy Harvesting Rectifier Circuit for Ultra-Low Power Applications.

This paper reviews and analyses the design of popular radio frequency energy harvesting systems and proposes a method to qualitatively and quantitatively analyze their circuit architectures using new square-wave approximation method. This approach helps in simplifying design analysis. Using this analysis, we can establish no load output voltage characteristics, upper limit on rectifier efficiency, and maximum power characteristics of a rectifier. This paper will help guide the design of RF energy harvesting rectifier circuits for radio frequency identification (RFIDs), the Internet of Things (IoTs), wearable, and implantable medical device applications. Different application scenarios are explained in the context of design challenges, and corresponding design considerations are discussed in order to evaluate their performance. The pros and cons of different rectifier topologies are also investigated. In addition to presenting the popular rectifier topologies, new measurement results of these energy harvester topologies, fabricated in 65nm, 130nm and 180nm CMOS technologies are also presented.

Decoding accelerometry for classification and prediction of critically ill patients with severe brain injury.

Our goal is to explore quantitative motor features in critically ill patients with severe brain injury (SBI). We hypothesized that computational decoding of these features would yield information on underlying neurological states and outcomes. Using wearable microsensors placed on all extremities, we recorded a median 24.1 (IQR: 22.8-25.1) hours of high-frequency accelerometry data per patient from a prospective cohort (n = 69) admitted to the ICU with SBI. Models were trained using time-, frequency-, and wavelet-domain features and levels of responsiveness and outcome as labels. The two primary tasks were detection of levels of responsiveness, assessed by motor sub-score of the Glasgow Coma Scale (GCSm), and prediction of functional outcome at discharge, measured with the Glasgow Outcome Scale-Extended (GOSE). Detection models achieved significant (AUC: 0.70 [95% CI: 0.53-0.85]) and consistent (observation windows: 12 min-9 h) discrimination of SBI patients capable of purposeful movement (GCSm > 4). Prediction models accurately discriminated patients of upper moderate disability or better (GOSE > 5) with 2-6 h of observation (AUC: 0.82 [95% CI: 0.75-0.90]). Results suggest that time series analysis of motor activity yields clinically relevant insights on underlying functional states and short-term outcomes in patients with SBI.

A Neuromorphic Proto-Object Based Dynamic Visual Saliency Model With a Hybrid FPGA Implementation.

Computing and attending to salient regions of a visual scene is an innate and necessary preprocessing step for both biological and engineered systems performing high-level visual tasks including object detection, tracking, and classification. Computational bandwidth and speed are improved by preferentially devoting computational resources to salient regions of the visual field. The human brain computes saliency effortlessly, but modeling this task in engineered systems is challenging. We first present a neuromorphic dynamic saliency model, which is bottom-up, feed-forward, and based on the notion of proto-objects with neurophysiological spatio-temporal features requiring no training. Our neuromorphic model outperforms state-of-the-art dynamic visual saliency models in predicting human eye fixations (i.e., ground truth saliency). Secondly, we present a hybrid FPGA implementation of the model for real-time applications, capable of processing 112×84 resolution frames at 18.71 Hz running at a 100 MHz clock rate - a 23.77× speedup from the software implementation. Additionally, our fixed-point model of the FPGA implementation yields comparable results to the software implementation.

A syndromic surveillance tool to detect anomalous clusters of COVID-19 symptoms in the United States.

Coronavirus SARS-COV-2 infections continue to spread across the world, yet effective large-scale disease detection and prediction remain limited. COVID Control: A Johns Hopkins University Study, is a novel syndromic surveillance approach, which collects body temperature and COVID-like illness (CLI) symptoms across the US using a smartphone app and applies spatio-temporal clustering techniques and cross-correlation analysis to create maps of abnormal symptomatology incidence that are made publicly available. The results of the cross-correlation analysis identify optimal temporal lags between symptoms and a range of COVID-19 outcomes, with new taste/smell loss showing the highest correlations. We also identified temporal clusters of change in taste/smell entries and confirmed COVID-19 incidence in Baltimore City and County. Further, we utilized an extended simulated dataset to showcase our analytics in Maryland. The resulting clusters can serve as indicators of emerging COVID-19 outbreaks, and support syndromic surveillance as an early warning system for disease prevention and control.

A Compact Free-Floating Device for Passive Charge-Balanced Neural Stimulation using PEDOT/CNT microelectrodes.

Wirelessly powered implants are increasingly being developed as free-floating single-channel devices to interface with neurons directly at stimulation sites. In order to stimulate neurons in a manner that is safe to both the electrode and the surrounding tissue, charge accumulation over time needs to be avoided. The implementation of conventional charge balancing methods often leads to an increase in system complexity, power consumption or area, all of which are critical parameters in ultra-small wireless devices. The proposed charge balancing method described in this work, which relies on bipolar capacitive integrated electrodes, does not increase these parameters. The standalone wirelessly powered stimulating implant is implemented in a 130nm CMOS technology and measures 0.009 mm3.

A Miniature Wireless Silicon-on-Insulator Image Sensor for Brain Fluorescence Imaging.

Optical recording of genetically encoded calcium indicator (GECI) allows neuroscientists to study the activity of genetically labeled neuron populations, but our current tools lack the resolution, stability and are often too invasive. Here we present the design concepts, prototypes, and preliminary measurement results of a super-miniaturized wireless image sensor built using a 32nm Silicon-on-Insulator process. SOI process is optimal for wireless applications, and we can further thin the substrate to reduce overall device thickness to ~25µm and operate the pixels using back-side illumination. The proposed device is 300µm × 300µm. Our prototype is built on a 3 × 3mm die.

Proto-Object Based Saliency Model With Texture Detection Channel.

The amount of visual information projected from the retina to the brain exceeds the information processing capacity of the latter. Attention, therefore, functions as a filter to highlight important information at multiple stages of the visual pathway that requires further and more detailed analysis. Among other functions, this determines where to fixate since only the fovea allows for high resolution imaging. Visual saliency modeling, i.e. understanding how the brain selects important information to analyze further and to determine where to fixate next, is an important research topic in computational neuroscience and computer vision. Most existing bottom-up saliency models use low-level features such as intensity and color, while some models employ high-level features, like faces. However, little consideration has been given to mid-level features, such as texture, for visual saliency models. In this paper, we extend a biologically plausible proto-object based saliency model by adding simple texture channels which employ nonlinear operations that mimic the processing performed by primate visual cortex. The extended model shows statistically significant improved performance in predicting human fixations compared to the previous model. Comparing the performance of our model with others on publicly available benchmarking datasets, we find that our biologically plausible model matches the performance of other models, even though those were designed entirely for maximal performance with little regard to biological realism.

Closed-loop bioelectronic medicine for diabetes management.

Modulation of the nervous system by delivering electrical or pharmaceutical agents has contributed to the development of novel treatments to serious health disorders. Recent advances in multidisciplinary research has enabled the emergence of a new powerful therapeutic approach called bioelectronic medicine. Bioelectronic medicine exploits the fact that every organ in our bodies is neurally innervated and thus electrical interfacing with peripheral nerves can be a potential pathway for diagnosing or treating diseases such as diabetes. In this context, a plethora of studies have confirmed the important role of the nervous system in maintaining a tight regulation of glucose homeostasis. This has initiated new research exploring the opportunities of bioelectronic medicine for improving glucose control in people with diabetes, including regulation of gastric emptying, insulin sensitivity, and secretion of pancreatic hormones. Moreover, the development of novel closed-loop strategies aims to provide effective, specific and safe interfacing with the nervous system, and thereby targeting the organ of interest. This is especially valuable in the context of chronic diseases such as diabetes, where closed-loop bioelectronic medicine promises to provide real-time, autonomous and patient-specific therapies. In this article, we present an overview of the state-of-the-art for closed-loop neuromodulation systems in relation to diabetes and discuss future related opportunities for management of this chronic disease.

Detection and Confirmation of Multiple Human Targets Using Pixel-Wise Code Aperture Measurements.

Compressive video measurements can save bandwidth and data storage. However, conventional approaches to target detection require the compressive measurements to be reconstructed before any detectors are applied. This is not only time consuming but also may lose information in the reconstruction process. In this paper, we summarized the application of a recent approach to vehicle detection and classification directly in the compressive measurement domain to human targets. The raw videos were collected using a pixel-wise code exposure (PCE) camera, which condensed multiple frames into one frame. A combination of two deep learning-based algorithms (you only look once (YOLO) and residual network (ResNet)) was used for detection and confirmation. Optical and mid-wave infrared (MWIR) videos from a well-known database (SENSIAC) were used in our experiments. Extensive experiments demonstrated that the proposed framework was feasible for target detection up to 1500 m, but target confirmation needs more research.

A closed-loop, all-electronic pixel-wise adaptive imaging system for high dynamic range videography.

Digital cameras expose and readout all pixels in accordance with a global sample clock. This rigid global control of exposure and sampling is problematic for capturing scenes with large variance in brightness and motion, and may cause regions of motion blur, under- and overexposure. To address these issues, we developed a CMOS imaging system that automatically adjusts each pixel's exposure and sampling rate to fit local motion and brightness. This system consists of an image sensor with pixel-addressable exposure configurability in combination with a real-time, per-pixel exposure controller. It operates in a closed-loop to sample, detect and optimize each pixel's exposure and sampling rate for optimal acquisition. Per-pixel exposure control is implemented using all-integrated electronics without external optical modulation. This reduces system complexity and power consumption compared to existing solutions. Implemented using standard 130nm CMOS process, the chip has 256 × 256 pixels and consumes 7.31mW. To evaluate performance, we used this system to capture scenes with complex lighting and motion conditions that would lead to loss of information for globally-exposed cameras. These results demonstrate the advantage of pixel-wise adaptive imaging for a range of computer vision tasks such as segmentation, motion estimation and object recognition.

The Microbead: A 0.009 mm3 Implantable Wireless Neural Stimulator.

Wirelessly powered implants are increasingly being developed to interface with neurons in the brain. They often rely on microelectrode arrays, which are limited by their ability to cover large cortical surface areas and long-term stability because of their physical size and rigid configuration. Yet some clinical and research applications prioritize a distributed neural interface over one that offers high channel count. One solution to make large scale, fully specifiable, electrical stimulation/recording possible, is to disconnect the electrodes from the base, so that they can be arbitrarily placed freely in the nervous system. In this work, a wirelessly powered stimulating implant is miniaturized using a novel electrode integration technique, and its implanted depth maximized using new optimization design methods for the transmitter and receiver coils. The stimulating device is implemented in a 130 nm CMOS technology with the following characteristics: 300 µm × 300 µm × 80 µm size; optimized two-coil inductive link; and integrated circuit, electrodes and coil. The wireless and stimulation capability of the implant is demonstrated in a conductive medium, as well as in-vivo. To the best of our knowledge, the fabricated free-floating miniaturized implant has the best depth-to-volume ratio making it an excellent tool for minimally-invasive distributed neural interface, and thus could eventually complement or replace the rigid arrays that are currently the state-of-the-art in brain set-ups.

Deep Learning-Based Target Tracking and Classification for Low Quality Videos Using Coded Aperture Cameras.

Compressive sensing has seen many applications in recent years. One type of compressive sensing device is the Pixel-wise Code Exposure (PCE) camera, which has low power consumption and individual control of pixel exposure time. In order to use PCE cameras for practical applications, a time consuming and lossy process is needed to reconstruct the original frames. In this paper, we present a deep learning approach that directly performs target tracking and classification in the compressive measurement domain without any frame reconstruction. In particular, we propose to apply You Only Look Once (YOLO) to detect and track targets in the frames and we propose to apply Residual Network (ResNet) for classification. Extensive simulations using low quality optical and mid-wave infrared (MWIR) videos in the SENSIAC database demonstrated the efficacy of our proposed approach.