RI2AP: Robust and Interpretable 2D Anomaly Prediction in Assembly Pipelines.

Predicting anomalies in manufacturing assembly lines is crucial for reducing time and labor costs and improving processes. For instance, in rocket assembly, premature part failures can lead to significant financial losses and labor inefficiencies. With the abundance of sensor data in the Industry 4.0 era, machine learning (ML) offers potential for early anomaly detection. However, current ML methods for anomaly prediction have limitations, with F1 measure scores of only 50% and 66% for prediction and detection, respectively. This is due to challenges like the rarity of anomalous events, scarcity of high-fidelity simulation data (actual data are expensive), and the complex relationships between anomalies not easily captured using traditional ML approaches. Specifically, these challenges relate to two dimensions of anomaly prediction: predicting when anomalies will occur and understanding the dependencies between them. This paper introduces a new method called Robust and Interpretable 2D Anomaly Prediction (RI2AP) designed to address both dimensions effectively. RI2AP is demonstrated on a rocket assembly simulation, showing up to a 30-point improvement in F1 measure compared to current ML methods. This highlights its potential to enhance automated anomaly prediction in manufacturing. Additionally, RI2AP includes a novel interpretation mechanism inspired by a causal-influence framework, providing domain experts with valuable insights into sensor readings and their impact on predictions. Finally, the RI2AP model was deployed in a real manufacturing setting for assembling rocket parts. Results and insights from this deployment demonstrate the promise of RI2AP for anomaly prediction in manufacturing assembly pipelines.

Ferroelectric capacitors and field-effect transistors as in-memory computing elements for machine learning workloads.

This study discusses the feasibility of Ferroelectric Capacitors (FeCaps) and Ferroelectric Field-Effect Transistors (FeFETs) as In-Memory Computing (IMC) elements to accelerate machine learning (ML) workloads. We conducted an exploration of device fabrication and proposed system-algorithm co-design to boost performance. A novel FeCap device, incorporating an interfacial layer (IL) and Hf 0.5 Zr 0.5 O 2 (HZO), ensures a reduction in operating voltage and enhances HZO scaling while being compatible with CMOS circuits. The IL also enriches ferroelectricity and retention properties. When integrated into crossbar arrays, FeCaps and FeFETs demonstrate their effectiveness as IMC components, eliminating sneak paths and enabling selector-less operation, leading to notable improvements in energy efficiency and area utilization. However, it is worth noting that limited capacitance ratios in FeCaps introduced errors in multiply-and-accumulate (MAC) computations. The proposed co-design approach helps in mitigating these errors and achieves high accuracy in classifying the CIFAR-10 dataset, elevating it from a baseline of 10% to 81.7%. FeFETs in crossbars, with a higher on-off ratio, outperform FeCaps, and our proposed charge-based sensing scheme achieved at least an order of magnitude reduction in power consumption, compared to prevalent current-based methods.

CL3: Generalization of Contrastive Loss for Lifelong Learning.

Lifelong learning portrays learning gradually in nonstationary environments and emulates the process of human learning, which is efficient, robust, and able to learn new concepts incrementally from sequential experience. To equip neural networks with such a capability, one needs to overcome the problem of catastrophic forgetting, the phenomenon of forgetting past knowledge while learning new concepts. In this work, we propose a novel knowledge distillation algorithm that makes use of contrastive learning to help a neural network to preserve its past knowledge while learning from a series of tasks. Our proposed generalized form of contrastive distillation strategy tackles catastrophic forgetting of old knowledge, and minimizes semantic drift by maintaining a similar embedding space, as well as ensures compactness in feature distribution to accommodate novel tasks in a current model. Our comprehensive study shows that our method achieves improved performances in the challenging class-incremental, task-incremental, and domain-incremental learning for supervised scenarios.

Subspace distillation for continual learning.

An ultimate objective in continual learning is to preserve knowledge learned in preceding tasks while learning new tasks. To mitigate forgetting prior knowledge, we propose a novel knowledge distillation technique that takes into the account the manifold structure of the latent/output space of a neural network in learning novel tasks. To achieve this, we propose to approximate the data manifold up-to its first order, hence benefiting from linear subspaces to model the structure and maintain the knowledge of a neural network while learning novel concepts. We demonstrate that the modeling with subspaces provides several intriguing properties, including robustness to noise and therefore effective for mitigating Catastrophic Forgetting in continual learning. We also discuss and show how our proposed method can be adopted to address both classification and segmentation problems. Empirically, we observe that our proposed method outperforms various continual learning methods on several challenging datasets including Pascal VOC, and Tiny-Imagenet. Furthermore, we show how the proposed method can be seamlessly combined with existing learning approaches to improve their performances. The codes of this article will be available at https://github.com/csiro-robotics/SDCL.

Fetal Health Classification from Cardiotocograph for Both Stages of Labor-A Soft-Computing-Based Approach.

To date, cardiotocography (CTG) is the only non-invasive and cost-effective tool available for continuous monitoring of the fetal health. In spite of a marked growth in the automation of the CTG analysis, it still remains a challenging signal processing task. Complex and dynamic patterns of fetal heart are poorly interpreted. Particularly, the precise interpretation of the suspected cases is fairly low by both visual and automated methods. Also, the first and second stage of labor produce very different fetal heart rate (FHR) dynamics. Thus, a robust classification model takes both stages into consideration separately. In this work, the authors propose a machine-learning-based model, which was applied separately to both the stages of labor, using standard classifiers such as SVM, random forest (RF), multi-layer perceptron (MLP), and bagging to classify the CTG. The outcome was validated using the model performance measure, combined performance measure, and the ROC-AUC. Though AUC-ROC was sufficiently high for all the classifiers, the other parameters established a better performance by SVM and RF. For suspicious cases the accuracies of SVM and RF were 97.4% and 98%, respectively, whereas sensitivity was 96.4% and specificity was 98% approximately. In the second stage of labor the accuracies were 90.6% and 89.3% for SVM and RF, respectively. Limits of agreement for 95% between the manual annotation and the outcome of SVM and RF were (-0.05 to 0.01) and (-0.03 to 0.02). Henceforth, the proposed classification model is efficient and can be integrated into the automated decision support system.

A machine learning pipeline to classify foetal heart rate deceleration with optimal feature set.

Deceleration is considered a commonly practised means to assess Foetal Heart Rate (FHR) through visual inspection and interpretation of patterns in Cardiotocography (CTG). The precision of deceleration classification relies on the accurate estimation of corresponding event points (EP) from the FHR and the Uterine Contraction Pressure (UCP). This work proposes a deceleration classification pipeline by comparing four machine learning (ML) models, namely, Multilayer Perceptron (MLP), Random Forest (RF), Naïve Bayes (NB), and Simple Logistics Regression. Towards an automated classification of deceleration from EP using the pipeline, it systematically compares three approaches to create feature sets from the detected EP: (1) a novel fuzzy logic (FL)-based approach, (2) expert annotation by clinicians, and (3) calculated using National Institute of Child Health and Human Development guidelines. The classification results were validated using different popular statistical metrics, including receiver operating characteristic curve, intra-class correlation coefficient, Deming regression, and Bland-Altman Plot. The highest classification accuracy (97.94%) was obtained with MLP when the EP was annotated with the proposed FL approach compared to RF, which obtained 63.92% with the clinician-annotated EP. The results indicate that the FL annotated feature set is the optimal one for classifying deceleration from FHR.

DIET-SNN: A Low-Latency Spiking Neural Network With Direct Input Encoding and Leakage and Threshold Optimization.

Bioinspired spiking neural networks (SNNs), operating with asynchronous binary signals (or spikes) distributed over time, can potentially lead to greater computational efficiency on event-driven hardware. The state-of-the-art SNNs suffer from high inference latency, resulting from inefficient input encoding and suboptimal settings of the neuron parameters (firing threshold and membrane leak). We propose DIET-SNN, a low-latency deep spiking network trained with gradient descent to optimize the membrane leak and the firing threshold along with other network parameters (weights). The membrane leak and threshold of each layer are optimized with end-to-end backpropagation to achieve competitive accuracy at reduced latency. The input layer directly processes the analog pixel values of an image without converting it to spike train. The first convolutional layer converts analog inputs into spikes where leaky-integrate-and-fire (LIF) neurons integrate the weighted inputs and generate an output spike when the membrane potential crosses the trained firing threshold. The trained membrane leak selectively attenuates the membrane potential, which increases activation sparsity in the network. The reduced latency combined with high activation sparsity provides massive improvements in computational efficiency. We evaluate DIET-SNN on image classification tasks from CIFAR and ImageNet datasets on VGG and ResNet architectures. We achieve top-1 accuracy of 69% with five timesteps (inference latency) on the ImageNet dataset with 12× less compute energy than an equivalent standard artificial neural network (ANN). In addition, DIET-SNN performs 20- 500× faster inference compared to other state-of-the-art SNN models.

LWSNet - a novel deep-learning architecture to segregate Covid-19 and pneumonia from x-ray imagery.

Automatic detection of lung diseases using AI-based tools became very much necessary to handle the huge number of cases occurring across the globe and support the doctors. This paper proposed a novel deep learning architecture named LWSNet (Light Weight Stacking Network) to separate Covid-19, cold pneumonia, and normal chest x-ray images. This framework is based on single, double, triple, and quadruple stack mechanisms to address the above-mentioned tri-class problem. In this framework, a truncated version of standard deep learning models and a lightweight CNN model was considered to conviniently deploy in resource-constraint devices. An evaluation was conducted on three publicly available datasets alongwith their combination. We received 97.28%, 96.50%, 97.41%, and 98.54% highest classification accuracies using quadruple stack. On further investigation, we found, using LWSNet, the average accuracy got improved from individual model to quadruple model by 2.31%, 2.55%, 2.88%, and 2.26% on four respective datasets.

Application of Machine Learning and Deep Learning Techniques for COVID-19 Screening Using Radiological Imaging: A Comprehensive Review.

Lung, being one of the most important organs in human body, is often affected by various SARS diseases, among which COVID-19 has been found to be the most fatal disease in recent times. In fact, SARS-COVID 19 led to pandemic that spreads fast among the community causing respiratory problems. Under such situation, radiological imaging-based screening [mostly chest X-ray and computer tomography (CT) modalities] has been performed for rapid screening of the disease as it is a non-invasive approach. Due to scarcity of physician/chest specialist/expert doctors, technology-enabled disease screening techniques have been developed by several researchers with the help of artificial intelligence and machine learning (AI/ML). It can be remarkably observed that the researchers have introduced several AI/ML/DL (deep learning) algorithms for computer-assisted detection of COVID-19 using chest X-ray and CT images. In this paper, a comprehensive review has been conducted to summarize the works related to applications of AI/ML/DL for diagnostic prediction of COVID-19, mainly using X-ray and CT images. Following the PRISMA guidelines, total 265 articles have been selected out of 1715 published articles till the third quarter of 2021. Furthermore, this review summarizes and compares varieties of ML/DL techniques, various datasets, and their results using X-ray and CT imaging. A detailed discussion has been made on the novelty of the published works, along with advantages and limitations.

Compute in-Memory with Non-Volatile Elements for Neural Networks: A Review from a Co-Design Perspective.

Deep learning has become ubiquitous, touching daily lives across the globe. Today, traditional computer architectures are stressed to their limits in efficiently executing the growing complexity of data and models. Compute-in-memory (CIM) can potentially play an important role in developing efficient hardware solutions that reduce data movement from compute-unit to memory, known as the von Neumann bottleneck. At its heart is a cross-bar architecture with nodal non-volatile-memory elements that performs an analog multiply-and-accumulate operation, enabling the matrix-vector-multiplications repeatedly used in all neural network workloads. The memory materials can significantly influence final system-level characteristics and chip performance, including speed, power, and classification accuracy. With an over-arching co-design viewpoint, this review assesses the use of cross-bar based CIM for neural networks, connecting the material properties and the associated design constraints and demands to application, architecture, and performance. Both digital and analog memory are considered, assessing the status for training and inference, and providing metrics for the collective set of properties non-volatile memory materials will need to demonstrate for a successful CIM technology.

RoMIA: a framework for creating Robust Medical Imaging AI models for chest radiographs.

Artificial Intelligence (AI) methods, particularly Deep Neural Networks (DNNs), have shown great promise in a range of medical imaging tasks. However, the susceptibility of DNNs to producing erroneous outputs under the presence of input noise and variations is of great concern and one of the largest challenges to their adoption in medical settings. Towards addressing this challenge, we explore the robustness of DNNs trained for chest radiograph classification under a range of perturbations reflective of clinical settings. We propose RoMIA, a framework for the creation of Robust Medical Imaging AI models. RoMIA adds three key steps to the model training and deployment flow: (i) Noise-added training, wherein a part of the training data is synthetically transformed to represent common noise sources, (ii) Fine-tuning with input mixing, in which the model is refined with inputs formed by mixing data from the original training set with a small number of images from a different source, and (iii) DCT-based denoising, which removes a fraction of high-frequency components of each image before applying the model to classify it. We applied RoMIA to create six different robust models for classifying chest radiographs using the CheXpert dataset. We evaluated the models on the CheXphoto dataset, which consists of naturally and synthetically perturbed images intended to evaluate robustness. Models produced by RoMIA show 3%-5% improvement in robust accuracy, which corresponds to an average reduction of 22.6% in misclassifications. These results suggest that RoMIA can be a useful step towards enabling the adoption of AI models in medical imaging applications.

An Audit of Diagnostic Disparity between Intraoperative Frozen Section Diagnosis and Final Histopathological Diagnosis of Central Nervous System Lesions at a Tertiary Care Center.

Introduction Evaluation of intraoperative squash smear and frozen section (FS) in central nervous system (CNS) neoplasms is consistently practiced for rapid assessment and has several advantages to its credence. It is an invaluable tool to ensure adequacy of tissue obtained to establish the diagnosis. Moreover, it aids in guiding the surgeon for critical decisions regarding the extent of resection. Although molecular markers have been integrated with morphology in the revised 2016 World Health Organization classification of brain tumors, precise morphological assessment still remains the foundation for the diagnosis and rapid intraoperative assessment of morphological details is equally critical and rewarding. Objective This study aims to audit the diagnostic disparity between intraoperative diagnoses based on a combination of squash cytology and FS in cases of CNS lesions with gold standard, final diagnosis based on examination of formalin fixed paraffin embedded hematoxylin, and eosin-stained tissue sections. Materials and Methods All intraoperative squash cytology and FS reported for CNS lesions from January 2017 to December 2020 were reviewed. The cases were categorized into three groups-group 1: when diagnosis of intraoperative diagnosis based on a combination of squash cytology and FS was same as the final histopathological diagnosis (concordant), group 2: partially concordant, and group 3: discordant cases. Statistical Analysis Descriptive statistics was used to classify the data and diagnostic accuracy was calculated. Results Complete concordance was present in 69.96% (191/273) cases, 20.1% (55/273) cases showed partial concordance, and 9.89% (27/273) cases were discordant with histopathological diagnosis. Out of the 27 discordant cases, misclassification of tumor type was the most common category (11 cases, 40%), followed by grading mismatch (7 cases, 25.9%), and misdiagnosis of tumor versus nontumor conditions (9 cases, 33.3%). Conclusion Our study shows that combination of intraoperative squash cytology and FS shows a high percentage of accuracy in arriving at intraoperative diagnosis in cases of intracranial lesions. Regular audits of discordant cases should be conducted by surgeons and pathologists as part of a quality assurance measure to sensitize themselves with the potential pitfalls, minimizing misinterpretation and helping in providing a more conclusive opinion to the operating surgeons.

Low precision decentralized distributed training over IID and non-IID data.

Decentralized distributed learning is the key to enabling large-scale machine learning (training) on the edge devices utilizing private user-generated local data, without relying on the cloud. However, practical realization of such on-device training is limited by the communication and compute bottleneck. In this paper, we propose and show the convergence of low precision decentralized training that aims to reduce the computational complexity and communication cost of decentralized training. Many feedback-based compression techniques have been proposed in the literature to reduce communication costs. To the best of our knowledge, there is no work that applies and shows compute efficient training techniques such as quantization, pruning etc., for peer-to-peer decentralized learning setups. Since real-world applications have a significant skew in the data distribution, we design "Range-EvoNorm" as the normalization activation layer which is better suited for low precision training over non-IID data. Moreover, we show that the proposed low precision training can be used in synergy with other communication compression methods decreasing the communication cost further. Our experiments indicate that 8-bit decentralized training has minimal accuracy loss compared to its full precision counterpart even with non-IID data. However, when low precision training is accompanied by communication compression through sparsification we observe a 1-2% drop in accuracy. The proposed low precision decentralized training decreases computational complexity, memory usage, and communication cost by ∼4× and compute energy by a factor of ∼20×, while trading off less than a 1% accuracy for both IID and non-IID data. In particular, for higher skew values, we observe an increase in accuracy (by ∼0.5%) with low precision training, indicating the regularization effect of the quantization.

Noise resilient leaky integrate-and-fire neurons based on multi-domain spintronic devices.

The capability of emulating neural functionalities efficiently in hardware is crucial for building neuromorphic computing systems. While various types of neuro-mimetic devices have been investigated, it remains challenging to provide a compact device that can emulate spiking neurons. In this work, we propose a non-volatile spin-based device for efficiently emulating a leaky integrate-and-fire neuron. By incorporating an exchange-coupled composite free layer in spin-orbit torque magnetic tunnel junctions, multi-domain magnetization switching dynamics is exploited to realize gradual accumulation of membrane potential for a leaky integrate-and-fire neuron with compact footprints. The proposed device offers significantly improved scalability compared with previously proposed spin-based neuro-mimetic implementations while exhibiting high energy efficiency and good controllability. Moreover, the proposed neuron device exhibits a varying leak constant and a varying membrane resistance that are both dependent on the magnitude of the membrane potential. Interestingly, we demonstrate that such device-inspired dynamic behaviors can be incorporated to construct more robust spiking neural network models, and find improved resiliency against various types of noise injection scenarios. The proposed spintronic neuro-mimetic devices may potentially open up exciting opportunities for the development of efficient and robust neuro-inspired computational hardware.

ProKnow: Process knowledge for safety constrained and explainable question generation for mental health diagnostic assistance.

Virtual Mental Health Assistants (VMHAs) are utilized in health care to provide patient services such as counseling and suggestive care. They are not used for patient diagnostic assistance because they cannot adhere to safety constraints and specialized clinical process knowledge (ProKnow) used to obtain clinical diagnoses. In this work, we define ProKnow as an ordered set of information that maps to evidence-based guidelines or categories of conceptual understanding to experts in a domain. We also introduce a new dataset of diagnostic conversations guided by safety constraints and ProKnow that healthcare professionals use (ProKnow-data). We develop a method for natural language question generation (NLG) that collects diagnostic information from the patient interactively (ProKnow-algo). We demonstrate the limitations of using state-of-the-art large-scale language models (LMs) on this dataset. ProKnow-algo incorporates the process knowledge through explicitly modeling safety, knowledge capture, and explainability. As computational metrics for evaluation do not directly translate to clinical settings, we involve expert clinicians in designing evaluation metrics that test four properties: safety, logical coherence, and knowledge capture for explainability while minimizing the standard cross entropy loss to preserve distribution semantics-based similarity to the ground truth. LMs with ProKnow-algo generated 89% safer questions in the depression and anxiety domain (tested property: safety). Further, without ProKnow-algo generations question did not adhere to clinical process knowledge in ProKnow-data (tested property: knowledge capture). In comparison, ProKnow-algo-based generations yield a 96% reduction in our metrics to measure knowledge capture. The explainability of the generated question is assessed by computing similarity with concepts in depression and anxiety knowledge bases. Overall, irrespective of the type of LMs, ProKnow-algo achieved an averaged 82% improvement over simple pre-trained LMs on safety, explainability, and process-guided question generation. For reproducibility, we will make ProKnow-data and the code repository of ProKnow-algo publicly available upon acceptance.

BlocTrain: Block-Wise Conditional Training and Inference for Efficient Spike-Based Deep Learning.

Spiking neural networks (SNNs), with their inherent capability to learn sparse spike-based input representations over time, offer a promising solution for enabling the next generation of intelligent autonomous systems. Nevertheless, end-to-end training of deep SNNs is both compute- and memory-intensive because of the need to backpropagate error gradients through time. We propose BlocTrain, which is a scalable and complexity-aware incremental algorithm for memory-efficient training of deep SNNs. We divide a deep SNN into blocks, where each block consists of few convolutional layers followed by a classifier. We train the blocks sequentially using local errors from the classifier. Once a given block is trained, our algorithm dynamically figures out easy vs. hard classes using the class-wise accuracy, and trains the deeper block only on the hard class inputs. In addition, we also incorporate a hard class detector (HCD) per block that is used during inference to exit early for the easy class inputs and activate the deeper blocks only for the hard class inputs. We trained ResNet-9 SNN divided into three blocks, using BlocTrain, on CIFAR-10 and obtained 86.4% accuracy, which is achieved with up to 2.95× lower memory requirement during the course of training, and 1.89× compute efficiency per inference (due to early exit strategy) with 1.45× memory overhead (primarily due to classifier weights) compared to end-to-end network. We also trained ResNet-11, divided into four blocks, on CIFAR-100 and obtained 58.21% accuracy, which is one of the first reported accuracy for SNN trained entirely with spike-based backpropagation on CIFAR-100.

Deep neural network to detect COVID-19: one architecture for both CT Scans and Chest X-rays.

Since December 2019, the novel COVID-19's spread rate is exponential, and AI-driven tools are used to prevent further spreading [1]. They can help predict, screen, and diagnose COVID-19 positive cases. Within this scope, imaging with Computed Tomography (CT) scans and Chest X-rays (CXRs) are widely used in mass triage situations. In the literature, AI-driven tools are limited to one data type either CT scan or CXR to detect COVID-19 positive cases. Integrating multiple data types could possibly provide more information in detecting anomaly patterns due to COVID-19. Therefore, in this paper, we engineered a Convolutional Neural Network (CNN) -tailored Deep Neural Network (DNN) that can collectively train/test both CT scans and CXRs. In our experiments, we achieved an overall accuracy of 96.28% (AUC = 0.9808 and false negative rate = 0.0208). Further, major existing DNNs provided coherent results while integrating CT scans and CXRs to detect COVID-19 positive cases.

Quantifying the Brain Predictivity of Artificial Neural Networks With Nonlinear Response Mapping.

Quantifying the similarity between artificial neural networks (ANNs) and their biological counterparts is an important step toward building more brain-like artificial intelligence systems. Recent efforts in this direction use neural predictivity, or the ability to predict the responses of a biological brain given the information in an ANN (such as its internal activations), when both are presented with the same stimulus. We propose a new approach to quantifying neural predictivity by explicitly mapping the activations of an ANN to brain responses with a non-linear function, and measuring the error between the predicted and actual brain responses. Further, we propose to use a neural network to approximate this mapping function by training it on a set of neural recordings. The proposed method was implemented within the TensorFlow framework and evaluated on a suite of 8 state-of-the-art image recognition ANNs. Our experiments suggest that the use of a non-linear mapping function leads to higher neural predictivity. Our findings also reaffirm the observation that the latest advances in classification performance of image recognition ANNs are not matched by improvements in their neural predictivity. Finally, we examine the impact of pruning, a widely used ANN optimization, on neural predictivity, and demonstrate that network sparsity leads to higher neural predictivity.

Neuromorphic learning with Mott insulator NiO.

Habituation and sensitization (nonassociative learning) are among the most fundamental forms of learning and memory behavior present in organisms that enable adaptation and learning in dynamic environments. Emulating such features of intelligence found in nature in the solid state can serve as inspiration for algorithmic simulations in artificial neural networks and potential use in neuromorphic computing. Here, we demonstrate nonassociative learning with a prototypical Mott insulator, nickel oxide (NiO), under a variety of external stimuli at and above room temperature. Similar to biological species such as Aplysia, habituation and sensitization of NiO possess time-dependent plasticity relying on both strength and time interval between stimuli. A combination of experimental approaches and first-principles calculations reveals that such learning behavior of NiO results from dynamic modulation of its defect and electronic structure. An artificial neural network model inspired by such nonassociative learning is simulated to show advantages for an unsupervised clustering task in accuracy and reducing catastrophic interference, which could help mitigate the stability-plasticity dilemma. Mott insulators can therefore serve as building blocks to examine learning behavior noted in biology and inspire new learning algorithms for artificial intelligence.