Bridging AI and Clinical Practice: Integrating Automated Sleep Scoring Algorithm with Uncertainty-Guided Physician Review.

Purpose: This study aims to enhance the clinical use of automated sleep-scoring algorithms by incorporating an uncertainty estimation approach to efficiently assist clinicians in the manual review of predicted hypnograms, a necessity due to the notable inter-scorer variability inherent in polysomnography (PSG) databases. Our efforts target the extent of review required to achieve predefined agreement levels, examining both in-domain (ID) and out-of-domain (OOD) data, and considering subjects' diagnoses. Patients and Methods: A total of 19,578 PSGs from 13 open-access databases were used to train U-Sleep, a state-of-the-art sleep-scoring algorithm. We leveraged a comprehensive clinical database of an additional 8832 PSGs, covering a full spectrum of ages (0-91 years) and sleep-disorders, to refine the U-Sleep, and to evaluate different uncertainty-quantification approaches, including our novel confidence network. The ID data consisted of PSGs scored by over 50 physicians, and the two OOD sets comprised recordings each scored by a unique senior physician. Results: U-Sleep demonstrated robust performance, with Cohen's kappa (K) at 76.2% on ID and 73.8-78.8% on OOD data. The confidence network excelled at identifying uncertain predictions, achieving AUROC scores of 85.7% on ID and 82.5-85.6% on OOD data. Independently of sleep-disorder status, statistical evaluations revealed significant differences in confidence scores between aligning vs discording predictions, and significant correlations of confidence scores with classification performance metrics. To achieve κ ≥ 90% with physician intervention, examining less than 29.0% of uncertain epochs was required, substantially reducing physicians' workload, and facilitating near-perfect agreement. Conclusion: Inter-scorer variability limits the accuracy of the scoring algorithms to ~80%. By integrating an uncertainty estimation with U-Sleep, we enhance the review of predicted hypnograms, to align with the scoring taste of a responsible physician. Validated across ID and OOD data and various sleep-disorders, our approach offers a strategy to boost automated scoring tools' usability in clinical settings.

Deep Generative Models: The winning key for large and easily accessible ECG datasets?

Large high-quality datasets are essential for building powerful artificial intelligence (AI) algorithms capable of supporting advancement in cardiac clinical research. However, researchers working with electrocardiogram (ECG) signals struggle to get access and/or to build one. The aim of the present work is to shed light on a potential solution to address the lack of large and easily accessible ECG datasets. Firstly, the main causes of such a lack are identified and examined. Afterward, the potentials and limitations of cardiac data generation via deep generative models (DGMs) are deeply analyzed. These very promising algorithms have been found capable not only of generating large quantities of ECG signals but also of supporting data anonymization processes, to simplify data sharing while respecting patients' privacy. Their application could help research progress and cooperation in the name of open science. However several aspects, such as a standardized synthetic data quality evaluation and algorithm stability, need to be further explored.

Identification of Brugada syndrome based on P-wave features: an artificial intelligence-based approach.

AIMS: Brugada syndrome (BrS) is an inherited disease associated with an increased risk of ventricular arrhythmias. Recent studies have reported the presence of an altered atrial phenotype characterized by abnormal P-wave parameters. The aim of this study was to identify BrS based exclusively on P-wave features through an artificial intelligence (AI)-based model. METHODS AND RESULTS: Continuous 5 min 12-lead ECG recordings were obtained in sinus rhythm from (i) patients with spontaneous or ajmaline-induced BrS and no history of AF and (ii) subjects with suspected BrS and negative ajmaline challenge. The recorded ECG signals were processed and divided into epochs of 15 s each. Within these epochs, P-waves were first identified and then averaged. From the averaged P-waves, a total of 67 different features considered relevant to the classification task were extracted. These features were then used to train nine different AI-based supervised classifiers. A total of 2228 averaged P-wave observations, resulting from the analysis of 33 420 P-waves, were obtained from 123 patients (79 BrS+ and 44 BrS-). Averaged P-waves were divided using a patient-wise split, allocating 80% for training and 20% for testing, ensuring data integrity and reducing biases in AI-based model training. The BrS+ patients presented with longer P-wave duration (136 ms vs. 124 ms, P < 0.001) and higher terminal force in lead V1 (2.5 au vs. 1.7 au, P < 0.01) compared with BrS- subjects. Among classifiers, AdaBoost model had the highest values of performance for all the considered metrics, reaching an accuracy of over 81% (sensitivity 86%, specificity 73%). CONCLUSION: An AI machine-learning model is able to identify patients with BrS based only on P-wave characteristics. These findings confirm the presence of an atrial hallmark and open new horizons for AI-guided BrS diagnosis.

U-Sleep's resilience to AASM guidelines.

AASM guidelines are the result of decades of efforts aiming at standardizing sleep scoring procedure, with the final goal of sharing a worldwide common methodology. The guidelines cover several aspects from the technical/digital specifications, e.g., recommended EEG derivations, to detailed sleep scoring rules accordingly to age. Automated sleep scoring systems have always largely exploited the standards as fundamental guidelines. In this context, deep learning has demonstrated better performance compared to classical machine learning. Our present work shows that a deep learning-based sleep scoring algorithm may not need to fully exploit the clinical knowledge or to strictly adhere to the AASM guidelines. Specifically, we demonstrate that U-Sleep, a state-of-the-art sleep scoring algorithm, can be strong enough to solve the scoring task even using clinically non-recommended or non-conventional derivations, and with no need to exploit information about the chronological age of the subjects. We finally strengthen a well-known finding that using data from multiple data centers always results in a better performing model compared with training on a single cohort. Indeed, we show that this latter statement is still valid even by increasing the size and the heterogeneity of the single data cohort. In all our experiments we used 28528 polysomnography studies from 13 different clinical studies.

Multi-scored sleep databases: how to exploit the multiple-labels in automated sleep scoring.

STUDY OBJECTIVES: Inter-scorer variability in scoring polysomnograms is a well-known problem. Most of the existing automated sleep scoring systems are trained using labels annotated by a single-scorer, whose subjective evaluation is transferred to the model. When annotations from two or more scorers are available, the scoring models are usually trained on the scorer consensus. The averaged scorer's subjectivity is transferred into the model, losing information about the internal variability among different scorers. In this study, we aim to insert the multiple-knowledge of the different physicians into the training procedure. The goal is to optimize a model training, exploiting the full information that can be extracted from the consensus of a group of scorers. METHODS: We train two lightweight deep learning-based models on three different multi-scored databases. We exploit the label smoothing technique together with a soft-consensus (LSSC) distribution to insert the multiple-knowledge in the training procedure of the model. We introduce the averaged cosine similarity metric (ACS) to quantify the similarity between the hypnodensity-graph generated by the models with-LSSC and the hypnodensity-graph generated by the scorer consensus. RESULTS: The performance of the models improves on all the databases when we train the models with our LSSC. We found an increase in ACS (up to 6.4%) between the hypnodensity-graph generated by the models trained with-LSSC and the hypnodensity-graph generated by the consensus. CONCLUSION: Our approach definitely enables a model to better adapt to the consensus of the group of scorers. Future work will focus on further investigations on different scoring architectures and hopefully large-scale-heterogeneous multi-scored datasets.

Wearables in the home-based assessment of abnormal movements in Parkinson's disease: a systematic review of the literature.

At present, the standard practices for home-based assessments of abnormal movements in Parkinson's disease (PD) are based either on subjective tools or on objective measures that often fail to capture day-to-day fluctuations and long-term information in real-life conditions in a way that patient's compliance and privacy are secured. The employment of wearable technologies in PD represents a great paradigm shift in healthcare remote diagnostics and therapeutics monitoring. However, their applicability in everyday clinical practice seems to be still limited. We carried out a systematic search across the Medline Database. In total, 246 publications, published until 1 June 2020, were identified. Among them, 26 reports met the inclusion criteria and were included in the present review. We focused more on clinically relevant aspects of wearables' application including feasibility and efficacy of the assessment, the number, type and body position of the wearable devices, type of PD motor symptom, environment and duration of assessments and validation methodology. The aim of this review is to provide a systematic overview of the current knowledge and state-of-the-art of the home-based assessment of motor symptoms and fluctuations in PD patients using wearable technology, highlighting current problems and laying foundations for future works.

Addressing bias in big data and AI for health care: A call for open science.

Artificial intelligence (AI) has an astonishing potential in assisting clinical decision making and revolutionizing the field of health care. A major open challenge that AI will need to address before its integration in the clinical routine is that of algorithmic bias. Most AI algorithms need big datasets to learn from, but several groups of the human population have a long history of being absent or misrepresented in existing biomedical datasets. If the training data is misrepresentative of the population variability, AI is prone to reinforcing bias, which can lead to fatal outcomes, misdiagnoses, and lack of generalization. Here, we describe the challenges in rendering AI algorithms fairer, and we propose concrete steps for addressing bias using tools from the field of open science.

DeepSleepNet-Lite: A Simplified Automatic Sleep Stage Scoring Model With Uncertainty Estimates.

Deep learning is widely used in the most recent automatic sleep scoring algorithms. Its popularity stems from its excellent performance and from its ability to process raw signals and to learn feature directly from the data. Most of the existing scoring algorithms exploit very computationally demanding architectures, due to their high number of training parameters, and process lengthy time sequences in input (up to 12 minutes). Only few of these architectures provide an estimate of the model uncertainty. In this study we propose DeepSleepNet-Lite, a simplified and lightweight scoring architecture, processing only 90-seconds EEG input sequences. We exploit, for the first time in sleep scoring, the Monte Carlo dropout technique to enhance the performance of the architecture and to also detect the uncertain instances. The evaluation is performed on a single-channel EEG Fpz-Cz from the open source Sleep-EDF expanded database. DeepSleepNet-Lite achieves slightly lower performance, if not on par, compared to the existing state-of-the-art architectures, in overall accuracy, macro F1-score and Cohen's kappa (on Sleep-EDF v1-2013 ±30mins: 84.0%, 78.0%, 0.78; on Sleep-EDF v2-2018 ±30mins: 80.3%, 75.2%, 0.73). Monte Carlo dropout enables the estimate of the uncertain predictions. By rejecting the uncertain instances, the model achieves higher performance on both versions of the database (on Sleep-EDF v1-2013 ±30mins: 86.1.0%, 79.6%, 0.81; on Sleep-EDF v2-2018 ±30mins: 82.3%, 76.7%, 0.76). Our lighter sleep scoring approach paves the way to the application of scoring algorithms for sleep analysis in real-time.

The SleepFit Tablet Application for Home-Based Clinical Data Collection in Parkinson Disease: User-Centric Development and Usability Study.

BACKGROUND: Parkinson disease (PD) is a common, multifaceted neurodegenerative disorder profoundly impacting patients' autonomy and quality of life. Assessment in real-life conditions of subjective symptoms and objective metrics of mobility and nonmotor symptoms such as sleep disturbance is strongly advocated. This information would critically guide the adaptation of antiparkinsonian medications and nonpharmacological interventions. Moreover, since the spread of the COVID-19 pandemic, health care practices are being reshaped toward a more home-based care. New technologies could play a pivotal role in this new approach to clinical care. Nevertheless, devices and information technology tools might be unhandy for PD patients, thus dramatically limiting their widespread employment. OBJECTIVE: The goals of the research were development and usability evaluation of an application, SleepFit, for ecological momentary assessment of objective and subjective clinical metrics at PD patients' homes, and as a remote tool for researchers to monitor patients and integrate and manage data. METHODS: An iterative and user-centric strategy was employed for the development of SleepFit. The core structure of SleepFit consists of (1) an electronic finger-tapping test; (2) motor, sleepiness, and emotional subjective scales; and (3) a sleep diary. Applicable design, ergonomic, and navigation principles have been applied while tailoring the application to the specific patient population. Three progressively enhanced versions of the application (alpha, v1.0, v2.0) were tested by a total of 56 patients with PD who were asked to perform multiple home assessments 4 times per day for 2 weeks. Patient compliance was calculated as the proportion of completed tasks out of the total number of expected tasks. Satisfaction on the latest version (v2.0) was evaluated as potential willingness to use SleepFit again after the end of the study. RESULTS: From alpha to v1.0, SleepFit was improved in graphics, ergonomics, and navigation, with automated flows guiding the patients in performing tasks throughout the 24 hours, and real-time data collection and consultation were made possible thanks to a remote web portal. In v2.0, the kiosk-mode feature restricts the use of the tablet to the SleepFit application only, thus preventing users from accidentally exiting the application. A total of 52 (4 dropouts) patients were included in the analyses. Overall compliance (all versions) was 88.89% (5707/6420). SleepFit was progressively enhanced and compliance increased from 87.86% (2070/2356) to 89.92% (2899/3224; P=.04). Among the patients who used v2.0, 96% (25/26) declared they would use SleepFit again. CONCLUSIONS: SleepFit can be considered a state-of-the-art home-based system that increases compliance in PD patients, ensures high-quality data collection, and works as a handy tool for remote monitoring and data management in clinical research. Thanks to its user-friendliness and modular structure, it could be employed in other clinical studies with minimum adaptation efforts. TRIAL REGISTRATION: ClinicalTrials.gov NCT02723396; https://clinicaltrials.gov/ct2/show/NCT02723396.

Personalized Sleep Spindle Detection in Whole Night Polysomnography.

The present study proposes a new personalized sleep spindle detection algorithm, suggesting the importance of an individualized approach. We identify an optimal set of features that characterize the spindle and exploit a support vector machine to distinguish between spindle and nonspindle patterns. The algorithm is assessed on the open source DREAMS database, that contains only selected part of the polysomnography, and on whole night polysomnography recordings from the SPASH database. We show that on the former database the personalization can boost sensitivity, from 84.2% to 89.8%, with a slight increase in specificity, from 97.6% to 98.1%. On a whole night polysomnography instead, the algorithm reaches a sensitivity of 98.6% and a specificity of 98.1%, thanks to the personalization approach. Future work will address the integration of the spindle detection algorithm within a sleep scoring automated procedure.

Temporal dependency in automatic sleep scoring via deep learning based architectures: An empirical study.

The present study evaluates how effectively a deep learning based sleep scoring system does encode the temporal dependency from raw polysomnography signals. An exhaustive range of neural networks, including state of the art architecture, have been used in the evaluation. The architectures have been assessed using a single-channel EEG Fpz-Cz from the open source Sleep-EDF expanded database. The best performing model reached an overall accuracy of 85.2% and a Cohen's kappa of 0.8, with an F1-score of stage N1 equal to 50.2%. We have introduced a new metric, δnorm, to better evaluate temporal dependencies. A simple feed forward architecture not only achieves comparable performance to most up-to-date complex architectures, but also does better encode the continuous temporal characteristics of sleep.Clinical relevance - A better understanding of the capability of the network in encoding sleep temporal patterns could lead to improve the automatic sleep scoring.

A New Prospective, Home-Based Monitoring of Motor Symptoms in Parkinson's Disease.

BACKGROUND: Subjective symptoms, which are retrospectively assessed during clinical interviews in the office, may be influenced by patient recall in Parkinson's disease (PD). Prospective collection of subjective data might be an effective tool to overcome this bias. OBJECTIVE: We investigated the correspondence between prospectively and retrospectively assessed motor symptoms in PD. METHODS: Forty-two consecutive patients (9 females, 67±9.8 years old) with mild to moderate PD reported their symptoms four times a day for two weeks, using the "SleepFit" application (app) for tablets. This app incorporates a new Visual Analogue Scale assessing global mobility (m-VAS), and the Scales for Outcome in Parkinson Assessment Diary Card (SCOPA-DC). At day 14, the Movement Disorders Society Unified Parkinson's Disease Rating Scale (MDS-UPDRS) parts II and IV questionnaires were completed at the hospital. Agreement (root mean square difference) and the tendency to under- or overestimate their symptoms by patients (relative difference after normalization) were calculated to compare prospectively vs. retrospectively collected information. RESULTS: Although agreement was good for overall scores (m-VAS: 10.0%; SCOPA-DC: 18.3%), and for single motor symptoms (involuntary movements, hand dexterity, walking, changing position; each <20%), some individuals with more advanced disease, higher fatigue or worse sleep quality showed poor symptom recall in retrospect. Moreover, a subgroup of patients (16.7%) either over- or underestimated symptom severity. CONCLUSIONS: Regular, prospective monitoring of motor symptoms is suitable in PD patients. SleepFit might be a useful tool in routine practice to identify patients tending to under- or overestimate their symptoms, and for their follow-up.

Automated sleep scoring: A review of the latest approaches.

Clinical sleep scoring involves a tedious visual review of overnight polysomnograms by a human expert, according to official standards. It could appear then a suitable task for modern artificial intelligence algorithms. Indeed, machine learning algorithms have been applied to sleep scoring for many years. As a result, several software products offer nowadays automated or semi-automated scoring services. However, the vast majority of the sleep physicians do not use them. Very recently, thanks to the increased computational power, deep learning has also been employed with promising results. Machine learning algorithms can undoubtedly reach a high accuracy in specific situations, but there are many difficulties in their introduction in the daily routine. In this review, the latest approaches that are applying deep learning for facilitating and accelerating sleep scoring are thoroughly analyzed and compared with the state of the art methods. Then the obstacles in introducing automated sleep scoring in the clinical practice are examined. Deep learning algorithm capabilities of learning from a highly heterogeneous dataset, in terms both of human data and of scorers, are very promising and should be further investigated.