Factors affecting inter-rater agreement in human classification of eye movements: a comparison of three datasets.

Manual classification of eye-movements is used in research and as a basis for comparison with automatic algorithms in the development phase. However, human classification will not be useful if it is unreliable and unrepeatable. Therefore, it is important to know what factors might influence and enhance the accuracy and reliability of human classification of eye-movements. In this report we compare three datasets of human manual classification, two from earlier datasets and one, our own dataset, which we present here for the first time. For inter-rater reliability, we assess both the event-level F1-score and sample-level Cohen's κ, across groups of raters. The report points to several possible influences on human classification reliability: eye-tracker quality, use of head restraint, characteristics of the recorded subjects, the availability of detailed scoring rules, and the characteristics and training of the raters.

Sleep classification using Consumer Sleep Technologies and AI: A review of the current landscape.

Classifying sleep stages in real-time represents considerable potential, for instance in enabling interactive noise masking in noisy environments when persons are in a state of light sleep or to support clinical staff in analyzing sleep patterns etc. However, the current gold standard for classifying sleep stages, Polysomnography (PSG), is too cumbersome to apply outside controlled hospital settings and requires manual as well as highly specialized knowledge to classify sleep stages. Using data from Consumer Sleep Technologies (CSTs) to inform machine learning algorithms represent a promising opportunity for automating the process of classifying sleep stages, also in settings outside the confinements of clinical expert settings. This study reviews 27 papers that use CSTs in combination with Artificial Intelligence (AI) models to classify sleep stages. AI models and their performance are described and compared to synthesize current state of the art in sleep stage classification with CSTs. Furthermore, gaps in the current approaches are shown and how these AI models could be improved in the near-future. Lastly, the challenges of designing interactions for users that are asleep are highlighted pointing towards avenues of more interactive sleep interventions based on AI-infused CSTs solutions.