The first step is the hardest: pitfalls of representing and tokenizing temporal data for large language models.

OBJECTIVES: Large language models (LLMs) have demonstrated remarkable generalization and across diverse tasks, leading individuals to increasingly use them as personal assistants due to their emerging reasoning capabilities. Nevertheless, a notable obstacle emerges when including numerical/temporal data into these prompts, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. This article discusses the challenges of representing and tokenizing temporal data. It argues that naively passing timeseries to LLMs can be ineffective due to the modality gap between numbers and text. MATERIALS AND METHODS: We conduct a case study by tokenizing a sample mobile sensing dataset using the OpenAI tokenizer. We also review recent works that feed timeseries data into LLMs for human-centric tasks, outlining common experimental setups like zero-shot prompting and few-shot learning. RESULTS: The case study shows that popular LLMs split timestamps and sensor values into multiple nonmeaningful tokens, indicating they struggle with temporal data. We find that preliminary works rely heavily on prompt engineering and timeseries aggregation to "ground" LLMs, hinting that the "modality gap" hampers progress. The literature was critically analyzed through the lens of models optimizing for expressiveness versus parameter efficiency. On one end of the spectrum, training large domain-specific models from scratch is expressive but not parameter-efficient. On the other end, zero-shot prompting of LLMs is parameter-efficient but lacks expressiveness for temporal data. DISCUSSION: We argue tokenizers are not optimized for numerical data, while the scarcity of timeseries examples in training corpora exacerbates difficulties. We advocate balancing model expressiveness and computational efficiency when integrating temporal data. Prompt tuning, model grafting, and improved tokenizers are highlighted as promising directions. CONCLUSION: We underscore that despite promising capabilities, LLMs cannot meaningfully process temporal data unless the input representation is addressed. We argue that this paradigm shift in how we leverage pretrained models will particularly affect the area of biomedical signals, given the lack of modality-specific foundation models.

EarSet: A Multi-Modal Dataset for Studying the Impact of Head and Facial Movements on In-Ear PPG Signals.

Photoplethysmography (PPG) is a simple, yet powerful technique to study blood volume changes by measuring light intensity variations. However, PPG is severely affected by motion artifacts, which hinder its trustworthiness. This problem is pressing in earables since head movements and facial expressions cause skin and tissue displacements around and inside the ear. Understanding such artifacts is fundamental to the success of earables for accurate cardiovascular health monitoring. However, the lack of in-ear PPG datasets prevents the research community from tackling this challenge. In this work, we report on the design of an ear tip featuring a 3-channels PPG and a co-located 6-axis motion sensor. This, enables sensing PPG data at multiple wavelengths and the corresponding motion signature from both ears. Leveraging our device, we collected a multi-modal dataset from 30 participants while performing 16 natural motions, including both head/face and full body movements. This unique dataset will greatly support research towards making in-ear vital signs sensing more accurate and robust, thus unlocking the full potential of the next-generation PPG-equipped earables.

A multidevice and multimodal dataset for human energy expenditure estimation using wearable devices.

We present a multi-device and multi-modal dataset, called WEEE, collected from 17 participants while they were performing different physical activities. WEEE contains: (1) sensor data collected using seven wearable devices placed on four body locations (head, ear, chest, and wrist); (2) respiratory data collected with an indirect calorimeter serving as ground-truth information; (3) demographics and body composition data (e.g., fat percentage); (4) intensity level and type of physical activities, along with their corresponding metabolic equivalent of task (MET) values; and (5) answers to questionnaires about participants' physical activity level, diet, stress and sleep. Thanks to the diversity of sensors and body locations, we believe that the dataset will enable the development of novel human energy expenditure (EE) estimation techniques for a diverse set of application scenarios. EE refers to the amount of energy an individual uses to maintain body functions and as a result of physical activity. A reliable estimate of people's EE thus enables computing systems to make inferences about users' physical activity and help them promoting a healthier lifestyle.