Word Forms Reflect Trade-Offs Between Speaker Effort and Robust Listener Recognition.

How do cognitive pressures shape the lexicons of natural languages? Here, we reframe George Kingsley Zipf's proposed "law of abbreviation" within a more general framework that relates it to cognitive pressures that affect speakers and listeners. In this new framework, speakers' drive to reduce effort (Zipf's proposal) is counteracted by the need for low-frequency words to have word forms that are sufficiently distinctive to allow for accurate recognition by listeners. To support this framework, we replicate and extend recent work using the prevalence of subword phonemic sequences (phonotactic probability) to measure speakers' production effort in place of Zipf's measure of length. Across languages and corpora, phonotactic probability is more strongly correlated with word frequency than word length. We also show this measure of ease of speech production (phonotactic probability) is strongly correlated with a measure of perceptual difficulty that indexes the degree of competition from alternative interpretations in word recognition. This is consistent with the claim that there must be trade-offs between these two factors, and is inconsistent with a recent proposal that phonotactic probability facilitates both perception and production. To our knowledge, this is the first work to offer an explanation why long, phonotactically improbable word forms remain in the lexicons of natural languages.

Exploring complex and integrated information during sleep.

The Integrated Information Theory is a theoretical framework that aims to elucidate the nature of consciousness by postulating that it emerges from the integration of information within a system, and that the degree of consciousness depends on the extent of information integration within the system. When consciousness is lost, the core complex of consciousness proposed by the Integrated Information Theory disintegrates, and Φ measures, which reflect the level of integrated information, are expected to diminish. This study examined the predictions of the Integrated Information Theory using the global brain network acquired via functional magnetic resonance imaging during various tasks and sleep. We discovered that the complex located within the frontoparietal network remained constant regardless of task content, while the regional distribution of the complex collapsed in the initial stages of sleep. Furthermore, Φ measures decreased as sleep progressed under limited analysis conditions. These findings align with predictions made by the Integrated Information Theory and support its postulates.

Comparative analyses of the information content of letters, characters, and inter-word spaces across writing systems.

One difference among writing systems is how orthographic cues are used to demarcate words; although most alphabetic scripts use inter-word spaces, some Asian scripts do not explicitly mark word boundaries (e.g., Chinese). It is unclear whether these differences are arbitrary or whether they are designed to maximize reading efficiency. Here, we show that spaces inserted between words in non-demarcated scripts provide less information about word boundaries than spaces in demarcated scripts. Furthermore, despite the fact that less information is contained by inter-word spaces than characters/letters of the same size, the information content of inter-word spaces in demarcated scripts is closer to that of characters/letters compared to the information content of inter-word spaces that are inserted in non-demarcated scripts. These results suggest that the conventions used to demarcate word boundaries are sufficient to support efficient reading. Our findings provide new insights into the universals and variation across writing systems and shed light on the mental processes that support skilled reading.

Information theory reveals physiological manifestations of COVID-19 that correlate with symptom density of illness.

Algorithms for the detection of COVID-19 illness from wearable sensor devices tend to implicitly treat the disease as causing a stereotyped (and therefore recognizable) deviation from healthy physiology. In contrast, a substantial diversity of bodily responses to SARS-CoV-2 infection have been reported in the clinical milieu. This raises the question of how to characterize the diversity of illness manifestations, and whether such characterization could reveal meaningful relationships across different illness manifestations. Here, we present a framework motivated by information theory to generate quantified maps of illness presentation, which we term "manifestations," as resolved by continuous physiological data from a wearable device (Oura Ring). We test this framework on five physiological data streams (heart rate, heart rate variability, respiratory rate, metabolic activity, and sleep temperature) assessed at the time of reported illness onset in a previously reported COVID-19-positive cohort (N = 73). We find that the number of distinct manifestations are few in this cohort, compared to the space of all possible manifestations. In addition, manifestation frequency correlates with the rough number of symptoms reported by a given individual, over a several-day period prior to their imputed onset of illness. These findings suggest that information-theoretic approaches can be used to sort COVID-19 illness manifestations into types with real-world value. This proof of concept supports the use of information-theoretic approaches to map illness manifestations from continuous physiological data. Such approaches could likely inform algorithm design and real-time treatment decisions if developed on large, diverse samples.

Response inhibition in premotor cortex corresponds to a complex reshuffle of the mesoscopic information network.

Recent studies have explored functional and effective neural networks in animal models; however, the dynamics of information propagation among functional modules under cognitive control remain largely unknown. Here, we addressed the issue using transfer entropy and graph theory methods on mesoscopic neural activities recorded in the dorsal premotor cortex of rhesus monkeys. We focused our study on the decision time of a Stop-signal task, looking for patterns in the network configuration that could influence motor plan maturation when the Stop signal is provided. When comparing trials with successful inhibition to those with generated movement, the nodes of the network resulted organized into four clusters, hierarchically arranged, and distinctly involved in information transfer. Interestingly, the hierarchies and the strength of information transmission between clusters varied throughout the task, distinguishing between generated movements and canceled ones and corresponding to measurable levels of network complexity. Our results suggest a putative mechanism for motor inhibition in premotor cortex: a topological reshuffle of the information exchanged among ensembles of neurons.

In this study, we investigated the dynamics of information transfer among functionally identified neural modules during cognitive motor control. Our focus was on mesoscopic neural activities in the dorsal premotor cortex of rhesus monkeys engaged in a Stop-signal task. Leveraging multivariate transfer entropy and graph theory, we uncovered insights on how behavioral control shapes the topology of information transmission in a local brain network. Task phases modulated the strength and hierarchy of information exchange between modules, revealing the nuanced interplay between neural populations during generated and canceled movements. Notably, during successful inhibition, the network displayed a distinctive configuration, unveiling a novel mechanism for motor inhibition in the premotor cortex: a topological reshuffle of information among neuronal ensembles.

Testing dynamic correlations and nonlinearity in bivariate time series through information measures and surrogate data analysis.

The increasing availability of time series data depicting the evolution of physical system properties has prompted the development of methods focused on extracting insights into the system behavior over time, discerning whether it stems from deterministic or stochastic dynamical systems. Surrogate data testing plays a crucial role in this process by facilitating robust statistical assessments. This ensures that the observed results are not mere occurrences by chance, but genuinely reflect the inherent characteristics of the underlying system. The initial process involves formulating a null hypothesis, which is tested using surrogate data in cases where assumptions about the underlying distributions are absent. A discriminating statistic is then computed for both the original data and each surrogate data set. Significantly deviating values between the original data and the surrogate data ensemble lead to the rejection of the null hypothesis. In this work, we present various surrogate methods designed to assess specific statistical properties in random processes. Specifically, we introduce methods for evaluating the presence of autodependencies and nonlinear dynamics within individual processes, using Information Storage as a discriminating statistic. Additionally, methods are introduced for detecting coupling and nonlinearities in bivariate processes, employing the Mutual Information Rate for this purpose. The surrogate methods introduced are first tested through simulations involving univariate and bivariate processes exhibiting both linear and nonlinear dynamics. Then, they are applied to physiological time series of Heart Period (RR intervals) and respiratory flow (RESP) variability measured during spontaneous and paced breathing. Simulations demonstrated that the proposed methods effectively identify essential dynamical features of stochastic systems. The real data application showed that paced breathing, at low breathing rate, increases the predictability of the individual dynamics of RR and RESP and dampens nonlinearity in their coupled dynamics.

Rebuilding the Habitable Zone from the Bottom up with Computational Zones.

Computation, if treated as a set of physical processes that act on information represented by states of matter, encompasses biological systems, digital systems, and other constructs and may be a fundamental measure of living systems. The opportunity for biological computation, represented in the propagation and selection-driven evolution of information-carrying organic molecular structures, has been partially characterized in terms of planetary habitable zones (HZs) based on primary conditions such as temperature and the presence of liquid water. A generalization of this concept to computational zones (CZs) is proposed, with constraints set by three principal characteristics: capacity (including computation rates), energy, and instantiation (or substrate, including spatial extent). CZs naturally combine traditional habitability factors, including those associated with biological function that incorporate the chemical milieu, constraints on nutrients and free energy, as well as element availability. Two example applications are presented by examining the fundamental thermodynamic work efficiency and Landauer limit of photon-driven biological computation on planetary surfaces and of generalized computation in stellar energy capture structures (a.k.a. Dyson structures). It is suggested that CZs that involve nested structures or substellar objects could manifest unique observational signatures as cool far-infrared emitters. While these latter scenarios are entirely hypothetical, they offer a useful, complementary introduction to the potential universality of CZs.

Building Test Batteries Based on Analyzing Random Number Generator Tests within the Framework of Algorithmic Information Theory.

The problem of testing random number generators is considered and a new method for comparing the power of different statistical tests is proposed. It is based on the definitions of random sequence developed in the framework of algorithmic information theory and allows comparing the power of different tests in some cases when the available methods of mathematical statistics do not distinguish between tests. In particular, it is shown that tests based on data compression methods using dictionaries should be included in test batteries.

Toward a universal theory of consciousness.

While falsifiability has been broadly discussed as a desirable property of a theory of consciousness, in this paper, we introduce the meta-theoretic concept of "Universality" as an additional desirable property for a theory of consciousness. The concept of universality, often assumed in physics, posits that the fundamental laws of nature are consistent and apply equally everywhere in the universe and remain constant over time. This assumption is crucial in science, acting as a guiding principle for developing and testing theories. When applied to theories of consciousness, universality can be defined as the ability of a theory to determine whether any fully described dynamical system is conscious or non-conscious. Importantly, for a theory to be universal, the determinant of consciousness needs to be defined as an intrinsic property of a system as opposed to replying on the interpretation of the external observer. The importance of universality originates from the consideration that given that consciousness is a natural phenomenon, it could in principle manifest in any physical system that satisfies a certain set of conditions whether it is biological or non-biological. To date, apart from a few exceptions, most existing theories do not possess this property. Instead, they tend to make predictions as to the neural correlates of consciousness based on the interpretations of brain functions, which makes those theories only applicable to brain-centric systems. While current functionalist theories of consciousness tend to be heavily reliant on our interpretations of brain functions, we argue that functionalist theories could be converted to a universal theory by specifying mathematical formulations of the constituent concepts. While neurobiological and functionalist theories retain their utility in practice, we will eventually need a universal theory to fully explain why certain types of systems possess consciousness.

Identification of potential biomarkers for bone metastasis using human cancer metastasis database.

Objective: Information theory has been successfully employed to identify optimal pathway networks, mutual information (MI), and entropy as a dynamic response in statistical methods and estimate input and output information in systems biology. This research aims to investigate potentially integrated gene signatures for bone metastasis using graph-based information theory from the dynamic interaction interphase. Methods: The expression dataset with the series ID GSE26964 for bone metastasis from prostate cancer was retrieved. The dataset was segregated for differentially expressed genes (DEGs) using the Human Cancer Metastasis Database. MI was considered to capture non-linear connections to classify the key DEGs from the collected dataset using gene-gene statistical analysis and then a protein-protein interaction network (PPIN). The PPIN was used to calculate centrality metrics, bottlenecks, and functional annotations. Results: A total of 531 DEGs were identified. Thirteen genes were classified as highly correlated based on their gene expression data matrix. The extended PPIN of the 13 genes comprised 53 nodes and 372 edges. A total of four DEGs were identified as hubs. One novel gene was identified with strong network connectivity. Conclusion: The novel biomarkers for metastasis may provide information on cancer metastasis to the bone by implying MI and information theory.

Information content and optimization of self-organized developmental systems.

A key feature of many developmental systems is their ability to self-organize spatial patterns of functionally distinct cell fates. To ensure proper biological function, such patterns must be established reproducibly, by controlling and even harnessing intrinsic and extrinsic fluctuations. While the relevant molecular processes are increasingly well understood, we lack a principled framework to quantify the performance of such stochastic self-organizing systems. To that end, we introduce an information-theoretic measure for self-organized fate specification during embryonic development. We show that the proposed measure assesses the total information content of fate patterns and decomposes it into interpretable contributions corresponding to the positional and correlational information. By optimizing the proposed measure, our framework provides a normative theory for developmental circuits, which we demonstrate on lateral inhibition, cell type proportioning, and reaction-diffusion models of self-organization. This paves a way toward a classification of developmental systems based on a common information-theoretic language, thereby organizing the zoo of implicated chemical and mechanical signaling processes.

An information-theoretic analysis of targeted regressions during reading.

Regressions, or backward saccades, are common during reading, accounting for between 5% and 20% of all saccades. And yet, relatively little is known about what causes them. We provide an information-theoretic operationalization for two previous qualitative hypotheses about regressions, which we dub reactivation and reanalysis. We argue that these hypotheses make different predictions about the pointwise mutual information or pmi between a regression's source and target. Intuitively, the pmi between two words measures how much more (or less) likely one word is to be present given the other. On one hand, the reactivation hypothesis predicts that regressions occur between words that are associated, implying high positive values of pmi. On the other hand, the reanalysis hypothesis predicts that regressions should occur between words that are not associated with each other, implying negative, low values of pmi. As a second theoretical contribution, we expand on previous theories by considering not only pmi but also expected values of pmi, E[pmi], where the expectation is taken over all possible realizations of the regression's target. The rationale for this is that language processing involves making inferences under uncertainty, and readers may be uncertain about what they have read, especially if a previous word was skipped. To test both theories, we use contemporary language models to estimate pmi-based statistics over word pairs in three corpora of eye tracking data in English, as well as in six languages across three language families (Indo-European, Uralic, and Turkic). Our results are consistent across languages and models tested: Positive values of pmi and E[pmi] consistently help to predict the patterns of regressions during reading, whereas negative values of pmi and E[pmi] do not. Our information-theoretic interpretation increases the predictive scope of both theories and our studies present the first systematic crosslinguistic analysis of regressions in the literature. Our results support the reactivation hypothesis and, more broadly, they expand the number of language processing behaviors that can be linked to information-theoretic principles.

The physical and evolutionary energy landscapes of devolved protein sequences corresponding to pseudogenes.

Protein evolution is guided by structural, functional, and dynamical constraints ensuring organismal viability. Pseudogenes are genomic sequences identified in many eukaryotes that lack translational activity due to sequence degradation and thus over time have undergone "devolution." Previously pseudogenized genes sometimes regain their protein-coding function, suggesting they may still encode robust folding energy landscapes despite multiple mutations. We study both the physical folding landscapes of protein sequences corresponding to human pseudogenes using the Associative Memory, Water Mediated, Structure and Energy Model, and the evolutionary energy landscapes obtained using direct coupling analysis (DCA) on their parent protein families. We found that generally mutations that have occurred in pseudogene sequences have disrupted their native global network of stabilizing residue interactions, making it harder for them to fold if they were translated. In some cases, however, energetic frustration has apparently decreased when the functional constraints were removed. We analyzed this unexpected situation for Cyclophilin A, Profilin-1, and Small Ubiquitin-like Modifier 2 Protein. Our analysis reveals that when such mutations in the pseudogene ultimately stabilize folding, at the same time, they likely alter the pseudogenes' former biological activity, as estimated by DCA. We localize most of these stabilizing mutations generally to normally frustrated regions required for binding to other partners.

Stimulus information guides the emergence of behavior-related signals in primary somatosensory cortex during learning.

Neurons in the primary cortex carry sensory- and behavior-related information, but it remains an open question how this information emerges and intersects together during learning. Current evidence points to two possible learning-related changes: sensory information increases in the primary cortex or sensory information remains stable, but its readout efficiency in association cortices increases. We investigated this question by imaging neuronal activity in mouse primary somatosensory cortex before, during, and after learning of an object localization task. We quantified sensory- and behavior-related information and estimated how much sensory information was used to instruct perceptual choices as learning progressed. We find that sensory information increases from the start of training, while choice information is mostly present in the later stages of learning. Additionally, the readout of sensory information becomes more efficient with learning as early as in the primary sensory cortex. Together, our results highlight the importance of primary cortical neurons in perceptual learning.

On the Accurate Estimation of Information-Theoretic Quantities from Multi-Dimensional Sample Data.

Using information-theoretic quantities in practical applications with continuous data is often hindered by the fact that probability density functions need to be estimated in higher dimensions, which can become unreliable or even computationally unfeasible. To make these useful quantities more accessible, alternative approaches such as binned frequencies using histograms and k-nearest neighbors (k-NN) have been proposed. However, a systematic comparison of the applicability of these methods has been lacking. We wish to fill this gap by comparing kernel-density-based estimation (KDE) with these two alternatives in carefully designed synthetic test cases. Specifically, we wish to estimate the information-theoretic quantities: entropy, Kullback-Leibler divergence, and mutual information, from sample data. As a reference, the results are compared to closed-form solutions or numerical integrals. We generate samples from distributions of various shapes in dimensions ranging from one to ten. We evaluate the estimators' performance as a function of sample size, distribution characteristics, and chosen hyperparameters. We further compare the required computation time and specific implementation challenges. Notably, k-NN estimation tends to outperform other methods, considering algorithmic implementation, computational efficiency, and estimation accuracy, especially with sufficient data. This study provides valuable insights into the strengths and limitations of the different estimation methods for information-theoretic quantities. It also highlights the significance of considering the characteristics of the data, as well as the targeted information-theoretic quantity when selecting an appropriate estimation technique. These findings will assist scientists and practitioners in choosing the most suitable method, considering their specific application and available data. We have collected the compared estimation methods in a ready-to-use open-source Python 3 toolbox and, thereby, hope to promote the use of information-theoretic quantities by researchers and practitioners to evaluate the information in data and models in various disciplines.

Information Theory, Living Systems, and Communication Engineering.

Mainstream research on information theory within the field of living systems involves the application of analytical tools to understand a broad range of life processes. This paper is dedicated to an opposite problem: it explores the information theory and communication engineering methods that have counterparts in the data transmission process by way of DNA structures and neural fibers. Considering the requirements of modern multimedia, transmission methods chosen by nature may be different, suboptimal, or even far from optimal. However, nature is known for rational resource usage, so its methods have a significant advantage: they are proven to be sustainable. Perhaps understanding the engineering aspects of methods of nature can inspire a design of alternative green, stable, and low-cost transmission.

Entropy-based detection of Twitter echo chambers.

Echo chambers, i.e. clusters of users exposed to news and opinions in line with their previous beliefs, were observed in many online debates on social platforms. We propose a completely unbiased entropy-based method for detecting echo chambers. The method is completely agnostic to the nature of the data. In the Italian Twitter debate about the Covid-19 vaccination, we find a limited presence of users in echo chambers (about 0.35% of all users). Nevertheless, their impact on the formation of a common discourse is strong, as users in echo chambers are responsible for nearly a third of the retweets in the original dataset. Moreover, in the case study observed, echo chambers appear to be a receptacle for disinformative content.

Determination of the point resolution of high-resolution transmission electron microscope using the through-focus technique.

From the viewpoint of evaluating the instrumental performance of high-resolution electron microscopy (HREM), the Scherzer condition was investigated using information theory. As a result, the optimum defocus amount Δf can be expressed based on [Formula: see text] , and the formula [Formula: see text] is obtained. Furthermore, a procedure for measuring point resolution using the through-focus technique is developed, and a new method for determining the spherical aberration coefficient using the variance of Δf is introduced in the procedure.

Information density as a predictor of communication dynamics.

In a recent paper, Aceves and Evans computed information and semantic density measures for hundreds of languages, and showed that these measures predict the pace and breadth of ideas in communication. Here, we summarize their key findings and situate them in a broader debate about the adaptive nature of language.