Emergence of input selective recurrent dynamics via information transfer maximization.

Network structures of the brain have wiring patterns specialized for specific functions. These patterns are partially determined genetically or evolutionarily based on the type of task or stimulus. These wiring patterns are important in information processing; however, their organizational principles are not fully understood. This study frames the maximization of information transmission alongside the reduction of maintenance costs as a multi-objective optimization challenge, utilizing information theory and evolutionary computing algorithms with an emphasis on the visual system. The goal is to understand the underlying principles of circuit formation by exploring the patterns of wiring and information processing. The study demonstrates that efficient information transmission necessitates sparse circuits with internal modular structures featuring distinct wiring patterns. Significant trade-offs underscore the necessity of balance in wiring pattern development. The dynamics of effective circuits exhibit moderate flexibility in response to stimuli, in line with observations from prior visual system studies. Maximizing information transfer may allow for the self-organization of information processing functions similar to actual biological circuits, without being limited by modality. This study offers insights into neuroscience and the potential to improve reservoir computing performance.

An information-theoretic analysis of targeted regressions during reading.

Regressions, or backward saccades, are common during reading, accounting for between 5% and 20% of all saccades. And yet, relatively little is known about what causes them. We provide an information-theoretic operationalization for two previous qualitative hypotheses about regressions, which we dub reactivation and reanalysis. We argue that these hypotheses make different predictions about the pointwise mutual information or pmi between a regression's source and target. Intuitively, the pmi between two words measures how much more (or less) likely one word is to be present given the other. On one hand, the reactivation hypothesis predicts that regressions occur between words that are associated, implying high positive values of pmi. On the other hand, the reanalysis hypothesis predicts that regressions should occur between words that are not associated with each other, implying negative, low values of pmi. As a second theoretical contribution, we expand on previous theories by considering not only pmi but also expected values of pmi, E[pmi], where the expectation is taken over all possible realizations of the regression's target. The rationale for this is that language processing involves making inferences under uncertainty, and readers may be uncertain about what they have read, especially if a previous word was skipped. To test both theories, we use contemporary language models to estimate pmi-based statistics over word pairs in three corpora of eye tracking data in English, as well as in six languages across three language families (Indo-European, Uralic, and Turkic). Our results are consistent across languages and models tested: Positive values of pmi and E[pmi] consistently help to predict the patterns of regressions during reading, whereas negative values of pmi and E[pmi] do not. Our information-theoretic interpretation increases the predictive scope of both theories and our studies present the first systematic crosslinguistic analysis of regressions in the literature. Our results support the reactivation hypothesis and, more broadly, they expand the number of language processing behaviors that can be linked to information-theoretic principles.

Synaptic Information Storage Capacity Measured With Information Theory.

Variation in the strength of synapses can be quantified by measuring the anatomical properties of synapses. Quantifying precision of synaptic plasticity is fundamental to understanding information storage and retrieval in neural circuits. Synapses from the same axon onto the same dendrite have a common history of coactivation, making them ideal candidates for determining the precision of synaptic plasticity based on the similarity of their physical dimensions. Here, the precision and amount of information stored in synapse dimensions were quantified with Shannon information theory, expanding prior analysis that used signal detection theory (Bartol et al., 2015). The two methods were compared using dendritic spine head volumes in the middle of the stratum radiatum of hippocampal area CA1 as well-defined measures of synaptic strength. Information theory delineated the number of distinguishable synaptic strengths based on nonoverlapping bins of dendritic spine head volumes. Shannon entropy was applied to measure synaptic information storage capacity (SISC) and resulted in a lower bound of 4.1 bits and upper bound of 4.59 bits of information based on 24 distinguishable sizes. We further compared the distribution of distinguishable sizes and a uniform distribution using Kullback-Leibler divergence and discovered that there was a nearly uniform distribution of spine head volumes across the sizes, suggesting optimal use of the distinguishable values. Thus, SISC provides a new analytical measure that can be generalized to probe synaptic strengths and capacity for plasticity in different brain regions of different species and among animals raised in different conditions or during learning. How brain diseases and disorders affect the precision of synaptic plasticity can also be probed.

A measure centrality index for systematic empirical comparison of consciousness theories.

Consciousness science is marred by disparate constructs and methodologies, making it challenging to systematically compare theories. This foundational crisis casts doubts on the scientific character of the field itself. Addressing it, we propose a framework for systematically comparing consciousness theories by introducing a novel inter-theory classification interface, the Measure Centrality Index (MCI). Recognizing its gradient distribution, the MCI assesses the degree of importance a specific empirical measure has for a given consciousness theory. We apply the MCI to probe how the empirical measures of the Global Neuronal Workspace Theory (GNW), Integrated Information Theory (IIT), and Temporospatial Theory of Consciousness (TTC) would fare within the context of the other two. We demonstrate that direct comparison of IIT, GNW, and TTC is meaningful and valid for some measures like Lempel-Ziv Complexity (LZC), Autocorrelation Window (ACW), and possibly Mutual Information (MI). In contrast, it is problematic for others like the anatomical and physiological neural correlates of consciousness (NCC) due to their MCI-based differential weightings within the structure of the theories. In sum, we introduce and provide proof-of-principle of a novel systematic method for direct inter-theory empirical comparisons, thereby addressing isolated evolution of theories and confirmatory bias issues in the state-of-the-art neuroscience of consciousness.

Consciousness transitions during epilepsy seizures through the lens of integrated information theory.

Consciousness is one of the most complex aspects of human experience. Studying the mechanisms involved in the transitions among different levels of consciousness remains as one of the greatest challenges in neuroscience. In this study we use a measure of integrated information (ΦAR) to evaluate dynamic changes during consciousness transitions. We applied the measure to intracranial electroencephalography (SEEG) recordings collected from 6 patients that suffer from refractory epilepsy, taking into account inter-ictal, pre-ictal and ictal periods. We analyzed the dynamical evolution of ΦAR in groups of electrode contacts outside the epileptogenic region and compared it with the Consciousness Seizure Scale (CCS). We show that changes on ΦAR are significantly correlated with changes in the reported states of consciousness.

Life and fate of Ervin Bauer (1890-1938), the eminent scholar and foundational theoretical biologist.

Ervin Bauer (1890-1938) was the first to build a general molecular-based biological theory. He defined the basic principles of theoretical biology from a thermodynamic perspective, focusing on the capacity of biological systems to produce and support the state of sustainable non-equilibrium. His central work "Theoretical Biology" (1935) was written long before modern advances in molecular biology, genetics, and information theory. Ervin Bauer and his wife Stefánia were executed in Stalin's Great Terror. This paper presents a brief introduction to Ervin Bauer's life and includes his short biography.

Multi-view graph pooling with coarsened graph disentanglement.

Multi-view graph pooling utilizes information from multiple perspectives to generate a coarsened graph, exhibiting superior performance in graph-level tasks. However, existing methods mainly focus on the types of multi-view information to improve graph pooling operations, lacking explicit control over the pooling process and theoretical analysis of the relationships between views. In this paper, we rethink the current paradigm of multi-view graph pooling from an information theory perspective, subsequently introducing GDMGP, an innovative method for multi-view graph pooling derived from the principles of graph disentanglement. This approach effectively simplifies the original graph into a more structured, disentangled coarsened graph, enhancing the clarity and utility of the graph representation. Our approach begins with the design of a novel view mapper that dynamically integrates the node and topology information of the original graph. This integration enhances its information sufficiency. Next, we introduce a view fusion mechanism based on conditional entropy to accurately regulate the task-relevant information in the views, aiming to minimize information loss in the pooling process. Finally, to further enhance the expressiveness of the coarsened graph, we disentangle the fused view into task-relevant and task-irrelevant subgraphs through mutual information minimization, retaining the task-relevant subgraph for downstream tasks. We theoretically demonstrate that the performance of the coarsened graph generated by our GDMGP is superior to that of any single input view. The effectiveness of GDMGP is further validated by experimental results on seven public datasets.

Quantifying the efficacy of voltage protocols in characterising ion channel kinetics: A novel information-theoretic approach.

Voltage-clamp experiments are commonly utilised to characterise cellular ion channel kinetics. In these experiments, cells are stimulated using a known time-varying voltage, referred to as the voltage protocol, and the resulting cellular response, typically in the form of current, is measured. Parameters of models that describe ion channel kinetics are then estimated by solving an inverse problem which aims to minimise the discrepancy between the predicted response of the model and the actual measured cell response. In this paper, a novel framework to evaluate the information content of voltage-clamp protocols in relation to ion channel model parameters is presented. Additional quantitative information metrics that allow for comparisons among various voltage protocols are proposed. These metrics offer a foundation for future optimal design frameworks to devise novel, information-rich protocols. The efficacy of the proposed framework is evidenced through the analysis of seven voltage protocols from the literature. By comparing known numerical results for inverse problems using these protocols with the information-theoretic metrics, the proposed approach is validated. The essential steps of the framework are: (i) generate random samples of the parameters from chosen prior distributions; (ii) run the model to generate model output (current) for all samples; (iii) construct reduced-dimensional representations of the time-varying current output using proper orthogonal decomposition (POD); (iv) estimate information-theoretic metrics such as mutual information, entropy equivalent variance, and conditional mutual information using non-parametric methods; (v) interpret the metrics; for example, a higher mutual information between a parameter and the current output suggests the protocol yields greater information about that parameter, resulting in improved identifiability; and (vi) integrate the information-theoretic metrics into a single quantitative criterion, encapsulating the protocol's efficacy in estimating model parameters.

Robust and consistent measures of pattern separation based on information theory and demonstrated in the dentate gyrus.

Pattern separation is a valuable computational function performed by neuronal circuits, such as the dentate gyrus, where dissimilarity between inputs is increased, reducing noise and increasing the storage capacity of downstream networks. Pattern separation is studied from both in vivo experimental and computational perspectives and, a number of different measures (such as orthogonalisation, decorrelation, or spike train distance) have been applied to quantify the process of pattern separation. However, these are known to give conclusions that can differ qualitatively depending on the choice of measure and the parameters used to calculate it. We here demonstrate that arbitrarily increasing sparsity, a noticeable feature of dentate granule cell firing and one that is believed to be key to pattern separation, typically leads to improved classical measures for pattern separation even, inappropriately, up to the point where almost all information about the inputs is lost. Standard measures therefore both cannot differentiate between pattern separation and pattern destruction, and give results that may depend on arbitrary parameter choices. We propose that techniques from information theory, in particular mutual information, transfer entropy, and redundancy, should be applied to penalise the potential for lost information (often due to increased sparsity) that is neglected by existing measures. We compare five commonly-used measures of pattern separation with three novel techniques based on information theory, showing that the latter can be applied in a principled way and provide a robust and reliable measure for comparing the pattern separation performance of different neurons and networks. We demonstrate our new measures on detailed compartmental models of individual dentate granule cells and a dentate microcircuit, and show how structural changes associated with epilepsy affect pattern separation performance. We also demonstrate how our measures of pattern separation can predict pattern completion accuracy. Overall, our measures solve a widely acknowledged problem in assessing the pattern separation of neural circuits such as the dentate gyrus, as well as the cerebellum and mushroom body. Finally we provide a publicly available toolbox allowing for easy analysis of pattern separation in spike train ensembles.

The Missing Person problem through the lens of information theory.

Missing person cases typically require a genetic kinship test to determine the relationship between an unidentified individual and the relatives of the missing person. When not enough genetic evidence has been collected the lack of statistical power of these tests might lead to unreliable results. This is particularly true when just a few distant relatives are available for genotyping. In this contribution, we considered a Bayesian network approach for kinship testing and proposed several information theoretic metrics in order to quantitatively evaluate the information content of pedigrees. We show how these statistics are related to the widely used likelihood ratio values and could be employed to efficiently prioritize family members in order to optimize the statistical power in missing person problems. Our methodology seamlessly integrates with Bayesian modeling approaches, like the GENis platform that we have recently developed for high-throughput missing person identification tasks. Furthermore, our approach can also be easily incorporated into Elston-Stewart forensic frameworks. To facilitate the application of our methodology, we have developed the forensIT package, freely available on CRAN repository, which implements all the methodologies described in our manuscript.

Modifying the false discovery rate procedure based on the information theory under arbitrary correlation structure and its performance in high-dimensional genomic data.

BACKGROUND: Controlling the False Discovery Rate (FDR) in Multiple Comparison Procedures (MCPs) has widespread applications in many scientific fields. Previous studies show that the correlation structure between test statistics increases the variance and bias of FDR. The objective of this study is to modify the effect of correlation in MCPs based on the information theory. We proposed three modified procedures (M1, M2, and M3) under strong, moderate, and mild assumptions based on the conditional Fisher Information of the consecutive sorted test statistics for controlling the false discovery rate under arbitrary correlation structure. The performance of the proposed procedures was compared with the Benjamini-Hochberg (BH) and Benjamini-Yekutieli (BY) procedures in simulation study and real high-dimensional data of colorectal cancer gene expressions. In the simulation study, we generated 1000 differential multivariate Gaussian features with different levels of the correlation structure and screened the significance features by the FDR controlling procedures, with strong control on the Family Wise Error Rates. RESULTS: When there was no correlation between 1000 simulated features, the performance of the BH procedure was similar to the three proposed procedures. In low to medium correlation structures the BY procedure is too conservative. The BH procedure is too liberal, and the mean number of screened features was constant at the different levels of the correlation between features. The mean number of screened features by proposed procedures was between BY and BH procedures and reduced when the correlations increased. Where the features are highly correlated the number of screened features by proposed procedures reached the Bonferroni (BF) procedure, as expected. In real data analysis the BY, BH, M1, M2, and M3 procedures were done to screen gene expressions of colorectal cancer. To fit a predictive model based on the screened features the Efficient Bayesian Logistic Regression (EBLR) model was used. The fitted EBLR models based on the screened features by M1 and M2 procedures have minimum entropies and are more efficient than BY and BH procedures. CONCLUSION: The modified proposed procedures based on information theory, are much more flexible than BH and BY procedures for the amount of correlation between test statistics. The modified procedures avoided screening the non-informative features and so the number of screened features reduced with the increase in the level of correlation.

Towards generalizable Graph Contrastive Learning: An information theory perspective.

Graph Contrastive Learning (GCL) is increasingly employed in graph representation learning with the primary aim of learning node/graph representations from a predefined pretext task that can generalize to various downstream tasks. Meanwhile, the transition from a specific pretext task to diverse and unpredictable downstream tasks poses a significant challenge for GCL's generalization ability. Most existing GCL approaches maximize mutual information between two views derived from the original graph, either randomly or heuristically. However, the generalization ability of GCL and its theoretical principles are still less studied. In this paper, we introduce a novel metric GCL-GE, to quantify the generalization gap between predefined pretext and agnostic downstream tasks. Given the inherent intractability of GCL-GE, we leverage concepts from information theory to derive a mutual information upper bound that is independent of the downstream tasks, thus enabling the metric's optimization despite the variability in downstream tasks. Based on the theoretical insight, we propose InfoAdv, a GCL framework to directly enhance generalization by jointly optimizing GCL-GE and InfoMax. Extensive experiments validate the capability of InfoAdv to enhance performance across a wide variety of downstream tasks, demonstrating its effectiveness in improving the generalizability of GCL.

Entropy removal of medical diagnostics.

Shannon entropy is a core concept in machine learning and information theory, particularly in decision tree modeling. To date, no studies have extensively and quantitatively applied Shannon entropy in a systematic way to quantify the entropy of clinical situations using diagnostic variables (true and false positives and negatives, respectively). Decision tree representations of medical decision-making tools can be generated using diagnostic variables found in literature and entropy removal can be calculated for these tools. This concept of clinical entropy removal has significant potential for further use to bring forth healthcare innovation, such as quantifying the impact of clinical guidelines and value of care and applications to Emergency Medicine scenarios where diagnostic accuracy in a limited time window is paramount. This analysis was done for 623 diagnostic tools and provided unique insights into their utility. For studies that provided detailed data on medical decision-making algorithms, bootstrapped datasets were generated from source data to perform comprehensive machine learning analysis on these algorithms and their constituent steps, which revealed a novel and thorough evaluation of medical diagnostic algorithms.

The Information Theory of Aging.

Information storage and retrieval is essential for all life. In biology, information is primarily stored in two distinct ways: the genome, comprising nucleic acids, acts as a foundational blueprint and the epigenome, consisting of chemical modifications to DNA and histone proteins, regulates gene expression patterns and endows cells with specific identities and functions. Unlike the stable, digital nature of genetic information, epigenetic information is stored in a digital-analog format, susceptible to alterations induced by diverse environmental signals and cellular damage. The Information Theory of Aging (ITOA) states that the aging process is driven by the progressive loss of youthful epigenetic information, the retrieval of which via epigenetic reprogramming can improve the function of damaged and aged tissues by catalyzing age reversal.

On distance-based topological indices and co-indices of fractal-type molecular graphs and their respective graph entropies.

In graph theory, a topological index is a numerical value that is in good correlation with certain physical properties of a molecule. It serves as an indicator of how a chemical structure behaves. The Shannon's entropy describes a comparable loss of data in information transmission networks. It has found use in the field of information theory. Inspired by the concept of Shannon's entropy, we have calculated some topological descriptors for fractal and Cayley-type dendrimer trees. We also find the entropy that is predicted by these indices.

Phi fluctuates with surprisal: An empirical pre-study for the synthesis of the free energy principle and integrated information theory.

The Free Energy Principle (FEP) and Integrated Information Theory (IIT) are two ambitious theoretical approaches. The first aims to make a formal framework for describing self-organizing and life-like systems in general, and the second attempts a mathematical theory of conscious experience based on the intrinsic properties of a system. They are each concerned with complementary aspects of the properties of systems, one with life and behavior, the other with meaning and experience, so combining them has potential for scientific value. In this paper, we take a first step towards such a synthesis by expanding on the results of an earlier published evolutionary simulation study, which show a relationship between IIT-measures and fitness in differing complexities of tasks. We relate a basic information theoretic measure from the FEP, surprisal, to this result, finding that the surprisal of simulated agents' observations is inversely related to the general increase in fitness and integration over evolutionary time. Moreover, surprisal fluctuates together with IIT-based consciousness measures in within-trial time. This suggests that the consciousness measures used in IIT indirectly depend on the relation between the agent and the external world, and that it should therefore be possible to relate them to the theoretical concepts used in the FEP. Lastly, we suggest a future approach for investigating this relationship empirically.

Information theory-based direct causality measure to assess cardiac fibrillation dynamics.

Understanding the mechanism sustaining cardiac fibrillation can facilitate the personalization of treatment. Granger causality analysis can be used to determine the existence of a hierarchical fibrillation mechanism that is more amenable to ablation treatment in cardiac time-series data. Conventional Granger causality based on linear predictability may fail if the assumption is not met or given sparsely sampled, high-dimensional data. More recently developed information theory-based causality measures could potentially provide a more accurate estimate of the nonlinear coupling. However, despite their successful application to linear and nonlinear physical systems, their use is not known in the clinical field. Partial mutual information from mixed embedding (PMIME) was implemented to identify the direct coupling of cardiac electrophysiology signals. We show that PMIME requires less data and is more robust to extrinsic confounding factors. The algorithms were then extended for efficient characterization of fibrillation organization and hierarchy using clinical high-dimensional data. We show that PMIME network measures correlate well with the spatio-temporal organization of fibrillation and demonstrated that hierarchical type of fibrillation and drivers could be identified in a subset of ventricular fibrillation patients, such that regions of high hierarchy are associated with high dominant frequency.

Integrated information theory (IIT) 4.0: Formulating the properties of phenomenal existence in physical terms.

This paper presents Integrated Information Theory (IIT) 4.0. IIT aims to account for the properties of experience in physical (operational) terms. It identifies the essential properties of experience (axioms), infers the necessary and sufficient properties that its substrate must satisfy (postulates), and expresses them in mathematical terms. In principle, the postulates can be applied to any system of units in a state to determine whether it is conscious, to what degree, and in what way. IIT offers a parsimonious explanation of empirical evidence, makes testable predictions concerning both the presence and the quality of experience, and permits inferences and extrapolations. IIT 4.0 incorporates several developments of the past ten years, including a more accurate formulation of the axioms as postulates and mathematical expressions, the introduction of a unique measure of intrinsic information that is consistent with the postulates, and an explicit assessment of causal relations. By fully unfolding a system's irreducible cause-effect power, the distinctions and relations specified by a substrate can account for the quality of experience.