Revisiting neural information, computing and linking capacity.

Neural information theory represents a fundamental method to model dynamic relations in biological systems. However, the notion of information, its representation, its content and how it is processed are the subject of fierce debates. Since the limiting capacity of neuronal links strongly depends on how neurons are hypothesized to work, their operating modes are revisited by analyzing the differences between the results of the communication models published during the past seven decades and those of the recently developed generalization of the classical information theory. It is pointed out that the operating mode of neurons is in resemblance with an appropriate combination of the formerly hypothesized analog and digital working modes; furthermore that not only the notion of neural information and its processing must be reinterpreted. Given that the transmission channel is passive in Shannon's model, the active role of the transfer channels (the axons) may introduce further transmission limits in addition to the limits concluded from the information theory. The time-aware operating model enables us to explain why (depending on the researcher's point of view) the operation can be considered either purely analog or purely digital.

On the Role of Speed in Technological and Biological Information Transfer for Computations.

In all kinds of implementations of computing, whether technological or biological, some material carrier for the information exists, so in real-world implementations, the propagation speed of information cannot exceed the speed of its carrier. Because of this limitation, one must also consider the transfer time between computing units for any implementation. We need a different mathematical method to consider this limitation: classic mathematics can only describe infinitely fast and small computing system implementations. The difference between mathematical handling methods leads to different descriptions of the computing features of the systems. The proposed handling also explains why biological implementations can have lifelong learning and technological ones cannot. Our conclusion about learning matches published experimental evidence, both in biological and technological computing.

Towards Generalizing the Information Theory for Neural Communication.

Neuroscience extensively uses the information theory to describe neural communication, among others, to calculate the amount of information transferred in neural communication and to attempt the cracking of its coding. There are fierce debates on how information is represented in the brain and during transmission inside the brain. The neural information theory attempts to use the assumptions of electronic communication; despite the experimental evidence that the neural spikes carry information on non-discrete states, they have shallow communication speed, and the spikes' timing precision matters. Furthermore, in biology, the communication channel is active, which enforces an additional power bandwidth limitation to the neural information transfer. The paper revises the notions needed to describe information transfer in technical and biological communication systems. It argues that biology uses Shannon's idea outside of its range of validity and introduces an adequate interpretation of information. In addition, the presented time-aware approach to the information theory reveals pieces of evidence for the role of processes (as opposed to states) in neural operations. The generalized information theory describes both kinds of communication, and the classic theory is the particular case of the generalized theory.

How Amdahl's Law limits the performance of large artificial neural networks : why the functionality of full-scale brain simulation on processor-based simulators is limited.

With both knowing more and more details about how neurons and complex neural networks work and having serious demand for making performable huge artificial networks, more and more efforts are devoted to build both hardware and/or software simulators and supercomputers targeting artificial intelligence applications, demanding an exponentially increasing amount of computing capacity. However, the inherently parallel operation of the neural networks is mostly simulated deploying inherently sequential (or in the best case: sequential-parallel) computing elements. The paper shows that neural network simulators, (both software and hardware ones), akin to all other sequential-parallel computing systems, have computing performance limitation due to deploying clock-driven electronic circuits, the 70-year old computing paradigm and Amdahl's Law about parallelized computing systems. The findings explain the limitations/saturation experienced in former studies.