Effects and prediction of cognitive load on encoding model of brain response to auditory and linguistic stimuli in educational multimedia.

Multimedia is extensively used for educational purposes. However, certain types of multimedia lack proper design, which could impose a cognitive load on the user. Therefore, it is essential to predict cognitive load and understand how it impairs brain functioning. Participants watched a version of educational multimedia that applied Mayer's principles, followed by a version that did not. Meanwhile, their electroencephalography (EEG) was recorded. Subsequently, they participated in a post-test and completed a self-reported cognitive load questionnaire. The audio envelope and word frequency were extracted from the multimedia, and the temporal response functions (TRFs) were obtained using a linear encoding model. We observed that the behavioral data are different between the two groups and the TRFs of the two multimedia versions were different. We saw changes in the amplitude and latencies of both early and late components. In addition, correlations were found between behavioral data and the amplitude and latencies of TRF components. Cognitive load decreased participants' attention to the multimedia, and semantic processing of words also occurred with a delay and smaller amplitude. Hence, encoding models provide insights into the temporal and spatial mapping of the cognitive load activity, which could help us detect and reduce cognitive load in potential environments such as educational multimedia or simulators for different purposes.

A neurocomputational model of decision and confidence in object recognition task.

How does the brain process natural visual stimuli to make a decision? Imagine driving through fog. An object looms ahead. What do you do? This decision requires not only identifying the object but also choosing an action based on your decision confidence. In this circumstance, confidence is making a bridge between seeing and believing. Our study unveils how the brain processes visual information to make such decisions with an assessment of confidence, using a model inspired by the visual cortex. To computationally model the process, this study uses a spiking neural network inspired by the hierarchy of the visual cortex in mammals to investigate the dynamics of feedforward object recognition and decision-making in the brain. The model consists of two modules: a temporal dynamic object representation module and an attractor neural network-based decision-making module. Unlike traditional models, ours captures the evolution of evidence within the visual cortex, mimicking how confidence forms in the brain. This offers a more biologically plausible approach to decision-making when encountering real-world stimuli. We conducted experiments using natural stimuli and measured accuracy, reaction time, and confidence. The model's estimated confidence aligns remarkably well with human-reported confidence. Furthermore, the model can simulate the human change-of-mind phenomenon, reflecting the ongoing evaluation of evidence in the brain. Also, this finding offers decision-making and confidence encoding share the same neural circuit.

A fully spiking coupled model of a deep neural network and a recurrent attractor explains dynamics of decision making in an object recognition task.

Objective.Object recognition and making a choice regarding the recognized object is pivotal for most animals. This process in the brain contains information representation and decision making steps which both take different amount of times for different objects. While dynamics of object recognition and decision making are usually ignored in object recognition models, here we proposed a fully spiking hierarchical model, explaining the process of object recognition from information representation to making decision.Approach.Coupling a deep neural network and a recurrent attractor based decision making model beside using spike time dependent plasticity learning rules in several convolutional and pooling layers, we proposed a model which can resemble brain behaviors during an object recognition task. We also measured human choices and reaction times in a psychophysical object recognition task and used it as a reference to evaluate the model.Main results.The proposed model explains not only the probability of making a correct decision but also the time that it takes to make a decision. Importantly, neural firing rates in both feature representation and decision making levels mimic the observed patterns in animal studies (number of spikes (p-value < 10-173) and the time of the peak response (p-value < 10-31) are significantly modulated with the strength of the stimulus). Moreover, the speed-accuracy trade-off as a well-known characteristic of decision making process in the brain is also observed in the model (changing the decision bound significantly affect the reaction time (p-value < 10-59) and accuracy (p-value < 10-165)).Significance.We proposed a fully spiking deep neural network which can explain dynamics of making decision about an object in both neural and behavioral level. Results showed that there is a strong and significant correlation (r= 0.57) between the reaction time of the model and of human participants in the psychophysical object recognition task.

Interpersonal alignment of neural evidence accumulation to social exchange of confidence.

Private, subjective beliefs about uncertainty have been found to have idiosyncratic computational and neural substrates yet, humans share such beliefs seamlessly and cooperate successfully. Bringing together decision making under uncertainty and interpersonal alignment in communication, in a discovery plus pre-registered replication design, we examined the neuro-computational basis of the relationship between privately held and socially shared uncertainty. Examining confidence-speed-accuracy trade-off in uncertainty-ridden perceptual decisions under social vs isolated context, we found that shared (i.e. reported confidence) and subjective (inferred from pupillometry) uncertainty dynamically followed social information. An attractor neural network model incorporating social information as top-down additive input captured the observed behavior and demonstrated the emergence of social alignment in virtual dyadic simulations. Electroencephalography showed that social exchange of confidence modulated the neural signature of perceptual evidence accumulation in the central parietal cortex. Our findings offer a neural population model for interpersonal alignment of shared beliefs.

Optical coherence tomography confirms non-malignant pigmented lesions in phacomatosis pigmentokeratotica using a support vector machine learning algorithm.

INTRODUCTION: Phacomatosis pigmentokeratotica (PPK), an epidermal nevus syndrome, is characterized by the coexistence of nevus spilus and nevus sebaceus. Within the nevus spilus, an extensive range of atypical nevi of different morphologies may manifest. Pigmented lesions may fulfill the ABCDE criteria for melanoma, which may prompt a physician to perform a full-thickness biopsy. MOTIVATION: Excisions result in pain, mental distress, and physical disfigurement. For patients with a significant number of nevi with morphologic atypia, it may not be physically feasible to biopsy a large number of lesions. Optical coherence tomography (OCT) is a non-invasive imaging modality that may be used to visualize non-melanoma and melanoma skin cancers. MATERIALS AND METHOD: In this study, we used OCT to image pigmented lesions with morphologic atypia in a patient with PPK and assessed their quantitative optical properties compared to OCT cases of melanoma. We implement a support vector machine learning algorithm with Gabor wavelet transformation algorithm during post-image processing to extract optical properties and calculate attenuation coefficients. RESULTS: The algorithm was trained and tested to extract and classify textural data. CONCLUSION: We conclude that implementing this post-imaging machine learning algorithm to OCT images of pigmented lesions in PPK has been able to successfully confirm benign optical properties. Additionally, we identified remarkable differences in attenuation coefficient values and tissue optical characteristics, further defining separating benign features of pigmented lesions in PPK from malignant features.

Deeper neural network models better reflect how humans cope with contrast variation in object recognition.

Visual inputs are far from ideal in everyday situations such as in the fog where the contrasts of input stimuli are low. However, human perception remains relatively robust to contrast variations. To provide insights about the underlying mechanisms of contrast invariance, we addressed two questions. Do contrast effects disappear along the visual hierarchy? Do later stages of the visual hierarchy contribute to contrast invariance? We ran a behavioral experiment where we manipulated the level of stimulus contrast and the involvement of higher-level visual areas through immediate and delayed backward masking of the stimulus. Backward masking led to significant drop in performance in our visual categorization task, supporting the role of higher-level visual areas in contrast invariance. To obtain mechanistic insights, we ran the same categorization task on three state-of the-art computational models of human vision each with a different depth in visual hierarchy. We found contrast effects all along the visual hierarchy, no matter how far into the hierarchy. Moreover, that final layers of deeper hierarchical models, which had been shown to be best models of final stages of the visual system, coped with contrast effects more effectively. These results suggest that, while contrast effects reach the final stages of the hierarchy, those stages play a significant role in compensating for contrast variations in the visual system.

Resolving the neural mechanism of core object recognition in space and time: A computational approach.

The underlying mechanism of object recognition- a fundamental brain ability- has been investigated in various studies. However, balancing between the speed and accuracy of recognition is less explored. Most of the computational models of object recognition are not potentially able to explain the recognition time and, thus, only focus on the recognition accuracy because of two reasons: lack of a temporal representation mechanism for sensory processing and using non-biological classifiers for decision-making processing. Here, we proposed a hierarchical temporal model of object recognition using a spiking deep neural network coupled to a biologically plausible decision-making model for explaining both recognition time and accuracy. We showed that the response dynamics of the proposed model can resemble those of the brain. Firstly, in an object recognition task, the model can mimic human's and monkey's recognition time as well as accuracy. Secondly, the model can replicate different speed-accuracy trade-off regimes as observed in the literature. More importantly, we demonstrated that temporal representation of different abstraction levels (superordinate, midlevel, and subordinate) in the proposed model matched the brain representation dynamics observed in previous studies. We conclude that the accumulation of spikes, generated by a hierarchical feedforward spiking structure, to reach abound can well explain not even the dynamics of making a decision, but also the representations dynamics for different abstraction levels.

Explaining Integration of Evidence Separated by Temporal Gaps with Frontoparietal Circuit Models.

Perceptual decisions rely on accumulating sensory evidence over time. However, the accumulation process is complicated in real life when evidence resulted from separated cues over time. Previous studies demonstrate that participants are able to integrate information from two separated cues to improve their performance invariant to an interval between the cues. However, there is no neural model that can account for accuracy and confidence in decisions when there is a time interval in evidence. We used behavioral and EEG datasets from a visual choice task -Random dot motion- with separated evidence to investigate three candid distributed neural networks. We showed that decisions based on evidence accumulation by separated cues over time are best explained by the interplay of recurrent cortical dynamics of centro-parietal and frontal brain areas while an uncertainty-monitoring module included in the model.

Assessment of instantaneous cognitive load imposed by educational multimedia using electroencephalography signals.

The use of multimedia learning is increasing in modern education. On the other hand, it is crucial to design multimedia contents that impose an optimal amount of cognitive load, which leads to efficient learning. Objective assessment of instantaneous cognitive load plays a critical role in educational design quality evaluation. Electroencephalography (EEG) has been considered a potential candidate for cognitive load assessment among neurophysiological methods. In this study, we experiment to collect EEG signals during a multimedia learning task and then build a model for instantaneous cognitive load measurement. In the experiment, we designed four educational multimedia in two categories to impose different levels of cognitive load by intentionally applying/violating Mayer's multimedia design principles. Thirty university students with homogenous English language proficiency participated in our experiment. We divided them randomly into two groups, and each watched a version of the multimedia followed by a recall test task and filling out a NASA-TLX questionnaire. EEG signals are collected during these tasks. To construct the load assessment model, at first, power spectral density (PSD) based features are extracted from EEG signals. Using the minimum redundancy - maximum relevance (MRMR) feature selection approach, the best features are selected. In this way, the selected features consist of only about 12% of the total number of features. In the next step, we propose a scoring model using a support vector machine (SVM) for instantaneous cognitive load assessment in 3s segments of multimedia. Our experiments indicate that the selected feature set can classify the instantaneous cognitive load with an accuracy of 84.5 ± 2.1%. The findings of this study indicate that EEG signals can be used as an appropriate tool for measuring the cognitive load introduced by educational videos. This can be help instructional designers to develop more effective content.

Excitatory deep brain stimulation quenches beta oscillations arising in a computational model of the subthalamo-pallidal loop.

Parkinson's disease (PD) is associated with abnormal [Formula: see text] band oscillations (13-30 Hz) in the cortico-basal ganglia circuits. Abnormally increased striato-pallidal inhibition and strengthening the synaptic coupling between subthalamic nucleus (STN) and globus pallidus externa (GPe), due to the loss of dopamine, are considered as the potential sources of [Formula: see text] oscillations in the basal ganglia. Deep brain stimulation (DBS) of the basal ganglia subregions is known as a way to reduce the pathological [Formula: see text] oscillations and motor deficits related to PD. Despite the success of the DBS, its underlying mechanism is poorly understood and, there is controversy about the inhibitory or excitatory role of the DBS in the literature. Here, we utilized a computational network model of basal ganglia which consists of STN, GPe, globus pallidus interna, and thalamic neuronal population. This model can reproduce healthy and pathological [Formula: see text] oscillations similar to what has been observed in experimental studies. Using this model, we investigated the effect of DBS to understand whether its effect is excitatory or inhibitory. Our results show that the excitatory DBS is able to quench the pathological synchrony and [Formula: see text] oscillations, while, applying inhibitory DBS failed to quench the PD signs. In light of simulation results, we conclude that the effect of the DBS on its target is excitatory.

Task-dependent neural representations of visual object categories.

What do we perceive in a glance of an object? If we are questioned about it, will our perception be affected? How does the task demand influence visual processing in the brain and, consequently, our behaviour? To address these questions, we conducted an object categorisation experiment with three tasks, one at the superordinate level ('animate/inanimate') and two at the basic levels ('face/body' and 'animal/human face') along with a passive task in which participants were not required to categorise objects. To control bottom-up information and eliminate the effect of sensory-driven dissimilarity, we used a particular set of animal face images as the identical target stimuli across all tasks. We then investigated the impact of top-down task demands on behaviour and brain representations. Behavioural results demonstrated a superordinate advantage in the reaction time, while the accuracy was similar for all categorisation levels. The event-related potentials (ERPs) for all categorisation levels were highly similar except for about 170 ms and after 300 ms from stimulus onset. In these time windows, the animal/human face categorisation, which required fine-scale discrimination, elicited a differential ERP response. Similarly, decoding analysis over all electrodes showed the highest peak value of task decoding around 170 ms, followed by a few significant timepoints, generally after 300 ms. Moreover, brain responses revealed task-related neural modulation during categorisation tasks compared with the passive task. Overall, these findings demonstrate different task-related effects on the behavioural response and brain representations. The early and late components of neural modulation could be linked to perceptual and top-down processing of object categories, respectively.

Dissociable Contribution of Extrastriate Responses to Representational Enhancement of Gaze Targets.

Before saccadic eye movements, our perception of the saccade targets is enhanced. Changes in the visual representation of saccade targets, which presumably underlie this perceptual benefit, emerge even before the eye begins to move. This perisaccadic enhancement has been shown to involve changes in the response magnitude, selectivity, and reliability of visual neurons. In this study, we quantified multiple aspects of perisaccadic changes in the neural response, including gain, feature tuning, contrast response function, reliability, and correlated activity between neurons. We then assessed the contributions of these various perisaccadic modulations to the population's enhanced perisaccadic representation of saccade targets. We found a partial dissociation between the motor information, carried entirely by gain changes, and visual information, which depended on all three types of modulation. These findings expand our understanding of the perisaccadic enhancement of visual representations and further support the existence of multiple sources of motor modulation and visual enhancement within extrastriate visual cortex.

A Recurrent Temporal Model for Semantic Levels Categorization Based on Human Visual System.

Humans can categorize an object in different semantic levels. For example, a dog can be categorized as an animal (superordinate), a terrestrial animal (basic), or a dog (subordinate). Recent studies have shown that the duration of stimulus presentation can affect the mechanism of categorization in the brain. Rapid stimulus presentation will not allow top-down influences to be applied on the visual cortex, whereas in the nonrapid, top-down influences can be established and the final result will be different. In this paper, a spiking recurrent temporal model based on the human visual system for semantic levels of categorization is introduced. We showed that the categorization problem for up-right and inverted images can be solved without taking advantage of feedback, but for the occlusion and deletion problems, top-down feedback is necessary. The proposed computational model has three feedback paths that express the effects of expectation and the perceptual task, and it is described by the type of problem that the model seeks to solve and the level of categorization. Depending on the semantic level of the asked question, the model changes its neuronal structure and connections. Another application of recursive paths is solving the expectation effect problem, that is, compensating the reduce in firing rate by the top-down influences due to the available features in the object. In addition, in this paper, a psychophysical experiment is performed and top-down influences are investigated through this experiment. In this experiment, by top-down influences, the speed and accuracy of the categorization of the subjects increased for all three categorization levels. In both the presence and absence of top-down influences, the remarkable point is the superordinate advantage.

A temporal hierarchical feedforward model explains both the time and the accuracy of object recognition.

Brain can recognize different objects as ones it has previously experienced. The recognition accuracy and its processing time depend on different stimulus properties such as the viewing conditions, the noise levels, etc. Recognition accuracy can be explained well by different models. However, most models paid no attention to the processing time, and the ones which do, are not biologically plausible. By modifying a hierarchical spiking neural network (spiking HMAX), the input stimulus is represented temporally within the spike trains. Then, by coupling the modified spiking HMAX model, with an accumulation-to-bound decision-making model, the generated spikes are accumulated over time. The input category is determined as soon as the firing rates of accumulators reaches a threshold (decision bound). The proposed object recognition model accounts for both recognition time and accuracy. Results show that not only does the model follow human accuracy in a psychophysical task better than the well-known non-temporal models, but also it predicts human response time in each choice. Results provide enough evidence that the temporal representation of features is informative, since it can improve the accuracy of a biologically plausible decision maker over time. In addition, the decision bound is able to adjust the speed-accuracy trade-off in different object recognition tasks.

Inherent Importance of Early Visual Features in Attraction of Human Attention.

Local contrasts attract human attention to different areas of an image. Studies have shown that orientation, color, and intensity are some basic visual features which their contrasts attract our attention. Since these features are in different modalities, their contribution in the attraction of human attention is not easily comparable. In this study, we investigated the importance of these three features in the attraction of human attention in synthetic and natural images. Choosing 100% percent detectable contrast in each modality, we studied the competition between different features. Psychophysics results showed that, although single features can be detected easily in all trials, when features were presented simultaneously in a stimulus, orientation always attracts subject's attention. In addition, computational results showed that orientation feature map is more informative about the pattern of human saccades in natural images. Finally, using optimization algorithms we quantified the impact of each feature map in construction of the final saliency map.

Sequence-dependent sensitivity explains the accuracy of decisions when cues are separated with a gap.

Most decisions require information gathering from a stimulus presented with different gaps. However, the neural mechanism underlying this integration is ambiguous. Recently, it has been claimed that humans can optimally integrate the information of two discrete pulses independent of the temporal gap between them. Interestingly, subjects' performance on such a task, with two discrete pulses, is superior to what a perfect accumulator can predict. Although numerous neuronal and descriptive models have been proposed to explain the mechanism of perceptual decision-making, none can explain human behavior on this two-pulse task. In order to investigate the mechanism of decision-making on the noted tasks, a set of modified drift-diffusion models based on different hypotheses were used. Model comparisons clarified that, in a sequence of information arriving at different times, the accumulated information of earlier evidence affects the process of information accumulation of later evidence. It was shown that the rate of information extraction depends on whether the pulse is the first or the second one. Moreover, our findings suggest that a drift diffusion model with a dynamic drift rate can also explain the stronger effect of the second pulse on decisions as shown by Kiani et al. (Journal of Neuroscience, 33 (42), 16483-16489, 2013).

Beyond core object recognition: Recurrent processes account for object recognition under occlusion.

Core object recognition, the ability to rapidly recognize objects despite variations in their appearance, is largely solved through the feedforward processing of visual information. Deep neural networks are shown to achieve human-level performance in these tasks, and explain the primate brain representation. On the other hand, object recognition under more challenging conditions (i.e. beyond the core recognition problem) is less characterized. One such example is object recognition under occlusion. It is unclear to what extent feedforward and recurrent processes contribute in object recognition under occlusion. Furthermore, we do not know whether the conventional deep neural networks, such as AlexNet, which were shown to be successful in solving core object recognition, can perform similarly well in problems that go beyond the core recognition. Here, we characterize neural dynamics of object recognition under occlusion, using magnetoencephalography (MEG), while participants were presented with images of objects with various levels of occlusion. We provide evidence from multivariate analysis of MEG data, behavioral data, and computational modelling, demonstrating an essential role for recurrent processes in object recognition under occlusion. Furthermore, the computational model with local recurrent connections, used here, suggests a mechanistic explanation of how the human brain might be solving this problem.

Confidence Representation of Perceptual Decision by EEG and Eye Data in a Random Dot Motion Task.

The Confidence of a decision could be considered as the internal estimate of decision accuracy. This variable has been studied extensively by different types of recording data such as behavioral, electroencephalography (EEG), eye and electrophysiology data. Although the value of the reported confidence is considered as one of the most important parameters in decision making, the confidence reporting phase might be considered as a restrictive element in investigating the decision process. Thus, decision confidence should be extracted by means of other provided types of information. Here, we proposed eight confidence related properties in EEG and eye data which are significantly descriptive of the defined confidence levels in a random dot motion (RDM) task. As a matter of fact, our proposed EEG and eye data properties are capable of recognizing more than nine distinct levels of confidence. Among our proposed features, the latency of the pupil maximum diameter through the stimulus presentation was established to be the most associated one to the confidence levels. Through the time-dependent analysis of these features, we recognized the time interval of 500-600 ms after the stimulus onset as an important time in correlating features to the confidence levels.

Residual Information of Previous Decision Affects Evidence Accumulation in Current Decision.

Bias in perceptual decisions can be generally defined as an effect which is controlled by factors other than the decision-relevant information (e.g., perceptual information in a perceptual task, when trials are independent). The literature on decision-making suggests two main hypotheses to account for this kind of bias: internal bias signals are derived from (a) the residual of motor signals generated to report a decision in the past, and (b) the residual of sensory information extracted from the stimulus in the past. Beside these hypotheses, this study suggests that making a decision in the past per se may bias the next decision. We demonstrate the validity of this assumption, first, by performing behavioral experiments based on the two-alternative forced-choice (TAFC) discrimination of motion direction paradigms and, then, we modified the pure drift-diffusion model (DDM) based on the accumulation-to-bound mechanism to account for the sequential effect. In both cases, the trace of the previous trial influences the current decision. Results indicate that the probability of being correct in the current decision increases if it is in line with the previously made decision even in the presence of feedback. Moreover, a modified model that keeps the previous decision information in the starting point of evidence accumulation provides a better fit to the behavioral data. Our findings suggest that the accumulated evidence in the decision-making process after crossing the bound in the previous decision can affect the parameters of information accumulation for the current decision in consecutive trials.

Spatiotemporal analysis of category and target-related information processing in the brain during object detection.

To recognize a target object, the brain implements strategies which involve a combination of externally sensory-driven and internally task-driven mechanisms. While several studies have suggested a role for frontal brain areas in enhancing task-related representations in visual cortices, especially in the lateral-occipital cortex, they have remained silent about the type of information transferred to visual areas. However, the recently developed methods of representational connectivity analysis, allowed us to track the movement of different types of information in the brain. Accordingly, we designed an EEG object detection experiment and inspected the spatiotemporal dynamics of category- and target-related information across the brain. Results showed that the prefrontal area initiated the processing of target-related information. This information was then transferred to posterior brain areas during stimulus presentation probably to facilitate object detection and to direct the decision-making procedure. We also observed that, as compared to category-related information, the target-related information could predict the behavioral performance more accurately, suggesting the dominant representation of internal compared to external information in brain signals. These results provided new evidence about the role of prefrontal cortices in the processing of task-related information in the brain during object detection.