Hybrid visual and memory search for scenes and objects with variable viewpoints.

In hybrid search, observers search visual arrays for any of several target types held in memory. The key finding in hybrid search is that response times (RTs) increase as a linear function of the number of items in a display (visual set size), but RTs increase linearly with the log of the memory set size. Previous experiments have shown this result for specific targets (find exactly this picture of a boot on a blank background) and for broad categorical targets (find any animal). Arguably, these are rather unnatural situations. In the real world, objects are parts of scenes and are seen from multiple viewpoints. The present experiments generalize the hybrid search findings to scenes (Experiment 1) and multiple viewpoints (Experiment 2). The results replicated the basic pattern of hybrid search results: RTs increased logarithmically with the number of scene photos/categories held in memory. Experiment 3 controls the experiment for which viewpoints were seen in an initial learning phase. The results replicate the findings of Experiment 2. Experiment 4 compares hybrid search for specific viewpoints, variable viewpoints, and categorical targets. Search difficulty increases from specific viewpoints to variable viewpoints and then to categorical targets. The results of the four experiments show the generality of logarithmic search through memory in hybrid search.

Sequence Modeling of Passive Sensing Data for Treatment Response Prediction in Major Depressive Disorder.

Major depressive disorder (MDD) is a prevalent mental health condition and has become a pressing societal challenge. Early prediction of treatment response may aid in the rehabilitation engineering of depression, which is of great practical significance for the relief of suffering and burden of MDD. In this paper, we present a sequence modeling approach that uses data collected by passive sensing techniques to predict patients with an outcome of treatment responded defined by the reduction in clinical administrated scales. Hundreds of patients with MDD have been recruited from outpatient clinics at 4 psychiatric sites. Each has been delivered with a self-developed app to passively record their daily phone usage and physical data with minimal human action. An unavoidable dilemma in passive sensing is missing values. To overcome that, the proposed approach combined feature extraction and sequence modeling methods to fully utilize the pattern of missing values from longitudinal data. With no treatment constraints, it enables us to predict the treatment response of MDD 8-10 weeks before the completion of the treatment course, leaving time for preventative measures. Our work explored the feasibility of treatment response prediction using longitudinal passive sensing data and sparse ground truth, and also has the potential for preventing depression by forecasting treatment outcomes weeks in advance.

A conceptual framework of cognitive-affective theory of mind: towards a precision identification of mental disorders.

To explore the minds of others, which is traditionally referred to as Theory of Mind (ToM), is perhaps the most fundamental ability of humans as social beings. Impairments in ToM could lead to difficulties or even deficits in social interaction. The present study focuses on two core components of ToM, the ability to infer others' beliefs and the ability to infer others' emotions, which we refer to as cognitive and affective ToM respectively. Charting both typical and atypical trajectories underlying the cognitive-affective ToM promises to shed light on the precision identification of mental disorders, such as depressive disorders (DD) and autism spectrum disorder (ASD). However, most prior studies failed to capture the underlying processes involved in the cognitive-affective ToM in a fine-grained manner. To address this problem, we propose an innovative conceptual framework, referred to as visual theory of mind (V-ToM), by constructing visual scenes with emotional and cognitive meanings and by depicting explicitly a four-stage process of how humans make inferences about the beliefs and emotions of others. Through recording individuals' eye movements while looking at the visual scenes, our model enables us to accurately measure each stage involved in the computation of cognitive-affective ToM, thereby allowing us to infer about potential difficulties that might occur in each stage. Our model is based on a large sample size (n > 700) and a novel audio-visual paradigm using visual scenes containing cognitive-emotional meanings. Here we report the obtained differential features among healthy controls, DD and ASD individuals that overcome the subjectivity of conventional questionnaire-based assessment, and therefore could serve as valuable references for mental health applications based on AI-aided digital medicine.

An improved multi-input deep convolutional neural network for automatic emotion recognition.

Current decoding algorithms based on a one-dimensional (1D) convolutional neural network (CNN) have shown effectiveness in the automatic recognition of emotional tasks using physiological signals. However, these recognition models usually take a single modal of physiological signal as input, and the inter-correlates between different modalities of physiological signals are completely ignored, which could be an important source of information for emotion recognition. Therefore, a complete end-to-end multi-input deep convolutional neural network (MI-DCNN) structure was designed in this study. The newly designed 1D-CNN structure can take full advantage of multi-modal physiological signals and automatically complete the process from feature extraction to emotion classification simultaneously. To evaluate the effectiveness of the proposed model, we designed an emotion elicitation experiment and collected a total of 52 participants' physiological signals including electrocardiography (ECG), electrodermal activity (EDA), and respiratory activity (RSP) while watching emotion elicitation videos. Subsequently, traditional machine learning methods were applied as baseline comparisons; for arousal, the baseline accuracy and f1-score of our dataset were 62.9 ± 0.9% and 0.628 ± 0.01, respectively; for valence, the baseline accuracy and f1-score of our dataset were 60.3 ± 0.8% and 0.600 ± 0.01, respectively. Differences between the MI-DCNN and single-input DCNN were also compared, and the proposed method was verified on two public datasets (DEAP and DREAMER) as well as our dataset. The computing results in our dataset showed a significant improvement in both tasks compared to traditional machine learning methods (t-test, arousal: p = 9.7E-03 < 0.01, valence: 6.5E-03 < 0.01), which demonstrated the strength of introducing a multi-input convolutional neural network for emotion recognition based on multi-modal physiological signals.

Top-down control of attention by stereoscopic depth.

Stereoscopic depth has a mixed record as a guiding attribute in visual attention. Visual search can be efficient if the target lies at a unique depth; whereas automatic segmentation of search arrays into different depth planes does not appear to be pre-attentive. These prior findings describe bottom-up, stimulus-driven depth guidance. Here, we ask about the top-down selection of depth information. To assess the ability to direct attention to specific depth planes, Experiment 1 used the centroid judgment paradigm which permits quantitative measures of selective processing of items of different depths or colors. Experiment 1 showed that a subset of observers could deploy specific attention filters for each of eight depth planes, suggesting that at least some observers can direct attention to a specific depth plane quite precisely. Experiment 2 used eight depth planes in a visual search experiment. Observers were encouraged to guide their attention to far or near depth planes with an informative but imperfect cue. The benefits of this probabilistic cue were small. However, this may not be a specific problem with guidance by stereoscopic depth. Equivalently poor results were obtained with color. To check and prove that depth guidance in search is possible, Experiment 3 presented items in only two depth planes. In this case, information about the target depth plane allows observers to search more efficiently, replicating earlier work. We conclude that top-down guidance by stereoscopic depth is possible but that it is hard to apply the full range of our stereoscopic ability in search.

Binocularity and visual search-Revisited.

Binocular rivalry is a phenomenon of visual competition in which perception alternates between two monocular images. When two eye's images only differ in luminance, observers may perceive shininess, a form of rivalry called binocular luster. Does dichoptic information guide attention in visual search? Wolfe and Franzel (Perception & Psychophysics, 44(1), 81-93, 1988) reported that rivalry could guide attention only weakly, but that luster (shininess) "popped out," producing very shallow Reaction Time (RT) × Set Size functions. In this study, we have revisited the topic with new and improved stimuli. By using a checkerboard pattern in rivalry experiments, we found that search for rivalry can be more efficient (16 ms/item) than standard, rivalrous grating (30 ms/item). The checkerboard may reduce distracting orientation signals that masked the salience of rivalry between simple orthogonal gratings. Lustrous stimuli did not pop out when potential contrast and luminance artifacts were reduced. However, search efficiency was substantially improved when luster was added to the search target. Both rivalry and luster tasks can produce search asymmetries, as is characteristic of guiding features in search. These results suggest that interocular differences that produce rivalry or luster can guide attention, but these effects are relatively weak and can be hidden by other features like luminance and orientation in visual search tasks.