Opportunities and Pitfalls in Applying Emotion Recognition Software for Persons With a Visual Impairment: Simulated Real Life Conversations.

BACKGROUND: A large part of the communication cues exchanged between persons is nonverbal. Persons with a visual impairment are often unable to perceive these cues, such as gestures or facial expression of emotions. In a previous study, we have determined that visually impaired persons can increase their ability to recognize facial expressions of emotions from validated pictures and videos by using an emotion recognition system that signals vibrotactile cues associated with one of the six basic emotions. OBJECTIVE: The aim of this study was to determine whether the previously tested emotion recognition system worked equally well in realistic situations and under controlled laboratory conditions. METHODS: The emotion recognition system consists of a camera mounted on spectacles, a tablet running facial emotion recognition software, and a waist belt with vibrotactile stimulators to provide haptic feedback representing Ekman's six universal emotions. A total of 8 visually impaired persons (4 females and 4 males; mean age 46.75 years, age range 28-66 years) participated in two training sessions followed by one experimental session. During the experiment, participants engaged in two 15 minute conversations, in one of which they wore the emotion recognition system. To conclude the study, exit interviews were conducted to assess the experiences of the participants. Due to technical issues with the registration of the emotion recognition software, only 6 participants were included in the video analysis. RESULTS: We found that participants were quickly able to learn, distinguish, and remember vibrotactile signals associated with the six emotions. A total of 4 participants felt that they were able to use the vibrotactile signals in the conversation. Moreover, 5 out of the 6 participants had no difficulties in keeping the camera focused on the conversation partner. The emotion recognition was very accurate in detecting happiness but performed unsatisfactorily in recognizing the other five universal emotions. CONCLUSIONS: The system requires some essential improvements in performance and wearability before it is ready to support visually impaired persons in their daily life interactions. Nevertheless, the participants saw potential in the system as an assistive technology, assuming their user requirements can be met.

Conveying facial expressions to blind and visually impaired persons through a wearable vibrotactile device.

In face-to-face social interactions, blind and visually impaired persons (VIPs) lack access to nonverbal cues like facial expressions, body posture, and gestures, which may lead to impaired interpersonal communication. In this study, a wearable sensory substitution device (SSD) consisting of a head mounted camera and a haptic belt was evaluated to determine whether vibrotactile cues around the waist could be used to convey facial expressions to users and whether such a device is desired by VIPs for use in daily living situations. Ten VIPs (mean age: 38.8, SD: 14.4) and 10 sighted persons (SPs) (mean age: 44.5, SD: 19.6) participated in the study, in which validated sets of pictures, silent videos, and videos with audio of facial expressions were presented to the participant. A control measurement was first performed to determine how accurately participants could identify facial expressions while relying on their functional senses. After a short training, participants were asked to determine facial expressions while wearing the emotion feedback system. VIPs using the device showed significant improvements in their ability to determine which facial expressions were shown. A significant increase in accuracy of 44.4% was found across all types of stimuli when comparing the scores of the control (mean±SEM: 35.0±2.5%) and supported (mean±SEM: 79.4±2.1%) phases. The greatest improvements achieved with the support of the SSD were found for silent stimuli (68.3% for pictures and 50.8% for silent videos). SPs also showed consistent, though not statistically significant, improvements while supported. Overall, our study shows that vibrotactile cues are well suited to convey facial expressions to VIPs in real-time. Participants became skilled with the device after a short training session. Further testing and development of the SSD is required to improve its accuracy and aesthetics for potential daily use.

Learning Enhances Sensory and Multiple Non-sensory Representations in Primary Visual Cortex.

We determined how learning modifies neural representations in primary visual cortex (V1) during acquisition of a visually guided behavioral task. We imaged the activity of the same layer 2/3 neuronal populations as mice learned to discriminate two visual patterns while running through a virtual corridor, where one pattern was rewarded. Improvements in behavioral performance were closely associated with increasingly distinguishable population-level representations of task-relevant stimuli, as a result of stabilization of existing and recruitment of new neurons selective for these stimuli. These effects correlated with the appearance of multiple task-dependent signals during learning: those that increased neuronal selectivity across the population when expert animals engaged in the task, and those reflecting anticipation or behavioral choices specifically in neuronal subsets preferring the rewarded stimulus. Therefore, learning engages diverse mechanisms that modify sensory and non-sensory representations in V1 to adjust its processing to task requirements and the behavioral relevance of visual stimuli.

Adaptive behavior of neighboring neurons during adaptation-induced plasticity of orientation tuning in VI.

BACKGROUND: Sensory neurons display transient changes of their response properties following prolonged exposure to an appropriate stimulus (adaptation). In adult cat primary visual cortex, orientation-selective neurons shift their preferred orientation after being adapted to a non-preferred orientation. The direction of those shifts, towards (attractive) or away (repulsive) from the adapter depends mostly on adaptation duration. How the adaptive behavior of a neuron is related to that of its neighbors remains unclear. RESULTS: Here we show that in most cases (75%), cells shift their preferred orientation in the same direction as their neighbors. We also found that cells shifting preferred orientation differently from their neighbors (25%) display three interesting properties: (i) larger variance of absolute shift amplitude, (ii) wider tuning bandwidth and (iii) larger range of preferred orientations among the cluster of cells. Several response properties of V1 neurons depend on their location within the cortical orientation map. Our results suggest that recording sites with both attractive and repulsive shifts following adaptation may be located in close proximity to iso-orientation domain boundaries or pinwheel centers. Indeed, those regions have a more diverse orientation distribution of local inputs that could account for the three properties above. On the other hand, sites with all cells shifting their preferred orientation in the same direction could be located within iso-orientation domains. CONCLUSIONS: Our results suggest that the direction and amplitude of orientation preference shifts in V1 depend on location within the orientation map. This anisotropy of adaptation-induced plasticity, comparable to that of the visual cortex itself, could have important implications for our understanding of visual adaptation at the psychophysical level.

Visual cells remember earlier applied target: plasticity of orientation selectivity.

BACKGROUND: A canonical proposition states that, in mature brain, neurons responsive to sensory stimuli are tuned to specific properties installed shortly after birth. It is amply demonstrated that that neurons in adult visual cortex of cats are orientation-selective that is they respond with the highest firing rates to preferred oriented stimuli. METHODOLOGY/PRINCIPAL FINDINGS: In anesthetized cats, prepared in a conventional fashion for single cell recordings, the present investigation shows that presenting a stimulus uninterruptedly at a non-preferred orientation for twelve minutes induces changes in orientation preference. Across all conditions orientation tuning curves were investigated using a trial by trial method. Contrary to what has been previously reported with shorter adaptation duration, twelve minutes of adaptation induces mostly attractive shifts, i.e. toward the adapter. After a recovery period allowing neurons to restore their original orientation tuning curves, we carried out a second adaptation which produced three major results: (1) more frequent attractive shifts, (2) an increase of their magnitude, and (3) an additional enhancement of responses at the new or acquired preferred orientation. Additionally, we also show that the direction of shifts depends on the duration of the adaptation: shorter adaptation in most cases produces repulsive shifts, whereas adaptation exceeding nine minutes results in attractive shifts, in the same unit. Consequently, shifts in preferred orientation depend on the duration of adaptation. CONCLUSION/SIGNIFICANCE: The supplementary response improvements indicate that neurons in area 17 keep a memory trace of the previous stimulus properties, thereby upgrading cellular performance. It also highlights the dynamic nature of basic neuronal properties in adult cortex since repeated adaptations modified both the orientation tuning selectivity and the response strength to the preferred orientation. These enhanced neuronal responses suggest that the range of neuronal plasticity available to the visual system is broader than anticipated.

Synchrony between orientation-selective neurons is modulated during adaptation-induced plasticity in cat visual cortex.

BACKGROUND: Visual neurons respond essentially to luminance variations occurring within their receptive fields. In primary visual cortex, each neuron is a filter for stimulus features such as orientation, motion direction and velocity, with the appropriate combination of features eliciting maximal firing rate. Temporal correlation of spike trains was proposed as a potential code for linking the neuronal responses evoked by various features of a same object. In the present study, synchrony strength was measured between cells following an adaptation protocol (prolonged exposure to a non-preferred stimulus) which induce plasticity of neurons' orientation preference. RESULTS: Multi-unit activity from area 17 of anesthetized adult cats was recorded. Single cells were sorted out and (1) orientation tuning curves were measured before and following 12 min adaptation and 60 min after adaptation (2) pairwise synchrony was measured by an index that was normalized in relation to the cells' firing rate. We first observed that the prolonged presentation of a non-preferred stimulus produces attractive (58%) and repulsive (42%) shifts of cell's tuning curves. It follows that the adaptation-induced plasticity leads to changes in preferred orientation difference, i.e. increase or decrease in tuning properties between neurons. We report here that, after adaptation, the neuron pairs that shared closer tuning properties display a significant increase of synchronization. Recovery from adaptation was accompanied by a return to the initial synchrony level. CONCLUSION: We conclude that synchrony reflects the similarity in neurons' response properties, and varies accordingly when these properties change.