A Critique of Automated Approaches to Code Facial Expressions: What Do Researchers Need to Know?

Facial expression recognition software is becoming more commonly used by affective scientists to measure facial expressions. Although the use of this software has exciting implications, there are persistent and concerning issues regarding the validity and reliability of these programs. In this paper, we highlight three of these issues: biases of the programs against certain skin colors and genders; the common inability of these programs to capture facial expressions made in non-idealized conditions (e.g., "in the wild"); and programs being forced to adopt the underlying assumptions of the specific theory of emotion on which each software is based. We then discuss three directions for the future of affective science in the area of automated facial coding. First, researchers need to be cognizant of exactly how and on which data sets the machine learning algorithms underlying these programs are being trained. In addition, there are several ethical considerations, such as privacy and data storage, surrounding the use of facial expression recognition programs. Finally, researchers should consider collecting additional emotion data, such as body language, and combine these data with facial expression data in order to achieve a more comprehensive picture of complex human emotions. Facial expression recognition programs are an excellent method of collecting facial expression data, but affective scientists should ensure that they recognize the limitations and ethical implications of these programs.

The association of negative mood with automatic and effortful facial expression mimicry.

The natural process of mimicking the facial expressions of others is well established, as are the deficits in this reflexive behavior for individuals with clinical disorders such as depression. This study examines the extent of this deficit in non-clinical individuals with high transient negative mood, and whether it extends to both automatic and effortful emotion expression behavior. One hundred and thirty-six participants were shown happy, sad, and neutral faces, while electromyography (EMG) recorded facial muscle responses. Automatic (reflexive) mimicry was assessed while participants simply viewed facially expressive photographs, while effortful mimicry was monitored when individuals were told to intentionally copy the expressions in the photographs. Results indicated that high levels of negative mood were primarily associated with deficits in effortful mimicry of happy expressions, although some similar evidence was found in automatic mimicry of happy faces. Surprisingly, there were also ties between negative moods and inaccuracies in effortful mimicry of sad expressions (but not automatic mimicry). Inaccurate automatic and effortful mimicry were also tied with lower self-reported social support and greater loneliness. These results indicate that even in healthy individuals, transient and minor changes in negative mood are tied to deficiencies in facial mimicry at both the automatic and effortful level.

The subcomponents of affect scale (SAS): validating a widely used affect scale.

OBJECTIVE: There is a need for a brief affect scale that also encompasses different components of affect relevant for researchers interested in physiological and health outcomes. The Subcomponents of Affect Scale (SAS) meets this need. This 18-item scale has nine positive and nine negative affect items encompassing six subscales (calm, well-being, vigour, depression, anxiety, anger). Previous research using the SAS has demonstrated its predictive validity, but no work has tested its subscale structure or longitudinal validity. DESIGN: Data from the Common Cold Project in which individuals (N = 610) completed the SAS over the course of seven days were used. RESULTS: Confirmatory factor analysis demonstrated the reliability of the subscale structure of the SAS across seven days (positive affect subscale structure: CFIs ≥ 0.98; negative affect subscale structure: CFIs ≥ 0.94 with day 6 CFI = 0.91) and tests of factorial invariance showed the scale is valid to use over time. CONCLUSIONS: These results confirm the psychometric validity of the subscale structure of the SAS and imply that the subscales can be used longitudinally, allowing for its use in health research as well as non-health research that can benefit from its subscale structure and longitudinal capabilities.

How and why could smiling influence physical health? A conceptual review.

ABSTRACTSmiling has been a topic of interest to psychologists for decades, with a myriad of studies tying this behavior to well-being. Despite this, we know surprisingly little about the nature of the connections between smiling and physical health. We review the literature connecting both naturally occurring smiles and experimentally manipulated smiles to physical health and health-relevant outcomes. This work is discussed in the context of existing affect and health-relevant theoretical models that help explain the connection between smiling and physical health including the facial feedback hypothesis, the undoing hypothesis, the generalized unsafety theory of stress, and polyvagal theory. We also describe a number of plausible pathways, some new and relatively untested, through which smiling may influence physical health such as trait or state positive affect, social relationships, stress buffering, and the oculocardiac reflex. Finally, we provide a discussion of possible future directions, including the importance of cultural variation and replication. Although this field is still in its infancy, the findings from both naturally occurring smile studies and experimentally manipulated smile studies consistently suggest that smiling may have a number of health-relevant benefits including beneficially impacting our physiology during acute stress, improved stress recovery, and reduced illness over time.

Say cheese? The connections between positive facial expressions in student identification photographs and health care seeking behavior.

This study examined whether positive facial expressions in student identification photographs were connected with a health-relevant behavior: visits to a health care center in the last year for preventive and non-preventive (e.g. illness, injury) purposes. Identification photographs were coded for degree of smile. Smiling participants were more likely to have sought preventive care versus those not smiling in their photographs, but there was no difference in non-preventive (i.e. ill health) visits. This study shows for the first time that smiling in photographs may be related to healthy behavior and complements past work connecting smiling to positive psychosocial and health outcomes.

Contrasting Experimentally Device-Manipulated and Device-Free Smiles.

Researchers in psychology have long been interested in not only studying smiles, but in examining the downstream effects of experimentally manipulated smiles. To experimentally manipulate smiles unobtrusively, participants typically hold devices (e.g., pens or chopsticks) in their mouths in a manner that activates the muscles involved in smiling. Surprisingly, despite decades of research using these methods, no study has tested to what degree these methods activate the same muscles as more natural, device-free smiles. Our study fills this gap in the literature by contrasting the magnitude of muscle activation in device-free smiles against the popular chopstick/pen manipulation. We also contrast these methods against the Smile Stick, a new device specifically designed to manipulate smiles in a comfortable and hygienic fashion. One hundred fifty-nine participants each participated in three facial expression manipulations that were held for 1 min: smile manipulation via Smile Stick, smile manipulation via chopsticks, and device-free smile. Facial electromyography was used to measure the intensity of the activation of the two main types of muscles involved in genuine, Duchenne smiling: the orbicularis oculi (a muscle group around the eyes) and the zygomaticus major (a muscle group in the cheeks). Furthermore, following each manipulation, participants rated their experience of the manipulation (i.e., comfort, fatigue, and difficulty), experienced affect (positive and negative), and levels of arousal. Results indicated that the Smile Stick and chopsticks performed equally across all measurements. Device-free smiles were rated as most comfortable but also the most fatiguing, and procured the greatest levels of positive affect and lowest levels of negative affect. Furthermore, device-free smiles resulted in significantly higher levels of both zygomaticus major (by ∼40%) and orbicularis oculi (by ∼15%) muscle activation than either the Smile Stick or chopsticks. The two devices were not different from each other in muscle activation. This study reveals that while device-free smiling procures the greatest changes in muscle activation and affect change, smiling muscle groups are activated by device manipulations, and expected changes in affect do occur, albeit to a lesser degree than device-free smiling. It also indicates that the Smile Stick is an acceptable and comparable alternative to disposable chopsticks.

Misremembering pain: A memory blindness approach to adding a better end.

How people remember feeling in the past informs future decisions; however, memory for past emotion is subject to a number of biases. Previous research on choice blindness has shown that people often fail to notice when they are exposed to misinformation about their own decisions, preferences, and memories. This type of misinformation can influence how they later remember past events. In the present study, we examined the memory blindness effect in a new domain: memory for pain. Participants (N = 269) underwent a cold-pressor task and rated how much pain, distress, and positive and negative affect they had experienced. Later, participants were shown their pain ratings and asked to explain them. Some of the participants were shown lower pain ratings than they had actually made. In a second session, participants recalled how painful the task had been and how much distress and positive and negative affect they remembered experiencing. The results indicated that the majority of participants who were exposed to misinformation failed to detect the manipulation, and subsequently remembered the task as being less painful. The participants in the misinformation condition were not overall more willing to repeat the study tasks, but the participants who recalled less distress, less negative affect, and more positive affect were more willing to repeat the study tasks again in a future experiment. These results demonstrate the malleability of memory for painful experiences and that willingness to repeat aversive experiences may depend more on memory for affective reactions to the original experience than on memory for the physical pain itself.

When is affect variability bad for health? The association between affect variability and immune response to the influenza vaccination.

OBJECTIVES: This study addresses methodological and theoretical questions about the association between affect and physical health. Specifically, we examine the role of affect variability and its interaction with mean levels of affect to predict antibody (Ab) levels in response to an influenza vaccination. METHODS: Participants (N=83) received the vaccination and completed daily diary measures of affect four times a day for 13days. At one and four months post-vaccination, blood was collected from the participants to assess Ab levels. RESULTS: Findings indicate that affect variability and its interaction with mean levels of affect predict an individual's immune response. Those high in mean positive affect (PA) who had more PA variability were more likely to have a lower Ab response in comparison to those who had high mean PA and less PA variability. Although it did not interact with mean negative affect (NA), NA variability on its own was associated with Ab response, whereby those with less NA variability mounted a more robust immune response. CONCLUSION: Affect variability is related to immune response to an influenza vaccination and, in some cases, interacts with mean levels of affect. These oscillations in affective experiences are critical to consider in order to unpack the intricacies of how affect influences health. These findings suggest that future researchers should consider the important role of affect variability on physical health-relevant outcomes as well as examine the moderating effect of mean affect levels.