In the Pursuit of Effective Affective Computing: The Relationship Between Features and Registration.

For facial expression recognition systems to be applicable in the real world, they need to be able to detect and track a previously unseen person's face and its facial movements accurately in realistic environments. A highly plausible solution involves performing a "dense" form of alignment, where 60-70 fiducial facial points are tracked with high accuracy. The problem is that, in practice, this type of dense alignment had so far been impossible to achieve in a generic sense, mainly due to poor reliability and robustness. Instead, many expression detection methods have opted for a "coarse" form of face alignment, followed by an application of a biologically inspired appearance descriptor such as the histogram of oriented gradients or Gabor magnitudes. Encouragingly, recent advances to a number of dense alignment algorithms have demonstrated both high reliability and accuracy for unseen subjects [e.g., constrained local models (CLMs)]. This begs the question: Aside from countering against illumination variation, what do these appearance descriptors do that standard pixel representations do not? In this paper, we show that, when close to perfect alignment is obtained, there is no real benefit in employing these different appearance-based representations (under consistent illumination conditions). In fact, when misalignment does occur, we show that these appearance descriptors do work well by encoding robustness to alignment error. For this work, we compared two popular methods for dense alignment-subject-dependent active appearance models versus subject-independent CLMs-on the task of action-unit detection. These comparisons were conducted through a battery of experiments across various publicly available data sets (i.e., CK+, Pain, M3, and GEMEP-FERA). We also report our performance in the recent 2011 Facial Expression Recognition and Analysis Challenge for the subject-independent task.

Infant affective responses to mother's still face at 6 months differentially predict externalizing and internalizing behaviors at 18 months.

This study investigated (a) stability and change in infant affective responses to the still-face interaction, (b) whether maternal depression affected infant responses, and (c) whether responses to the still-face interaction predicted toddler problem behaviors. Infants (63 girls and 66 boys) of European American mothers (67 depressed and 62 nondepressed) were observed in the still-face interaction at 2, 4, and 6 months. Affect and gaze were coded on a 1-s time base. There were stable individual differences in gazing away and in rates of negative affect. Developmental change occurred only for gazing away, which increased. At 18 months, infants who failed to smile at 6 months in the still-face interaction showed more externalizing-type behaviors than did other toddlers. Infants who failed to cry at 6 months showed fewer internalizing-type behaviors. Mothers' current depressive symptoms and infants' earlier responses to the still-face interaction made independent, comparable contributions to problem behaviors at 18 months.

Automated tracking of facial features in patients with facial neuromuscular dysfunction.

Facial neuromuscular dysfunction severely impacts adaptive and expressive behavior and emotional health. Appropriate treatment is aided by quantitative and efficient assessment of facial motion impairment. We validated a newly developed method of quantifying facial motion, automated face analysis (AFA), by comparing it with an established manual marking method, the Maximal Static Response Assay (MSRA). In the AFA, motion of facial features is tracked automatically by computer vision without the need for placement of physical markers or restrictions of rigid head motion. Nine patients (seven women and two men) with a mean age of 39.3 years and various facial nerve disorders (five with Bell's palsy, three with trauma, and one with tumor resection) participated. The patients were videotaped while performing voluntary facial action tasks (brow raise, eye closure, and smile). For comparison with MSRA, physical markers were placed on facial landmarks. Image sequences were digitized into 640 x 480 x 24-bit pixel arrays at 30 frames per second (1 pixel congruent with0.3 mm). As defined for the MSRA, the coordinates of the center of each marker were manually recorded in the initial and final digitized frames, which correspond to repose and maximal response. For the AFA, these points were tracked automatically in the image sequence. Pearson correlation coefficients were used to evaluate consistency of measurement between manual (the MSRA) and automated (the AFA) tracking methods, and paired t tests were used to assess the mean difference between methods for feature tracking. Feature measures were highly consistent between methods, Pearson's r = 0.96 or higher, p < 0.001 for each of the action tasks. The mean differences between the methods were small; the mean error between methods was comparable to the error within the manual method (less than 1 pixel). The AFA demonstrated strong concurrent validity with the MSRA for pixel-wise displacement. Tracking was fully automated and provided motion vectors, which may be useful in guiding surgical and rehabilitative approaches to restoring facial function in patients with facial neuromuscular disorders.

Human facial expressions as adaptations: Evolutionary questions in facial expression research.

The importance of the face in social interaction and social intelligence is widely recognized in anthropology. Yet the adaptive functions of human facial expression remain largely unknown. An evolutionary model of human facial expression as behavioral adaptation can be constructed, given the current knowledge of the phenotypic variation, ecological contexts, and fitness consequences of facial behavior. Studies of facial expression are available, but results are not typically framed in an evolutionary perspective. This review identifies the relevant physical phenomena of facial expression and integrates the study of this behavior with the anthropological study of communication and sociality in general. Anthropological issues with relevance to the evolutionary study of facial expression include: facial expressions as coordinated, stereotyped behavioral phenotypes, the unique contexts and functions of different facial expressions, the relationship of facial expression to speech, the value of facial expressions as signals, and the relationship of facial expression to social intelligence in humans and in nonhuman primates. Human smiling is used as an example of adaptation, and testable hypotheses concerning the human smile, as well as other expressions, are proposed.

Automated face analysis by feature point tracking has high concurrent validity with manual FACS coding.

The face is a rich source of information about human behavior. Available methods for coding facial displays, however, are human-observer dependent, labor intensive, and difficult to standardize. To enable rigorous and efficient quantitative measurement of facial displays, we have developed an automated method of facial display analysis. In this report, we compare the results with this automated system with those of manual FACS (Facial Action Coding System, Ekman & Friesen, 1978a) coding. One hundred university students were videotaped while performing a series of facial displays. The image sequences were coded from videotape by certified FACS coders. Fifteen action units and action unit combinations that occurred a minimum of 25 times were selected for automated analysis. Facial features were automatically tracked in digitized image sequences using a hierarchical algorithm for estimating optical flow. The measurements were normalized for variation in position, orientation, and scale. The image sequences were randomly divided into a training set and a cross-validation set, and discriminant function analyses were conducted on the feature point measurements. In the training set, average agreement with manual FACS coding was 92% or higher for action units in the brow, eye, and mouth regions. In the cross-validation set, average agreement was 91%, 88%, and 81% for action units in the brow, eye, and mouth regions, respectively. Automated face analysis by feature point tracking demonstrated high concurrent validity with manual FACS coding.

Gender differences in emotional expressivity and self-regulation during early infancy.

Eighty-one 6-month-old infants and their mothers were videotaped in Tronick's face-to-face still-face paradigm to evaluate gender differences in infant and maternal emotional expressivity and regulation. Male infants had greater difficulty than female infants in maintaining affective regulation during each episode, including the still face. Mother-son dyads had higher synchrony scores than mother-daughter dyads but took longer in repairing interactive errors. In addition, maternal affect, matching, rate of change between matching and mismatching states, and synchrony in the play preceding the still face differentially mediated male and female infants' responses to the still face and reunion play. The developmental implications of these gender differences are discussed.

Specific impairment of smiling increases the severity of depressive symptoms in patients with facial neuromuscular disorders.

Depressive symptoms and related emotional distress are prevalent among patients with facial neuromuscular disorders, and the psychological distress impacts the functional disabilities associated with the facial impairment. A specific impairment in the ability to smile may elevate the risk for depression, with patients experiencing a reduced physiological feedback associated with smiling as well as the social consequences of the inability to communicate positive emotion. We tested the hypothesis that specific impairments in the ability to smile increase the severity of depressive symptoms in patients with facial neuromuscular disorders. Twenty-nine consecutive patients (mean age, 50.2 years; SD, 17.0 years; range, 18-81 years) with a facial neuromuscular disorder, who volunteered and completed all of the assessment measures participated. Facial neuromuscular impairments were assessed using multiple measures of facial motility and dysfunction, and emotional functioning was assessed using self-report measures of depression, anxiety, and positive and negative affect. Severity of global facial impairment was statistically controlled in evaluating the association between specific impairment in smiling and the degree of depressive symptoms. Separate hierarchical linear regression analyses indicated the specific impairment of smiling contributed to the prediction of depression (R(2) =.41, df = 3,25, p =.00) and anxiety (R(2) =.35, df = 3,25, p =.00), controlling first for the contribution of global impairment and facial physical disability. The specific impairment of smiling did not contribute to the prediction of positive emotional experience. Specific impairment of smiling and physical disability, but not global impairment of facial motion, were key predictors of depression in patients with facial neuromuscular disorders. The results emphasize the need to assess and treat depression and anxiety in patients with a facial neuromuscular disorder.

Psychological distress: linking impairment with disability in facial neuromotor disorders.

The relationship between facial neuromotor system impairment, disability, and psychological adjustment is not well understood. This study was designed to explore the relation between impairment and disability and the impact of psychological adjustment on the relation for individuals with disorders of the facial neuromotor system. We studied outpatients (n=48; mean age, 49.0; SD=16.3; range, 18 to 84 years) with a facial neuromotor disorder and acute or chronic facial paralysis. Measures of impairment (Facial Motion Assay, House-Brackmann scale, and Facial Grading System), disability (Facial Disability Index, physical and social well-being subscales), and psychological adjustment (Beck Anxiety Inventory, Beck Depression Inventory) were administered. Bivariate correlations between impairment and disability measures indicated impairment was positively correlated with physical and social disability (r=0.44, p < 0.01; r=0.39, p < 0.05, respectively). Stepwise regression analysis to predict disability indicated physical disability was predicted by impairment and the interaction of impairment and psychological distress (R2=0.425; F=12.57; df=2, 34; p=0.002). Psychological distress, and not impairment and the interaction of impairment and distress, was the single predictor of social disability (R2=0.274; F=13.23; df=1, 35; p=0.001). Psychological distress was a moderator of the relation between impairment and physical disability and a mediator of the relation between impairment and social disability for individuals with facial neuromotor disorders. Assessment and interventions targeted for psychological distress in addition to interventions targeted for impairments appear warranted to effectively reduce the disability associated with facial neuromotor disorders.

The development of early externalizing problems among children from low-income families: a transformational perspective.

The present study examined pathways leading to early externalizing problems from age 1 to 3 1/2 in a design that took advantage of our knowledge of normative progression and normative socialization as well as findings from research on risk. A sample of 130 low-income participants was followed longitudinally from 12 to 42 months using observational measures of developmentally salient parenting and child disruptive behavior to predict early externalizing problems. Results are best accommodated by concepts such as transformation and transaction from developmental psychology. For boys, both child and parent variables predicted later externalizing. For girls and boys, the interaction between child noncompliance and maternal rejection was significant.

Mothers' affective behavior with infant siblings: stability and change.

To evaluate within-family differences in maternal affective behavior toward siblings, face-to-face interactions were observed between 39 mothers and their firstborn and second-born infants at 2 months of age. Mother and infant affect was coded on a 1-s time base with behavioral descriptors. Mothers were more positive with second-born infants, and second-born infants were more positive than were firstborns. The siblings' affective behaviors were unrelated, but maternal positive affect was both moderately stable between siblings and correlated with each infant's affect. Thus, in the context of stable individual differences in maternal positive affect, siblings experienced unique affective interactions with their mothers as early as 2 months.

Reflux symptoms in 100 normal infants: diagnostic validity of the infant gastroesophageal reflux questionnaire.

To identify the prevalence of reflux symptoms in normal infants, to characterize the diagnostic validity of a previously described 138-item Infant Gastroesophageal Reflux Questionnaire (I-GERQ) for separating normal infants from those with gastroesophageal reflux disease (GERD), and to identify potentially provocative caretaking practices, we administered the questionnaire to 100 infants attending a well-baby clinic (normals) and to 35 infants referred to the Gastroenterology Division for evaluation for GERI) and testing positive on esophageal pH probe or biopsy (GERD infants). Differences were analyzed by Chi-square, and odds ratios were defined. The diagnostic validity of a 25-point I-GERQ GERD score based on 11 items on the questionnaire was evaluated by calculating its sensitivity, specificity, and positive and negative predictive values. We found that normal infants had a high prevalence of reflux symptoms, such as daily regurgitation (40%), respiratory symptoms, crying more than an hour a day (17%), arching (10%), or daily hiccups (36%) but that many symptoms were significantly more prevalent in the GERD than in the normal infants (Chi-square P < .05), and odds ratios were above 3 for nearly 20 items. The positive and negative predictive values for the 25-point I-GERQ score were 1.00 and .94-.98, respectively. Environmental smoke exposure did not quite reach significance as a provocative factor for GERD. Although normal infants have a high prevalence of symptoms suggesting GERD, a simple questionnaire-based score is a valid diagnostic test with high positive and negative predictive values.

A combination of vocal fo dynamic and summary features discriminates between three pragmatic categories of infant-directed speech.

To assess the relative contribution of dynamic and summary features of vocal fundamental frequency (f0) to the statistical discrimination of pragmatic categories in infant-directed speech, 49 mothers were instructed to use their voice to get their 4-month-old baby's attention, show approval, and provide comfort. Vocal f0 from 621 tokens was extracted using a Computerized Speech Laboratory and custom software. Dynamic features were measured with convergent methods (visual judgment and quantitative modeling of f0 contour shape). Summary features were f0 mean, standard deviation, and duration. Dynamic and summary features both individually and in combination statistically discriminated between each of the pragmatic categories. Classification rates were 69% and 62% in initial and cross-validation DFAs, respectively.

Behaviors associated with onset of gastroesophageal reflux episodes in infants. Prospective study using split-screen video and pH probe.

To identify behaviors associated with the onset of gastroesophageal reflux episodes in infants both systematically and prospectively, each of 10 patients (aged 2 to 32 weeks) was studied during 2 hours of intraluminal esophageal pH probe monitoring, using a split-screen audiovisual recording technique. Videotape analysis of eight infants who had scoreable reflux events revealed six discrete behaviors closely associated temporally (P < .001 to < .05) with the onset of reflux events: "discomfort" (crying or frowning), "emission" (of liquid or gas, i.e., regurgitation, drooling, or burping), yawning, stridor, stretching, and mouthing. Three behaviors (hiccuping, sneezing, and thumb-sucking) were infrequent but were significantly associated with onset of reflux events in one or two patients each. A tenth behavior, coughing or gagging, was significantly associated with onset of reflux events in two patients, but not in the rest, despite relatively frequent occurrence. Exploration of temporal relations between reflux and each behavior suggested that discomfort, emission, mouthing, and cough-gag may have caused reflux episodes, and that all 10 of the behaviors may have been caused by reflux episodes. These findings and a "quiet period" immediately preceding episodes in six of the infants suggest interesting pathophysiologic mechanisms in infants which require further evaluation.

Reliability and validity of an infant gastroesophageal reflux questionnaire.

To improve history-taking of infants with suspected gastroesophageal reflux, we developed an Infant Gastroesophageal Reflux Questionnaire consisting of 161 items covering demographics, symptoms (regurgitation, weight deficit, respiratory difficulties, fussiness, apnea, and pain or bleeding of esophagitis), and possible causes (feeding volume and frequency, allergy, infection, colic, central nervous system abnormalities, positioning, and smoke exposure). The questionnaire was completed by primary caretakers of 69 infants aged 1 to 58 weeks suspected of having reflux. Median time to complete the questionnaire was 20 minutes. The median internal consistency of 29 pairs of redundant questions was 0.94. Median test-retest consistency of 110 items for nine respondents was 0.88. Median interobserver consistency, evaluated for 129 items in 35 questionnaires also filled out by secondary caretakers, was 0.85. The median accuracy of four externally validated items was 1.00. This questionnaire can aid pediatricians in making decisions regarding diagnoses and treatment in this common but complex disorder.

Prevalence and correlates of postpartum depression in first-time mothers.

The prevalence and correlates of postpartum depression were examined in a large (N = 1,033) sample of married, primiparous, middle-class mothers of full-term, healthy infants; 9.3% met modified Research Diagnostic Criteria for depression. However, 39% of the nondepressed women also reported at least 1 somatic symptom. Depressed women whose symptoms were current obtained elevated scores on the Center for Epidemiological Studies-Depression scale, as did some women who did not meet depression criteria. Depressed women different from nondepressed women on several socioeconomic status indicators and the occurrence of obstetric complications, even in this low-risk sample. These data have implications for the assessment of depression in postpartum women.

Specificity of infants' response to mothers' affective behavior.

Mother-infant face-to-face interaction is central to infant socioemotional development. Little has been known about the mechanisms that mediate the mother's influence. Findings are reviewed from a series of laboratory studies that suggest the major functional components of a mother's behavior are its affective quality and its contingent relationship to her baby's behavior. Quality of mother's affective expression accounted for individual differences in the behavior of thirteen 7-month-old infants living in multiproblem families. Infants' response was specific to the type of affective expression mothers displayed. Flat, withdrawn maternal affective expression was associated with infant distress. Intrusive maternal expression was associated with increased gaze aversion. Lack of contingent responsiveness was common to all but four mothers. Findings suggest that withdrawn or intrusive maternal affective expression, together with lack of contingent responsiveness, may in part be responsible for the risk-status of infants in multiproblem families.

Infant-mother face-to-face interaction: age and gender differences in coordination and the occurrence of miscoordination.

To evaluate the extent to which infants and mothers are able to coordinate their behavior, the interactions of 54 mother-infant pairs--18 each at 3, 6, and 9 months of age--were videotaped. Coordination was evaluated with 2 measures: (1) matching--the extent to which mother and infant engage in the same behavior at the same time; and (2) synchrony--the extent to which mother and infant change their behavior with respect to one another. Mother-infant pairs increase their degree of coordination with infant age, but the proportion of time they are coordinated is small. Mother-son pairs spend more time in coordinated states than mother-daughter pairs. The results suggest that interactions be characterized in terms of their movement from coordinated to miscoordinated states rather than only in terms of their degree of coordination. The gender differences are discussed in terms of their importance for the developmental differences in females and males.

Individual and family factors associated with risk of institutionalization of mentally retarded adults.

A demographic description of the individual and family characteristics of mentally retarded persons referred for placement at a state residential facility in 1982 was presented. Three outcome dispositions (continued home placement, institutional placement, and community residential placement) were considered. Findings suggest that measurements of risk of institutionalization must be expanded beyond personal characteristics of retarded individuals to include stress factors within the family.

Three-month-old infants' reaction to simulated maternal depression.

To investigate the nature of the young infant's social competence, the effect of depressed maternal expression during face-to-face interaction was examined using an experimental analogue of maternal depression. Subjects were 12 female and 12 male infants, ages 96-110 days, and their mothers. 2 counter-balanced experimental treatments consisted of 3 min of normal maternal interaction and 3 min of stimulated depressed interaction. A control treatment consisted of 2 3-min epochs of normal maternal interaction. Interactions were videotaped and infant behavior described on a 5-sec time base that maintained order of occurrence. Infants in the depressed condition structured their behavior differently and were more negative than infants in the normal condition. Infants in the depressed condition produced higher proportions of protest, wary, and brief positive. Infants in the depressed condition cycled among protest, wary, and look away. Infants in the normal condition cycled among monitor, brief positive, and play. In addition, differences in negativity were likely to continue briefly after mothers switched from depressed to normal interaction. The data indicate that infants have a specific, appropriate, negative reaction to simulated depression in their mothers. These results question formulations based on alternate hypotheses and suggest that the infant has communicative intent in its interactions.