Others are asking participants to group faces into as many categories as they think are needed to capture the emotions, or getting participants from different cultures to label pictures in their own language. Software firms tend not to allow their algorithms such scope for free association. A typical artificial intelligence AI program for emotion detection is fed millions of images of faces and hundreds of hours of video footage in which each emotion has been labelled, and from which it can discern patterns.
Affectiva says it has trained its software on more than 7 million faces from 87 countries, and that this gives it an accuracy in the 90th percentile. The company declined to comment on the science underlying its algorithm. Researchers on both sides of the debate are sceptical of this kind of software, however, citing concerns over the data used to train algorithms and the fact that the science is still debated.
He has not heard back. Martinez concedes that automated emotion detection might be able to say something about the average emotional response of a group. Affectiva, for example, sells software to marketing agencies and brands to help predict how a customer base might react to a product or marketing campaign. If this software makes a mistake, the stakes are low — an advert might be slightly less effective than hoped.
Last year, Hungary, Latvia and Greece piloted a system for prescreening travellers that aims to detect deception by analysing microexpressions in the face. Settling the emotional-expressions debate will require different kinds of investigation. Then use machines to record and analyse real-world footage. Barrett thinks that more data and analytical techniques could help researchers to learn something new, instead of revisiting tired data sets and experiments.
She throws down a challenge to the tech companies eager to exploit what she and many others increasingly see as shaky science. Chen, C. Natl Acad. USA , E—E PubMed Article Google Scholar. Ekman, P. Science , 86—88 Article Google Scholar. Crawford, K. Google Scholar. Susskind, J. Barrett, L.
Interest 20 , 1—68 Benitez-Quiroz, C. USA , — Chen, Z. Elfenbein, H. Cowen, A. Interest 20 , 69—90 Download references. News 04 NOV Research Highlight 03 NOV News 22 OCT News Feature 10 NOV World View 02 NOV Career Feature 25 OCT Article 03 NOV NYU School of Medicine. Washington University in St. Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily. The Americans, on the other hand, continued to display their negative emotions in front of the experimenter.
Scientists concluded that the reason why the Japanese and American participants acted differently in the presence of other people was because of the display rules they had learned in their cultures. Display rules not only tell us how and when to express emotions but also influence how we see and understand emotions in others.
For example, in some cultures, people are used to seeing faces that show a lot of strong emotions. In others, where display rules say that emotions should not always be strongly expressed, people are used to faces with less expression.
When you see emotional expressions of similar strength often enough in your own culture, this influences the way you read those emotions from facial expressions. So, what if you want to figure out what someone is feeling.
Which part of the face gives you your biggest clue? The answer to this question comes from experiments that used eye trackers. An eye tracker is a special device that monitors eye movements and, therefore, can tell scientists exactly where a person is looking. These experiments showed that depending on where people are from, they focus their attention on different parts of the face when trying to figure out what others are feeling.
East Asian participants, for example, mostly look for clues from the eyes. Western participants find their clues more from the whole face, including the eyes, the eyebrows, the nose, and the mouth. This suggests that people from different cultures express their emotions using different facial signals, and also, different cultures analyze facial gestures differently [ 7 ]. One reason for these cultural differences is that display rules also influence the way we process information from the face and the way we categorize this information into emotions.
What if you saw a face where the eyes and the mouth are showing different emotions for example, the eyes are sad, but the mouth is happy? Would you pay more attention to what the eyes are displaying or what the mouth is displaying?
One experiment showed that people will have different answers depending on their culture [ 8 ]. Japanese people mostly read the emotion signaled by the eyes, while Americans focused more on the mouth region to read the emotion. In fact, you can see a similar pattern when you look at emoticons emojis used in different cultures. Emoticons are symbols that use letters, punctuation marks, or numbers to express emotions.
In Asia, for example, most emoticons use different eye shapes to express different emotions. In western cultures, most emoticons use different mouth shapes to express emotions Figure 2. Reading faces is a superpower that you can get better at the more you learn about it and the more you practice it. There are also different games you could play with your friends to see who amongst you is better at reading emotions.
For example, you could make pictures or cards with faces expressing different emotions happy, angry, sad, surprised, afraid, disgusted. Then someone can pull a card and try to enact the emotion from the card without using any words—just their faces.
Others should guess which emotion the person with the card is trying to portray. As a bonus, you will get to practice your acting skills! In this test, participants rate emotional expressions based only on pictures of the eyes, so without seeing the rest of the face. This is relatively difficult and often feels like guessing, but it turns out that most people are quite skilled in doing this task. A nice online version of this test can be found at this link.
However, as scientists have shown, we may be using the map a bit differently depending on where we are from, despite using the same brain systems. The cultural differences in our beliefs, behaviors, and display rules will influence the way we both show and read emotions. These cultural differences will even affect which facial clues we choose to focus our attention on when reading emotions.
Emotions motivate to approach rewards or avoid punishments and they play a critical role in everyday human social interaction. Emotional facial expression is a core aspect of emotion processing in humans Scherer and Ellgring, ; Keltner and Cordaro, ; Sander et al.
In order to measure emotional facial expressions, researchers typically use either certain observation techniques or record the activity of specific muscles with facial electromyography EMG; Mauss and Robinson, ; Wolf, Recent advances in technology have enabled emotion researchers to obtain AU activity and consecutive emotion measurements automatically through analysis of video and photo recordings Pantic and Rothkrantz, ; Cohn and Sayette, Compared to human observation, automatic facial coding is less time consuming and always blind to the research hypothesis for an overview of analysis systems see Poria et al.
Even in comparison to electrode-based measures, it is less invasive and less susceptible to motion artifacts Schulte-Mecklenbeck et al. Furthermore, video-based measurements do not require preparation or application of electrodes and hence are more flexible for data collection e. For these reasons, automatic facial coding may be the preferable measurement technique to detect emotional facial responses in a broad spectrum of research fields.
Converging evidence shows that automatic facial coding AFC provides sensitive and specific scores for emotional intensities, as well as associated AUs, in highly standardized and prototypical facial expression inventories for static photographs Bijlstra and Dotsch, ; Mavadati et al. Summarizing these results, pleasant facial expressions happy are detected with higher probabilities compared to unpleasant facial expressions anger, sadness, disgust, or anxiety and misattributions of specific emotions e.
Furthermore, AFC of mimicked pleasant and unpleasant facial expressions correlate strongly with EMG measurements within the same participants Beringer et al.
However, these detection patterns are typically even stronger pronounced in untrained human observers Nummenmaa and Calvo, ; Calvo and Nummenmaa, Findings indicate that AFC is a suitable measurement alternative to human observers, in particular if recordings are made under optimal conditions e. Photos and videos of well-trained actors, showing specific emotions in an exaggerated, FACS-coordinated manner are indeed useful for basic testing of the measuring systems.
However, they do not necessarily reflect naturally occurring emotional facial reactions. The use of such validation material can be informative in terms of the upper limit performance for these six basic emotions, but may not be suitable for testing the sensitivity of detecting spontaneously occurring emotional responses.
Although this is a necessary first step, it does not yet prove that measurement sensitivity is sufficient for spontaneously and naturally occurring emotional expressions e. The other study demonstrated good prediction of unpleasant versus pleasant facial responses with an AU-based machine learning procedure Haines et al. Unfortunately, in both studies there was no neutral picture category as a comparative condition.
In providing scores for valence and arousal, the FR follows psychological models of emotion that highlight the importance of a two-dimensional affective space Russell, ; Russell and Barrett, ; Barrett and Bliss-Moreau, ; but there are other models that include additional dimensions, e. Valence ranges from pleasant to unpleasant, whereas the arousal dimension ranges from not arousing to highly arousing emotional states.
In turn, these dimensions usually elicit approach and withdrawal behavior or behavioral tendencies, and activate the corresponding motor preparedness Davidson, ; Bradley et al.
Valence and arousal are thought to portray primarily independent processes, in that arousal does not simply correspond to the intensity of a current pleasant or unpleasant affective state Kuppens et al. Additionally, there is evidence that specific neural structures are involved in processing pleasant and unpleasant arousal levels Gerdes et al.
Facial reactions are known to mirror valence evaluations and occur unintentionally in the presence of emotional stimuli Neumann et al. Valence-type reactions are indicated by facial reactions and changes in autonomic activity, such as variations to sweat glands or heart rate, which are associated with arousal processes Siegel et al.
However, enhanced arousal levels modulate the intensity of facial reactions Fujimura et al. EMG of the corrugator and zygomaticus muscles is frequently used to measure the processing of emotion Cacioppo et al. The corrugator is related linearly with the self-reporting of hedonic valence, manifesting in an increase of activity for unpleasant emotions and a decrease for pleasant emotional states Hess and Blairy, ; Rymarczyk et al. In particular, corrugator activity distinguishes strongly between different pleasant and unpleasant facial expressions Wolf et al.
The zygomaticus on the other hand is selectively activated in pleasant states elicited by emotional images Lang et al.
There are notable differences in the rationale of AFC and EMG-measurements: While EMG, in particular, measurements of the corrugator and the zygomaticus muscles, are expected to correlate with the core affective dimension of valence, AFC is typically trained to recognize intensities of basic emotional facial expressions. Correspondingly, the valence parameter generated by AFC is also grounded in this logic. However, the basic emotion approach can also be projected in the core affect framework Posner et al.
Research regarding indicators of emotional arousal focuses on peripheral physiological measurements. A recent meta-analysis Siegel et al. In general, physiological indicators are more highly modulated by emotional compared to neutral stimuli.
Skin Conductance SC in particular is not a very specific measure for different basic emotions, as increases in SC activity are induced by multiple emotional states Kreibig, However, SC is a highly sensitive measure of emotional arousal compared to respiration or heart rate Mendes, SC also correlates strongly with verbal reports of arousal during the viewing of emotional pictures Lang et al.
Furthermore, SC shows high coherence to continuous self-reports of emotional arousal elicited by dynamic emotional videos Golland et al. Emotional arousal measured by SC increases while viewing high arousing images, both pleasant and unpleasant, compared to low arousing or neutral pictures Bradley et al.
While standardized inventories provide a clear-cut norm for the evaluation of AFC i. Importantly, previous studies have used test material e. Hence, we argue that a critical standard would be to test FR against other well-established psychophysiological indicators of emotion like EMG and SC. In order to use FR to score natural expressions, a test under more naturalistic conditions is needed.
The presented study directly compares the measurement performance of FR indicators of emotional expressions from human participants with measurements from physiological channels in a naturalistic setting. This, however, has not yet been attempted so we set out to close this research gap. In order to induce emotional expressions in our participants, standardized emotion-eliciting pictures were presented in a typical free viewing paradigm.
This will provide essential information on the relative usefulness of AFC in emotion research. Thus, we used the different measures to analyze spontaneous emotional reactions to pleasant, unpleasant and neutral images varying in arousal from the International Affective Picture System IAPS; Lang et al. We hypothesized that both measures differ between responses to pleasant, neutral, and unpleasant stimuli as a function of emotional valence. In addition, we tested the hypothesis that overall facial movement — i.
We hypothesize that both measures show elevated signals for arousing pleasant and unpleasant compared to neutral pictures. The relationships between measurement sensitivity, specificity indicators and self-report ratings were assessed. In general, it has been shown that EMG and SC are both highly sensitive indicators of emotional valence and arousal e.
Hence, it is expected that both electrode-based measures correlate substantially and specifically with the corresponding self-report dimension. Concerning FR measures, a similar association pattern should be observed if video-based measures perform as sensitively and specifically as established psychophysiological emotion measurement procedures.
Accordingly, FR measures of valence and arousal are thought to correlate sensitively and specifically with corresponding self-report of valence and arousal.
A total of 43 volunteers 14 males participated in the experiment. Eight participants were left-handed. Ethnicity was mostly European, with three participants of African descent, one of Asian descent, and two from the Middle East. General exclusion criteria included being under 18 years of age, use of psychoactive medication, acute episode of mental disorders, or severe somatic diseases, as well as those who have a beard or wear glasses. Three participants were excluded prior to the analyses due to computer failures.
Participants with corrected vision were asked to wear contact lenses during the experiment. Furthermore, all participants signed informed consent before the data collection.
The experiment was approved by University Research Ethics Committee. A socio-demographic questionnaire e. Each of the 10 groups of pictures were represented by 6 IAPS scenes. Because neutral scenes typically induce less variable responses, fewer pictures were selected for this category. The rational for scene selection was two-fold: First, pleasant, neutral, and unpleasant scenes should clearly differ in valence.
Second, pleasant and unpleasant scenes should not differ in arousal, but should have higher arousal levels than neutral scenes. Following informed consent and completion of the questionnaires, participants used a medical skin exfoliant on areas of their faces in order to improve EMG measurement signal where electrodes were next attached.
Participants were told to make a neutral facial expression for 10 s at the beginning of the experiment. This time interval served as individual calibration period for FR measurements. The experimental trials were presented in two subsequent blocks see Figure 1 for an illustration. In order to familiarize participants with the specific task, 5 practice trials preceded both blocks. Presentation order was randomized such that a maximum of three pictures from the emotion stimulus category were shown in a row to avoid habituation effects.
After the first block, a short break was incorporated before block two started. Afterward, the participants were asked to evaluate the pictures. The 60 pictures were shown in the exact same order for 3 s and were immediately followed by two visual rating scales Bradley and Lang, Both scales were inverted to improve interpretability. Figure 1. One exemplary trail for each of the two experimental blocks. High-precision software E-Prime; Version 2.
Optimal illumination with diffused frontal light was maintained throughout. EMG electrodes were placed on the zygomaticus major and corrugator supercilii on the left facial hemisphere, following the recommendations of Fridlund and Cacioppo SC electrodes were mounted on the left hand palm. Electrodes were filled with isotonic gel. EMG signals were rectified then integrated with a time constant of 5. EMG measurements were analyzed combined as the difference between the mean activities of zygomaticus and the corrugator EMG Delta.
Positive values of this combined measure indicate activation of the zygomaticus and deactivation of the corrugator muscle and can be interpreted as pleasant valence measure. Conversely, negative values indicate activation of the corrugator and deactivation of the zygomaticus muscles and can be interpreted as an unpleasant valence measure. This rationale improved comparability between EMG measurements and video-based assessment of valence parameters i.
A separate analysis of corrugator and zygomaticus muscle activity is reported in Supplementary Material A. SC activities were measured and preprocessed following the recommendations of Boucsein et al.
Signals were filtered using Butterworth Zero Phase Filters with a low cutoff of 0. The visual pattern classifier is based on deep learning networks to extract visual features from pictures or videos and calculate intensity estimations for each specific emotion. In accordance with neuro-computational models human face processing Dailey et al.
On the most integrated level, FR provides scores for valence and arousal. FR software calculates FR Valence pleasant to unpleasant as the difference between pleasant and unpleasant emotion intensities. FR Arousal inactive to active is an index of overall AU activation 3. FR measurements were calibrated per participant as recommended by the software manual. The East-Asian or elderly face-model was applied where appropriate instead of the general face-model. For better interpretability, both scales were multiplied by Figure 2.
Example of the automatic facial coding analysis of the Facereader software Noldus Information Technology. The net represents the digital face model which establishes distance measures between distinct facial features. Based on this information, activity of specific action units is estimated. Right In a next step, the current profile of action unit activities is integrated to higher order emotion measures in this case the basic emotion happiness, pleasant valence, and relatively high arousal.
The averages of psycho-physiological and video-based measurements as well as self-report ratings were calculated for all pictures of one stimulus category pleasant, neutral, and unpleasant. To account for changes over time, activities were averaged in 1-s intervals for 5 s after stimulus onset. We applied Greenhouse and Geisser correction where appropriate.
In addition to univariate analysis of the different measures, Pearson correlations between self-report ratings of valence and arousal, measures of FR, EMG, and SC were reported. All data was averaged per picture over participants and z -standardized for each physiological and behavioral measurement for their most active time windows EMG: 1—3 s; SC, AFC: 3—5 s so that all correlations would improve in comparability.
Analysis of the emotional self-report scales showed the expected pattern for valence and arousal rating of the stimulus material see Table 1 4. Table 1. Hence, effects of stimulus category were analyzed separately for each time window see Table 2A. Overall, FR Valence detected moderate differences between responses to pleasant and unpleasant or between pleasant and neutral pictures. No differences between reactions to neutral and unpleasant pictures can be reported, which might indicate a lowered sensitivity of FR Valence in the detection of unpleasant facial expression or a negative trend for neutral responses.
Explorative comparison of FR Valence against the baseline i. Table 2. Figure 3. Error bars are standard errors of the mean. Green areas highlight time windows with significant stimulus category effects. Hence, effects of stimulus category were analyzed separately for each time window see also Table 2B. Taken together, EMG signals differentiated between all picture categories and varied rather strongly between pleasant and unpleasant pictures.
Furthermore, EMG signals differed between picture categories immediately after stimulus presentation, whereas FR Valence showed an unexpected long latency of 2 s. In order to detect time-dependent effects of the stimulus categories , time windows are analyzed separately see Table 3A. Thus, unpleasant compared to pleasant emotional scenes elicited stronger overall movement in the face indicated by FR Arousal see also Figure 3C.
Table 3. Hence, effects of stimulus category were analyzed separately for each time window see Table 3B. Surprisingly, all stimulus categories induced more activation measured by FR Arousal, which had the highest activation in response to unpleasant pictures and the lowest activation for pleasant pictures.
In contrast to FR Arousal, SC activity increased when viewing emotional arousing pictures and decreased for neutral pictures. In order to provide further information on measurement performance of FR Valence and EMG Delta, correlations between both measures and self-report ratings of emotional valence were calculated. Ratings and measurements of all participants were averaged per stimulus.
These results show that FR Valence is a sensitive indicator for emotional valence and corresponds highly with EMG activity patterns regarding pleasant stimuli. However, it did not predict reactions toward unpleasant emotional content. Table 4. Figure 4. Values indicate z -standardized mean values per stimulus. Thus, unpleasant ratings were associated with higher FR Arousal activity.
This demonstrates that FR Arousal as an activity parameter is more predictive in terms of valence than arousal ratings, whereas SC activity is a sensitive and specific indicator of emotional arousal.
This is the first systematic evaluation of a state-of-the-art AFC software i. We identified great potential for its use as a research tool, with some noteworthy limitations.
0コメント