Time for an about face on facial coding

We launched our latest technology last week in Amsterdam at #IIEX.

Sentient Expression® is a technology with multi-layered, deep learning AI. It creates valid and reliable inference data on the nature of emotions being expressed. It is strengthened by combining it with implicit cognitive measures from Sentient Prime® at the individual level, to reveal not only what people are feeling, but why. 

The reaction of the audience was telling. Many were inspired by the potential of big emotion data to provide rapid, deep insight into human behavior. While some expressed important skepticism about the method and technology.

1. “The race for facial recognition in advertising has begun”.

There is a very important distinction to make as facial coding becomes mainstream. Facial action coding is not facial recognition. Software or manual coders of facial and bodily movements are not “recognizing faces”. These methods are quantifying systematic movements of the face, body or vocal bursts within a specific context and offering insight into the emotional experience of expressors. This is important for critics to understand, because it is within “facial recognition” that software has been shown to possess systematic biases in identification – not within “facial action coding”.

2. “How do we account for differences between cultures that are prone to express more or less emotion on their faces?”

This is such an important question and it is one that Insights professionals must address when using any research method. Indeed, we are well versed in the need to calibrate our quantitative survey-based data according to systematic “response scale use bias” across cultures. The same principles apply here. 

All data needs to be interpreted relative to the natural mean and variance on the scale of the measurement method being used. So, just as you would interpret the Likert-scale responses of a population that never uses the end points of your scale, so too would you interpret the mitigated emotionally expressive behavior of a population. Further, norms within specific testing contexts for distinct groups of people should be used to understand the relative magnitude of the expressive behavior of a group.

3. Can you effectively use facial coding earlier in the creative process?

Yes. In fact, facial action coding is highly valuable in early stage animatic/storyboard testing. Animated scenes and stories evoke emotional responses that are expressed through facial and bodily movements and are accurately captured by automated coding systems. This application of facial coding requires knowledgeable researchers to appropriately interpret the data. For example, if your animated story boards have captions the data you’re observing in your facial action coding software is likely not “confusion”, but rather, the furrows on the brows of your participants are likely signs of “concentration” as your participants read the words. Just as is true with all of the other data we gather on participants; the appropriate interpretation of the data requires expertise with the research method.

Recent popular press articles have conflated facial action coding with facial recognition, and erroneously applied conclusions from articles to stoke fear about artificial intelligence. Furthermore, some prominent voices in our industry are amplifying the overreaching conclusion from Feldman-Barrett et. al. (2019) that facial action coding is not valid for any application.

YouTube has over 2 billion users, and over 1 billion hours of content are watched on the channel every day. In homes, offices and coffeeshops around the world, hundreds of millions of people sit down at their computers and make emotional expressions into their cameras every day. This emotionally expressive data is big data, and it represents the next great resource for understanding the drivers of the human emotional experience.

Furthermore, advances in emotional expression theory (Cowen and Keltner, 2019) and the automated measurement of expressive behavior (McCandless, Taylor, Clough and Reid, 2020), have made analysis of this vast renewable resource tractable. So, for an industry that remains viable only if it continues to produce novel insight into the drivers of human behavior, turning our back on these advancements because we misinterpret a conclusion, or because current business models are threatened by automated emotion coding, is highly unadvisable.

In order to remain relevant and rapidly advance human knowledge on the derivers of behavior, the industry must take a deeper look at the scientific literature on quantifying emotionally expressive behavior and think critically about its potential for providing answers to our business questions. Leaders within marketing research and insights would do the industry a great service by getting deeper into the literature to evaluate the validity, reliability, sensitivity and value of quantifying emotionally expressive behavior. 

To learn more, and get deeper into the scientific advancements in quantifying emotionally expressive behavior, read our white paper on the topic:

The Systematic Measurement of Emotionally Expressive Behavior