It’s all very well having an understanding of what emotion analysis is, and how it can be determined using computer facial detection and analysis. However it is of equal importance to understand how this technology is actually being used in the real-world, outside of the Kairos laboratory. For that matter, what are some of the potential uses for emotion analysis that with further development could become real game changers?

In the first article in this series, The Universally Recognized Facial Expressions of Emotion I analysed the theories that have been posited in relation to whether there actually are universally recognized facial expressions of emotion, and if so, which expressions can be clearly evident from observing a person’s face. The second article, What is Emotion Analysis? took the theory and established what Kairos’ view was, and what the Kairos Emotion API measured.

The technology, as developed by Kairos, its competitors and academic research scientists, has begun to find a place in the real-world, and there are massive opportunities for further advancement. Some of the sectors that are either already using the technology, or where there is clearly great potential, are as follows.

Academic Research

In The Universally Recognized Facial Expressions of Emotion we saw a number of cases where researchers explicitly looked at the validity of there being universal facial expressions of emotion. There have been a number of other studies over the years, that have used facial emotion analysis as a tool to collect evidence to help determine whether a particular idea hypothesised is valid or not.

One such study from researchers at the Centre for the Advancement of Psychological Science and Law in the Department of Psychology at the University of British Columbia focussed on how easily it was for people to hide and misrepresent their emotions from their faces. They wrote a paper on Secrets and Lies: Involuntary leakage in deceptive facial expressions as a function of emotional intensity. The researchers recorded participants’ expressions as they viewed disgusting, sad, frightening and happy images (at both low and high intensity). Perhaps unsurprisingly the results of this analysis was that high intensity emotion was more difficult to conceal than low intensity emotion. The “leakage” of emotional signals occurred most frequently with fearful and happiness expressions.

There are numerous other academic studies using real-time emotion analysis as part of their focus, for instance a Real-time Emotion Identification for Socially Intelligent Robots. A study undertaken to investigate possibilities of facial emotion analysis in gaming is a feasibility study in using facial expressions analysis to evaluate player experiences.

Advertising

Emotion analysis has been used to test the effectiveness of various advertisements. A group of testers are filmed while watching the ads, and their faces analysed for particular emotional expressions. The ad makers are very interested in whether the viewers are exhibiting the correct emotions for what was intended to be the responses to their ad.

Although Simple Media in the UK uses different techniques to Kairos’ Emotion Analysis, they have used their version of emotion analysis, combined with eye tracking, to analyse arousal and attention in a winning XFactor advertisement.

Emotional responses to advertising have also been the focus of a number of studies. In one, researchers asked over 1,000 American, British, French, and German respondents to watch a number of web videos, from a library of 115 commercials in the categories of pet care, chocolate, gum, and instant foods. The respondents were observed on webcam, using facial emotion analysis software to track and code their facial responses as they watched each commercial.

Among the companies who have used emotion analysis as they test consumers’ reactions to their advertising or products are such heavyweights as Honda Motor Co, Proctor and Gamble, Coca Cola and Unilever.

Media Testing

A number of media companies have discovered that using emotion analysis software is helpful to them as they test the potential (or actual) success of new programming. For instance, CBS uses emotion analysis software at its Las Vegas laboratory in Television City, whenever it tests new shows. A number of other television companies undertake similar analysis. As the Discover Channel publicly says “We trust the team at Television City to make respondents feel at home during their time there and make them comfortable in sharing their true reactions to our program ideas and full-length episodes.” That testimonial demonstrates one of the key benefits of analysing the faces of the participants - they can relax and watch the program, leaving the software to give their responses for them, much easier than having to fill in a report.

Retail

One retail-related example of using emotion analysis has been with Simple Usability, a UK behavioural research consultancy. They provide shopper-researchers with portable eye tracking glasses which helps them observe shopper behavior. Amongst other things these create heat maps demonstrating the shoppers’ interest in what they are looking at.

Simple Usability also uses facial detection technology to categorise demographic profiles of shoppers lingering at displays. While the demographic analytics here are more reminiscent of the sort of data that you can receive from IMRSV for Marketers, than from the Emotion Analysis API, Simple Usability has been using their facial detection technology to determine customers’ attention.

Of course, it helps retailers to understand shoppers’ emotional reactions to your displays. Armed with this information retailers can optimize their displays to attract, and then keep the attention of customers. It can also be used to recognize dissatisfied and unhappy customers, and to try and modify the pitch to them to satisfy them better.

This is probably best demonstrated at the moment in Russia, where a large cosmetics chain, Ulybka Radugi, is keeping records of their customers facial expressions when they make a purchase. Once this data is on record, it helps Ulbyka Radugi create custom campaigns and promotions aimed at individual shoppers. The project's goal was to "make the consumer experience more compelling than shopping online." They tie a customer's emotional state to the type of cosmetics they wear. The software combines the data insights with real-time analytics; using information from facial expressions, shoppers can be targeted with SMS text messages offering discounts or ads as soon as they leave the store.

Gaming

The Xbox One has some capability for emotion analysis built-in. While they probably consider this most useful for immersive game development, Microsoft has also been developing non-gaming applications, including the ability to analyse emotions when watching ads on TV.

The Xbox technology is still in its relatively early days, but by scrutinising your face, the software can detect what sort of face you're pulling--happy, neutral, scared etc, as well as whether you're engaged or if you're looking away. As time goes on, imagine the potential of this in games, as the developers try to push the players into an all-immersive environment - your avatar could take on your emotions.

Legal

Law enforcement agencies have developed quite an interest in facial detection and recognition technology. While there are many possible uses for this (particularly in the area of facial recognition) there are possibilities for emotion analysis. While there is still much research to be done in this area, the aim is to enable a computer to analyze nonverbal cues on video and make a judgment about a subject's truthfulness, with sufficient accuracy correlating facial expressions to deception, to meet legal needs.

The television program - Lie to Me, is based upon where this technology is heading. Indeed the show freely admits it is based upon the work of Paul Ekman (the father of research into The Universally Recognized Facial Expressions of Emotion) who amongst all his other roles has served as an advisor to police departments and anti-terrorism groups (including the Transportation Security Administration) and acted as a scientific consultant in the production of the television series. Ekman has stated that the professional and scientific elements on the show are around 90 percent accurate.

How Could You Use Emotion Analysis in Your Development?

The Kairos Emotion API is aimed at developers. It is designed to enable developers to come up with the apps of their dreams … to help them record the emotion visualized on peoples’ faces and then utilize that information in a way that will help the end users of their app. What types of apps could you develop? What consumer problems could you use the Emotions API to solve?

This is still a very new field of study. The potential for expansion is huge. While some are still worried about the “Big Brother” effect, of collecting emotional data from real people’s faces, the reality is that there is no identification of specific people.

I am sure there are many creative and amazing possible uses of emotional data that are simply waiting for someone to think of them. What awesome new and innovative app could you create using this technology?