them. In our work [9], we tried to overcome this difficulty by trying to represent emotions comprehensively using 2 different methods.

​

Universal Basic Emotions – or Not?

One thing is clear: the universality of emotions is a popular and widely used modus operandi amongst researchers, academics and general public. There are lot of research articles published based on multiple experiments trying to prove the universality of basic emotions and their cross-cultural application. However, there are many (if not more) published articles that refute this claim, those are also based on extensive experimentation and analyses. So, currently the science of emotions is unclear to the universality of human emotions. The first recorded study on emotions was done by Charles Darwin, when in his study called The expression of emotions in man and animals (1872, republished in 1998), he purported the idea of universality of human emotions.

​

The opinions of 2 main experts from psychology and neuroscience, Paul Ekman and Lisa Feldman Barrett, are rooted at the opposite ends regarding the universality of the basic human emotions. While P. Ekman contends that human emotions are basic and universal [10,11] (and that there are 6 of them viz. happiness, sadness, fear, surprise, disgust and anger), L.F. Barrett asserts that the emotions are constructed by the brain depending on a lot of variables [12,13] (including culture, perception and source of stimulation) and that there is no universality to it. Their arguments and analysis are sound and each has its advantages and shortcomings depending on the situation. Both have their arguments based on experiments and models, however, neither is completely true nor completely false. The problem, in my opinion, is that we were trying very hard to root the emotions into being basic, however, emotions have been shown, time and again, to be very subjective in nature. How can then such a concept be defined such that it encompasses all the different cultures, geographies, viewpoints, and still be unequivocal? There is a huge research potential in this direction.

​

Emotion Recognition – a Necessity

Application point of view, there are many important scenarios where automatic emotion recognition is crucial. Human Computer Interaction (HCI) involves a verbal or non-verbal interchange between machine and human. It is easy for us to understand what the machine is telling us (because we designed it), however, it is difficult for the machines to understand our response. Automatic emotion recognition should help the machine's limited capability to understand emotional response from humans. Another similar application is in the driver-assist technology. It is well known that human performance decreases during prolonged sleep deprivation [14]. Due to this, the probability of accidents due to driver's drowsiness is increased. Automatic emotion recognition can help detect the emotions that are responsible for this kind of tiredness early on and help avoid road-accidents. Emotion recognition is very important from health applications as well. Emotion recognition systems can help detect early onset of psychological disorders (like autism, bi-polar and anxiety) and help diagnose their nature. Content monitoring on social networks like Facebook, Twitter, Instagram has become of utmost importance to filter unwanted content (violence, graphical images, etc). Analyzing the affective (evoked emotions) content of the image or the tweet is essential in recognizing the potential harm it can cause. Human behavior analysis is an important research area where human emotions plays a crucial role.

​

Although, computationally, it is very difficult to formalize what happiness is or what do we mean when we say that a person is angry, the current mobile market is rife with many applications that claim to be recognizing our emotions. These applications, many of which claim that they employ AI (Artificial Intelligence) or ML (Machine Learning), predominantly use the facial expressions to detect the apparent emotions from the face. These applications run on algorithms that have been pre-trained with images of faces that depict different emotions. Pre-training helps to build a robust and effective systems, although the bottleneck is the data. Due to the current technological revolution, generating data (specifically labeled data) has become very easy and cheap.

​

Emotion Recognition from Images

So goes the saying: “A picture is worth a thousand words” - irrespective of the correctness of claim made in the idiom, human vision is one of the five sensory mechanisms (vision, smell, taste, hearing, touch) through which we can visualize, understand and navigate through the physical world around us. Vision helps us apprehend the incoming obstacle to move around efficiently, helps us recognize the objects surrounding us (including people) and view the physical world (its complexity and grandeur) in the visible spectrum.

​

A photographic camera is the main source of generating images today. With revolutionary advances in chip manufacturing, design and electronic systems, cameras have become ubiquitous and available at very cheap rates across multitude of devices including mobile phones, tablets, laptops, surveillance systems, satellites, automobiles, etc. This has led to generation of billions of photos which capture almost every aspect of our daily lives. There are lot of images with people showing various emotions, performing diverse activities in different situations, backgrounds and social gatherings. A few of those images are shown in Figure 1. Just by taking a short look at each those A photographic camera is the main source of generating images today. With revolutionary advances in chip manufacturing, design and electronic systems, cameras have become ubiquitous and available at very cheap rates across multitude of devices including mobile phones, tablets, laptops, surveillance systems, satellites, automobiles, etc. This has led to generation of billions of photos which capture almost every aspect of our daily lives. There are lot of images with people showing various emotions, performing diverse activities in different situations, backgrounds and social gatherings. A few of those images are shown in Figure 1. Just by taking a short look at each those images, we process multitude of information:

• What activity is being done - skiing, playing, painting, etc.

• What is the location - mountain, living-room, office, etc.

• Who is present and what is that person doing - a woman holding a wii-like console, couple of hikers in the snow, depressed woman, artist painting, etc.

• What is the mood - engaged in the painting, happy with the dog, depressed in the office, etc.

​

This is just from the top of my head, when each person looks at the images, different objects, relationships between them, different aspects, reactions, emotions, etc are revealed. A single image is a rich source of information; and current machine learning (and/or computer vision) algorithms have made it possible to process that information and quantify it. It is possible to recognize (or perceive) a person’s apparent emotion from an image.

​

For example, look at Figure 3. The person (in this case a boy on a skateboard) is marked in a red bounding-box. We observe that the person is on a skateboard and looking ahead with focus. This shows that he is engaged in the activity. Also, by his body posture one can say that he is performing a ‘trick’ on his skateboard which seems risky, so we can say that he is also excited at that instant (due to the adrenaline rush that he might be getting while doing this ‘trick’). One more thing to note is that, due to the curvature of the road, it is necessary for him to keep anticipating the road ahead, else he might lose balance. Overall, he is having 3 distinct feelings viz. Anticipation, Excitement and Engagement. Another way of understanding the emotional states is to evaluate the intensity of 3