Emotion recognition has become a very controversial issue in brain-computer interfaces (BCIs). Moreover, numerous studies have been conducted in order to recognize emotions. Also, there are several important definitions and theories about human emotions. In this paper we try to cover important topics related to the field of emotion recognition. We review several studies which are based on analyzing electroencephalogram (EEG) signals as a biological marker in emotion changes. Considering low cost, good time and spatial resolution, EEG has become very common and is widely used in most BCI applications and studies. First, we state some theories and basic definitions related to emotions. Then some important steps of an emotion recognition system like different kinds of biologic measurements (EEG, electrocardiogram [EEG], respiration rate, etc), offline vs online recognition methods, emotion stimulation types and common emotion models are described. Finally, the recent and most important studies are reviewed.
Thagard P. Mind: Introduction to Cognitive Science. MIT press; 2005.
Oately, K. Best Laid Schemes: The Psychology of Emotions. Cambridge, Cambridge University Press; 1992.
Nussbaum M. Upheavals of thought. Cambridge, Cambridge University Press; 2001.
Ekman P, Friesen W, Osullivan M, Chan A, Diacoyannitarlatzis I, Heider K, et al. Universals and cultural differences in the judgments of facial expressions of emotion. J Pers Soc Psychol. 1987; 41:732–73.
Kessous L, Castellano G, Caridakis G. Multimodal emotion recognition in speech-based interaction using facial expression, body gesture and acoustic analysis. Journal on Multimodal User Interfaces. 2009; 3: 33-48.
Kaynak O, Alpaydin E, Oja E, Xu L, Raouzaiou A, Ioannou S, et al. An Intelligent Scheme for Facial Expression Recognition. in Artificial Neural Networks and Neural Information Processing— ICANN/ICONIP. 2003; 2714: 182-182.
Cheonshu P, Jungwoo R, Joochan S, Hyunkyu C. An Emotion Expression System for the Emotional Robot. In: IEEE International Symposium on Consumer Electronics, 2007. ISCE 2007; 20-23 June 2007; Irving, TX, USA: IEEE. p.1-6.
Rani P, Sarkar N. A New Approach to Implicit Human-Robot Interaction Using Affective Cues. In: Mobile Robots: towards New Applications; 2006.
Apolloni B, Howlett R, Jain L, Sim KB, Jang IH, Park CH. The Development of Interactive Feature Selection and GA Feature Selection Method for Emotion Recognition. In Knowledge-Based Intelligent Information and Engineering Systems, Lecture Notes in Computer Science book series (LNCS, volume 4694); 2007. p.73-81.
Kim K, Bang S, Kim S. Emotion recognition system using short term monitoring of physiological signals. Medical and Biological Engineering and Computing. 2004; 42: 419-427.
Jonghwa K, Ande E. Emotion Recognition Based on Physiological Changes in Music Listening. In: IEEE Transactions on Pattern Analysis and Machine Intelligence; 02 February 2008; p. 2067-2083.
Yuen CT, San WS, Seong TCh, Rizon M. Classification of Human Emotions from EEG Signals using Statistical Features and Neural Network. International Journal of Integrated Engineering 2009; 3.
Li M, Lu BL. Emotion classification based on gamma-band EEG, Conf Proc IEEE Eng Med Biol Soc. 2009;2009:1323-6.
Schaaff K, Schultz T. Towards emotion recognition from electroencephalographic signals, Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Conference on; 10-12 Sept. 2009
Petrantonakis PC, Hadjileontiadis LJ. EEG-Based Emotion Recognition Using Hybrid Filtering and Higher Order Crossings. In: 2009 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops; 10-12 September 2009; Amsterdam. p. 1-6.
Khalili Z, Moradi MH. Emotion Recognition System Using Brain and Peripheral Signals: Using Correlation Dimension to Improve the Results of EEG. In: International Joint Conference on Neural Networks, IJCNN 2009, Atlanta, Georgia, USA; 14-19 June 2009.
Macaš M, Vavrecka M, Gerla V. Classification of the emotional states based on the EEG signal processing. In: International Conference on Information Technology and Applications in Biomedicine, 2009. ITAB 2009. 9th; 4-7 Nov. 2009.
Hosseini SA, Khalilzadeh MA. Emotional stress recognition system using EEG and psychophysiological signals: Using New Labelling Process of EEG Signals in Emotional Stress State. In: Biomedical Engineering and Computer Science (ICBECS) 2010 International Conference on; 23-25 April 2010.
Petrantonakis PC, Hadjileontiadis LJ. Adaptive Extraction of Emotion-Related EEG Segments Using Multidimensional Directed Information in Time-Frequency Domain. In: Conf Proc IEEE Eng Med Biol Soc. 2010; 2010:1-4.
Petrantonakis PC, Hadjileontiadis LJ. Emotion Recognition From EEG Using Higher Order Crossings. In: IEEE Transactions on Information Technology in Biomedicine; 23 October 2009. p. 186 – 197.
Petrantonakis PC, Hadjileontiadis LJ. A Novel Emotion Elicitation Index Using Frontal Brain Asymmetry for Enhanced EEG-Based Emotion Recognition. In: IEEE Transactions on Information Technology in Biomedicine; 27 May 2011. p. 737 – 746.
Hosseini SA, Khalilzadeh MA, Naghibi-Sistani MB. Higher Order Spectra Analysis of EEG Signals in Emotional Stress States. In: Information Technology and Computer Science (ITCS), 2010 Second International Conference on, Kiev, Ukraine; 24-25 July 2010.
Petrantonakis PC. An Emotion Elicitation Metric for the Valence/Arousal and Six Basic Emotions Affective Models: A comparative Study. In: Information Technology and Applications in Biomedicine (ITAB), 2010 10th IEEE International Conference on; Corfu, Greece; 3-5 Nov 2010.
Hidalgo-Mu˜noz AR, López MM, Pereira AT, Santos IM, Tomé AM. Spectral turbulence measuring as feature extraction method from EEG on affective computing. Biomedical Signal Processing and Control. 2013; 8: 945– 950.
Goodman RN, Rietschel JC, Lo LiCh, Costanzo ME, Hatfield BD. Stress, emotion regulation and cognitive performance: The predictive contributions of trait and state relative frontal EEG alpha asymmetry. International Journal of Psychophysiology. 2013; 87: 115–123.
Weinreich A, Stephani T, Schubert T. Emotion effects within frontal alpha oscillation in a picture oddball paradigm, International Journal of Psychophysiology. 2016; 110: 200–206.
Othmana M, Wahaba A, Karima I, Dzulkiflib MA, Fakhri I, Alshaikli T. EEG emotion recognition based on the dimensional models of emotions. Procedia - Social and Behavioral Sciences. 2013; 97:30 – 37.
Lahane P, Sangaiah AK. An Approach to EEG Based Emotion Recognition and Classification using Kernel Density Estimation. Procedia Computer Science. 2015; 48: 574 – 581.
Bozhkov L, Georgieva P, Santos I, Pereira A, Silva C. EEG-based subject independent aﬀective computing models. Procedia Computer Science. 2015; 53: 375–382.
Balconi M, Grippa E, Vanutelli ME. What hemodynamic (fNIRS), electrophysiological (EEG) and autonomic integrated measures can tell us about emotional processing, Brain and Cognition. 2015; 95: 67–76.
Bozhkov L, Koprinkova-Hristova P, Georgieva P. Reservoir computing for emotion valence discrimination from EEG signals. Neurocomputing. 2017; 231: 28–40.
Su J, Duan D, Zhang X, Lei H, Wang Ch, Guo H, et al. The effect of negative emotion on multiple object tracking task: An ERP study. Neuroscience Letters. 2017; 641:15–20
Choi D, Sekiya T, Minote N, Watanuki S. Relative left frontal activity in reappraisal and suppression of negative emotion: Evidence from frontal alpha asymmetry (FAA). International Journal of Psychophysiology. 2016; 109:37-44.
Li Y, Cao D, Wei L, Tang Y, Wang J. Abnormal functional connectivity of EEG gamma band in patients with depression during emotional face processing. Clinical neurophysiology. 2015; 126:2078–2089.
Mattavelli G, Rosanova M, Casali AG, Papagno C, Romero Lauro LJ. Timing of emotion representation in right and left occipital region: Evidence from combined TMS-EEG. Brain and Cognition. 2016; 106: 13–22.
Mavratzakis A, Herbert C, Walla P. Emotional facial expressions evoke faster orienting responses, but weaker emotional responses at neural and behavioural levels compared to scenes: A simultaneous EEG and facial EMG study. NeuroImage. 2016; 124:931–946.
Neath-Tavares KN, Itier, RJ. Neural processing of fearful and happy facial expressions during emotion-relevant and emotion-irrelevant tasks: a fixation-to-feature approach. Biological Psychology. 2016; 119:122-140.
Solomon B, O’Toole L, Hong M, Dennis TA. Negative affectivity and EEG asymmetry interact to predict emotional interference on attention in early school-aged children. Brain and Cognition. 2014; 87: 173–180.
Brenner CA, Rumak SP, Burns AMN, Kieffaber PD. The role of encoding and attention in facial emotion memory: An EEG investigation. International Journal of Psychophysiology. 2014; 93: 398–410.
Tseng YL, Yang HH, Savostyanov AN, Chien VSC, Liou M. Voluntary attention in Asperger’s syndrome: Brain electrical oscillation and phase-synchronization during facial emotion recognition. Research in Autism Spectrum Disorders. 2015; 13–14: 32–51.
Brenner CA, Rumak SP, Burns AMN. Facial Emotion Memory in Schizophrenia: From Encoding to Maintenance-related EEG. Clinical Neurophysiology. 2015; 127:1366-73.
Balconia M, Brambilla E, Falbo L. BIS/BAS, cortical oscillations and coherence in response to emotional cues. Brain Research Bulletin. 2009; 80: 151–157.
Balconi M, Pozzoli Uberto. Arousal effect on emotional face comprehension Frequency band changes in different time intervals. Physiology & Behavior. 2009. 97:455–462.
Zhang Y, Ji X, Zhang S. An approach to EEG-based emotion recognition using combined feature extraction method. Neuroscience Letters. 2016. 633: 152–157.
Murugappan M, Nagarajan R, Sazali Y. Combining Spatial Filtering and Wavelet Transform for Classifying Human Emotions Using EEG Signals. Journal of Medical and Biological Engineering. 2011; 31(1): 45-51.
Nie D, Wang XW, Shi LC. EEG-based Emotion Recognition during Watching Movies. In: Neural Engineering (NER), 2011 5th International IEEE/EMBS Conference on; Cancun, Mexico; 27 April-1 May 2011.
Osaka K, Tanioka T, Clocsin R. Electroencephalograph Estimation Method of Measuring Empathic Understanding. In: Natural Language Processing and Knowledge Engineering, 2007. NLP-KE 2007. International Conference on, 30 Aug.-1 Sept. 2007, Beijing; 2007; p. 514-519.
Murugappan M, Rizon M, Nagarajan R. Lifting scheme for human emotion recognition using EEG. In: Information Technology, 2008. ITSim 2008. International Symposium on; Kuala Lumpur, Malaysia; 26-28 Aug. 2008.
Khosrowabadi R, Heijnen M, Wahab A. The Dynamic Emotion Recognition System Based on Functional Connectivity of Brain Regions. In: Intelligent Vehicles Symposium (IV), 2010 IEEE; San Diego, CA, USA; 21-24 June 2010.
Nasoz F, Alvarez K, Lisetti CL, Finkelstein N. Emotion recognition from physiological signals using wireless sensors for presence technologies. Cogn Tech Work. 2004; 6: 4–14.
Koelstra S, Muhl C, Soleymani M. DEAP: A Database for Emotion Analysis; Using Physiological Signals, In: IEEE Transactions on Affective Computing. 2012; 3: 18–31.
Murugappan M, Rizon M, Nagarajan R, Yaacob S. EEG feature extraction for classifying emotions using FCM and FKM. International journal of Computers and Communications. 2007; 1(2) :21-25.
Koelstra S, Patras I. Fusion of facial expressions and EEG for implicit affective tagging. Image and Vision Computing. 2013; 31: 164–174.
Lee G, Kwon M, KavuriSri S, Lee M. Emotion recognition based on 3D fuzzy visual and EEG features in movie clips. Neurocomputing. 2014;144 :560-568.
Mekler A, Gorbunov I, Gavrilov V. Systemic processes in the brain: The EEG study on the emotions of different hierarchical levels and signs. In: International Journal of Psychophysiology. Nov 2014; 94(2):191-191.
Khezri M, Firoozabadi M, Sharafat AR. Reliable Emotion Recognition System Based on Dynamic Adaptive Fusion of Forehead Biopotentials and Physiological Signals. Computer Methods and Programs in Biomedicine. 2015;122(2):149-64.
Kumar N, Khaund K, Shyamanta M. Hazarika. Bispectral Analysis of EEG for Emotion Recognition. Procedia Computer Science. 2016; 84: 31– 35.
Atkinson J, Campos D. Improving BCI-based emotion recognition by combining EEG feature selection and kernel classiﬁers. Expert Systems with Applications. 2016; 47: 35–41.
Chai X, Wang Q, Zhao Y, Liu X, Bai O, Li Y. Unsupervised domain adaptation techniques based on autoencoder for non-stationary EEG-based emotion recognition. Computers in Biology and Medicine. 2016; 79: 205-214.
Huang X, Kortelainen J, Zhao G, Li X, Moilanen A, Seppänen T, Pietikäinen M. Multi-modal emotion analysis from facial expressions and electroencephalogram. Computer Vision and Image Understanding. 2016; 147: 114–124.
Lin YP, Wang CH, Wu TL, Jeng SK, Chen JH. EEG-Based Emotion Recognition in Music Listening, In: Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE International Conference on;19-24 April 2009; Taipei, Taiwan; 2009.
Khosrowabadi R, Wahab A, Keng K. Affective computation on EEG correlates of emotion from musical and vocal stimuli, Neural Networks, 2009. IJCNN 2009. International Joint Conference on; 14-19 June 2009.
Kuncheva LI, Christy T, Pierce I, Mansoor SP. Multi-modal Biometric Emotion Recognition Using Classifier Ensembles. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems; Springer, Berlin, Heidelberg; 2011. p. 317-326.
Sourina O, Liu Y. A fractal-based algorithm of emotion recognition from EEG using arousal-valence model, biosignals. Proceedings of the International Conference on Bio-inspired Systems and Signal Processing; 26-29 January 2011; Rome, Italy; 2011.
Lin YP, Wang CH, Wu TL, Jeng SK, Chen JH. EEG-based emotion recognition in music listening: a comparison of schemes for multiclass support vector machine. In: Acoustics, Speech and Signal Processing, 2009. ICASSP 2009. IEEE International Conference on; 19-24 April 2009.
Liu Y, Sourina O, Nguyen MK. Real-time EEG-based Human Emotion Recognition and Visualization. Cyberworlds (CW); 2010 International Conference on; 20-22 Oct. 2010.
Lin YP, Wang CH, Wu TY, Jeng SK, Chen JH. Multilayer Perceptron for EEG Signal Classification during Listening to Emotional Music. In: TENCON 2007 - 2007 IEEE Region 10 Conference; 30 Oct-2 Nov. 2007
Lin YP, Wang CH, Wu TY, Jeng SK, Chen JH. Support Vector Machine for EEG Signal Classification during Listening to Emotional Music. In: Multimedia Signal Processing, 2008 IEEE 10th Workshop on, 8-10 Oct; 2008.
Hoseingholizade S, Hashemi Golpaygani MR, Saburruh Monfared A. Studying emotion through nonlinear processing of EEG. Procedia - Social and Behavioral Sciences. 2012; 32: 163– 169.
Daly I, Malik A, Hwang F, Roesch E, Weaver J, Kirke A, et al. Neural correlates of emotional responses to music: An EEG study. Neurosci Lett. 2014; 24; 573:52-7.
Daly I, Williams D, Hallowell J, Hwang F, Kirke A, Malik A, et al. Music-induced emotions can be predicted from a combination of brain activity and acoustic features. Brain and Cognition. 2015; 101: 1–11.
Akar SA, Kara S, Agambayev S, Bilgiç V. Non linear analysis of EEGs of patients with major depression during different emotional states, Computers in Biology and Medicine. 2015; 67:49–60.
Maity AK, Pratihar R, Anubrato M, Dey S, Vishal A, Shankha S, et al. Multifractal Detrended Fluctuation Analysis of alpha and theta EEG rhythms with musical stimuli. Chaos, Solitons and Fractals. 2015; 81: 52–67.
Banerjee A, Sanyala S, Patranabis A, Banerjee K, Guhathakurta T, Senguptaa R, et al. Study on Brain Dynamics by Non Linear Analysis of Music Induced EEG Signals. Physica A. 2016; 444:110–120.
Bhatti AM, Majid M, Anwar SM, Khan B. Human emotion recognition and analysis in response to audio music using brain signals. Computers in Human Behavior. 2016; 65: 267-275.
Shahabi H, Moghimi S. Toward automatic detection of brain responses to emotional music through analysis of EEG effective connectivity. Computers in Human Behavior. 2016; 58: 231-239.
Chanel G, Ansari-Asl K, Pun T. Valence-arousal evaluation using physiological signals in an emotion recall paradigm. In: Systems, Man and Cybernetics, 2007. ISIC. IEEE International Conference on; 7-10 Oct. 2007.
Ahmed MAK, Basori AH. The influence of beta signal toward emotion classification for facial expression control through EEG sensors. Procedia - Social and Behavioral Sciences. 2013 ;97: 730 –736
Shin YB, Woo SH, Kim DH, Kim J, Kim JJ, Park JY. The effect on emotions and brain activity by the direct/indirect lighting in the residential environment. Neuroscience Letters. 2015; 584: 28–32.
Placid G, Avola D, Petracca A, Sgallari F, Spezialetti M. Basis for the implementation of an EEG-based single-trial binary brain computer interface through the disgust produced by remembering unpleasant odors. Neurocomputing. 2015;160: 308–318.
Iacoviello D, Petracca A, Spezialetti M, Placidi G. A Real-time classiﬁcation algorithm for EEG-based BCI driven by self-induced emotions. Computer Methods and Programs in Biomedicine. 2015; 122: 293-303.
Chanel G, Rebetez C, Bétrancourt M, Pun T. Emotion Assessment From Physiological Signals for Adaptation of Game Difﬁculty. In: IEEE transactions on systems, man, and cybernetics—part a: systems and humans; November 2011.
Sourina O, Wang Q, Liu Y, Nguyen M. A real-time fractal based brain state recognition from EEG and its applications. Biosignals. SciTePress. 2011; 82–90.
Yisi L, Sourina O, Minh N. Real-time EEG-based human emotion recognition and visualization. In: Proceedings of international conference on cyber worlds (CW); 2010; p.262–269.
Zhang Q, Lee M. Emotion development system by interacting with human EEG and natural scene understanding. Cognitive Systems Research. 2012; 14: 37–49.
Sourina O, Kulish VV, Sourin A. Novel Tools for Quantification of Brain Responses to Music Stimuli. In: 13th International Conference on Biomedical Engineering. IFMBE Proceedings. 2009.
Yin Z, Zhao M, Wang Y, Yang j, Zhang J. Recognition of emotions using Multimodal physiological Signals and an Ensemble deep learning model. Computer Methods and Programs in Biomedicine. 2017; 140: 93–110.
Yuvaraja R, Murugappana M, Ibrahim NM, Sundaraj K, Omar MI, Mohamad K, et al. Detection of emotions in Parkinson’s disease using higher order spectral features from brain’s electrical activity. Biomedical Signal Processing and Control. 2014; 14: 108–116
Brennan AM, Harris AWF, Williams LM. Neural processing of facial expressions of emotion in ﬁrst onset psychosis. Psychiatry Res. 2014 Nov 30; 219 (3):477-85.
Yeung MK, Han YMY, Sze SL, Chan AS. Altered right frontal cortical connectivity during facial emotion recognition in children with autism spectrum disorders. Research in Autism Spectrum Disorders. 2014; 8: 1567–1577.
Tsolaki AC, Kosmidou VE, Kompatsiaris IY, Papadaniil C, Hadjileontiadis L, Tsolaki M, Age-induced differences in brain neural activation elicited by visual emotional stimuli: a high-density EEG study. Neuroscience. 2017; 340:268-278.
Uusberg A, Uibo H, Kreegipuu K, Allik J. EEG alpha and cortical inhibition in affective attention. International Journal of Psychophysiology. 2013; 89: 26–36.
Uusberg A, Thiruchselvam R, Gross JJ. Using distraction to regulate emotion: Insights from EEG theta dynamics. International Journal of Psychophysiology. 2014; 91: 254–260
Urbanek M, Harvey M, Gowan JM, Agrawal N. Regulation of emotions in psychogenic nonepileptic seizures, Epilepsy & Behavior. 2014; 37: 110–115.
Croft RJ, McKernan F, Gray M, Churchyard A, Georgiou-Karistianis N. Emotion perception and electrophysiological correlates in Huntington’s Disease. Clinical Neurophysiology. 2014; 125:1618–1625.
Soleymani M, Lichtenauer J, Pun T, Pantic M. A Multimodal Database for Affect Recognition and Implicit Tagging. In: IEEE transactions on affective computing; January-March 2012.
Zhang Q, Lee M. Analysis of positive and negative emotions in natural scene using brain activity and GIST. Neurocoputing. 2009; 72: 1302–1306.
Zheng WL, Lu BL. Investigating Critical Frequency Bands and Channels for EEG-based Emotion Recognition with Deep Neural Networks. In: IEEE Transactions on Autonomous Mental Development (IEEE TAMD); 2015; 7(3): 162-175
Bocharov, AV, Gennady G K, Alexander N S. Depression and implicit emotion processing: An EEG study. Neurophysiologie Clinique/Clinical Neurophysiology; 2017.
Bong SZ, Wan K, Murugappan M, Ibrahim NM, Rajamanickam Y, Mohamad K. Implementation of wavelet packet transform and non-linear analysis for emotion classification in stroke patient using brain signals. Biomedical Signal Processing and Control. 2017 Jul 31; 36:102-12.
Weifeng Liu, Lianbo Zhang, Dapeng Tao, Jun Cheng, Reinforcement Online Learning for Emotion Prediction by Using Physiological Signals, Pattern Recognition Letters. 2017.
Yue Wu, Yang Wei, John Tudor, A real-time wearable emotion detection headband based on EEG measurement, Sensors and Actuators: A Physical. 2017; 263: 614-621.