• Logo
  • SBMUJournals

Integration of Wavelet and Recurrence Quantification Analysis in Emotion Recognition of Bilinguals

Ateke Goshvarpour, Atefeh Goshvarpour, Ataollah Abbasi




Background: This study offers a robust framework for the classification of autonomic signals into five affective states during the picture viewing. To this end, the following emotion categories studied: five classes of the arousal-valence plane (5C), three classes of arousal (3A), and three categories of valence (3V). For the first time, the linguality information also incorporated into the recognition procedure. Precisely, the main objective of this paper was to present a fundamental approach for evaluating and classifying the emotions of monolingual and bilingual college students.
Methods: Utilizing the nonlinear dynamics, the recurrence quantification measures of the wavelet coefficients extracted. To optimize the feature space, different feature selection approaches, including generalized discriminant analysis (GDA), principal component analysis (PCA), kernel PCA, and linear discriminant analysis (LDA), were examined. Finally, considering linguality information, the classification was performed using a probabilistic neural network (PNN).
Results: Using LDA and the PNN, the highest recognition rates of 95.51%, 95.7%, and 95.98% were attained for the 5C, 3A, and 3V, respectively. Considering the linguality information, a further improvement of the classification rates accomplished.
Conclusion: The proposed methodology can provide a valuable tool for discriminating affective states in practical applications within the area of human-computer interfaces.


Autonomic Nervous System; Emotions; Signal Processing; Bilingualism; Nonlinear Dynamics.


Park S, Kim K. Physiological reactivity and facial expression to emotion-inducing films in patients with schizophrenia. Archives of Psychiatric Nursing. 2011;25(6):e37-e47.

Drusch K, Stroth S, Kamp D, Frommann N, Wolwer W. Effects of training of affect recognition on the recognition and visual exploration of emotional faces in schizophrenia. Schizophrenia Research. 2014;159(2-3):485-90.

Swangnetr M. Analysis of patient-robot interaction using statistical and signal processing methods. Raleigh, North Carolina: North Carolina State University; 2010.

Hazlett R, editor Measuring emotional valence during interactive experiences: Boys at video gameplay. CHI '06: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2006; New York, NY, USA: ACM Press.

Yannakakis G, Hallam J. Entertainment modeling through physiology in physical play. International Journal of Human-Computer Studies. 2008;66(10):741-55.

Frantzidis C, Bratsas C, Papadelis C, Konstantinidis E, Pappas C, Bamidis PD. Toward emotion aware computing: An integrated approach using multichannel neurophysiological recordings and affective visual stimuli. IEEE Transactions on Information Technology in Biomedicine. 2010;14(3):589-97.

Valenza G, Citi L, Lanata A, Scilingi E, Barbieri R. Revealing real-time emotional responses: a personalized assessment based on heartbeat dynamics. Scientific Reports. 2014;4:4998.

Goshvarpour A, Abbasi A, Goshvarpour A. Affective visual stimuli: Characterization of the picture sequences impacts using nonlinear approaches. Basic and Clinical Neuroscience. 2015;6(4):209-21.

Nardelli M, Valenza G, Greco A, Lanata A, Scilingo E. Recognizing emotions induced by affective sounds through heart rate variability. IEEE Transactions on Affective Computing. 2015;6:385-94.

Pantic M, Rothkrantz L. Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2000;22(12):1424-45.

Cohn J, Kanade T. Use of automated facial image analysis for measurement of emotion expression. In: Coan J, Allen J, editors. The handbook of emotion elicitation and assessment: Oxford University Press Series in Affective Science; 2006.

Jaimes A, Sebe N. Multimodal human-computer interaction: A survey. Computer Vision and Image Understanding. 2007;108(1-2):116-34.

Breazeal C. Designing social robots. Cambridge, MA: MIT Press; 2001.

Sebe N, Bakker E, Cohen I, Gevers T, Huang T. Bimodal emotion recognition. 5th International conference on methods and techniques in behavioral research; Wageningen, Netherlands 2005.

Schuller B, Arsic D, Wallhoff F, Rigoll G, editors. Emotion recognition in the noise applying large acoustic feature sets. Proceedings of speech prosody International symposium on computer architecture; 2006; Dresden.

Castellano G, Villalba S, Camurri A, editors. Recognising human emotions from body movement and gesture dynamics. Proceedings of the 2nd International Conference on Affective Computing and Intelligent Interaction; 2007; Berlin, Heidelberg: Springer-Verlag.

Caridakis G, Castellano G, Kessous L, Raouzaiou A, Malatesta L, Asteriadis S, et al., editors. Multimodal emotion recognition from expressive faces, body gestures and speech. Artificial Intelligence and Innovations 2007: from Theory to Applications 2007; Berlin, Heidelberg: Springer-Verlag.

Gunes H, Piccardi M. Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications. 2007;30(4):1334-45.

Kreibig S. Autonomic nervous system activity in emotion: a review. Biological Psychology. 2010;84:394-421.

Picard R, Vyzas E, Healey J. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transaction on Pattern Analysis and Machine Intelligence. 2001;23(10):1175-91.

Kim K, Bang S, Kim S. Emotion recognition system using short-term monitoring of physiological signals. Medical and Biological Engineering and Computing. 2004;42(3):419-27.

Haag A, Goronzy S, Schaich P, Williams J. Emotion recognition using bio-sensors: first steps towards an automatic system. Affective Dialogue Systems. 2004;1:36-48.

Yoo S, Lee C, Park Y, Kim N, Lee B, Jeong K, editors. Neural network based emotion estimation using heart rate variability and skin resistance. 1st International Conference in Advances in Natural Computation; 2005: Springer-Verlag Berlin Heidelberg.

Choi A, Woo W, editors. Physiological sensing and feature extraction for emotion recognition by exploiting acupuncture spots. First International Conference in Affective Computing and Intelligent Interaction; 2005: Springer-Verlag Berlin Heidelberg.

Rani P, Liu C, Sarkar N, Vanman E. An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Analysis and Applications. 2006;9(1):58-69.

Rainville P, Bechara A, Naqvi N, Damasio A. Basic emotions are associated with distinct patterns of cardiorespiratory activity. International Journal of Psychophysiology. 2006;61:5-18.

Li L, Chen J, editors. Emotion recognition using physiological signals. 16th International Conference in Advances in Artificial Reality and Tele-Existence; 2006: Springer-Verlag Berlin Heidelberg.

Zhai J, Barreto A, editors. Stress detection in computer users based on digital signal processing of noninvasive physiological variables. 28th Annual International Conference Engineering in Medicine and Biology Society; 2006; New York City, USA: IEEE.

Liu C, Conn K, Sarkar N, Stone W. Physiology-based affect recognition for computer-assisted intervention of children with autism spectrum disorder. International Journal of Human-Computer Studies. 2008;66(9):662-77.

Katsis C, Katertsidis N, Ganiatsas G, Fotiadis D. Toward emotion recognition in car-racing drivers: a biosignal processing approach. IEEE Transaction on Systems, Man, and Cybernetics, Part A: Systems and Humans. 2008;38(3):502-12.

Chang C-Y, Zheng J-Y, Wang C-J. Based on support vector regression for emotion recognition using physiological signals. The 2010 International Joint Conference on Neural Networks (IJCNN); Barcelona: IEEE; 2010.

Katsis C, Katertsidis N, Fotiadis D. An Integrated System Based on Physiological Signals for the Assessment of Affective States in Patients with Anxiety Disorders. Biomedical Signal Processing and Control. 2011;6:261-8.

Niu X, Chen L, Chen Q, editors. Research on genetic algorithm based on emotion recognition using physiological signals. International Conference on Computational Problem-Solving (ICCP); 2011; Chengdu: IEEE.

Greco A, Valenza G, Lanata A, Rota G, Scilingo E. Electrodermal activity in bipolar patients during affective elicitation. IEEE Journal of Biomedical and health Informatics. 2014;18(6):1865-73.

Jang E-H, Park B-J, Park M-S, Kim S-H, Sohn J-H. Analysis of physiological signals for recognition of boredom, pain, and surprise emotions. Journal of Physiological Anthropology. 2015;34(1):25.

Xu Y, Liu G-Y. A method of emotion recognition based on ECG signal. International Conference on Computational Intelligence and Natural Computing: IEEE; 2009. p. 202-5.

Xu Y, Liu G, Hao M, Wen W, Huang X. Analysis of affective ECG signals toward emotion recognition. Journal of Electronics (China). 2010;27(1):8-14.

Valenza G, Lanata A, Scilingo E. The role of nonlinear dynamics in affective valence and arousal recognition. IEEE Transactions on Affective Computing. 2012;3(2):237-49.

Valenza G, Allegrini P, Lanata A, Scilingo E. Dominant Lyapunov exponent and approximate entropy in heart rate variability during emotional visual elicitation. Frontiers in Neuroengineering. 2012;5:1-7.

Jerritta S, Murugappan M, Wan K, Yaacob S. Emotion detection from QRS complex of ECG signals using hurst exponent for different age groups. Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII); Geneva, Switzerland2013. p. 849-54.

Jerritta S, Murugappan M, Wan K, Yaacob S. Classification of emotional states from electrocardiogram signals: a non-linear approach based on hurst. BioMedical Engineering OnLine. 2013;12:44.

Jerritta S, Murugappan M, Wan K, Yaacob S. Electrocardiogram-based emotion recognition system using empirical mode decomposition and discrete Fourier transform. Expert Systems. 2014;31(2):110-20.

Fischer A, Manstead A. Gender and culture differences in emotion. Emotion. 2004;4(1):87-94.

Isaacowitz D, Löckenhoff C, Lane R, Wright R, Sechrest L, Riedel R, et al. Age differences in recognition of emotion in lexical stimuli and facial expressions. Psychology and Aging. 2007;22:147-59.

Rozenkrants B, Polich J. Affective ERP processing in a visual oddball task: Arousal, valence and gender. Clinical Neurophysiology. 2008;119(10):2260-5.

Ruffman T, Henry J, Livingstone V, Phillips L. A meta-analytic review of emotion recognition and aging: Implications for neuropsychological models of aging. Neuroscience and Biobehavioral Reviews. 2008;32:863-81.

Chen L-F, Liu Z-T, Wu M, Ding M, Dong F-Y, Hirota K. Emotion-age-gender-nationality based intention understanding in human-robot interaction using two-layer fuzzy support vector regression. International Journal of Social Robotics. 2015; 7(5):709-29.

Harris C, Aycicegi A, Gleason J. Taboo words and reprimands elicit greater autonomic reactivity in a first language than in a second language. Applied Psycholinguistics. 2003;24:561-79.

Caldwell–Harris C, Aycicegi–Dinn A. Emotion and lying in a non-native language. International Journal of Psychophysiology. 2009;71:193-204.

Lang P, Bradley M, Cuthbert B. International affective picture system (IAPS): digitized photographs, instruction manual and affective ratings. Gainesville, FL: University of Florida; 2005.

Daubechies I. The wavelet transform, time frequency localization and signal analysis. IEEE Transactions on Information Theory. 1990;36:961-1005.

Marwan N, Romano M, Thiel M, kurths J. Recurrence plots for the analysis of complex systems. Physics Reports. 2007;438:237-329.

Fisher R. The use of multiple measurements in taxonomic problems. Annals of Eugenics. 1936;7:179-88.

Hotelling H. Analysis of a complex of statistical variables into principal components. Journal of Educational Psychology. 1933;24:417-41.

Van der Maaten L. An introduction to dimensionality reduction using MATLAB. The Netherlands: Universiteit Maastricht; 2007.

Scholkopf B, Smola A, Muller K. Nonlinear component analysis as a kernel eigenvalue problem. Neural Computation. 1998;10(5):1299-319.

Shawe-Taylor J, Christianini N. Kernel methods for pattern analysis. Cambridge, UK: Cambridge University Press; 2004.

Baudat G, Anouar F. Generalized discriminant analysis using a kernel approach. Neural Computation. 2000;12:2385-404.

Belhachat F, Izeboudjen N. Application of a probabilistic neural network for classification of cardiac arrhythmias. 13th International Research/Expert Conference on Trends in the Development of Machinery and Associated Technology (TMT); Hammamet, Tunisia 2009. p. 397-400.

Goshvarpour A, Abbasi A, Goshvarpour A. Affective autonomic differences between monolinguals and bilinguals: elicited by pictorial stimuli. Biomedical Engineering: Applications, Basis and Communications. 2017;29(2):1750008.

Lazar J, Stern A, Cohen R. Decision-making in foreign language reduces emotional arousal. Psychology 2014;5:2180-8.

Wang L, Liu G, Yang Z. The emotion recognition for grief based on nonlinear features of GSR. Journal of Computational Information Systems. 2014;10(4):1639-49.


  • There are currently no refbacks.