Design of a Computational Model to Detect Hybrid Emotion Through Facial Expressions in Videos Using CNN LSTM
Sahaya Sugirtha Cindrella S
Department of Computer Applications, Faculty of Science and Humanities, SRM Institute of Science and Technology, Kattankulathur, Chengalpattu, Tamil Nadu, India.
Department of Computer Applications, Faculty of Science and Humanities, SRM Institute of Science and Technology, Kattankulathur, Chengalpattu, Tamil Nadu, India.
In many applications of human-computer interaction, emotion prediction is essential. To enhance emotion categorization, we present a hybrid deep learning model in this study that blends convolutional neural networks (CNN) with long short-term memory (LSTM) networks. The pre-processing step refines the input data using Q-based score normalization to ensure ideal feature scale and distribution. Emotional states are robustly classified when CNN is employed to extract spatial data, and LSTM captures temporal relationships. Our model's ability to identify intricate emotion patterns is demonstrated through training and evaluation on a benchmark emotion dataset. According to experimental results, our suggested CNN-LSTM model performs exceptionally well on the test dataset, attaining 100% accuracy, precision, recall, and F1-score. These exceptional results highlight the power of combining CNN and LSTM in handling emotion prediction's spatial and continuous aspects. Q-based score normalization further enhances the model's performance by ensuring a well-distributed feature space, ultimately improving classification consistency. This study underscores the potential of hybrid deep learning architectures in improving emotion recognition applications. Our findings can be applied in diverse domains such as emotional computing, mental analytics, and human-computer interaction.
M. Bakiaraj and B. Subramani, “Optimized hybrid deep learning pipelines for processing heterogeneous facial expression datasets,” Measurement: Sensors, vol. 31, p. 100938, Feb. 2024, doi: 10.1016/j.measen.2023.100938.
M. N. Ab Wahab, A. Nazir, A. T. Zhen Ren, M. H. Mohd Noor, M. F. Akbar, and A. S. A. Mohamed, “Efficientnet-Lite and Hybrid CNN-KNN Implementation for Facial Expression Recognition on Raspberry Pi,” IEEE Access, vol. 9, pp. 134065–134080, 2021, doi: 10.1109/access.2021.3113337.
H. V. Manalu and A. P. Rifai, “Detection of human emotions through facial expressions using hybrid convolutional neural network-recurrent neural network algorithm,” Intelligent Systems with Applications, vol. 21, p. 200339, Mar. 2024, doi: 10.1016/j.iswa.2024.200339.
R. K. Mishra, S. Urolagin, J. A. Arul Jothi, and P. Gaur, “Deep hybrid learning for facial expression binary classifications and predictions,” Image and Vision Computing, vol. 128, p. 104573, Dec. 2022, doi: 10.1016/j.imavis.2022.104573.
G. Morshed, H. Ujir, and I. Hipiny, “Customer’s spontaneous facial expression recognition,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 22, no. 3, p. 1436, Jun. 2021, doi: 10.11591/ijeecs. v22.i3.pp1436-1445.
S. M. González-Lozoya, J. de la Calleja, L. Pellegrin, H. J. Escalante, Ma. A. Medina, and A. Benitez-Ruiz, “Recognition of facial expressions based on CNN features,” Multimedia Tools and Applications, vol. 79, no. 19–20, pp. 13987–14007, Feb. 2020, doi: 10.1007/s11042-020-08681-4.
F. Ma, Y. Li, S. Ni, S.-L. Huang, and L. Zhang, “Data Augmentation for Audio-Visual Emotion Recognition with an Efficient Multimodal Conditional GAN,” Applied Sciences, vol. 12, no. 1, p. 527, Jan. 2022, doi: 10.3390/app12010527.
S. Safwat, A. Mahmoud, I. Eldesouky Fattoh, and F. Ali, “Hybrid Deep Learning Model Based on GAN and RESNET for Detecting Fake Faces,” IEEE Access, vol. 12, pp. 86391–86402, 2024, doi: 10.1109/access.2024.3416910.
C. Gan, Y. Yang, Q. Zhu, D. K. Jain, and V. Struc, “DHF-Net: A hierarchical feature interactive fusion network for dialogue emotion recognition,” Expert Systems with Applications, vol. 210, p. 118525, Dec. 2022, doi: 10.1016/j.eswa.2022.118525.
E. M. G. Younis, S. Mohsen, E. H. Houssein, and O. A. S. Ibrahim, “Machine learning for human emotion recognition: a comprehensive review,” Neural Computing and Applications, vol. 36, no. 16, pp. 8901–8947, Feb. 2024, doi: 10.1007/s00521-024-09426-2.
Kumar, K. Srinivasan, W.-H. Cheng, and A. Y. Zomaya, “Hybrid context enriched deep learning model for fine-grained sentiment analysis in textual and visual semiotic modality social data,” Information Processing & Management, vol. 57, no. 1, p. 102141, Jan. 2020, doi: 10.1016/j.ipm.2019.102141.
R. Zhang and V. Y. Mariano, “Integration of Emotional Factors with GAN Algorithm in Stock Price Prediction Method Research,” IEEE Access, vol. 12, pp. 77368–77378, 2024, doi: 10.1109/access.2024.3406223.
F. Abdulhafidh Dael, Ö. Çağrı Yavuz, and U. Yavuz, “Stock Market Prediction Using Generative Adversarial Networks (GANs): Hybrid Intelligent Model,” Computer Systems Science and Engineering, vol. 47, no. 1, pp. 19–35, 2023, doi: 10.32604/csse.2023.037903.
Z. Liang et al., “EEGFuseNet: Hybrid Unsupervised Deep Feature Characterization and Fusion for High-Dimensional EEG With an Application to Emotion Recognition,” IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 29, pp. 1913–1925, 2021, doi: 10.1109/tnsre.2021.3111689.
K. Jahanbin and M. A. Z. Chahooki, “A Hybrid Deep Implicit Neural Model for Sentiment Analysis via Transfer Learning,” IEEE Access, vol. 12, pp. 131468–131486, 2024, doi: 10.1109/access.2024.3425819.
M. A. Razzaq et al., “A Hybrid Multimodal Emotion Recognition Framework for UX Evaluation Using Generalized Mixture Functions,” Sensors, vol. 23, no. 9, p. 4373, Apr. 2023, doi: 10.3390/s23094373.
M. Javed, Z. Zhang, F. H. Dahri, and A. A. Laghari, “Real-Time Deepfake Video Detection Using Eye Movement Analysis with a Hybrid Deep Learning Approach,” Electronics, vol. 13, no. 15, p. 2947, Jul. 2024, doi: 10.3390/electronics13152947.
R. Obiedat et al., “Sentiment Analysis of Customers’ Reviews Using a Hybrid Evolutionary SVM-Based Approach in an Imbalanced Data Distribution,” IEEE Access, vol. 10, pp. 22260–22273, 2022, doi: 10.1109/access.2022.3149482.
C. Luna-Jiménez, D. Griol, Z. Callejas, R. Kleinlein, J. M. Montero, and F. Fernández-Martínez, “Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning,” Sensors, vol. 21, no. 22, p. 7665, Nov. 2021, doi: 10.3390/s21227665.
M. F. Hashmi, B. K. K. Ashish, A. G. Keskar, N. D. Bokde, J. H. Yoon, and Z. W. Geem, “An Exploratory Analysis on Visual Counterfeits Using Conv-LSTM Hybrid Architecture,” IEEE Access, vol. 8, pp. 101293–101308, 2020, doi: 10.1109/access.2020.2998330.
M. P. Kalashami, M. M. Pedram, and H. Sadr, “EEG Feature Extraction and Data Augmentation in Emotion Recognition,” Computational Intelligence and Neuroscience, vol. 2022, pp. 1–16, Mar. 2022, doi: 10.1155/2022/7028517.
E. Lieskovská, M. Jakubec, R. Jarina, and M. Chmulík, “A Review on Speech Emotion Recognition Using Deep Learning and Attention Mechanism,” Electronics, vol. 10, no. 10, p. 1163, May 2021, doi: 10.3390/electronics10101163.
F. K. Bardak, M. N. Seyman, and F. Temurtaş, “Adaptive neuro-fuzzy based hybrid classification model for emotion recognition from EEG signals,” Neural Computing and Applications, vol. 36, no. 16, pp. 9189–9202, Feb. 2024, doi: 10.1007/s00521-024-09573-6.
J. M. Matthew and M. B. N. M. Mustafa, “Enhancement of Hybrid Deep Neural Network Using Activation Function for EEG Based Emotion Recognition,” Traitement du Signal, vol. 41, no. 4, pp. 1991–2002, Aug. 2024, doi: 10.18280/ts.410428.
Z. Su, Y. Feng, J. Liu, J. Peng, W. Jiang, and J. Liu, “An Audiovisual Correlation Matching Method Based on Fine-Grained Emotion and Feature Fusion,” Sensors, vol. 24, no. 17, p. 5681, Aug. 2024, doi: 10.3390/s24175681.
M. S. Islam, M. N. Islam, N. Hashim, M. Rashid, B. S. Bari, and F. A. Farid, “New Hybrid Deep Learning Approach Using BiGRU-BiLSTM and Multilayered Dilated CNN to Detect Arrhythmia,” IEEE Access, vol. 10, pp. 58081–58096, 2022, doi: 10.1109/access.2022.3178710.
R. Geethanjali and A. VALARMATHI, “A Novel Hybrid Deep Learning IChOA-CNN-LSTM Model for Modality-Enriched and Multilingual Emotion Recognition in Social Media A Novel Hybrid Deep Learning IChOA-CNN-LSTM Model,” Jul. 2024, doi: 10.21203/rs.3.rs-4609288/v1.
Xiaohu Wang, Yongmei Ren2, Ze Luo, Wei He, Jun Hong and Yinzhen Huang, “Deep learning-based EEG emotion recognition: Current trends and future perspectives”, Frontiers in Psychology, Vol. 14, No. 1126994, 2023.
Sara Bagherzadeh, Keivan Maghooli, Ahmad Shalbaf and Arash Maghsoudi, “A Hybrid EEG-based Emotion Recognition Approach Using Wavelet Convolutional Neural Networks and Support Vector Machine”, Basic clinical neuro science, Vol. 14, Issue 1, 2023.
L. Zhang, B. Xia, Y. Wang, W. Zhang, and Y. Han, “A Fine-Grained Approach for EEG-Based Emotion Recognition Using Clustering and Hybrid Deep Neural Networks,” Electronics, vol. 12, no. 23, p. 4717, Nov. 2023, doi: 10.3390/electronics12234717.
CRediT Author Statement
The authors confirm contribution to the paper as follows:
Conceptualization: Sahaya Sugirtha Cindrella S and Jayashree R;
Methodology: Sahaya Sugirtha Cindrella S;
Software: Sahaya Sugirtha Cindrella S;
Data Curation: Sahaya Sugirtha Cindrella S and Jayashree R;
Writing- Original Draft Preparation: Sahaya Sugirtha Cindrella S;
Visualization: Sahaya Sugirtha Cindrella S and Jayashree R;
Investigation: Jayashree R;
Supervision: Sahaya Sugirtha Cindrella S;
Validation: Jayashree R;
Writing- Reviewing and Editing: Sahaya Sugirtha Cindrella S and Jayashree R;
All authors reviewed the results and approved the final version of the manuscript.
Acknowledgements
Author(s) thanks to Dr.Jayashree R for this research completion and support.
Funding
No funding was received to assist with the preparation of this manuscript.
Ethics declarations
Conflict of interest
The authors have no conflicts of interest to declare that are relevant to the content of this article.
Availability of data and materials
Data sharing is not applicable to this article as no new data were created or analysed in this study.
Author information
Contributions
All authors have equal contribution in the paper and all authors have read and agreed to the published version of the manuscript.
Corresponding author
Jayashree R
Department of Computer Applications, Faculty of Science and Humanities, SRM Institute of Science and Technology, Kattankulathur, Chengalpattu, Tamil Nadu, India.
Open Access This article is licensed under a Creative Commons Attribution NoDerivs is a more restrictive license. It allows you to redistribute the material commercially or non-commercially but the user cannot make any changes whatsoever to the original, i.e. no derivatives of the original work. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/4.0/
Cite this article
Sahaya Sugirtha Cindrella S and Jayashree R, “Design of a Computational Model to Detect Hybrid Emotion Through Facial Expressions in Videos Using CNN LSTM”, Journal of Machine and Computing, vol.5, no.4, pp. 1984-1993, October 2025, doi: 10.53759/7669/jmc202505155.
