A Review Paper on Dual-Mode Emotion Recognition Systems Using Facial Analysis and Interactive Questioning

Emotion recognition has become a significant research area in computer vision and affective computing due to its growing applications in human computer interaction. Early approaches mainly relied on facial expression analysis; however, recent studies emphasize multimodal and contextual information for improved robustness. The review paper analyses recent advancements in emotion recognition systems with a focus on dual mode emotion approaches combining facial and speech-based emotion recognition. The study reviews deep learning techniques, system architectures, and commonly used datasets, particularly RAVDESS, for speech emotion recognition. It discusses an implemented facial expression recognition module to connect theory with practical use. The paper highlights existing research gaps concerning real-time performance, robustness, and computational efficiency. Finally, it outlines future research directions, focusing on efficient multimodal fusion, improved accuracy, and systems that can recognize emotion in real time.

Emotion recognition, facial expression recognition, speech emotion recognition, deep learning, multimodal emotion analysis, human-computer interaction

1. S. Li and W. Deng, “Deep facial expression recognition: A survey,” IEEE Transactions on Affective Computing, vol. 13, pp. 1195–1215, 2020, doi: 10.1109/TAFFC.2020.2981446. [Google Scholar] [Crossref]

2. Link- https://ieeexplore.ieee.org/document/9093037 [Google Scholar] [Crossref]

3. Y. Huang, F. Chen, S. Lv, and X. Wang, “Facial expression recognition: A survey,” Symmetry [2], vol. 11, no. 10, Art. no. 1189, 2019, doi: 10.3390/sym11101189. [Google Scholar] [Crossref]

4. Link-https://www.mdpi.com/2073-8994/11/10/1189 [Google Scholar] [Crossref]

5. M. Jaimini and N. Limbad, “A literature survey on facial expression recognition techniques using appearance-based features,” International Journal of Computer Trends and Technology (IJCTT), vol. 17, no. 2, pp. 161–165, 2014. Link- https://www.ijcttjournal.org/archives/ijctt-v17p131 [Google Scholar] [Crossref]

6. S. M. Mavadati, M. H. Mahoor, K. Bartlett, P. Trinh, and J. F. Cohn, “DISFA: A spontaneous facial action intensity database,” IEEE Transactions on Affective Computing, vol. 4, no. 2, pp. 151–160, 2013. [Google Scholar] [Crossref]

7. Link- https://doi.org/10.1109/T-AFFC.2013.4 [Google Scholar] [Crossref]

8. A. Mollahosseini, B. Hasani, and M. H. Mahoor, “AffectNet: A database for facial expression, valence, and arousal computing in the wild,” IEEE Transactions on Affective Computing, vol. 10, no. 1, pp. 18–31, 2017, doi: 10.1109/TAFFC.2017.2740923. [Google Scholar] [Crossref]

9. Link- https://ieeexplore.ieee.org/document/7811371 [Google Scholar] [Crossref]

10. S. Poria, N. Majumder, R. Mihalcea, and E. Hovy, “Emotion recognition in conversation: Research challenges, datasets, and recent advances,” IEEE Access, vol. 7, pp. 100943–100953, 2019, doi: 10.1109/ACCESS.2019.2929050. Link- https://arxiv.org/abs/1905.02947 [Google Scholar] [Crossref]

11. D. Ghosal, N. Majumder, S. Poria, N. Chhaya, and A. Gelbukh, “DialogueGCN: A graph convolutional neural network for emotion recognition in conversation,” arXiv preprint arXiv:1908.11540, 2019. Link- https://aclanthology.org/D19-1015/ [Google Scholar] [Crossref]

12. D. Hu, L. Wei, and X. Huai, “DialogueCRN: Contextual reasoning networks for emotion recognition in conversations,” arXiv preprint arXiv:2106.01978, 2021. Link- https://aclanthology.org/2021.acl-long.547/ [Google Scholar] [Crossref]

13. J. Z. Wen, H. – (Springer AI Review), “Deep emotion recognition in textual conversations: A survey,” Artificial Intelligence Review [9], Springer, 2024. Link- https://link.springer.com/article/10.1007/s10462-024-11010-y [Google Scholar] [Crossref]

14. G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schröder, “The SEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent,” IEEE Transactions on Affective Computing, vol. 3, no. 1, pp. 5–17, 2012. Link- https://doi.org/10.1109/T-AFFC.2011.20 [Google Scholar] [Crossref]

• Enhancing Formation Control of Multi Agent Systems Using Ann Based Technique
• Improving Sliding Mode Control with Chattering Reduction using Fuzzy Based Technique
• Cooking Quality, Fasting Blood Glucose, Glycemic Index and Load of High–Fiber Noodles Made from Wheat, Tiger Nut Residue and Cassava Flour Blends
• Matrix Rhythm Therapy Versus Interferential Therapy Combined with Lumbar Stabilization Exercises in Chronic Non-Specific Low Back Pain: A Randomized Comparative Trial
• Formulation and Sensory Evaluation of Functional Cake Prepared from Sweet Potato Powder