Sign Language Recognition Using Deep Learning: Advancements and Challenges

Abstract: Sign language recognition (SLR) has arisen as a major area of research in recent years, attempting to bridge the communication gap between the deaf and hard-of-hearing community and the hearing world. This research study addresses the construction and implementation of a manual alphabet recognition system utilising deep learning techniques, notably convolutional neural networks (CNNs). The work focuses on establishing an efficient and accurate system for converting Nigerian Sign Language manual alphabets into text. By integrating computer vision and machine learning methods, the proposed system seeks to overcome the communication gap between deaf and hearing individuals. The paper explains the technique adopted, including data collection, preprocessing, model architecture, and deployment using web-based tools. The system achieves a 95% success rate in recognizing static hand motions, proving its potential for real-world applications. However, issues in identifying dynamic motions and generalizing across varied user populations are observed. The report finishes with recommendations for future research, emphasizing the need for combining temporal analysis and expanding the system's capabilities to word and phrase recognition.

Deep Learning, Sign Language, Deaf, CNN

1. Alaftekin, M., Pacal, I. & Cicek, K. (2024) Real-time sign language recognition based on YOLO algorithm. Neural Computing & Applications 36,7609–7624 https://doi.org/10.1007/s00521-024-09503-6 [Google Scholar] [Crossref]

2. Asonye, E. I., Emma-Asonye, E., & Edward, M. (2018). Deaf in Nigeria: A Preliminary Survey of Isolated Deaf Communities. Sage Open, 8(2). https://doi.org/10.1177/2158244018786538 [Google Scholar] [Crossref]

3. Asonye, Emmanuel & Emma-Asonye, Ezinne & Edward, Mary. (2020). Linguistic Genocide against Development of Indigenous Signed Languages in Africa. [Google Scholar] [Crossref]

4. Cohen, S. (2020). Artificial intelligence and deep learning in pathology. Elsevier Health Sciences. https://doi.org/10.1016/C2018-0-02465-2 [Google Scholar] [Crossref]

5. Dabwan باسل دبوان, Basel & Jadhav, Mukti & Abosaq, Hamad & Olayah, Fekry & Yami, Mohammed & Ali, Yahya. (2024). Real-time System for Translating American Sign Language to Text Using Robust Techniques.1-6.10.1109/ICRASET59632.2023.10420110 [Google Scholar] [Crossref]

6. Deshpande, A., Shriwas, A., Deshmukh, V.J., & Kale, S. (2023). Sign Language Recognition System using CNN. 2023 International Conference on Intelligent and Innovative Technologies in Computing, Electrical and Electronics (IITCEE), 906-911. [Google Scholar] [Crossref]

7. Eleweke, C. (2002). A Review of Issues in Deaf Education Under Nigeria's 6-3-3-4 Education System. Journal of deaf studies and deaf education. 7. 74-82. 10.1093/deafed/7.1.74. [Google Scholar] [Crossref]

8. Gordon, R. G., Grimes, B. F., & Summer Institute of Linguistics. (2005). Ethnologue: Languages of the world (15th ed.). SIL International. [Google Scholar] [Crossref]

9. Greenberg, S., Blight, J., & Wong, A. Colour-based Gesture Recognition for American Sign Language via Hidden Markov Models. University of Waterloo, Canada. [Google Scholar] [Crossref]

10. Joudaki, S., Mohamad, D., Saba, T., Rehman, A., Al-Rodhaan, M., & Al-Dhelaan, A. (2014). Vision-based sign language classification: A directional review. IETE Technical Review, 31(5), 383-402. https://doi.org/10.1080/02564602.2014.961576 [Google Scholar] [Crossref]

11. Karbasi, M., Shah, A., & Landani, Z. (2015). An analysis of vision-based Malaysian sign: A review. International Journal of Advanced Research in Science, Engineering and Technology, 2(1), 395- 399. [Google Scholar] [Crossref]

12. Lillo-Martin, D., & Sandler, W. (2006). Sign language and linguistic universals. Cambridge University Press. Merriam-Webster. (n.d.). Manual alphabet. In Merriam-Webster.com dictionary. Retrieved October 22, 2024, from https://www.merriam-webster.com/dictionary/manual%20alphabet [Google Scholar] [Crossref]

13. Padden, C. (2003). How the alphabet came to be used in a sign language. Sign Language Studies, 4(1), 10-33. https://doi.org/10.1353/sls.2003.0026 [Google Scholar] [Crossref]

14. Pathan, R.K., Biswas, M. and Yasmin, S. (2023). Sign language recognition using the fusion of image and hand landmarks through multi-headed convolutional neural network. Sci Rep 13, 16975 (2023). https://doi.org/10.1038/s41598-023-43852-x [Google Scholar] [Crossref]

15. Paulraj, M. P., Yaacob, S., Azalan, M. S. Z., & Palaniappan, R. (2010). A phoneme based sign language recognition system using skin color segmentation. 6th International Colloquium on Signal Processing & Its Applications (CSPA), 1-5. IEEE. https://doi.org/10.1109/CSPA.2010.5545291 [Google Scholar] [Crossref]

16. Oguntimilehin, A., & Balogun, K. (2024). Real-Time Sign Language Fingerspelling Recognition using Convolutional Neural Network. The International Arab Journal of Information Technology, 21(1). https://doi.org/10.34028/iajit/21/1/14 [Google Scholar] [Crossref]

17. Shin, Jungpil & Matsuoka, Akitaka & Hasan, Md. Al & Srizon, Azmain. (2021). American Sign Language Alphabet Recognition by Extracting Feature from Hand Pose Estimation. Sensors (Basel, Switzerland). 21. 10.3390/s21175856. [Google Scholar] [Crossref]

18. Simon, C. (1982). International hand alphabet charts (2nd ed.). National Association of the Deaf. [Google Scholar] [Crossref]

19. Swee, T. T., Ariff, A. K., Salleh, S. H., Seng, S. K., & Huat, L. S. (2007). Wireless data gloves Malay sign language recognition system. 6th International Conference on Information, Communications & Signal Processing, 1-4. IEEE. https://doi.org/10.1109/ICICS.2007.4449599 [Google Scholar] [Crossref]

20. World Health Organization. (2024). Deafness and hearing loss. https://www.who.int/news-room/fact-sheets/detail/deafness- and-hearing-loss [Google Scholar] [Crossref]

21. Zhang, Yanqiong & Jiang, Xianwei. (2024). Recent Advances on Deep Learning for Sign Language Recognition. Computer Modeling in Engineering & Sciences. 139. 1-10.10.32604/cmes.2023.045731. [Google Scholar] [Crossref]

• Assess the Dilemma between the Use of Conventional and Unconventional Energy in Nigerian Shallow Water Oil Fields
• Modeling and Simulation of ‘Univariate and Multivariate analytics’ by applying ‘Deep Learning and Machine Learning’ Application of Support Vector Regression, Random Forest, K-Nearest Neighbors, Long Short-Term Memory and Gated Recurrent Units Algorithms f
• The Effects of Integrating Information and Communication Technology (ICT) In Teaching the Atomic Structure in Chemistry among Senior High School Students
• An Assessment of the Impact of Poor Indoor Air Quality on Public Health in Nigerian Urban Environments; Case Study of FCT, Abuja
• Optimizing Building Envelope Design for Cooling Loads Reduction in Abuja