A Hidden Markov Model Framework for POS Tagging of English–Punjabi Code-Mixed Social Media Text

Part-of-Speech (POS) tagging for code-mixed text is notably challenging due to frequent language switching, non-standard orthography, transliteration issues, and the prevalence of informal syntactic structures in user-generated content. This study presents a Hidden Markov Model (HMM)-based approach tailored to English–Punjabi code-mixed text, specifically addressing bilingual interactions in which Punjabi is written in Romanized script. A code-mixed corpus was strategically compiled from platforms such as YouTube, Facebook, and WhatsApp, and meticulously annotated at the token level. The resulting dataset comprises 900 sentences totalling 10,117 words, showcasing diverse mixing patterns and typical social media artefacts, including abbreviations, emojis, and irregular punctuation. The proposed framework conceptualizes POS tagging as a sequence labelling problem. It estimates emission and transition probabilities through the annotated corpus and employs the Viterbi algorithm to decode the most probable tag sequences. Experimental evaluations yield an overall tagging accuracy of 71.52%, establishing a probabilistic baseline for English–Punjabi code-mixed POS tagging. This work lays the groundwork for future research to integrate richer feature sets and leverage neural architectures to enhance performance.

code-mixed tex, part-of-speech tagging, hidden Markov model (HMM), natural language processing (NLP)

1. AlGhamdi, F., Molina, G., Diab, M., Solorio, T., Hawwari, A., Soto, V., & Hirschberg, J. (2016, November). Part of speech tagging for code switched data. In Proceedings of the Second Workshop on Computational Approaches to Code Switching (pp. 98-107). [Google Scholar] [Crossref]

2. Baig, A., Rahman, M. U., Kazi, H., & Baloch, A. (2020). Developing a pos tagged corpus of urdu tweets. Computers, 9(4), 90. https://doi.org/10.3390/computers9040090 [Google Scholar] [Crossref]

3. Bansal, N., Goyal, V., & Rani, S. (2020). Experimenting language identification for sentiment analysis of english punjabi code mixed social media text. International Journal of E-Adoption (IJEA), 12(1), 52-62. https://doi.org/10.4018/IJEA.2020010105 [Google Scholar] [Crossref]

4. Gill, M. S., Lehal, G. S., & Joshi, S. S. (2009). Part of speech tagging for grammar checking of punjabi. The Linguistic Journal, 4(1), 6-21. [Google Scholar] [Crossref]

5. Jamatia, A., Gambäck, B., & Das, A. (2015). Part-of-speech tagging for code-mixed english-hindi twitter and facebook chat messages. Association for Computational Linguistics. [Google Scholar] [Crossref]

6. Jamatia, A., Das, A., & Gamback, B. (2020). Deep learning-based language identification in English-Hindi-Bengali code-mixed social media corpora. Journal of Intelligent Systems, 28(3), 399-408. https://doi.org/10.1515/jisys-2017-0440 [Google Scholar] [Crossref]

7. Kumar, S., Kumar, M. A., & Soman, K. P. (2019). Deep learning based part-of-speech tagging for Malayalam Twitter data (Special issue: deep learning techniques for natural language processing). Journal of Intelligent Systems, 28(3), 423-435. https://doi.org/10.1515/jisys-2017-0520 [Google Scholar] [Crossref]

8. Nikiforos, M. N., & Kermanidis, K. L. (2020, May). A supervised part-of-speech tagger for the Greek language of the social web. In Proceedings of the Twelfth Language Resources and Evaluation Conference (pp. 3861-3867). https://aclanthology.org/2020.lrec-1.476/ [Google Scholar] [Crossref]

9. Pakray, P., Majumder, G., & Pathak, A. (2018, January). An hmm based pos tagger for pos tagging of code-mixed indian social media text. In Annual Convention of the Computer Society of India (pp. 495-504). Singapore: Springer Singapore. https://doi.org/10.1007/978-981-13-1343-1_41 [Google Scholar] [Crossref]

10. Pathak, D., Nandi, S., & Sarmah, P. (2023). Part-of-speech tagger for assamese using ensembling approach. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(10), 1-22. https://doi.org/10.1145/3617653 [Google Scholar] [Crossref]

11. Paul, A., Purkayastha, B. S., & Sarkar, S. (2015, September). Hidden Markov model based part of speech tagging for Nepali language. In 2015 international symposium on advanced computing and communication (ISACC) (pp. 149-156). IEEE. DOI: https://doi.org/10.1109/ISACC.2015.7377332 [Google Scholar] [Crossref]

12. Pimpale, P. B., & Patel, R. N. (2016). Experiments with POS tagging code-mixed Indian social media text. arXiv preprint arXiv:1610.09799. https://doi.org/10.48550/arXiv.1610.09799 [Google Scholar] [Crossref]

13. Raha, T., Mahata, S., Das, D., & Bandyopadhyay, S. (2019, December). Development of pos tagger for english-bengali code-mixed data. In Proceedings of the 16th International Conference on Natural Language Processing (pp. 143-149). https://aclanthology.org/2019.icon-1.17/ [Google Scholar] [Crossref]

14. Saharia, N., Das, D., Sharma, U., & Kalita, J. (2009, August). Part of speech tagger for Assamese text. In Proceedings of the ACL-IJCNLP 2009 conference short papers (pp. 33-36). DOI 10.4018/IJSE.2018010102 [Google Scholar] [Crossref]

15. Santiago-Benito, H., Cordova-Esparza, D. M., Castro-Sanchez, N. A., Terven, J., Romero-González, J. A., & Garcia-Ramirez, T. (2025). Automatic grammatical tagger for a Spanish–Mixtec parallel corpus. SoftwareX, 29, 101985. https://doi.org/10.1016/j.softx.2024.101985 [Google Scholar] [Crossref]

16. Sarkar, K., & Gayen, V. (2013). A trigram HMM-based POS tagger for Indian languages. In Proceedings of the international conference on frontiers of intelligent computing: theory and applications (FICTA) (pp. 205-212). Berlin, Heidelberg: Springer Berlin Heidelberg. [Google Scholar] [Crossref]

17. Sharma, S. K., & Lehal, G. S. (2011, June). Using Hidden Markov Model to improve the accuracy of Punjabi POS tagger. In 2011 IEEE international conference on computer science and automation engineering (Vol. 2, pp. 697-701). IEEE. DOI: https://doi.org/10.1109/CSAE.2011.5952600 [Google Scholar] [Crossref]

18. Silva, E. H. D., Pardo, T. A. S., Roman, N. T., & Felippo, A. D. (2021). Universal dependencies for tweets in brazilian portuguese: Tokenization and part of speech tagging. Anais. [Google Scholar] [Crossref]

19. Tiun, S., Ariffin, S. N. A. N., & Chew, Y. D. (2022, June). POS Tagging Model for Malay Tweets Using New POS Tagset and BiLTSM-CRF Approach. In ALTNLP (pp. 160-165). [Google Scholar] [Crossref]

20. Withanage, S. G., & Silva, T. (2020, November). A stochastic part of speech tagger for the sinhala language based on social media data mining. In 2020 20th International Conference on Advances in ICT for Emerging Regions (ICTer) (pp. 137-142). IEEE. DOI: https://doi.org/10.1109/ICTer51097.2020.9325456 [Google Scholar] [Crossref]

21. Vyas, Y., Gella, S., Sharma, J., Bali, K., & Choudhury, M. (2014, October). Pos tagging of english-hindi code-mixed social media content. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 974-979). [Google Scholar] [Crossref]

• Predictive Health Monitoring Systems for Electric Vehicle Powertrains Using Edge AI and CAN Bus Data
• Internship Portals: A Systematic Review of Current Platforms and Future Directions
• Towards Better Urban Mobility: A Comprehensive Assessment of Pedestrian Infrastructure in Naval, Biliran Province, Philippines
• An Affordable and Sustainable Efficient Color Sorting System Using Arduino and TCS3200 Sensor
• Financial Stress and Mobility Patterns: Implication for Transportation Policy Among Jeepney Passengers