Web Information Retrieval: A Literature Review of Search Engines, Semantic Search, Neural Information Retrieval, and Generative Artificial Intelligence

The growth in digital information available via the World Wide Web has made it more important than ever to develop efficient and intelligent information retrieval solutions. Despite the effectiveness of keyword-based search in matching terms accurately, traditional methods fall short in capturing intent and semantics. As a result, web information retrieval has gone through significant progress due to semantic search, machine learning, deep learning, knowledge graphs, transformers, and generative AI technologies. This literature review focuses on the developments in web information retrieval from 2016 to 2026. It discusses advancements in retrieval models, semantic search techniques, neural information retrieval, recommender systems, large language models, and generative AI-enabled retrieval systems. Major findings include a shift from keyword-based retrieval to context-based, intent-based, and knowledge-based retrieval approaches. Although there have been numerous developments in web information retrieval to ensure enhanced accuracy and user experience, the difficulties associated with scalability, misinformation, bias, interpretability, privacy, and computational efficiency continue to be considerable.

web information retrieval, search engines, semantic search, neural information retrieval, deep learning, generative AI, information retrieval systems

1. Chowdhury, T. (2026). Semantic Search With Vector Database: A Comprehensive Review of Models, Indexing and Applications. The Eastasouth Journal of Information System and Computer Science, 3(3), 1–15. [Google Scholar] [Crossref]

2. Chowdhury, T. (2026). Cloud-Based Information Retrieval for Big Data: A Survey of Architectures and Scalability Challenge. The Eastasouth Journal of Information System and Computer Science, 3(3), 1–14. [Google Scholar] [Crossref]

3. Ezhilarasi, K., & Kalavathy, G. M. (2018). Literature Survey: Analysis on Semantic Web Information Retrieval Methodologies. Proceedings of PECTEAM 2018. [Google Scholar] [Crossref]

4. Guo, J., Cai, Y., Fan, Y., Sun, F., Zhang, R., & Cheng, X. (2022). Semantic Models for the First-Stage Retrieval: A Comprehensive Review. ACM Transactions on Information Systems, 40(4), 1–42. [Google Scholar] [Crossref]

5. Kamil, M., & Çakir, D. (2025). Advances in Transformer-Based Semantic Search: Techniques, Benchmarks, and Future Directions. Turkish Journal of Mathematics and Computer Science, 17(1), 1–25. [Google Scholar] [Crossref]

6. Martinez-Rodriguez, J. L., Hogan, A., & Lopez-Arevalo, I. (2020). Information Extraction Meets the Semantic Web: A Survey. Semantic Web Journal, 11(2), 255–335. [Google Scholar] [Crossref]

7. Patil, S. D., & Aalam, Z. (2026). A Critical Review of Information Retrieval Techniques: Current Trends and Challenges. International Journal of Informatics and Communication Technology, 15(2), 456–464. [Google Scholar] [Crossref]

8. Rashid, J., & Nisar, M. W. (2016). A Study on Semantic Searching, Semantic Search Engines and Technologies Used for Semantic Search Engines. International Journal of Information Technology and Computer Science, 8(10), 82–89. [Google Scholar] [Crossref]

9. Zhang, Y., Rahman, M. M., Braylan, A., Dang, B., Chang, H. L., Kim, H., McNamara, Q., Angert, A., Banner, E., Khetan, V., McDonnell, T., Nguyen, A. T., Xu, D., Wallace, B. C., & Lease, M. (2016). Neural Information Retrieval: A Literature Review. arXiv Preprint arXiv:1611.06792. [Google Scholar] [Crossref]

10. Zhang, Y., Altingovde, I. S., Karagoz, P., Rahman, M. M., Braylan, A., Dang, B., Chang, H. L., Kim, H., McNamara, Q., Angert, A., Banner, E., Khetan, V., McDonnell, T., Nguyen, A. T., Xu, D., Wallace, B. C., & Lease, M. (2018). Neural Information Retrieval: at the End of the Early Years. Information Retrieval Journal, 21(2–3), 111–182. [Google Scholar] [Crossref]

11. Asadi, S., & Jamali, H. R. (2004). Shifts in Search Engine Development: A Review of Past, Present and Future Trends in Research on Search Engines. Webology, 1(2). [Google Scholar] [Crossref]

12. Baeza-Yates, R. (2003). Information Retrieval in the Web: Beyond Current Search Engines. International Journal of Approximate Reasoning, 34(2–3), 97–104. https://doi.org/10.1016/j.ijar.2003.07.002 [Google Scholar] [Crossref]

13. Choudhary, L., & Burdak, B. S. (2012). Role of Ranking Algorithms for Information Retrieval. arXiv. https://arxiv.org/abs/1208.1926 [Google Scholar] [Crossref]

14. Guo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., & Cheng, X. (2019). A Deep Look Into Neural Ranking Models for Information Retrieval. arXiv. https://arxiv.org/abs/1903.06902 [Google Scholar] [Crossref]

15. Hiwale, K., More, P., & Nayake, Y. (2024). A Evolution and Impact of Web Search Engines: A Comprehensive Review. Engineering and Technology Journal, 9(6). [Google Scholar] [Crossref]

16. Langville, A. N., & Meyer, C. D. (2005). A Survey of Eigenvector Methods for Web Information Retrieval. SIAM Review, 47(1), 135–161. https://doi.org/10.1137/S0036144503424786 [Google Scholar] [Crossref]

17. Lewandowski, D. (2005). Web Searching, Search Engines and Information Retrieval. Information Services & Use, 25(3–4), 137–147. https://doi.org/10.3233/ISU-2005-253-402 [Google Scholar] [Crossref]

18. Mitra, B., & Craswell, N. (2017). Neural Models for Information Retrieval. arXiv. https://arxiv.org/abs/1705.01509 [Google Scholar] [Crossref]

19. Moulahi, B., Tamine, L., & Ben Yahia, S. (2016). When Time Meets Information Retrieval: Past Proposals, Current Plans and Future Trends. Journal of Information Science, 42(6), 794–818. https://doi.org/10.1177/0165551515607277 [Google Scholar] [Crossref]

20. Pokorný, J. (2004). Web Searching and Information Retrieval. Computing in Science & Engineering, 6(4), 43–48. https://doi.org/10.1109/MCSE.2004.24 [Google Scholar] [Crossref]

21. Sanderson, M., & Croft, W. B. (2012). The History of Information Retrieval Research. Proceedings of the IEEE, 100(Special Centennial Issue), 1444–1451. https://doi.org/10.1109/JPROC.2012.2189916 [Google Scholar] [Crossref]

22. Srinivasa, S., & Bhatt, P. C. P. (2002). Introduction to Web Information Retrieval: A User Perspective. Resonance, 7(6), 29–41. https://doi.org/10.1007/BF02834389 [Google Scholar] [Crossref]

23. Ramachandran, S., Paulraj, S., Joseph,S., & Ramaraj, V. (2009). Enhanced Trustworthy and High-Quality Information Retrieval System for Web Search Engines. arXiv. https://arxiv.org/abs/0911.0914 [Google Scholar] [Crossref]

• Enhancing Formation Control of Multi Agent Systems Using Ann Based Technique
• Improving Sliding Mode Control with Chattering Reduction using Fuzzy Based Technique
• Cooking Quality, Fasting Blood Glucose, Glycemic Index and Load of High–Fiber Noodles Made from Wheat, Tiger Nut Residue and Cassava Flour Blends
• Matrix Rhythm Therapy Versus Interferential Therapy Combined with Lumbar Stabilization Exercises in Chronic Non-Specific Low Back Pain: A Randomized Comparative Trial
• Formulation and Sensory Evaluation of Functional Cake Prepared from Sweet Potato Powder