The growth in digital information available via the World Wide Web has made it more important than ever to

develop efficient and intelligent information retrieval solutions. Despite the effectiveness of keyword-based

search in matching terms accurately, traditional methods fall short in capturing intent and semantics. As a result,

web information retrieval has gone through significant progress due to semantic search, machine learning, deep

learning, knowledge graphs, transformers, and generative AI technologies. This literature review focuses on the

to be considerable.

Keywords— web information retrieval, search engines, semantic search, neural information retrieval, deep

learning, generative AI, information retrieval systems

Today the Web is the world's largest archive of information. It is billions of web pages, documents, multimedia

content, databases, and digital applications which produce an overwhelming amount of data that is growing day

after day. This is why the challenge of retrieving relevant information has become one of the key issues for

computer science and information systems today.

Information Retrieval (IR) is an operation which implies finding relevant information in large sets of structured

or unstructured data on the basis of users' queries. Until recently, web search engines operated mainly through

This literature review examines contemporary developments in web information retrieval, focusing on semantic

search, neural retrieval systems, deep learning models, knowledge-based retrieval, and AI-powered search

technologies.

The concept of information retrieval as a research area started with the development of methods of storing,

indexing, searching, and retrieving information from large collections of documents. The early information

retrieval methods utilized Boolean retrieval models, vector space model, and probabilistic ranking techniques.

Figure 1. Basic Components of a Web Information Retrieval System

The traditional search engines of the World Wide Web used methods based on keyword matching and ranking

The history of WebIR is directly associated with changes in search engines. The first search engines used only

keyword matching and indexation algorithms based on the information retrieval theory. Nevertheless, as the size

of the web grew, search engines began to use ranking algorithms in order to make the search more precise.

Modern search engines use various methods such as content, hyperlink analysis, user behavior data, and machine

learning techniques in order to give more accurate results. As a result, search engines have become an important

tool for searching and knowledge acquisition.

Another innovation that has come with WebIR is the link-based ranking algorithms. The earlier retrieval models

did not take into account any other factor apart from the relevance factor of textual documents, while the

hyperlinks in the web pages could act as an indication of the authority and relevance of the web pages. Using

relevance feedback make it possible for the users to customize their search and thus retrieve more relevant

results. Thus, it is clear that user behavior has become an important aspect in the design of web search engines.

The development of WebIR has also been impacted by the progress in machine learning and artificial

intelligence. The current research emphasizes the development of neural ranking models that use deep learning

for comprehending semantic similarities between queries and documents. In contrast to conventional keyword-

based systems, the new approach is able to comprehend context and resolve the problem of vocabulary mismatch

between queries and websites.

Another crucial trend in WebIR is the use of temporal information within the ranking processes. The dynamic

nature of web information makes the timeliness of the information very important. The use of temporal

on the web, there is a growing trend of including trust and quality evaluation components in retrieval systems.

Recently, due to the advancements in the domain of deep learning and natural language processing, neural

retrieval models have been introduced in information retrieval.

This study uses the method of structured narrative literature review, where PRISMA approach was included to

enhance the quality of the research process. Even though this paper does not represent the systematic literature

review, the presence of the clear selection criteria will reduce possible biases, and the research process will

become reproducible.

Literature Review was conducted from 2016 to 2026 and focused on peer-reviewed journals, conference papers,

survey papers, and reliable academic sources in the field of web information retrieval. The literature sources

2. Screening process where the duplicates from the gathered literature were deleted, and those not directly

relating to web information retrieval technology were excluded.

3. Assessment of the eligibility of the selected studies based on their publication quality, relevance to the research

objectives, publication date, and adequacy of technical analysis.

4. Inclusion and synthesis where the selected sources were categorized into several categories, such as traditional

retrieval models, semantic search, neural information retrieval, transformer retrieval, cloud retrieval, and

generative AI.

Unlike summarization of existing literature sources, this study also analyzes the developments and evolution of

the techniques, along with identifying common patterns, research gaps, and technical challenges associated with

different generations of information retrieval systems.

were highly scalable and efficient, such keyword-based systems did not have the capability of understanding the

semantic relationship among the concepts. The researchers found vocabulary mismatch as one of the biggest

problems that affected the retrieval process. The documents might contain different vocabularies than those used

in the query.

Semantic search is one of the greatest innovations in the domain of web information retrieval. As opposed to

conventional search techniques, semantic search focuses on the meaning behind a user query.

There is evidence to suggest that semantic search engines make use of ontology, knowledge graphs, linked data,

entity recognition, and contextual reasoning to increase their effectiveness in retrieving information. Such

engines identify relationships between concepts and return relevant responses based on semantics and not just

Deep Learning gave rise to a new approach called Neural Information Retrieval (Neural IR). Neural IR makes

use of artificial neural networks for acquiring representations of queries and documents from the raw data itself.

A survey literature on Neural IR pinpointed the evolution of deep neural architectures as a breakthrough in

information retrieval research. Neural models are capable of automatically learning semantic representations,

thereby eliminating the need for feature engineering.

Further research showed that neural information retrieval models are more effective than conventional models

in many retrieval tasks, especially in handling semantic similarity and context. Neural information retrieval

models are broadly divided into sparse retrieval models, dense retrieval models, and hybrid retrieval models.

Dense retrieval models make use of vector representations for documents and queries.

Modern discourse in this area also emphasizes the transition from keyword-based search to concept-based and

semantic understanding approaches.

comprehend the meaning of user queries. Semantic retrieval yields more relevant search results as compared to

keyword-based retrieval, especially when dealing with complicated searches. However, building and managing

Neural information retrieval enhances retrieval performance by generating dense vector representations of

Personalized retrieval systems necessitate large amounts of data about users, posing challenges related to data

privacy and security.

contextual inference, personalization, and AI-driven knowledge creation. Although innovations help make

information retrieval more effective and user-friendly, the issues of scalability, interpretability, fairness,

misinformation, and privacy should not be ignored.

2. Chowdhury, T. (2026). Cloud-Based Information Retrieval for Big Data: A Survey of Architectures and

Scalability Challenge. The Eastasouth Journal of Information System and Computer Science, 3(3), 1–

and Challenges. International Journal of Informatics and Communication Technology, 15(2), 456–464.

8. Rashid, J., & Nisar, M. W. (2016). A Study on Semantic Searching, Semantic Search Engines and

Technologies Used for Semantic Search Engines. International Journal of Information Technology and

Computer Science, 8(10), 82–89.

9. Zhang, Y., Rahman, M. M., Braylan, A., Dang, B., Chang, H. L., Kim, H., McNamara, Q., Angert, A.,

Banner, E., Khetan, V., McDonnell, T., Nguyen, A. T., Xu, D., Wallace, B. C., & Lease, M. (2016).

Neural Information Retrieval: A Literature Review. arXiv Preprint arXiv:1611.06792.

10. Zhang, Y., Altingovde, I. S., Karagoz, P., Rahman, M. M., Braylan, A., Dang, B., Chang, H. L., Kim,

H., McNamara, Q., Angert, A., Banner, E., Khetan, V., McDonnell, T., Nguyen, A. T., Xu, D., Wallace,

B. C., & Lease, M. (2018). Neural Information Retrieval: at the End of the Early Years. Information

13. Choudhary, L., & Burdak, B. S. (2012). Role of Ranking Algorithms for Information Retrieval. arXiv.

https://arxiv.org/abs/1208.1926

14. Guo, J., Fan, Y., Pang, L., Yang, L., Ai, Q., Zamani, H., Wu, C., Croft, W. B., & Cheng, X. (2019). A

Deep Look Into Neural Ranking Models for Information Retrieval. arXiv.

https://arxiv.org/abs/1903.06902

15. Hiwale, K., More, P., & Nayake, Y. (2024). A Evolution and Impact of Web Search Engines: A

Comprehensive Review. Engineering and Technology Journal, 9(6).

16. Langville, A. N., & Meyer, C. D. (2005). A Survey of Eigenvector Methods for Web Information

Retrieval. SIAM Review, 47(1), 135–161. https://doi.org/10.1137/S0036144503424786

17. Lewandowski, D. (2005). Web Searching, Search Engines and Information Retrieval. Information

20. Pokorný, J. (2004). Web Searching and Information Retrieval. Computing in Science & Engineering,

6(4), 43–48. https://doi.org/10.1109/MCSE.2004.24

21. Sanderson, M., & Croft, W. B. (2012). The History of Information Retrieval Research. Proceedings of

the IEEE, 100(Special Centennial Issue), 1444–1451. https://doi.org/10.1109/JPROC.2012.2189916

22. Srinivasa, S., & Bhatt, P. C. P. (2002). Introduction to Web Information Retrieval: A User Perspective.

Resonance, 7(6), 29–41.

https://doi.org/10.1007/BF02834389

23. Ramachandran, S., Paulraj, S., Joseph,S., & Ramaraj, V. (2009). Enhanced Trustworthy and High-

Quality Information Retrieval System for Web Search Engines. arXiv. https://arxiv.org/abs/0911.0914