A Brief History of Information Retrieval

While this thesis does not aim to provide an extensive overview of past or modern Information Retrieval techniques, the following section will provide a rapid overview of the field in order to contextualize the contributions of this thesis.¹If the reader finds this overview incomplete or yearns for more contextual depth, they are invited to search for further information within external sources. If the search is successful, great! If not, it only is further proof that modern search tools remain imperfect and that their improvement is a valid research motivation for this thesis.

Further finely scoped related work is also introduced in each individual chapter.

Early Pre-Digital Cataloging

Long before computers, humans developed methods to organize and retrieve information. One of the earliest examples is the Pinakes of Callimachus around the 3rd century BCE, often considered the first library catalog in the Western world. The Pinakes is said to have systematically cataloged the authors and works held in the Library of Alexandria, organizing texts by subject and arranging authors alphabetically — an early bibliographic method that helped lay the foundations for later library catalogs.

Through the centuries, librarians and scholars continued to develop catalogs and classification systems, from medieval manuscript indexes ²Conrau-Lewis (2018)Medieval Indices: Unlikely Leisurely Reading to Melvil Dewey’s Decimal Classification ³Dewey (1876)A Classification and Subject Index for Cataloguing and Arranging the Books and Pamphlets of a Library in 1876. These pre-digital innovations addressed the same core challenge as modern IR systems: how to store, organize, and find relevant information efficiently. In 1945, Vannevar Bush articulated a technological vision for the future of information retrieval in As We May Think ⁴Bush (1945)As We May Think. He describes the inadequacy of indexing-based systems for navigating the deluge of information:

Bush's vision was that new technology would enable moving away from this system. He proposes the Memex, a hypothetical electromechanical desk that could instantly retrieve microfilmed documents via associative links.

Founding of Information Retrieval (1950s–1960s)

Information Retrieval as a distinct field took shape in the 1950s and 1960s, alongside the advent of digital computers. In 1950, Calvin Mooers coined the term "information retrieval" ⁵Mooers (1950)The Theory of Digital Handling of Non-Numerical Information and Its Implications to Machine Economics, heralding the idea of using machines to search document collections. Early experiments followed quickly: in 1951, a master’s thesis at MIT by Philip Bagley ⁶Bagley (1951)Electronic Digital Machines for High-Speed Information Searching demonstrated one of the first computerized document retrieval experiments. Pioneers like Hans Peter Luhn at IBM began applying computers to automate indexing; for example, Luhn developed methods for auto-encoding documents and keyword-in-context indexing. By 1959 he published the influential paper Auto-Encoding of Documents for Information Retrieval Systems ⁷Luhn (1959)Auto-Encoding of Documents for Information Retrieval Systems, showing how computers could encode text and allow electronic searching of content. This period also saw growing support for IR research as the U.S. government recognized an information explosion funding projects on mechanized literature searching and the creation of citation indexes.

The Cranfield paradigm. A key milestone was the Cranfield experiments ⁸Cleverdon (1967)The Cranfield Tests on Index Language Devices, led by Pete Cleverdon in the early 1960s. The Cranfield studies introduced a rigorous experimental methodology for IR: assembling a test collection of documents, queries, and relevance judgments, and evaluating retrieval performance using metrics like precision and recall. This methodology became a cornerstone of IR research, establishing a scientific approach to evaluate how “relevant” the results returned by an IR system are, laying the foundation for future benchmarks.

Early retrieval models. During the same era, Gerard Salton formed the first major IR research group (at Harvard, later Cornell). Researchers began formulating the first theoretical models of IR. In 1960, Maron and Kuhns proposed a probabilistic indexing model ⁹Maron & Kuhns (1960)On Relevance, Probabilistic Indexing and Information Retrieval, foreshadowing later probabilistic retrieval approaches. Salton and others started developing what became the vector space model: by the late 1960s, Salton’s team had created the SMART system, representing documents and queries as vectors in a term-weighted space. In 1968 Salton published Automatic Information Organization and Retrieval ¹⁰Salton (1968)Automatic Information Organization and Retrieval, and by 1975 he fully articulated the vector space framework and term-weighting schemes ¹¹Salton et al. (1975)A Vector Space Model for Automatic Indexing. Another 1960s contribution came from Karen Spärck Jones, who introduced the concept of inverse document frequency (IDF) (Spärck Jones 1972) to statistically downweight common terms and boost rare, informative terms – a principle that soon became central to ranking algorithms.

The Statistical Revolution in IR (1970s–1990s)

Term weighting and probabilistic ranking. With the fundamentals laid, subsequent decades saw rapid improvement in statistical text retrieval methods. The 1970s refined the vector space and probabilistic models. Term frequency–inverse document frequency (TF–IDF) ¹²Spärck Jones (1972)A Statistical Interpretation of Term Specificity and Its Application in Retrieval weighting became a standard technique: terms are scored higher if they appear frequently in a document (term frequency) but are rare across the collection (high IDF)¹³ TF-IDF: Let $\mathcal{D}=\{d_1,\dots,d_N\}$ be a corpus of $N$ documents and let $t$ be a term (token). The TF–IDF weight of term $t$ in document $d$ is $$\mathrm{tfidf}(t,d,\mathcal{D}) \;=\; \mathrm{tf}(t,d)\cdot \mathrm{idf}(t,\mathcal{D}).$$ A common choice is $$\mathrm{tf}(t,d)=\frac{f_{t,d}}{\sum_{t'} f_{t',d}}$$ and $$\mathrm{idf}(t,\mathcal{D})=\log\!\left(\frac{N}{\mathrm{df}(t)}\right),$$ where $f_{t,d}$ is the raw count of occurrences of $t$ in $d$, $\sum_{t'} f_{t',d}$ is the total number of term occurrences in $d$ (so $\mathrm{tf}$ is a normalized frequency), and $\mathrm{df}(t)=\bigl|\{d_i\in\mathcal{D}: t\in d_i\}\bigr|$ is the document frequency (the number of documents containing $t$ at least once). Intuitively, $\mathrm{tf}$ upweights terms frequent in a given document, while $\mathrm{idf}$ downweights terms that appear in many documents. (Variants often use $\mathrm{tf}(t,d)=1+\log f_{t,d}$ and/or smoothing $\log\!\left(\frac{N+1}{\mathrm{df}(t)+1}\right)+1$ to avoid division by zero.) . On the probabilistic side, Stephen Robertson and colleagues extended early ideas into the Probabilistic Relevance Framework ¹⁴Robertson & Sparck Jones (1976)Relevance Weighting of Search Terms
Robertson (1977)The Probability Ranking Principle in IR
Robertson & Zaragoza (2009)The Probabilistic Relevance Framework: BM25 and Beyond in the 1970s and 1980s, leading to the development of ranking functions that estimate the probability a document is relevant to a query. These efforts culminated in the 1990s with Okapi BM25 ¹⁵Robertson & Walker (1994)Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval
Robertson et al. (1994)Okapi at TREC-3, developed in the Okapi/TREC line of work from the Robertson–Spärck Jones probabilistic framework and Robertson–Walker 2-Poisson approximations¹⁶BM25: Let $\mathcal{D}$ be a corpus, $d\in\mathcal{D}$ a document, and $t$ a term. The BM25 score of $d$ for a query $q$ is $$\mathrm{BM25}(q,d)=$$ $$\sum_{t\in q} \mathrm{idf}(t)\; \frac{f_{t,d}(k_1+1)}{f_{t,d}+k_1\!\left(1-b+b\,\frac{|d|}{\mathrm{avgdl}}\right)}$$ Here $f_{t,d}$ is the count of $t$ in $d$, $|d|$ is the document length (in tokens), and $\mathrm{avgdl}$ is the average document length in the corpus. $k_1>0$ controls term-frequency saturation, and $b\in[0,1]$ controls how strongly length normalization is applied. A common IDF is $$\mathrm{idf}(t)=\log\!\left(\frac{N-\mathrm{df}(t)+0.5}{\mathrm{df}(t)+0.5}\right),$$ where $N=|\mathcal{D}|$ and $\mathrm{df}(t)$ is the number of documents containing $t$. . By incorporating TF–IDF-like features into a probabilistic model, BM25 ¹⁷Robertson & Zaragoza (2009)The Probabilistic Relevance Framework: BM25 and Beyond
Manning et al. (2008)Introduction to Information Retrieval became one of the dominant ranking algorithms in search and remains widely used as a strong baseline in modern information retrieval systems.

Online search systems. Alongside algorithm advances, IR systems grew in scale and importance. The first online search systems appeared in the early 1970s as databases of bibliographic records: for example, MEDLINE ¹⁸Medicine (2024)MEDLINE History (1971) provided biomedical literature search, and DIALOG ¹⁹Engineering & Wiki (2019)Milestones: DIALOG Online Search System, 1966 offered commercial online search over multiple databases. These systems predated personal computers—users accessed them via terminals—but they demonstrated that searching thousands of documents via text queries was feasible and valuable. In 1978, the IR community established its premier conference, ACM SIGIR ²⁰Dattola (1978)ACM-SIGIR Conference on Information Storage and Retrieval, reflecting the field's maturation.

By the 1980s, personal computing and CD-ROMs enabled standalone IR systems (like library catalog CDs), and researchers explored new ideas such as relevance feedback through the Rocchio algorithm ²¹Rocchio (1971)Relevance Feedback in Information Retrieval (which allowed a system to adjust a query based on documents a user marked relevant) and clustering through the "cluster hypothesis" ²²Jardine & van Rijsbergen (1971)The Use of Hierarchic Clustering in Information Retrieval
van Rijsbergen (1979)Information Retrieval (relevant documents tend to cluster together). The inverted index ²³Manning et al. (2008)Introduction to Information Retrieval (storing, for each term, a list of documents containing it) became a core data structure, enabling fast keyword lookup even as collections grew to tens of thousands of documents. This era also saw experiments with user interfaces and visualization (e.g. plotting document similarity or using hypertext systems).

Standardizing evaluation. Evaluation at scale took a leap in 1992 with the start of the Text Retrieval Conference (TREC) ²⁴Harman (1993)Overview of the First Text REtrieval Conference. TREC built large standard test collections and yearly competitions for state-of-the-art retrieval methods. TREC not only pushed research towards scalable methods but also fostered the development of specialized tasks. Many techniques that became important, such as query expansion and relevance feedback, were evaluated and refined through TREC workshops.

Enter The Web (1990s–2010s)

Search at web scale. Web search emerged in the mid-1990s as a new frontier that dramatically expanded IR's scale and impact. It also marked an important step toward Vannevar Bush's vision of associative information access. In Bush's Memex, users would create and follow trails through related documents; on the Web, hyperlinks made such associative connections part of the infrastructure of global information access. Association however is not the same as retrieval. Links helped users navigate once useful pages were found, yet they did not by themselves determine which pages were authoritative, relevant, or worth visiting.

Early web search systems and directories—for example, Yahoo!'s human-curated directory in 1994 and AltaVista in 1995—helped users navigate the growing World Wide Web, but human-maintained directories were difficult to scale and keyword-based engines often returned many low-quality matches at web scale ²⁵Brin & Page (1998)The Anatomy of a Large-Scale Hypertextual Web Search Engine.

Link-based ranking. A major breakthrough came from the Stanford/Google work of the late 1990s, especially PageRank ²⁶Page et al. (1998)The PageRank Citation Ranking: Bringing Order to the Web
Brin & Page (1998)The Anatomy of a Large-Scale Hypertextual Web Search Engine, which treated the web as a graph and used hyperlinks as an additional ranking signal: a page linked from many other pages, especially pages that were themselves important, received a higher importance score. By combining link analysis with content relevance, anchor text, proximity, and other web-specific signals, Google ²⁷Brin & Page (1998)The Anatomy of a Large-Scale Hypertextual Web Search Engine produced more useful rankings than many earlier content-only systems. This represented a graph-based retrieval innovation, leveraging network topology in a way analogous to citation analysis in scholarly literature.

Learned and semantic ranking. In the 2000s, machine learning techniques became increasingly central to IR. Learning-to-rank algorithms ²⁸Burges et al. (2005)Learning to Rank Using Gradient Descent
Burges et al. (2006)Learning to Rank with Nonsmooth Cost Functions
Burges (2010)From RankNet to LambdaRank to LambdaMART: An Overview learned how to combine many ranking features—such as lexical matching scores, anchor-text evidence, document features, and user-behavior-derived signals—into ranking functions. Microsoft Research developed RankNet, LambdaRank, and LambdaMART ²⁹Burges et al. (2005)Learning to Rank Using Gradient Descent
Burges et al. (2006)Learning to Rank with Nonsmooth Cost Functions
Burges (2010)From RankNet to LambdaRank to LambdaMART: An Overview in the mid-2000s, with LambdaMART combining LambdaRank-style optimization with boosted decision trees. Commercial search engines increasingly incorporated machine-learned ranking methods, although the exact algorithms and deployment timelines were often proprietary.

Researchers also showed that clickthrough logs and query reformulations could provide useful implicit feedback for ranking ³⁰Joachims (2002)Optimizing Search Engines Using Clickthrough Data
Joachims et al. (2005)Accurately Interpreting Clickthrough Data as Implicit Feedback
Radlinski & Joachims (2005)Query Chains: Learning to Rank from Implicit Feedback.

Another 2000s-and-2010s trend was the use of semantic information and knowledge bases. For instance, Bing publicly described its Satori knowledge system ³¹Blog (2013)Understand Your World with Bing in 2013 as modeling entities—people, places, and things—and relationships among them. Google also moved in this direction through systems such as the Knowledge Graph and, in 2013, Hummingbird ³²Central (2025)A Guide to Google Search Ranking Systems, a major improvement to its ranking systems associated with better handling of longer and more conversational queries.

The Deep Learning Era of IR (2010s–2020s)

The 2010s shifted information retrieval from purely lexical matching toward neural semantic matching. Early neural IR systems such as DSSM ³³Huang et al. (2013)Learning Deep Structured Semantic Models for Web Search using Clickthrough Data mapped queries and documents into learned vector spaces, allowing systems to match texts that used different words but expressed related meanings. This was the beginning of dense retrieval: instead of representing a document only by the terms it contains, the system learns a continuous embedding that captures semantic similarity.

The BERT revolution. The transformer era accelerated this shift. BERT ³⁴Devlin et al. (2019)BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding showed that large pretrained language models could produce rich contextual representations of text, and by 2019 Google had publicly announced the use of BERT in Search ³⁵Nayak (2019)Understanding searches better than ever before to better handle longer, more conversational queries. In academic IR, BERT-style cross-encoders ³⁶Nogueira & Cho (2019)Passage Re-ranking with BERT
Yates et al. (2021)Pretrained Transformers for Text Ranking: BERT and Beyond became especially effective as rerankers: a first-stage retriever such as BM25 or a dense retriever retrieves a candidate set, and a neural model then scores each query–document pair more precisely.

Bi-encoders. At the same time, researchers developed neural retrievers that could operate efficiently at first stage. The key idea behind bi-encoders is to give up full query–document interaction at retrieval time: the query and each document are encoded separately, each text is compressed into a single vector, and retrieval becomes efficient nearest-neighbor search in embedding space ³⁷Johnson et al. (2017)Billion-scale Similarity Search with GPUs. Sentence-BERT ³⁸Reimers & Gurevych (2019)Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks made this recipe practical for BERT-style models by pooling contextualized token embeddings into sentence vectors that can be compared directly with cosine similarity. Its training objectives then teach the pooled vectors to behave geometrically: paraphrases and semantically related sentences should be close, while unrelated sentences should be far apart. DPR ³⁹Karpukhin et al. (2020)Dense Passage Retrieval for Open-Domain Question Answering adapted this principle to passage retrieval, using contrastive training so that a question vector is closer to its gold passage than to in-batch or mined negatives. These models helped establish the modern dense-retrieval template: encode documents once, index their vectors, and retrieve efficiently at query time.

By the early 2020s, the dominant neural IR systems were often multi-stage: one or several fast lexical, sparse, or dense retriever produces candidates; a stronger neural reranker refines them. Systems often combined lexical and semantic signals rather than replacing one with the other to capture a variety of signals ⁴⁰Nogueira & Cho (2019)Passage Re-ranking with BERT
Yates et al. (2021)Pretrained Transformers for Text Ranking: BERT and Beyond.

The Modern Stack: Hybrid, Contextual, and Agentic Retrieval

While the retrieve–rerank paradigm ⁴¹Nogueira & Cho (2019)Passage Re-ranking with BERT remains widely used in production systems, recent years have seen substantial improvements to every stage of the retrieval pipeline and the emergence of novel ideas.

Modern general-purpose embeddings. General-purpose embedding models (bi-encoders) remain one of the central model families in modern IR. Their goal is to map queries and documents into a shared representation space that supports a wide range of retrieval and semantic-matching tasks. Progress in this area has often been driven by benchmark optimization, especially on IR benchmarks such as BEIR ⁴²Thakur et al. (2021)BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models and broader embedding benchmarks such as MTEB ⁴³Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark. ⁴⁴ General-purpose document embeddings are useful across many tasks, including retrieval, classification, clustering, natural language inference, and semantic similarity. These goals are related, but not identical. In IR, a document representation does not necessarily need to encode all document content; it only needs to support matching with relevant queries. In principle, a retrieval representation could function more like an index key than a faithful semantic summary. In practice, however, modern embedding models often do encode substantial document content, and recent work suggests that information about the original text can mostly be recovered from embeddings (Morris et al. 2023).

Recent gains in embedding quality have come from several directions. One is improved training recipes and data construction. GTE ⁴⁵Li et al. (2023)Towards General Text Embeddings with Multi-stage Contrastive Learning trains unified text embeddings through multi-stage contrastive learning over diverse datasets. NV-Retriever ⁴⁶Moreira et al. (2024)NV-Retriever: Improving Text Embedding Models with Effective Hard-Negative Mining emphasizes the importance of hard-negative mining, especially positive-aware methods that reduce the risk of treating genuinely relevant passages as negatives during contrastive training. ReasonIR ⁴⁷Shao et al. (2025)ReasonIR: Training Retrievers for Reasoning Tasks trains retrievers on synthetic queries and hard negatives designed to teach relevance as usefulness for inference, rather than merely direct lexical overlap or surface-level semantic similarity.

A second major direction is the repurposing of large language models as embedding models. E5-Mistral ⁴⁸Wang et al. (2024)Improving Text Embeddings with Large Language Models shows that strong embedding models can be built by adapting LLMs and training them on large synthetic, instruction-style datasets. LLM2Vec ⁴⁹BehnamGhader et al. (2024)LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders converts decoder-only causal LLMs into bidirectional text encoders using bidirectional attention, masked next-token prediction, and contrastive learning. NV-Embed ⁵⁰Lee et al. (2024)NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models further introduces practical techniques such as latent-attention pooling, removal of the causal mask during contrastive training, and two-stage instruction tuning. More recent work also asks whether the bi-encoder paradigm itself can become more controllable. Promptriever ⁵¹Weller et al. (2025)Promptriever: Instruction-Trained Retrievers Can Be Prompted Like Language Models shows that dense retrievers can be prompted in natural language, while preserving the efficiency advantages of independently encoded queries and documents. Instead of treating the query as a fixed semantic object, Promptriever conditions retrieval on per-instance instructions that redefine what should count as relevant.

Expressive Matching Functions. Single-vector representation must compress all potentially relevant aspects of a document into one point in embedding space, and cosine similarity can express only a limited class of query–document relationships. The LIMIT benchmark ⁵²Weller et al. (2026)On the Theoretical Limitations of Embedding-Based Retrieval shows theoretically and empirically that single-vector embedding models cannot represent all possible top-$k$ relevance patterns unless the embedding dimension is sufficiently large; in practice, even strong embedding models fail on LIMIT, while lexical, multi-vector, and reranking approaches that leverage more expressive matching functions are less affected.

Cross-encoders ⁵³Nogueira & Cho (2019)Passage Re-ranking with BERT occupy the high-expressiveness end of this spectrum. Rather than encoding the query and document independently, they concatenate the query and candidate document and process them jointly with transformer self-attention. This allows every query token to attend to every document token, making the transformer itself a rich matching function. The cost is that documents cannot be precomputed as standalone vectors: each query–document pair must be scored separately, so these approaches are often used as a second reranking stage after first filtering with faster models.

LLM rerankers can be viewed as a compute-scaled version of the cross-encoder idea. Instead of using a relatively small supervised transformer reranker, an instruction-tuned LLM can be prompted to judge or rank candidate documents for a query. RankGPT ⁵⁴Sun et al. (2023)Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents showed that ChatGPT and GPT-4 can act as strong zero-shot rerankers, including in listwise settings where the model compares several candidates and outputs an ordering. This setup leverages both the expressive self-attention mechanism of large transformers and the broader linguistic, reasoning, and instruction-following priors acquired during LLM training. Recent work further explores whether reranking can benefit from explicit test-time computation. Rank1 ⁵⁵Weller et al. (2025)Rank1: Test-Time Compute for Reranking in Information Retrieval trains rerankers to produce reasoning traces before judging relevance, distilling traces from a stronger reasoning model into smaller reranking models.

Late-interaction models provide one way to increase expressiveness without moving all the way to full cross-encoder reranking. Instead of representing each query or document with a single vector, models such as ColBERT ⁵⁶Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT represent them as sets of contextualized token vectors. Documents can still be indexed in advance, but scoring is performed through token-level interactions, typically using a MaxSim operator that matches each query token against its best document-token counterpart.

Hypencoders ⁵⁷Killingback et al. (2025)Hypencoder: Hypernetworks for Information Retrieval propose another way to improve matching expressivity: the query encoder produces a small neural network that scores document representations, enabling query-specific relevance functions. Like late interaction, Hypencoder can be seen as an attempt to move beyond fixed cosine similarity while preserving some of the efficiency advantages of retrieval-time indexing.

Generative information retrieval (GenIR) departs more radically from standard matching-based retrieval: instead of comparing a query representation against indexed document representations, a generative model directly produces relevant document identifiers given a query, typically by having assimilated the corpus content in parametric memory ⁵⁸Tay et al. (2022)Transformer Memory as a Differentiable Search Index
Li et al. (2024)From Matching to Generation: A Survey on Generative Information Retrieval. While scaling GenIR to very large corpora remains difficult, it is an interesting concept because it exploits associative recall in a way standard retrieval models cannot.

Learned sparse retrieval. Sparse retrieval has also returned in neural form. Learned sparse models retain the efficiency and interpretability advantages of inverted indexes while learning term expansion and weighting from data. SPLADE ⁵⁹Formal et al. (2021)SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking is a representative example: it learns sparse lexical representations using explicit sparsity regularization and log-saturated term weights, making it suitable for first-stage retrieval with standard inverted-index infrastructure.

Recent work pushes sparse retrieval beyond surface vocabulary terms. Sparse-autoencoder-based retrieval models such as SPLARE ⁶⁰Formal et al. (2026)Learning Retrieval Models with Sparse Autoencoders learn sparse latent embeddings from LLM representations, aiming for more semantic and multilingual sparse features. The Latent Terms ⁶¹Clavié et al. (2026)Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies line of work similarly shows that dense retrievers can contain extractable sparse, BM25-ready latent features, suggesting that dense and sparse retrieval are not as separate as they first appear.

Context-aware document representations. Many retrieval-augmented generation systems split documents into chunks before embedding them, but this can remove the broader context needed to interpret each passage. Late Chunking ⁶²Günther et al. (2024)Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models addresses this by first encoding a longer text with a long-context embedding model and only then pooling token representations into chunk embeddings. As a result, each chunk vector can retain information from the surrounding document. In this line of thought, Contextual Document Embeddings ⁶³Morris & Rush (2024)Contextual Document Embeddings leverage priors on the corpus distribution when producing document embeddings to better express document specificity.

Modern Retrieval-Augmented Generation. Retrieval-Augmented Generation (RAG) (Lewis et al. 2020) reframes retrieval as a conditioning mechanism for language generation. Instead of returning only a ranked list of documents, a retriever selects a small set of passages, and a language model generates an answer conditioned on this external context.⁶⁴The original RAG paper introduced a trainable architecture that combined a pretrained seq2seq generator with a dense Wikipedia index, treating retrieved documents as latent variables marginalized during generation.(Lewis et al. 2020) Modern usage of the term is broader: RAG now often refers to a production pattern in which an instruction-tuned LLM is supplied with retrieved external context at prompt time, usually with additional components such as chunking, hybrid retrieval, reranking, citation generation, query rewriting, and sometimes multi-step agentic search. In practice, this changes the role of ranking: the goal is no longer merely to order documents for a human reader, but to select evidence that will fit within the model's context window and support a faithful synthesis. RAG in its current form has become one of the main industrial use cases of modern generative AI.

Agentic Search. Beyond generating from a fixed set of retrieved passages, modern LLMs can participate directly in the search process. WebGPT ⁶⁵Nakano et al. (2021)WebGPT: Browser-assisted Question-Answering with Human Feedback and ReAct ⁶⁶Yao et al. (2023)ReAct: Synergizing Reasoning and Acting in Language Models showed that language models can browse, call tools, inspect retrieved evidence, and decide which action to take next. This turns retrieval into an iterative loop: the model can decompose a task, issue searches, revise its query, and update its information need as evidence accumulates. IRCoT ⁶⁷Trivedi et al. (2023)Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions makes this loop explicit for multi-hop question answering by interleaving retrieval with chain-of-thought reasoning, while Self-RAG ⁶⁸Asai et al. (2024)Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection trains a model to decide when to retrieve and to critique both retrieved evidence and its own generations.

A related line of work questions the retrieval interface itself. Direct Corpus Interaction (DCI) ⁶⁹Li et al. (2026)Beyond Semantic Similarity: Rethinking Retrieval for Agentic Search via Direct Corpus Interaction argues that exposing a corpus only through a fixed top-$k$ retrieval API can be too restrictive for agents. Instead, an agent can interact directly with the raw corpus using general-purpose tools such as grep, file reads, shell commands, and lightweight scripts.

These new usage patterns change what retrievers should optimize for. When the retriever serves an agent rather than a human, the input is no longer just a standalone search query: it may include intermediate reasoning, subgoals, failed attempts, and partially discovered evidence. AgentIR ⁷⁰Chen et al. (2026)AgentIR: Reasoning-Aware Retrieval for Deep Research Agents formalizes this setting by treating deep-research agents as retrieval users. Instead of embedding only the final query, it jointly embeds the agent's reasoning trace and query, allowing retrieval to reflect the agent's evolving intent.

Despite all this progress, the core objective remains similar: finding relevant information hidden in the sum of human knowledge.