Personal Perspectives on Search

Vannevar Bush had the vision that searching using associative links would be the key insight to unlock new retrieval capabilities (Bush 1945). Time proved him right, well aided by the advent of computing. Statistical retrieval engines revolutionized fixed classification systems by linking information queries to documents through their words, not just their general topic. Deep neural networks further strengthened these associative links by moving beyond surface-level syntax and instead exploiting semantic proximity. The contributions of this thesis could be seen as further efforts to improve search by exploring the extent to which we can create associative links.

At a high level, with Visual Document Retrieval ¹Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Loison et al. (2026)ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
Teiletche et al. (2026)ModernVBERT: Towards Smaller Visual Document Retrievers, we demonstrate that cross-modal links can be leveraged to efficiently match query terms to image parts. In our work on contextual embeddings ²Conti et al. (2025)Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings, we show existing structural links between corpus documents can and should be exploited to facilitate retrieval. Through Large Language Model pretraining projects ³Faysse et al. (2025)CroissantLLM: A Truly Bilingual French-English Language Model
Meeus et al. (2024)Copyright Traps for Large Language Models, we explore parametric memory consolidation and study which types of knowledge can be surfaced (searched) from the LLM memory through prompting (associative recall). Associative links come in many forms. This thesis explores only a subset of them, but aims to show that broadening the types of links used for retrieval can open fruitful research directions. In particular, I believe that many promising directions remain underexplored beyond the beaten (but nevertheless useful) path of improving text embeddings primarily through larger contrastive datasets and larger models.

Future research directions in line with this vision are further detailed.

The Associative Memory Hierarchy Perspective

Viewing search solely through the lens of document retrieval is restrictive; a broader perspective is to understand search as memory access across a hierarchy of storage media. Humans answer questions multiple ways: by relying on internal memory, by attending to information currently in view (consulting notes), or by looking up information online or in a library. All of these memory systems differ in capacity, latency, reliability, update cost, and provenance. As mapped in Table 1, LLM systems exhibit an analogous hierarchy.

Model weights act as compressed parametric memory; the current context, together with the key-value cache, provides a transient working memory; and document stores, databases, tools, or the web provide external non-parametric memory (Table 1). ⁴This analogy is strengthened by recent work interpreting Transformers through the lens of associative memory. In particular, (Zhong et al. 2025) argues that both attention and feed-forward networks can be understood as memory systems, but with distinct retrieval and update properties. Attention constructs a dynamic associative memory over the current sequence, whereas feed-forward layers behave more like static learned associative stores.

Retrieval and Indexing. Under this view, standard document retrieval (for RAG use cases, for instance) is simply the promotion of information from external memory into working memory. Similarly, LLM prompting and autoregressive generation are the retrieval mechanisms that move information from parametric memory to working memory. Within the working memory, the attention operation performs in-context retrieval using learned associative links between query and key representations (which could be seen as indexes). To continue with the analogy, standard document indexing stores information from the working memory into external memory. Language Modeling through backpropagation inversely consolidates this information into the model weights, a sort of in-weights index triggered by future relevant activations.

This framework is useful in that it allows us to draw links between fields that today are mostly disjoint. My view is that some of the most interesting research problems in retrieval lie at their intersection. Work such as Transformer Memory as a Differentiable Search Index ⁵Tay et al. (2022)Transformer Memory as a Differentiable Search Index explicitly learns a corpus index in parametric memory. Instead of creating document embeddings through a model forward pass, works such as ⁶Eyuboglu et al. (2025)Cartridges: Lightweight and general-purpose long context representations via self-study
Kuratov et al. (2025)Cramming 1568 Tokens into a Single Vector and Back Again: Exploring the Limits of Embedding Space Capacity optimize the text representation through backpropagation, inspired by the prefix tuning approach ⁷Li & Liang (2021)Prefix-Tuning: Optimizing Continuous Prompts for Generation. Late Interaction ⁸Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT stems from the concept of better simulating the attention operation to allow for a matching operation more expressive than dot product similarity. Pushing this concept, Hypencoders ⁹Killingback et al. (2025)Hypencoder: Hypernetworks for Information Retrieval convert queries into neural networks designed to take document representations as inputs and output relevance scores; one more example of a clever research design in which the barrier between parametric and external memories are blurred, enabling downstream associative recall that transcends the possibilities given by cosine similarity.

Finally, modern agentic search systems achieve success thanks to their ability to leverage all forms of memory jointly (parametric, in-context, external). They combine strong parametric priors from the backbone LLM with tools to be able to navigate external memories (internet, corpora) and transfer information to their working memories where they can reason, all of this in a loop.

These systems can even be reinforced by blurring the distinction between in-context memory and external memory with works like Recursive Language Models ¹⁰Zhang et al. (2026)Recursive Language Models.

Breaking down artificial preconceptions and unifying the document retrieval field with the broader ideas from associative memory research is a powerful avenue for future improvement. Very concretely, long-context sparse attention mechanisms from million-token context windows are inches away from document retrieval over external memories: both fields study how to represent information and index it as dense embeddings, such that further associative recall is triggered given a relevant query. In Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility ¹¹Szilvasy et al. (2026)Self-Pruned Key-Value Attention: Learning When to Write by Predicting Future Utility, we devise ways to index information in a LLM KV Cache, but selectively learn when such information bears no signal and should not be persisted in memory. Such mechanisms would directly benefit most document retrieval systems.

Research Idea. Given more time, an idea I would have loved to explore is learning document embeddings solely through language modeling loss on unsupervised data. To do so, block sparse attention can be used to selectively include certain text sequences within the model training window, (softly) conditioned on the block's representation. Ideally, the block representation would be learned implicitly through soft gating techniques, similar to those used in our Self-Pruned KV paper. Blocks whose inclusion improves the language modeling objective would be assigned representations that make them more likely to be included, while unhelpful blocks would be filtered out. Importantly, block representations need not represent block content itself; they merely serve as an indexing mechanism. While it is unclear whether such an approach could outperform modern contrastive learning methods trained on supervised retrieval data, it would offer a different and potentially scalable route to document representation learning. In particular, it would make it possible to train retrieval-oriented representations on internet-scale corpora using only the language modeling objective, without relying on explicit supervision signals.

Towards Omnimodal Embeddings

Retrieval is moving from a predominantly text-centric paradigm toward an omnimodal one, in which useful evidence may be distributed across text, images, audio, video, code, structured data, and eventually even 3D assets. A first step in this direction was the rise of visual document retrieval: methods such as Document Screenshot Embedding (DSE) and ColPali showed that documents need not always be decomposed into OCR text and layout heuristics, but can instead be embedded directly from page images, preserving visual cues such as tables, figures, typography, and spatial organization ¹²Ma et al. (2024)Unifying Multimodal Retrieval via Document Screenshot Embedding
Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models. This line of work has since broadened from visual documents to richer forms of media. For instance, ColQwen-Omni ¹³Faysse et al. (2025)Introducing ColQwen-Omni: Retrieve in every modality extends the ColPali-style late-interaction retrieval paradigm toward audio and video on top of Qwen2.5-Omni ¹⁴Xu et al. (2025)Qwen2.5-Omni Technical Report, while models such as Omni-Embed-Nemotron, e5-omni, BiDirLM, and Gemini Embedding 2 explicitly target unified embedding spaces for heterogeneous modalities including text, image, audio, and video ¹⁵Xu et al. (2025)Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video
Chen et al. (2026)e5-omni: Explicit Cross-modal Alignment for Omni-modal Embeddings
Boizard et al. (2026)BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Shanbhogue et al. (2026)Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini.

A particularly important implication is that transfer across modalities can be positive rather than merely a source of interference. ModernVBERT provides an example in the visual-document setting: adding text-only query–document pairs to contrastive training improves visual document retrieval, suggesting that domain-specific supervision can sometimes be shared across modalities when the model has learned a sufficiently aligned representation space ¹⁶Teiletche et al. (2026)ModernVBERT: Towards Smaller Visual Document Retrievers. This points toward a more general principle for omnimodal search: useful retrieval behavior may be learned from one modality and reused in another, provided that the embedding geometry supports reliable cross-modal comparison.

As AI-powered search becomes more agentic, the competitive edge of a system will increasingly depend on its ability to access and exploit heterogeneous sources of evidence. Benchmarks such as BrowseComp and BrowseCompPlus already highlight the importance of persistence, iterative search, and variable test-time compute for finding hard-to-locate information on the web or in a corpus ¹⁷Wei et al. (2025)BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Chen et al. (2025)BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent. However, such agentic procedures still benefit from strong “first-layer'' retrieval models, which can cheaply select candidate evidence before more expensive reasoning or tool use is applied. Not all information is equally reachable: some sources remain costly to access, poorly indexed, private, ephemeral, or expressed in unfamiliar modalities. Exploiting such information at scale will therefore remain a central challenge and a quality differentiator.

Improving Search Scaffolding

Modern search systems are no longer just embedding models queried once against a static index. They are increasingly agentic systems that decompose questions, generate intermediate queries, retrieve candidates, read evidence, rerank sources, allocate context, synthesize answers, and decide when to stop. This makes the scaffold a central part of search performance. Benchmarks such as BrowseComp and BrowseComp-Plus make this visible: performance depends not only on the backbone LLM and retrieval model, but also on the procedure that controls iterative search and evidence use ¹⁸Wei et al. (2025)BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents
Chen et al. (2025)BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent. Similar lessons appear in work on ReAct, Self-Ask, IRCoT, HyDE, Search-R1, R1-Searcher, ReSearch, and Recursive Language Models, where changes in decomposition, query generation, tool use, or recursive evidence inspection can substantially improve retrieval-augmented reasoning without necessarily changing the underlying model ¹⁹Yao et al. (2022)ReAct: Synergizing Reasoning and Acting in Language Models
Press et al. (2022)Measuring and Narrowing the Compositionality Gap in Language Models
Trivedi et al. (2023)Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
Gao et al. (2023)Precise Zero-Shot Dense Retrieval without Relevance Labels
Jin et al. (2025)Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
Song et al. (2025)R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
Chen et al. (2025)ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning
Zhang et al. (2025)Recursive Language Models.

This suggests that scaffold engineering is currently one of the largest avenues for performance improvement in search. Improving the retriever or the backbone model often requires expensive training, large-scale data, or new infrastructure. By contrast, improving the scaffold can yield immediate gains. Creative ideas such as dumping model context in the filesystem and letting the model manipulate it ²⁰Zhang et al. (2026)Recursive Language Models can drastically improve performance in some tasks. Other benchmarks remain hardly solvable with current methods ²¹Tchuindjo et al. (2026)OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries and while perhaps not the most fun, I suspect smart scaffolding might be the most direct way forward.

As search becomes more agentic, evaluating systems rather than isolated models will become increasingly important.

Benchmarking Search Performance

After agentic systems and end-to-end evaluation, the next step forward in benchmarking is to veer away from benchmarks artificially constructed to be hard and multi-hop, such as BrowseComp (Wei et al. 2025) or OBLIQ-Bench ²²Tchuindjo et al. (2026)OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries, and instead evaluate search by its ability to improve decisions under uncertainty. Many practical search tasks are difficult to benchmark because relevance is underspecified: users search not merely to retrieve a document, but to update their beliefs and act.

Research Idea. Prediction markets offer a natural testbed for this idea. They provide dynamic, externally priced estimates about future events, such as elections, policy decisions, sports outcomes, company results, or breaking news. Since market prices aggregate dispersed information and are continuously updated, they can serve as a measurable proxy for the current public belief about an event ²³Hayek (1945)The Use of Knowledge in Society
Wolfers & Zitzewitz (2004)Prediction Markets
Wolfers & Zitzewitz (2006)Prediction Markets in Theory and Practice
Berg et al. (2008)Prediction Market Accuracy in the Long Run. A search system with genuine informational value should sometimes be able to improve on this baseline by finding deeper, fresher, or more decision-relevant evidence before it is fully reflected in the market.

Concretely, by holding the final decision-making model fixed and varying only the search system, each system could gather different sources about the same market, and the same downstream model would convert the gathered evidence into a calibrated forecast and trading decision. The edge over other market actors is reflected in the financial outcomes. The fact the market is dynamic and quickly patches information arbitrage weights the value of gathered information automatically. The evaluation target shifts from Did the system retrieve the annotated document? to Did the system retrieve information that updates priors (beliefs) in a way that is possible to leverage?

Given more flexibility, such environments could also become useful tests of source assessment, real-time reasoning, calibration, and more broadly general model intelligence in complex situations.

Search On

Intelligence is becoming a commodity, sold to consumers by trillion-dollar companies. Blindly trusting state-of-the-art LLM outputs is becoming an almost rational decision: as we improve modeling and search tooling, model responses are most often going to be correct and well grounded in verifiable sources.

Searching however, is often more than getting an exact answer to a specific question. To understand and truly value an answer, one must often first build prerequisite understanding, explore, reason, backtrack, synthesize... Other questions pop along the way. Bush says that we search by exploring the maze of human knowledge but to a curious mind, this twisted path is beautiful and interesting; it leads through side quests and down rabbit holes. Answers to why the brick road is yellow in the Wizard of Oz are surprisingly few Wikipedia links away from reading all about the Opium Wars.²⁴One of my favorite games I learnt from Professor Robert West consists in starting from a given Wikipedia article and navigating to another predetermined page using only in-article hyperlinks.

Searching is the gateway to knowledge, but the path stimulates curiosity and bootstraps creativity. Shiny search tools are invaluable to progress as a society, and search agents are clearly the future. Sometimes however, aimlessly digging through the disorganized pile of parchments in the library of Alexandria is still the best option. Search on!