Evaluating and Training Contextual Document Embeddings

Introduction

The ability to rapidly process and query large-scale textual corpora is a cornerstone of many industrial applications, ranging from the analysis of medical records and legal briefs to large-scale administrative archives. As these collections grow in size and complexity, advanced approaches to information retrieval (IR), particularly Retrieval-Augmented Generation (RAG) ¹Lewis et al. (2020)Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, have attracted widespread interest; yet dealing with long documents remains an open challenge.

While long context encoders have been recently developed ²Zhang et al. (2024)mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
Boizard et al. (2025)EuroBERT: Scaling Multilingual Encoders for European Languages along with long context embedding models ³Zhu et al. (2024)LongEmbed: Extending Embedding Models for Long Context Retrieval, modern document retrieval pipelines typically segment lengthy documents into smaller chunks to optimize the granularity for efficient retrieval and readability of the retrieved content ⁴Xu et al. (2024)Retrieval meets Long Context Large Language Models
Jiang et al. (2024)LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs. Traditionally, these chunks are then independently fed to an embedding model, and stored in a vector database for efficient future query matching. By doing so, these systems remove strong semantic and conceptual links between the split passages, directly affecting the resulting representations. An example is illustrated in Figure 2: embedding the sentence "He became emperor in 1804." without leveraging information about the person at hand (Napoléon) given in previous paragraphs will make matching queries related to Napoléon difficult.

Recognizing the significant business value of incorporating broader contextual information into retrieval, major companies have explored leveraging large generative language models (LLMs) to mitigate this limitation. Some approaches attempt to circumvent retrieval altogether by feeding millions of tokens into the model's context window at runtime ⁵Team et al. (2024)Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, while others reformulate individual passages by concatenating them with document-level summaries and context ⁶Anthropic (2024)Introducing Contextual Retrieval. However, these methods are prohibitively expensive at scale when dealing with corpora comprising thousands of documents.

Despite the critical importance of contextualized retrieval, standard benchmarks fail to capture this challenge. Evaluations traditionally focus on assessing the effectiveness of embedding models ⁷Thakur et al. (2021)BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark
Saad-Falcon et al. (2024)Benchmarking and Building Long-Context Retrieval Models with LoCo and M2-BERT, but they rely on datasets where document chunks are, by design, self-contained answers to the queries, which is a largely idealized scenario in practice ⁸Thakur et al. (2025)FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents. Consequently, benchmarks fail to highlight the limitations of current retrieval strategies in handling context-dependent passages. Worse, recent findings by ⁹Zhou et al. (2025)GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity? indicate that some widely-used benchmarks exhibit biases that favor standard context-agnostic retrieval methods. Companies such as Anthropic have acknowledged these issues and maintain proprietary contextual retrieval benchmarks that remain unavailable to the public¹⁰https://www.anthropic.com/news/contextual-retrieval, underscoring the gap between academic evaluations and real-world industrial needs.

Contribution 1: ConTEB. We introduce the Context-aware Text Embedding Benchmark, designed to assess the ability of retrieval systems to leverage information from the entire document when indexing and retrieving document chunks. ConTEB comprises both custom-designed tasks for fine-grained analysis, and practical retrieval evaluation settings spanning multiple document types, domains, and situations in which leveraging context is helpful to produce more meaningful chunk representations. We evaluate standard embedding methods on the benchmark and find they struggle when contextual awareness is required, while highlighting the promising contextual capabilities of approaches such as Late Chunking ¹¹Günther et al. (2024)Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models.

Contribution 2: Efficient Contextual Training. Improving upon the Late Chunking method ¹²Günther et al. (2024)Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models, we propose a novel embedding post-training method that optimizes information propagation between same-document chunks at indexing time to ensure embeddings are better contextualized. Our method largely boosts performance on ConTEB with minimal computational overhead. Through extensive ablations, we detail critical design choices and show our method displays increased robustness to sub-optimal chunking strategies and produces representations that scale better with corpus size.

We open-source all project artifacts, including the benchmark, models and training data¹³ https://github.com/illuin-tech/contextual-embeddings.

Problem Formulation & Related Work

Retrieval Frameworks

In this paper, we consider the traditional retrieval framework where a retrieval system given a query $q$, searches a corpus $\mathcal{D}$ for relevant documents. Each document $d \in \mathcal{D}$ is scored based on its content by first embedding the text into a vector space, and then computing a similarity measure. The similarity between a query $q$ and a document $d$ is defined as

where $\phi$ maps text into an $n$-dimensional vector space and $f: \mathbb{R}^n \times \mathbb{R}^n \to \mathbb{R}$ is a similarity function, such as cosine similarity or dot product.

In applied settings, individual documents are often too long to be practical for retrieval purposes ¹⁴Liu (2022)LlamaIndex
Zhong et al. (2025)Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation. Each document $d$ is thus divided into segments called chunks by a partitioning function $\mathcal{P}$ defined as

In the standard retrieval setting, the score is computed solely based on chunk content:

Additional information (priors) is however often available to the document embedding system. Typically, knowledge of the entire corpus $\mathcal{D}$, or of structural metadata $M_c$ such as neighboring document chunks obtained through $\mathcal{P}$, can be leveraged by a modified embedding function $\phi_2$, yielding the following similarity score:

This work is centered on efficiently integrating priors about the entire document when embedding a sub-document chunk.

Integrating Contextual Information

Neural embedding models for passage-level text representation, popularized by SentenceBERT ¹⁵Reimers & Gurevych (2019)Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, have enabled retrieval systems to move beyond lexical matching ¹⁶Robertson et al. (1994)Okapi at TREC-3. To include contextual information in these retrievers, previous works proposed methods that either operate offline during indexing, or online during querying when faced with a user request.

Indexing. The chunking strategy is a crucial design choice and often aims to optimize chunk self-containment. Fixed-size approaches with overlaps preserve continuity, while structure-aware chunking respects natural text boundaries, such as paragraphs or sentences. Semantic chunking, by contrast, splits text into topic-aligned segments. These methods appear in frameworks such as LlamaIndex ¹⁷Liu (2022)LlamaIndex and LangChain ¹⁸Chase (2022)LangChain, but different queries may need different chunk sizes. Thus, dynamic chunking techniques have emerged to adapt segmentation on the fly ¹⁹Zhong et al. (2025)Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation
Qian et al. (2024)Grounding Language Model with Chunking-Free In-Context Retrieval. Beyond optimizing chunking, some indexing approaches enrich chunks with broader context by prepending LLM-generated document summaries, contextual information or metadata ²⁰Anthropic (2024)Introducing Contextual Retrieval
Poliakov & Shvai (2024)Multi-Meta-RAG: Improving RAG for Multi-Hop Queries using Database Filtering with LLM-Extracted Metadata. Similarly, ²¹Morris & Rush (2024)Contextual Document Embeddings demonstrate that appending learned "corpus" embeddings to queries and documents can further improve retrieval. Other indexing-time techniques involve organizing chunks into higher-level data structures. For example, ²²Edge et al. (2024)From Local to Global: A Graph RAG Approach to Query-Focused Summarization and ²³Sarthi et al. (2024)RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval cluster related chunks into semantic graphs or tree hierarchies.

Querying. In contrast, query-time solutions rely on iterative or agentic loops to refine retrieval dynamically. LLMs can be used to iteratively update the query or request additional chunks based on partial results ²⁴Xiong et al. (2021)Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval
Trivedi et al. (2023)Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions, or even to run “self-checks” and seek extra context when needed ²⁵Asai et al. (2023)Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection. While these adaptive techniques can better address complex, multi-hop queries, they typically require much more computational resources during inference.

ConTEB: Context-aware Text Embedding Benchmark

Benchmark Design

Existing benchmarks often rely on (or assume) self-contained document chunks. This creates a misleading perception that contextualization offers little to no benefit, which in practice is rarely the case. To address this gap, the ConTEB benchmark philosophy is to explicitly be composed of tasks in which leveraging document-wide context should lead to performance improvements. Our benchmark originates from two sources: new datasets specifically created for ConTEB, and repurposed academic datasets. We take special care in selecting data sources spanning from multiple domains, including realistic industrial scenarios.

Why Context? Context can help resolve ambiguity, such as distinguishing between multiple meanings of a word or resolving pronouns and entity references (co-reference resolution). It is crucial when documents have a structured format, like legal or scientific texts, where understanding table of content hierarchy is key to aid intra-document disambiguation. Inversely, document-level contextual information is key when querying a corpus of documents that follow a strongly similar structure, such as annual company reports, to enable cross-document disambiguation.

Concept. To isolate the importance of contextual cues and diminish other confounding factors, we construct three benchmark tasks to study contextualization in controlled experimental settings ²⁶Allen-Zhu (2024)ICML 2024 Tutorial: Physics of Language Models.

We also evaluate more practical retrieval settings at larger scale where we suspect contextualization to help, and in which we rely on organic, pre-existing query-document pairs.

Benchmark Construction

The generic dataset curation pipeline is depicted in Figure 3. This three-stage process allows us to obtain queries linked to chunks belonging to long documents that contain context relevant to each chunk. We apply this pipeline to each of our data sources, with minor adjustments depending on the existing data at hand, that we detail in Section F.3.

1: Chunking. We select long documents spanning a variety of domains and chunk them through a structure-aware method²⁷RecursiveCharacterSplitter with a threshold of 1000 characters (Chase 2022) ²⁸Rajpurkar et al. (2016)SQuAD: 100,000+ Questions for Machine Comprehension of Text
Möller et al. (2020)COVID-QA: A Question Answering Dataset for COVID-19
Kočiský et al. (2017)The NarrativeQA Reading Comprehension Challenge
Chen et al. (2024)BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
Macé et al. (2026)ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval.

2: Pairing.

We use manual answer span annotations (SQuAD, ESG) or synthetically label them with an LLM (CovidQA, MLDR, NarrativeQA), to match queries with chunks obtained in Stage 1. This ensures queries are not solvable by design ²⁹Thakur et al. (2025)FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents. Alternatively, in our controlled experiment tasks, we generate queries pertaining to the chunks manually (Insurance) or synthetically using LLMs (Football, Geography).

3: Manual Verification & Modification.

The manually crafted questions in Insurance are designed to be ambiguous without prior knowledge of the document structure. This is manually verified in this phase. Going a step further, in Football and Geography, we reformulate chunks with the help of an LLM to remove explicit mentions of the original document's theme which all queries mention. We do so in all but the first chunks of each document, explicitly enforcing the need for context.

In addition to our contextual scenarios, we use NanoBEIR ³⁰Thakur et al. (2021)BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models to evaluate non-regression on standard non-contextualized embedding tasks.

By combining hard tasks in controlled environments, repurposed academic benchmarks, and real-world industrial queries, our benchmark provides a comprehensive assessment of retrieval models in both standard and context-dependent retrieval scenarios.

Training Dataset

Open training data is a key factor to ensure fair comparisons across methods and robust conclusions. In addition to our benchmark, we construct and release a training dataset composed of query and document chunk pairs. It includes the training splits of MLDR and NarrativeQA, repurposed with our previously detailed pipeline. To increase the number of queries, we further use GPT-4o to generate relevant supplementary synthetic queries. We also concatenate SQuAD chunks from the same Wikipedia article, keeping track of the original question-passage associations. The full dataset contains 9881 unique long documents (3698 tokens on average), corresponding to a total of 232'587 chunks and 307'241 queries (see Subsection F.3.6). Scaling the dataset to more sources, through diverse synthetic augmentations and refinement–based augmentation methods ³¹Lee et al. (2024)NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
Wang et al. (2024)Improving Text Embeddings with Large Language Models is left for future work.

Baselines

Training-Free. We evaluate a selection of off-the-shelf methods that are strong in their size categories such as a standard single-vector embedding model based on ModernBERT (modernbert-embed-large (Warner et al. 2024; Chaffin 2025)), its multi-vector ColBERT equivalent ³²Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Chaffin (2025)GTE-ModernColBERT and Okapi BM25 ³³Robertson et al. (1994)Okapi at TREC-3, a strong lexical matching method. Additionally, we compare against various contextualization approaches. Specifically, we include Anthropic’s contextual retrieval approach ³⁴Anthropic (2024)Introducing Contextual Retrieval³⁵We use Qwen-2.5-7B-Instruct as the generative model which we serve on a 80GB A100 GPU with vLLM and modernbert-embed-large as the embedding model, and evaluate Late Chunking ³⁶Günther et al. (2024)Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models without specific fine-tuning using modernbert-embed-large. These methods cover standard practices with varying levels of complexity and indexing budgets.³⁷We also evaluate RAPTOR (Sarthi et al. 2024) with Qwen-2.5-7B-Instruct and cde-small-v2 (Morris & Rush 2024) but find them to be poorly adapted to our problem settings.

Training-Based. For fair evaluation, we also fine-tune the sentence embedding method modernbert-embed-large on the training dataset with the same batch construction strategy as when training our main method, ensuring performance differences only stem from methodological design.

Training Contextual Embedders

In this work, we leverage recent advances in long-context embedding models ³⁸Zhang et al. (2024)mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference to improve upon existing approaches through novel training strategies.

Architecture

Late Chunking. Late Chunking ³⁹Günther et al. (2024)Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models (LC) is a training-free token pooling technique designed to enable information propagation across same-document chunks. Formally, given a document $d$ split into chunks $\{c_1, \dots, c_{N_d}\}$, dense retrievers compute independent representations:

In Late Chunking, chunks are concatenated and the whole sequence representation is computed in a single-forward pass:

where $H = [h_1, h_2, \dots, h_T]$ consists of token-level representations. We then apply average pooling within each original chunk to obtain chunk-wise representations:

This allows each chunk representation to benefit from contextualization over the full document before aggregation.

Late Interaction. Late Interaction (LI) models ⁴⁰Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Chen et al. (2024)BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation are retrieval methods that do not pool token representations and instead store all token embeddings of each document. This approach boosts performance, especially on long-context retrieval tasks ⁴¹Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
Zhu et al. (2024)LongEmbed: Extending Embedding Models for Long Context Retrieval, at the expense of storage cost. In this work, we propose extending Late Chunking approaches to LI models by applying standard LC but simply forgoing the final pooling and storing token embeddings depending on their original chunk memberships.

Setup.

As the base single-vector embedding model for our experiments, we use modernbert-embed-large ⁴²Chaffin (2025)ModernBERT-embed-large (396M parameters), which is fine-tuned for retrieval tasks using the method from (Nussbaum et al. 2024).

Respectively, we leverage GTE-ModernColBERT ⁴³Chaffin (2025)GTE-ModernColBERT (149M parameters) for our late interaction experiments. Both models are based on ModernBERT ⁴⁴Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference which supports a context length of up to 8,192 tokens, significantly surpassing the 512-token limit of traditional BERT models, and thereby enabling the processing of longer documents in a memory efficient manner, which is critical to our method.

Learning Objective

Late Chunking enables information "leakage" between chunks of the same document. While this training-free method showed promises, we construct a learning objective to explicitly optimize contextual embedding models for this setting. Our aim is twofold: optimizing chunk representations to integrate relevant document-level information, all while ensuring they retain their specificity with respect to other same-document chunks, in order to prevent embedding collapse.

Previous works ⁴⁵Karpukhin et al. (2020)Dense Passage Retrieval for Open-Domain Question Answering
Ni et al. (2021)Large Dual Encoders Are Generalizable Retrievers
Izacard et al. (2021)Unsupervised Dense Information Retrieval with Contrastive Learning
Li et al. (2023)Towards General Text Embeddings with Multi-stage Contrastive Learning
Wang et al. (2022)Text Embeddings by Weakly-Supervised Contrastive Pre-training
Nussbaum et al. (2024)Nomic Embed: Training a Reproducible Long Context Text Embedder have relied on various learning objectives inspired by the contrastive learning literature ⁴⁶Schroff et al. (2015)FaceNet: A Unified Embedding for Face Recognition and Clustering. A natural choice is the InfoNCE objective ⁴⁷Oord et al. (2018)Representation Learning with Contrastive Predictive Coding, which samples "negative" embeddings from other documents of the same batch.

In our approach, we combine these negatives with an auxiliary in-sequence contrastive loss, where chunks originating from the same document as the positive serve as hard negatives during training. Intuitively, training Late Chunking models contrastively with chunks from different documents encourages information propagation within each document and improves document identification. On the other hand, the contrastive term between same-document chunks ensures each chunk retains its specificity, and remains identifiable w.r.t. its neighbors. This aspect is further motivated by the fact that in practice, queried corpora often contain negative documents stemming from the same source. Figure 2 illustrates chunk roles across a training batch.

Training Loss. To balance the contribution of in-sequence and in-batch negatives, we define the weighted InfoNCE loss as:

where $\lambda_{\text{seq}} \in [0,1]$. Loss terms are defined as:

Here, $q$ denotes the query representation, and $k^+$ is the gold chunk representation, which belongs to $\mathcal{N}_{\text{seq}}$, the set of chunks from the same sequence as $k^+$. Temperature $\tau > 0$, and $\mathcal{N}_{\text{batch}}$ is the set of all in-batch samples that do not belong to $\mathcal{N}_{\text{seq}}$. This extends to late interaction models by replacing the dot product between query and chunk embeddings by ColBERT's MaxSim between the multiple query and document token embeddings.

By tuning $\lambda_{\text{seq}}$, we can adjust the relative importance of in-sequence versus in-batch contrastive learning (Figure 4) resulting in our InSeNT method.

Model training

Our training strategy (InSeNT) is designed to be lightweight and to occur on top of capable pre-trained embedding models without degrading their capabilities.

We use AdamW, a cosine decay learning rate scheduler with a 5% warm-up phase and a learning rate of $5e-5$ and train for 2 epochs on our training dataset. Batches are constructed by sampling 4 long documents per device, retrieving all corresponding chunks and concatenating them with a separator token in between. As documents in our training set contain more than 20 chunks on average, which are themselves often linked to one or multiple queries, a batch contains more than 100 query-positive-negative triplets to learn on.⁴⁸In MB+Training, data is sampled the same way for fair evaluation but flattened in batch, corresponding to per-device batch sizes of more than 100. A single epoch takes less than 1 H100 GPU hour.

Results

Document-wide context is essential. As seen in Table 2, methods leveraging contextual information widely outperform non-contextual methods across ConTEB tasks. These results highlight the critical role of context-aware embeddings in improving retrieval performance in such settings, whether through untrained Late Chunking approaches or expensive context-aware reformulation approaches. As expected, the gap is even more notable in ConTEB's controlled setting experiments.

Improving contextual information propagation. Our results clearly show that InSeNT variants outperform their untrained counterpart (+14.6 nDCG@10 for ModernBERT, +11.5 for ModernColBERT). Importantly, this is not due to the nature of the training data itself; the non-contextual ModernBERT model trained on the same data (ModernBERT + Training) does not improve upon the untrained baseline. Furthermore, the tasks that display the biggest improvements are the controlled setting tasks Insurance, Football, that are explicitly designed to elicit information given in previous paragraphs, and that are out-of-domain w.r.t. our training set.

Late Interaction. Interestingly, while LI models are good at long-context retrieving, they are poorly suited to out-of-the-box late chunking (-0.3 nDCG@10 w.r.t. ModernColBERT without LI). We posit that since token embeddings are never pooled, these models learn very local features and cannot leverage information from neighboring tokens. Once trained with our method, ModernColBERT+InSeNT displays large performance gains across the board (+11.5 nDCG@10 w.r.t. ModernColBERT + Late Chunking), showcasing an increased ability to leverage external context.

Context can add noise. The CovidQA task sticks out from the rest as untrained late chunking approaches severely degrade performance. Qualitative analysis, as well as the strong performance of the non-contextualized ModernColBERT method, indicate that the query-chunk pairing are often very extractive and match on technical medical terms, thus rendering context less useful. Our results show that naively applying late chunking in this setting adds noise and leads to notable performance drops (-21 nDCG@10), which are in large part recovered through our training method (+16 nDCG@10).

$\lambda_{seq}$ matters. The training objectives are to induce chunk representations to integrate document-level information (role of in-batch negatives) while maintaining their specificity with respect to other same document chunks (role of in-sequence negatives). By varying $\lambda_{seq}$ from Equation 9, we weight the importance of both objectives.

After training a series of models with varying $\lambda_{seq}$, we see on Figure 4 that training with only in-sequence or in-batch negatives yields the worse results, and the optimal $\lambda_{seq}$ varies depending on the task. When documents need to be disambiguated between one another (NanoBEIR, Geography), up-weighting in-batch negatives seems optimal. On tasks where the challenge lies in locating information within a given document (NarrativeQA, Covid-QA), in-sequence negatives play a large role, but still need to be combined to in-batch negatives. Striking the optimal trade-off is thus very use-case dependent, and we opt for $\lambda_{seq}=0.1$ after tuning on the validation split of our training dataset.

Efficiency-Performance. As shown in the Runtime column of Table 2, our approach is very capable on contextual tasks, yet does not add much computational overhead. In fact, we find slight indexing speed improvements, attributed to our approach's reduced need for padding in-batch sequences of different lengths. While Anthropic Contextual achieves similar performance on ConTEB, it relies on costly LLM-based summarization and chunk reformulation that are hardly scalable to huge corpora (120x slower).

Short-Context Performance. Careful hyperparameter tuning enables our best model to maintain strong performance on standard non-contextual benchmarks (NanoBEIR), demonstrating that long-context optimization does not compromise short-context retrieval. Interestingly, LI models suffer from more degradation, which we posit is due to the original reliance on very local features modified through our training. Mixing in non-contextual "replay" data during training or merging models ⁴⁹Li et al. (2024)Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Merging
Wang et al. (2025)LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging should further enable preserving the original embedding model's performance.

Ablations

Robustness to chunking. We assess our method's robustness to poor chunking strategies using SQuAD annotations. Each originally self-contained chunk is split into multiple progressively smaller sub-chunks while we keep track of the annotated answer span to identify the gold chunk. Eventually, these sub-chunks become too small to be self-contained and end up lacking sufficient information to be relevantly embedded on their own. Figure 5 (left) demonstrates that contextual embeddings greatly improve robustness w.r.t. suboptimal chunking. The model is able to elicit information from neighboring chunks to integrate contextual information within smaller sub-chunks, leading to a much more uniform retrieval performance across a wide range of chunk sizes.

Robustness to corpus size. Common in the industry are templated documents that differ mostly by a key aspect (year, company name) but contain otherwise very similar information. We study the dynamics of retrieval performance w.r.t. the amount of similar documents in the corpus by computing scaling laws in which we iteratively vary the number of unique documents (composed of multiple chunks) in the corpus. We observe in Figure 5 (right) that contextual embeddings scale vastly differently than their independently embedded counterpart. Intuitively, the greater the amount of similar documents and chunks in the corpus, the harder it is for a retrieval system to match the correct ones, but when embedding models are able to leverage external context, this effect is attenuated.

Information Propagation. We experiment with concatenating semantically similar yet independent short chunks as "artificial" long documents. The resulting model is contextual as it uses late chunking, but exhibits performance in line with non-contextual baselines (ModernBERT Large + Training). We posit training on arbitrarily concatenated chunks, which by design are not contextually linked, teaches the model not to use information from neighboring chunks. This highlights the necessity of sourcing organic long-context data during training to induce correct training dynamics. Details in Table 2 in Section F.5.

Conclusions

In this work, we introduced ConTEB, a benchmark designed to assess the effectiveness of retrieval models in leveraging document-wide contextual information. Our evaluation demonstrates that standard retrieval models struggle in context-dependent settings, while our proposed approach InSeNT, which combines Late Chunking and a novel training methodology, performs strongly on ConTEB without additional compute costs. These insights build towards the broader development of the Late Chunking paradigm in practice, and highlight the everlasting need for evaluation benchmarks that rigorously reflect how embedding models are used in real-world scenarios.

Future Work. Scaling our approach with recent decoder models with extended context lengths (e.g., 1M+ tokens ⁵⁰Yang et al. (2025)Qwen2.5-1M Technical Report) would enable embedding entire books or lengthy documents in a single forward pass, potentially unlocking new capabilities for large-scale document retrieval. It would also be interesting to observe the impact of our method on retrieval confidence ⁵¹Gisserot-Boukhlef et al. (2024)Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism. Finally, adapting our method to multi-modal embedding pipelines that have less control over the chunking strategy could further enhance retrieval systems in industrial applications with visually rich contextual documents ⁵²Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Ma et al. (2024)Unifying Multimodal Retrieval via Document Screenshot Embedding.

Limitations

While our approach enhances retrieval performance in context-dependent settings, limitations persist.

Context Length. Our method is applied to long-context encoders that currently support sequences of up to 8k tokens. While we have shown performance can extrapolate to sequences of up to 32k tokens, scaling this approach to handle 1M+ token contexts with decoder-based models would be an interesting research avenue and presents significant compute and memory challenges. It notably requires rethinking the data construction processes to ensure longer documents are effectively leveraged.

Non-contextual Performance. While our approach unlocks previously unattainable performance in contextual scenarios, it can come at the cost of slight short-context performance degradation. The optimal trade-off between non-contextual and contextual retrieval performance is highly usecase dependent Figure 4 and can be parametrized by practitioners using the $\lambda_{seq}$ parameter. Other approaches may be promising such as model merging ⁵³Li et al. (2024)Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Merging
Wang et al. (2025)LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Merging.

Data Generation. The creation of training and evaluation data relies on existing datasets and semi-synthetic generation pipelines. However, a fully automated and scalable method for generating high-quality queries that effectively induce non-trivial context utilization remains an open challenge.

Evaluation. While our model demonstrates strong cross-domain performance, further validation in real-world applications, various use cases, and multiple languages is necessary to further assess its robustness and generalizability.