ConTEB Appendix

Ethical Considerations

Bias. As our method introduces a novel way of leveraging document-wide context, the nature of information propagation between chunks remains uncertain. This may introduce biases that traditional embedding models do not encounter, necessitating further analysis.

Ecological Impact. Our post-training approach is computationally efficient, with total training and evaluation runs requiring fewer than 100 GPU hours on H100 hardware. By providing a cost-effective alternative to LLM-dependent contextualization techniques, we aim to reduce the environmental footprint of large-scale retrieval systems.

Social Impact. Improved retrieval capabilities can drive significant business benefits, particularly in industries that rely on processing extensive and structured documents, such as legal, medical, and financial sectors.

Acknowledgements

This work is partially supported by Illuin Technology, and by a grant from ANRT France. This work was performed using HPC resources from IDRIS through Grant AD011016393 and AD011015443 .

Antoine Bosselut gratefully acknowledges the support of the Swiss National Science Foundation (No. 215390), Innosuisse (PFFS-21-29), the EPFL Center for Imaging, Sony Group Corporation, and a Meta LLM Evaluation Research Grant.

ConTEB Details

This appendix describes the data generation process employed in this project. The methodology varies based on the dataset source, but generally, long documents are segmented into smaller chunks. If preexisting queries are available, they are mapped to relevant chunks using either provided answer spans (e.g., SQuAD) or tagged using GPT-4o. In cases where queries are unavailable, a large language model (LLM) generates them before associating them with the relevant text segments. This approach, illustrated in 3, is systematically applied across multiple datasets.

Wiki-based Datasets

Football and Geography are our two wiki-based datasets, focusing on the Sports and Geography domains.

Wikipedia Data Extraction The pipeline first retrieves Wikipedia summaries for a given person using the wikipediaapi library. The extracted summary is then split into paragraphs.

Text Rephrasing Each paragraph from the Wikipedia summary undergoes a rephrasing process to remove direct mentions of the person’s name while maintaining the original context. The rephrased text replaces names with pronouns such as ‘he’ or ‘she’. This transformation is performed using the GPT-4o model via the following prompt:

Here is a Wikipedia article: [Full Wikipedia Summary] Can you rephrase the following paragraph to remove all mention of the name of the person the article is about? You can leave other names as is and can replace the name with words such as 'he/she' or other generic paraphrases. [Paragraph to be rephrased]

Question Generation For each paragraph in the summary, the model generates three questions related to the person. The questions explicitly mention the person's name but do not include other named entities such as dates or proper nouns. The generation follows this structured prompt:

Here is a Wikipedia article:

[Full Wikipedia Summary]

Using specifically the following paragraph, can you ask 3 questions related to the person the article is about? Each question must mention the name of the person, but the question should not contain other named entities (dates, other proper nouns). Format the response as a Python list of strings and do not output anything else.

[Paragraph to be used for question generation]

NarrativeQA, COVID-QA, MLDR

NarrativeQA (literature), MLDR (encyclopedic) and Covid-QA (medical) consist of long documents, associated to existing sets of question-answer pairs.

We chunk these documents, and use GPT-4o to annotate which chunk, among the gold document, best contains information needed to answer the query. Since chunking is done a posteriori without considering the questions, chunks are not always self-contained and eliciting document-wide context can help build meaningful representations.

Synthetic Query Generation: To extend MLDR for our training dataset, OpenAI's GPT-4o model is prompted to generate 20-50 realistic queries per document, ensuring that each query aligns with the content of at least one chunk. This is on top of the queries that are already incuded in the dataset. Synthetic queries are included only in our training dataset.

Insurance

Insurance is composed of a long document with insurance-related statistics for each country of the European Union. Countries are often not referred to in-text, but only once in the section title. Therefore, certain chunks require knowledge of their position within the document to be properly disambiguated from others. Questions are manually crafted to require structural understanding for accurate chunk matching. This process, in addition to manual verification of the contextuality quality, makes Insurance a controlled dataset. Since questions are crafted after the chunking process, the annotation results directly from the manual question generation process.

SQuAD

SQuAD is an extractive QA dataset with questions associated to passages and annotated answer spans, that allow us to chunk individual passages into shorter sequences while preserving the original annotation.

ESG Reports

ESG Reports contains long documents from the fast-food industry, with manually annotated query-page pairs from the ViDoRe Benchmark v2 ¹Macé et al. (2026)ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval, originally thought for visual retrieving²https://huggingface.co/datasets/vidore/restaurant_esg_reports_beir. We convert all documents to text, chunk them, and re-annotate the resulting passages by hand, filtering out queries that relied solely on visual aspects (e.g., tables, graphs).

Training Data Statistics

Table 1 displays information about the training data. Our refined version of MLDR forms a large part of the training corpus. We can see that the majority of chunks are used as positives at least once, ensuring that the model is not biased towards the position of the chunk in the sequence.

Implementation Details

Sequence prefixes

ModernBERT-based models are trained with query and document prefixes. We apply the same approach in our training and inference frameworks. After several tests, we opt for using a single document prefix for the Late Chunking sequence, instead of adding a document prefix at the beginning of each chunk inside the same sequence. We separate chunks with [SEP] tokens to let the model understand the concept of chunks during its token embedding computation.

Late Interaction Models

We leverage the pylate ³Chaffin & Sourty (2024)PyLate: Flexible Training and Retrieval for Late Interaction Models library for the Late Interaction implementation. For training LI models with InSeNT, we adapt the LI mechanisms to incorporate it with Late Chunking in our own codebase. In particular, we do not use token skiplists at inference time, and use a single document prefix for the whole document sequence.

Additional Results

Training with concatenated short documents

Results of training an InSeNT model with concatenated short document data (using the Nomic dataset) are available in Table 2. Short docs are clustered from the nomic-supervised dataset (Nussbaum et al. 2024) following ⁴Morris & Rush (2024)Contextual Document Embeddings. This approach did not yield promising results, proving that natively long documents are necessary to induce relevant in-sequence signal.

Full ablation results on $\lambda_{seq}$

We show the results of the different values for $\lambda_{seq}$ on all our evaluation sets.

Extending context beyond 8192 tokens

ModernBERT was trained on documents of up to 8192 tokens ⁵Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. Its Late Interaction counterpart, GTE-ModernColBERT, was exclusively fine-tuned on documents of no more than 300 tokens. However, its generalization capabilities to longer documents have been shown by its developers ⁶Chaffin (2025)GTE-ModernColBERT, hinting at the fact that further research along those lines could be tried for both the bi-encoder and the LI variants.

Based on these results, we tried two approaches to handle documents longer than 8192 tokens with ModernBERT (necessary for the ESG reports dataset): computing Late Chunking with a context of max. 8192 tokens in an sliding window fashion (computing chunk embeddings in several forward passes of 8192 tokens, with 10 overlapping chunks between the various windows), and naively feeding the complete documents to the embedder.

To our surprise, the latter worked better by a large margin (43.1 on ESG as reported in 4, vs 25.4 for the sliding window approach), so we reported the results of this approach. Further studies could be led to better understand the dynamics underlying this extension.