ColPali: Efficient Document Retrieval with Vision Language Models

Introduction

Document Retrieval consists of matching a user query to relevant documents in a given corpus. It is central to many widespread industrial applications, either as a standalone ranking system (search engines) or as part of more complex information extraction or Retrieval Augmented Generation (RAG) pipelines.

Over recent years, pretrained language models have enabled large improvements in text embedding models. In practical industrial settings, however, the primary performance bottleneck for efficient document retrieval stems not from embedding model performance but from the prior data ingestion pipeline. Indexing a standard PDF document involves several steps. First, PDF parsers or Optical Character Recognition (OCR) systems are used to extract words from the pages. Document layout detection models can then be run to segment paragraphs, titles, and other page objects such as tables, figures, and headers. A chunking strategy is then defined to group text passages with some semantical coherence, and modern retrieval setups may even integrate a captioning step to describe visually rich elements in a natural language form, more suitable for embedding models.

In our experiments (Table 2), we typically find that optimizing the ingestion pipeline yields much better performance on visually rich document retrieval than optimizing the text embedding model.

Contribution 1: ViDoRe. In this work, we argue that document retrieval systems should not be evaluated solely on the capabilities of text embedding models ¹Bajaj et al. (2016)MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
Thakur et al. (2021)BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark, but should also consider the context and visual elements of the documents to be retrieved. To this end, we create and openly release ViDoRe, a comprehensive benchmark to evaluate systems on page-level document retrieval with a wide coverage of domains, visual elements, and languages. ViDoRe addresses practical document retrieval scenarios, where queries often necessitate both textual and visual understanding for accurate document matching. We highlight the shortcomings of current text-centric systems in these settings.²The ViDoRe benchmark leaderboard is hosted publicly at https://huggingface.co/spaces/vidore/vidore-leaderboard to encourage further developments.

Contribution 2: ColPali. We propose a novel concept and model architecture based on Vision Language Models (VLMs) to efficiently index documents purely from their visual features, allowing for subsequent fast query matching with late interaction mechanisms ³Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Our method, ColPali, significantly outperforms all other retrieval systems on ViDoRe while being fast and end-to-end trainable. These results demonstrate the potential and the many benefits of this novel Retrieval in Vision Space concept, which could significantly alter the way document retrieval is approached in the industry moving forward. We release all resources at https://hf.co/vidore.

Problem Formulation & Related Work

Problem Setting. In our setting, a retrieval system scores how relevant a document $d$ from corpus $\mathcal{D}$ is with respect to a query $q$. Computing the similarity score $s(q, d) \in \mathbb{R}$ for each of the $|\mathcal{D}|$ documents in the corpus creates a ranking we can use to extract the most relevant documents. In this work, we focus on page-level retrieval: given a query, is the correct document page retrieved by the system? For coherence with existing literature, we further use the term document to refer to individual pages, i.e. the atomic retrieved elements in our setting. As we focus on practical industrial retrieval applications (RAG, search engines) with potentially large corpora sizes, latency constraints are imposed on scoring systems. Most current retrieval systems can be decomposed into (1) an offline indexation phase in which a document index is built and (2) an online querying phase in which a query is matched to documents from the index and where low latency is vital to the user experience.

Under these industrial constraints, we identify three main properties an efficient document retrieval system should exhibit: (R1) strong retrieval performance, as measured by standard retrieval metrics; (R2) fast online querying, measured through average latencies; and (R3) high-throughput corpus indexation, i.e. the number of pages that can be embedded in a given time frame.

Textual Retrieval Methods

Document Retrieval in Text Space.

Statistical methods based on word frequency like TF-IDF ⁴Spärck Jones (1972)A Statistical Interpretation of Term Specificity and Its Application in Retrieval and BM25 ⁵Robertson et al. (1994)Okapi at TREC-3 are still widely used due to their simplicity and efficiency. More recently, neural embedding models based on fine-tuned large language models display state-of-the-art performance on a variety of text embedding tasks and top the retrieval leaderboards ⁶Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark.

Neural Retrievers. In bi-encoder models ⁷Reimers & Gurevych (2019)Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Karpukhin et al. (2020)Dense Passage Retrieval for Open-Domain Question Answering
Wang et al. (2022)Text Embeddings by Weakly-Supervised Contrastive Pre-training, documents are independently mapped offline to a dense vector space. Queries are embedded online and matched to documents through a fast cosine distance computation. A slower, but slightly more performant alternative, cross-encoder systems ⁸Wang et al. (2020)MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Cohere (2024)Introducing Rerank 3: A New Foundation Model for Efficient Enterprise Search & Retrieval concatenate query and document as a single input sequence and iteratively attribute matching scores to each possible combination. This enables full attention computation between query and document terms but comes at the cost of computational efficiency, as $|\mathcal{D}|$ encoding passes must be done online.

Multi-Vector retrieval via late interaction. In the late interaction paradigm introduced by ColBERT ⁹Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, an embedding is pre-computed and indexed per document token. At runtime, similarity can be computed with individual query token embeddings. The idea is to benefit from the rich interaction between individual query and document terms while taking advantage of the offline computation and fast query matching enabled by bi-encoders. See Subsection C.7.3 for more details.

Retrieval Evaluation. Although benchmarks and leaderboards have been developed to evaluate text embedding models ¹⁰Thakur et al. (2021)BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark, much of the performance improvements in industrial use cases of embedding models stem from the prior data ingestion pipeline. While documents often rely on visual elements to more efficiently convey information to human readers, text-only systems barely tap into these visual cues. Other work has also independently studied table or chart retrieval systems through repurposed Question Answering datasets ¹¹Zhang et al. (2019)Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrieval
Nowak et al. (2024)Multimodal Chart Retrieval: A Comparison of Text, Table and Image Based Approaches, but these studies assess only specialized methods for each task.

To our knowledge, no benchmark evaluates document retrieval systems in practical settings; in an end-to-end manner, across several document types and topics, and by evaluating the use of both textual and visual document features.

Integrating Visual features

Contrastive Vision Language Models. Mapping latent representations of textual content to corresponding representations of visual content has been done by aligning disjoint visual and text encoders through contrastive losses ¹²Radford et al. (2021)Learning Transferable Visual Models From Natural Language Supervision
Zhai et al. (2023)Sigmoid Loss for Language Image Pre-Training. While some OCR capabilities exist in these models, the visual component is often not optimized for text understanding.

The Fine-grained Interactive Language-Image Pre-training ¹³Yao et al. (2021)FILIP: Fine-grained Interactive Language-Image Pre-Training framework extends the late interaction mechanism to cross-modal Vision Language Models, relying on max similarity operations between text tokens and image patches.

Visually Rich Document Understanding.

To go beyond text, some document-focused models jointly encode text tokens alongside visual or document layout features ¹⁴Appalaraju et al. (2021)DocFormer: End-to-End Transformer for Document Understanding
Kim et al. (2021)OCR-free Document Understanding Transformer
Huang et al. (2022)LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Tang et al. (2022)Unifying Vision, Text, and Layout for Universal Document Processing. Large Language transformer Models (LLMs) with strong reasoning capabilities have recently been combined with Vision Transformers (ViTs) ¹⁵Dosovitskiy et al. (2020)An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale to create VLMs ¹⁶Alayrac et al. (2022)Flamingo: a Visual Language Model for Few-Shot Learning
Liu et al. (2023)Visual Instruction Tuning
Bai et al. (2023)Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
Laurençon et al. (2024)What matters when building vision-language models? where image patch vectors from contrastively trained ViT models ¹⁷Zhai et al. (2023)Sigmoid Loss for Language Image Pre-Training are fed as input embeddings to the LLM and concatenated with the text-token embeddings.

PaliGemma. The PaliGemma-3B model ¹⁸Beyer et al. (2024)PaliGemma: A versatile 3B VLM for transfer extends concepts from Pali3 ¹⁹Chen et al. (2023)PaLI-3 Vision Language Models: Smaller, Faster, Stronger, and projects SigLIP-So400m/14 ²⁰Alabdulmohsin et al. (2023)Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design patch embeddings into Gemma-2B's text vector space ²¹Team et al. (2024)Gemma: Open Models Based on Gemini Research and Technology. Along with its reasonable size w.r.t. other performant VLMs, an interesting property of PaliGemma's text model is that it is fine-tuned with full-block attention on the prefix (instruction text and image tokens). See Section C.7 for more details.

VLMs display enhanced capabilities in Visual Question Answering, captioning, and document understanding ²²Yue et al. (2023)MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI, but are not optimized for retrieval tasks.

The ViDoRe Benchmark

Existing benchmarks for contrastive vision-language models primarily evaluate retrieval for natural images ²³Lin et al. (2014)Microsoft COCO: Common Objects in Context
Borchmann et al. (2021)DUE: End-to-End Document Understanding Benchmark
Thapliyal et al. (2022)Crossmodal-3600: A Massively Multilingual Multimodal Evaluation Dataset. On the other hand, textual retrieval benchmarks ²⁴Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark are evaluated at the textual passage level and are not tailored for document retrieval tasks. We fill the gap with ViDoRe, a comprehensive benchmark for document retrieval using visual features.

Benchmark Design

ViDoRe is designed to comprehensively evaluate retrieval systems on their capacity to match queries to relevant documents at the page level. This benchmark encompasses multiple orthogonal subtasks, with focuses on various modalities - text, figures, infographics, tables; thematic domains - medical, business, scientific, administrative; or languages - English, French. Tasks also span varying levels of complexity, in order to capture signals from both weaker and stronger systems. As many systems require large amounts of time to index pages (captioning-based approaches can take dozens of seconds per page for instance), we limit the number of candidate documents for each retrieval task in order to evaluate even complex systems in a reasonable timeframe without sacrificing quality. For trainable retrieval systems, we provide a reference training set that can be used to facilitate comparisons.

Academic Tasks. We repurpose widely used visual question-answering benchmarks for retrieval tasks: for each page-question-answer triplet, we use the question as the query, and the associated page as the gold document (Table 1). These academic datasets either focus on single specific modalities ²⁵Mathew et al. (2020)DocVQA: A Dataset for VQA on Document Images
Mathew et al. (2021)InfographicVQA
Li et al. (2024)Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models or target more varied visually rich documents ²⁶Zhu et al. (2022)Towards Complex Document Understanding By Discrete Reasoning. Moreover, we consider TabFQuAD, a human-labeled dataset on tables extracted from French industrial PDF documents released with this work. Details can be found in Subsection C.3.1.

Practical tasks. We construct topic-specific retrieval benchmarks spanning multiple domains to go beyond repurposed QA datasets and evaluate retrieval in more realistic industrial situations (e.g. RAG). To achieve this, we collect publicly accessible PDF documents and generate queries pertaining to document pages using Claude-3 Sonnet, a high-quality proprietary vision-language model ²⁷Anthropic (2024)The Claude 3 Model Family: Opus, Sonnet, Haiku. In total, we collect 1,000 document pages per topic, which we associate with 100 queries extensively filtered for quality and relevance by human annotators. The corpus topics are intentionally specific to maximize syntactic proximity between documents, creating more challenging retrieval tasks and covering an array of orthogonal domains (Table 1).

²⁸Answers are generated alongside queries to (1) ground queries and improve their quality and (2) provide resources to foster future work.

Evaluation Metrics. We evaluate performance on our benchmark (Requirement R1) using standard metrics from the retrieval literature (nDCG, Recall@K, MRR). We report nDCG@5 values as the main performance metric in this work and release the complete sets of results along with the models²⁹https://huggingface.co/vidore. To validate compliance with practical industrial requirements (Subsection), we also consider query latencies (R2) and indexing throughputs (R3).

Assessing Current Systems

Unstructured. We evaluate retrieval systems representative of those found in standard industrial RAG pipelines. As is common practice, we rely on the Unstructured³⁰www.unstructured.io off-the-shelf tool in the highest resolution settings to construct high-quality text chunks from PDF documents. Unstructured orchestrates the document parsing pipeline, relying on deep learning vision models to detect titles and document layouts ³¹Ge et al. (2021)YOLOX: Exceeding YOLO Series in 2021, OCR engines ³²Smith (2007)An Overview of the Tesseract OCR Engine to extract text in non-native PDFs, specialized methods or models to detect and reconstruct tables, and implements a chunking strategy (by-title) that leverages the detected document structure to preserve section boundaries when concatenating texts. As is common practice, in our simplest Unstructured configuration (text-only), only textual elements are kept and figures, images, and tables are considered noisy information and are filtered out.

Unstructured + X. While Unstructured is a strong baseline by itself, we further augment Unstructured's output by integrating the visual elements. In (+ OCR), tables, charts, and images are run through an OCR engine, processed by Unstructured, and chunked independently. In (+ Captioning), we set up a fully-fledged captioning strategy ³³Zhao et al. (2023)Retrieving Multimodal Information for Augmented Generation: A Survey, in which we feed visual elements to a strong proprietary Vision Language Model (Claude-3 Sonnet ³⁴Anthropic (2024)The Claude 3 Model Family: Opus, Sonnet, Haiku) to obtain highly detailed textual descriptions of the elements. Both strategies aim to integrate visual elements in the retrieval pipeline but incur significant latency and resource costs (Subsection 3.5.2).

Embedding Model. To embed textual chunks, we evaluate Okapi BM25, the de facto standard sparse statistical retrieval method, and the dense encoder of BGE-M3 ³⁵Chen et al. (2024)BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation, a multilingual neural method with SOTA performance in its size category. Chunks are embedded and scored independently, and page-level scores are obtained by max-pooling over the page's chunk scores.³⁶We empirically validated the max-pooling strategy over sub-page chunks to be more effective than concatenating all page chunks before embedding pagewise.

Contrastive VLMs. We also evaluate the strongest available vision-language embedding models; Jina CLIP ³⁷Koukounas et al. (2024)Jina CLIP: Your CLIP Model Is Also Your Text Retriever, Nomic Embed Vision ³⁸Nomic (2024)Nomic Embed Vision: Expanding The Nomic Latent Space, and SigLIP-So400m/14 ³⁹Alabdulmohsin et al. (2023)Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design.

Results. From a performance perspective, the best results are obtained by combining the Unstructured parser with visual information, either from captioning strategies or by running OCR on the visual elements (Table 2). Little difference is seen between BM25 and BGE-M3 embeddings, highlighting the visual information bottleneck. Contrastive VLMs lag behind. Beyond retrieval performance (R1), the indexing latencies (R2) reported in Figure 2 illustrate that PDF parsing pipelines can be very lengthy, especially when incorporating OCR or captioning strategies. Querying latencies at runtime (R3) are very good for all evaluated systems ($\leq22$ ms on an NVIDIA L4) due to fast query encoding and cosine similarity matching.

Late interaction based Vision Retrieval

Architecture

Vision-Language Models. Encouraged by their strong document understanding capabilities, we propose adapting recent VLMs for retrieval. The key concept is to leverage the alignment between output embeddings of text and image tokens acquired during multi-modal fine-tuning.

To this extent, we introduce ColPali, a Paligemma-3B extension that is capable of generating ColBERT-style multi-vector representations of text and images (Figure 1). PaliGemma-3B is a strong candidate due to its small size, the many released checkpoints fine-tuned for different image resolutions and tasks, and its promising performance on various document understanding benchmarks. We add a projection layer to map each of the language model's output token embeddings (whether from text or image tokens) to a vector space of reduced dimension $D = 128$ as used in the ColBERT paper ⁴⁰Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT to keep lightweight bag-of-embedding representations.

Late Interaction.

Given query $q$ and document $d$, we denote as $\mathbf{E_q} \in \mathbb{R}^{{N_q} \times D}$ and $\mathbf{E_d} \in \mathbb{R}^{N_d \times D}$ their respective multi-vector representation in the common embedding space $\mathbb{R}^D$, where $N_q$ and $N_d$ are respectively the number of vectors in the query and in the document page embeddings. The late interaction operator, $\text{LI} \left( q, d \right)$, is the sum over all query vectors $\mathbf{E_{q}}^{(j)}$, of its maximum dot product $\langle \cdot | \cdot \rangle$ with each of the $N_d$ document embedding vectors $\mathbf{E_d}_{(1:N_d)}$.

Contrastive Loss. The Late Interaction operation is fully differentiable, enabling backpropagation. Let a batch $\left\{ q_k, d_k \right\}_{k \in [|1, b |]}$ composed of $b$ query-page pairs, where for all $k \in [|1, b|]$, the document page $d_k$ is the document corresponding to query $q_k$.

Following ⁴¹Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, we define our in-batch contrastive loss $\mathcal{L}$ as the softmaxed cross-entropy of the positive scores $s_k^+ = \text{LI} \left( q_k, d_k \right)$ w.r.t. the maximal in-batch negative scores $s_k^- = \max \limits_{l, l \neq k} \hspace{3mm} \text{LI} \left( q_k, d_l \right)$⁴²We reformulate the loss to leverage the numerically stable softplus function where $\texttt{softplus(x)} = \log \left( 1 + \exp \left( x \right)\right)$:

Model training

Dataset. Our training dataset of 118,695 query-page pairs comprises training sets of openly available academic datasets ($63\%$) and a synthetic dataset made up of pages from web-crawled PDF documents and augmented with VLM-generated (Claude-3 Sonnet) pseudo-questions ($37\%$). Dataset split details are given in Subsection C.3.3. Our training set is fully English by design, enabling us to study zero-shot generalization to non-English languages⁴³Multilingual data is present in the pretraining corpus of the language model (Gemma-2B) and potentially occurs during PaliGemma-3B's multimodal training.. We explicitly verify no multi-page PDF document is used both ViDoRe and in the train set to prevent evaluation contamination. A validation set is created with $2\%$ of the samples to tune hyperparameters. We openly release the training dataset⁴⁴https://huggingface.co/datasets/vidore/colpali_train_set for reproducibility and to encourage further research.

Parameters. All models are trained for 1 epoch on the train set. Unless specified otherwise, we train models in bfloat16 format, use low-rank adapters (LoRA, ⁴⁵Hu et al. (2021)LoRA: Low-Rank Adaptation of Large Language Models) with $\alpha=32$ and $r=32$ on the transformer layers from the language model, as well as the final randomly initialized projection layer, and use a paged\_adamw\_8bit optimizer. We train on an 8 GPU setup with data parallelism, a learning rate of $5e-5$ with linear decay with 2.5% warmup steps, and a batch size of 32.

Query Augmentation. As in ⁴⁶Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT, we append 5 <unused0> tokens to the query tokens to serve as a soft, differentiable query expansion or re-weighting mechanism.

Results

Performance (R1)

We show performance is achieved iteratively through the combination of three factors; (1) a carefully crafted task-specific dataset, (2) pairing a pretrained LLM to a vision model to better leverage text semantics from the image, and (3) using multi-vector embeddings rather than a single vector representation to better capture the vast amount of visual information present in a document.

Fine-tuning a Vision Model on a document-retrieval-oriented dataset: BiSigLIP. SigLIP⁴⁷https://huggingface.co/google/siglip-so400m-patch14-384 is a strong vision-language bi-encoder producing single-vector embeddings, pretrained on billions of image-text pairs from the English split of WebLI ⁴⁸Chen et al. (2023)PaLI-3 Vision Language Models: Smaller, Faster, Stronger. Further fine-tuning the textual component of this model on our document-oriented dataset (BiSigLIP) yields clear improvements across the board, particularly on figure retrieval (ArxivQA) and table retrieval tasks (TabFQuAD).

Feeding image patches to a LLM: BiPali. In the PaliGemma model architecture, SigLIP-generated patch embeddings are fed to a text language model and we can obtain LLM contextualized output patch embeddings.⁴⁹Note that the SigLIP model used in PaliGemma slightly differs in terms of number patches - 1024 patches for PaliGemma's vision encoder, and 729 for the standalone SigLIP model. This technique aligns the image token representations with the text token embeddings in the LLM's embeddings space, and augments the vision model embeddings with the language model's text understanding capabilities. We average pool these representations to obtain a single dense vector, effectively creating a PaliGemma bi-encoder model (BiPali). After fine-tuning on the training dataset, we obtain a model that performs slightly worse in English than the tuned BiSigLIP variant.⁵⁰This can be explained by the fact that contrary to SigLIP, the original PaliGemma is not trained on contrastive matching tasks, but rather on next token prediction. Our contrastive fine-tuning phase on 119K images to transform PaliGemma into a bi-encoder is 5 orders of magnitude smaller than SigLIP's original contrastive training. However, we see notable improvements in French tasks, indicating that BiPali's LLM (Gemma 2B) helps multilingual text understanding. This is particularly notable as our training dataset does not contain non-English samples.

Leveraging Multi-Vector Embeddings through Late Interaction: ColPali. One benefit of inputting image patch embeddings through a language model is that they are natively mapped to a latent space similar to the textual input (query). This enables leveraging the ColBERT strategy to construct one embedding per image patch token, and at inference compute all interactions between text tokens and image patches, resulting in a step-change improvement in performance compared to BiPali. Results in Table 2 show that our ColPali model also largely outperforms the strong baselines based on Unstructured and captioning, as well as all evaluated text-image embedding models. The difference is particularly stark on the more visually complex benchmark tasks, such as InfographicVQA, ArxivQA, and TabFQuAD, respectively representing infographics, figures, and tables. However, text-centric documents are also better retrieved by the ColPali models across all evaluated domains and languages, making our approach the overall best-performing document-retrieval model.

Negative Results. For extensiveness, we also train ColSigLIP, a late interaction variant of the BiSigLIP model, but obtain abysmal performance. We attribute this to the large gaps w.r.t. SigLIP's pre-training, in which only a pooled latent representation is used in the contrastive loss, which does not optimize the representations of individual patch and token embeddings. Similarly, we train a BiSigLIP$_{PaliGemma}$ variant, in which we retrieve the image representations from the SigLIP model that has been further updated by PaliGemma fine-tuning, and use the text representations from PaliGemma's text model. After fine-tuning on our dataset, performance is severely inferior to that of SigLIP$_{Vanilla}$, which simply encodes with SigLIP's original text and vision components. This indicates a logical misalignment between SigLIP embeddings and Gemma embeddings after PaliGemma training. We detail these results in Subsection C.5.1.

Latencies & Memory Footprint

Online Querying. (R2) Logically, querying latencies differ between ColPali and a BGE-M3 embedding model. For BGE, encoding takes about $22$ ms for 15 tokens, while encoding a query with ColPali's language model takes about $30$ ms⁵¹Computed for a batch size of $1$ (online), and averaged over 1000 queries. See Subsection C.4.4. For smaller corpus sizes, computing the late interaction operation induces only marginal overhead ($\approx 1$ ms per 1000 pages in the corpus), and the cosine similarity computation between bi-encoder vectors is even faster. Optimized late interaction engines ⁵²Santhanam et al. (2022)PLAID: An Efficient Engine for Late Interaction Retrieval
Lee et al. (2023)Rethinking the Role of Token Retrieval in Multi-Vector Retrieval make it easy to scale corpus sizes to millions of documents with reduced latency degradation.

Offline Indexing. (R3) Standard retrieval methods using bi-encoders represent each chunk as a single vector embedding, which is easy to store and fast to compute. However, processing a PDF to obtain the different chunks is the most time-consuming part (layout detection, OCR, chunking), and using captioning to handle multimodal data will only exacerbate this already lengthy process. On the other hand, ColPali directly encodes pages from their image representation. Although the model is larger than standard retrieval encoders, skipping preprocessing allows large indexing speedups⁵³Measured on an NVIDIA L4 GPU and averaged over 100 pages, with a batch size of 4 pages for ColPali and 8 text chunks for Bi-Encoders. On average, a page is divided into 2.1 chunks. See Subsection C.4.4. (Figure 2). As pages are embedded end-to-end in a single forward pass, VRAM usage depends exclusively on the sequence length (number of patches per image), which is fixed, enabling efficient batching strategies that fully leverage hardware acceleration. ColPali also benefits from most LLM efficiency improvements introduced in the ecosystem, such as Flash Attention ⁵⁴Dao (2023)FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning.

Storage Footprint. Our method requires storing a vector per image patch, along with 6 extra text tokens “Describe the image" concatenated to image patches. We project each PaliGemma vector to a lower dimensional space ($D=128$) to maximize efficiency, leading to a memory footprint of $257.5$ KB per page (Subsection C.4.3). Importantly, the memory footprint of the naive ColBERT indexing strategy can be drastically improved through compression and clustering mechanisms ⁵⁵Santhanam et al. (2022)PLAID: An Efficient Engine for Late Interaction Retrieval
Clavié et al. (2024)Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling.

Token pooling.

Token pooling ⁵⁶Clavié et al. (2024)Reducing the Footprint of Multi-Vector Retrieval with Minimal Performance Impact via Token Pooling is a CRUDE-compliant method (document addition/deletion-friendly) that aims to reduce the number of vectors in multi-vector embeddings. For ColPali, many image patches share redundant information, e.g. white background patches. By pooling these patches together, we can reduce the number of embeddings while retaining most information. Retrieval performance with hierarchical mean token pooling on image embeddings is shown in Figure 3. With a pool factor of 3, the total number of vectors is reduced by $66.7\%$ while $97.8\%$ of the original performance is maintained. We note that the Shift dataset—composed of the most text-dense documents—is a clear outlier, showing that more information-dense documents contain fewer redundant patches and may be prone to greater performance degradation with such pooling techniques.

Interpretability

By superimposing the late interaction heatmap on top of the original image, we can visualize the most salient image patches with respect to each term of the query, yielding interpretable insights into model focus zones. As epitomized in Figure 4, we observe ColPali exhibits strong OCR capabilities as both the words “hourly" and “hours" present a high similarity score with the query token <\_hour>. We also note particular focus on other non-trivial image features such as the x-axis representing hours being salient. Other visualization examples are shown in Section C.6.

Ablation study

We run various ablations to better understand the mechanisms at play. By default, result deltas reported below refer to nDCG@5 values averaged over all ViDoRe tasks. Detailed results in C.5.2.

Tradeoffs between model size and the number of image patches. We train a variant of PaliGemma with half the number of image patches (512). While we observe a clear performance degradation with respect to the 1024-patch ColPali model ($-24.8$ nDCG@5), memory usage is much lower.

As an alternative to PaliGemma, we train Idefics2-8B ⁵⁷Laurençon et al. (2024)What matters when building vision-language models?, a VLM with a similar architecture based on a Mistral-7B ⁵⁸Jiang et al. (2023)Mistral 7B language backbone and a SigLIP vision encoder paired with a perceiver resampler. The most notable differences from PaliGemma lie in the size of the language model (2B and 7B, respectively) and the number of image patches (between 512 and 2048 for PaliGemma, and 64 post-resampling for Idefics2⁵⁹With the option of adding 4 sub-image crops of 64 tokens each to the sequence, for a total of 320 tokens). Our results suggest that better language models enable more efficient representations of image embeddings: ColIdefics2 with 64 patches largely outperforms ColPali with 512 patches (+20.1 nDCG@5). However, ColIdefics2 (64) remains less accurate than ColPali (1024) ($-4.7$ nDCG@5) while being about twice as slow in terms of training and inference latency.

These results suggest there are tradeoffs between performance (R1), latencies during online querying (R2) and offline indexation phases (R3), and index memory size.

Unfreezing the vision component. We train a ColPali variant by also backpropagating through and updating the vision encoder and the projection layer. This leads to a slight performance degradation ($-0.7$ nDCG@5). These conclusions may change with larger scales of training data.

Impact of “query augmentation" tokens. In ColBERT, special tokens are concatenated to the input query to serve as soft query augmentation buffers. Training without these tokens, we observe no significant performance difference in the English benchmarks. However, performance on the French tasks seems to improve ($+9.8$ nDCG@G on Shift, $+6.3$ nDCG@5 on TabFQuAD, Table 5).

Impact of the Pairwise CE loss. Training with an in-batch negative contrastive loss, instead of the pairwise CE loss that only considers the hardest negative sample, leads to a slight performance degradation ($-1.6$ nDCG@5) on the aggregated benchmark.

Adapting models to new tasks. Contrary to more complex multi-step retrieval pipelines, ColPali can be trained end-to-end, directly optimizing the downstream retrieval task which greatly facilitates fine-tuning to boost performance on specialized domains, multilingual retrieval, or specific visual elements the model struggles with. To demonstrate, we add 1552 samples representing French tables and associated queries to the training set. This represents the only French data in the training set, with all other examples being kept unchanged. We see clear nDCG@5 improvements ($+2.6$) and even starker Recall@1 gains ($+5$) on the TabFQuAD benchmark, with no performance degradation on the rest of the benchmark tasks ($+0.4$ nDCG@5 overall).

Better VLMs lead to better visual retrievers. As improved VLMs are released, it is interesting to observe whether improved performance on generative tasks translates once these models are adapted for image retrieval tasks through ColPali training strategies. We train the recently released Qwen2-VL 2B ⁶⁰Wang et al. (2024)Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, a SOTA 2-billion-parameter generative VLM, with the same data and training strategy, obtaining ColQwen2-VL. To approximately match ColPali's memory requirements, we limit the number of image patches to 768, slightly fewer than ColPali's 1024 patches. We observe a clear improvement of $+5.3$ nDCG@5 over ColPali, showcasing a clear correlation between performance on generative benchmarks and retrieval metrics.

Out-of-domain generalization. Some of the datasets in the ViDoRe benchmark have train sets, which we have integrated within the ColPali train set (e.g. academic tasks). This is standard in embedding models ⁶¹Wang et al. (2024)Improving Text Embeddings with Large Language Models
Lee et al. (2024)NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models, and while ColPali also exhibits strong performance on tasks in which this is not the case (French data is never seen by the model during training, for instance), it remains interesting to evaluate model performance when training is done on a fully disjoint data distribution. We train a ColPali variant solely using the recent DocMatix dataset ⁶²Laurençon et al. (2024)Building and better understanding vision-language models: insights and future directions., a large-scale, synthetically annotated visual document question-answering dataset, which we subsample to obtain a comparably sized train set. Results on ViDoRe show the performance drop is minor ($-2.2$ nDCG@5), still outperforming the closest baseline method by over 12 points. These results show that ColPali generalizes well outside its training distribution and demonstrate that our results are not unreasonably boosted with respect to baselines (BGE-M3) that cannot be fine-tuned on the same data⁶³Training with data resembling what BGE-M3 models would see at inference time would require running complex extraction pipelines for the more than 100K documents in the training set, notably relying on external proprietary captioning models, which would be both too costly and time-consuming. This is not needed to train vision-based models.

Conclusions

In this work, we introduced the Visual Document Retrieval Benchmark (ViDoRe), which evaluates document retrieval systems in realistic settings involving visually complex documents. We demonstrated that current retrieval pipelines and contrastive vision-language models struggle to efficiently exploit visual information embedded in documents, leading to suboptimal performance. To address this, we presented ColPali, a novel retrieval method that leverages Vision-Language Models to create high-quality, multi-vector embeddings purely from visual document features. ColPali largely outperforms the best existing document retrieval methods while enabling faster corpus indexing times and maintaining low querying latencies, thus circumventing many pain points of modern document retrieval applications. We hope to drive industrial adoption, and to encourage future work by publicly releasing the ViDoRe benchmark, the data, the codebase, and all models and baselines from our work.

Future Work.

Beyond performance improvements that could be obtained through better data, backbone models or training strategies, our vision at term is to combine visual retrieval systems and visually grounded query answering to create end-to-end RAG systems that purely function from image features. This idea is supported by concurrent work ⁶⁴Ma et al. (2024)MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations showcasing the strong promises of VLMs for visual QA, and may eventually become a new industrial standard for document processing. In this line of work, reliability is key, and confidence estimation techniques for Information Retrieval methods could become central to implement abstention mechanisms ⁶⁵Gisserot-Boukhlef et al. (2024)Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism, and are particularly interesting given the information rich multi-vector scoring mechanisms of late interaction systems. Expanding benchmarking efforts to cover more languages, modalities, and tasks is also a crucial future research direction ⁶⁶Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks.