ModernVBERT: Towards Smaller Visual Document Retrievers

Introduction

The ability to quickly locate specific information in vast document collections is a core building block of digital systems today, supporting use cases that range from web search and virtual assistants to enterprise knowledge management. Neural information retrieval (IR) models, and in particular dense retrievers, have become the de facto backbone of modern search systems thanks to their strong semantic matching capabilities and good scalability properties ¹Reimers & Gurevych (2019)Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Karpukhin et al. (2020)Dense Passage Retrieval for Open-Domain Question Answering
Wang et al. (2022)Text Embeddings by Weakly-Supervised Contrastive Pre-training.

This trend is amplified by the widespread adoption of Retrieval-Augmented Generation (RAG) ²Lewis et al. (2020)Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, where a retriever is used to select a small set of relevant documents that condition a downstream generator. In such systems, the first-stage retrieval module is a well-known bottleneck: its recall directly upper-bounds the quality of the generated answers, while its latency and indexing costs partially drive the overall system efficiency ³Lin & Byrne (2022)Retrieval Augmented Visual Question Answering with Outside Knowledge. As a result, improving document retrieval, especially for long, complex files such as PDFs, scientific articles, and reports, is a key lever for making industrial RAG deployments more accurate and cost-effective.

Visual Document Retrieval. Historically, document retrieval in these settings has operated purely in the text space. To index PDFs or scans, practitioners first run heavy preprocessing pipelines that include Optical Character Recognition (OCR), layout analysis, and heuristic passage segmentation, before embedding the resulting text spans with a neural encoder.

This approach suffers from several limitations: OCR and layout parsing can be brittle and slow, complex visual elements such as tables, figures, and typography are often poorly captured, and any error or bias introduced during preprocessing is propagated to the retriever.

Visual Document Retrieval (VDR) has emerged as a compelling alternative to such text-based systems. Rather than indexing pre-extracted textual content, VDR models directly operate on page screenshots: given a user query, they retrieve relevant document pages by matching the query against image-based representations of the pages ⁴Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models. By bypassing OCR and layout parsing, VDR yields simpler end-to-end pipelines, significantly reduces indexing latency, and better exploits visual cues such as layout, figures, and fonts, while achieving strong performance on visually rich benchmarks like ViDoRe.

Limits of Generative VLM Repurposing. Most current VDR systems are obtained by repurposing large generative vision–language decoders ⁵Alayrac et al. (2022)Flamingo: a Visual Language Model for Few-Shot Learning as retrieval encoders via post-hoc contrastive fine-tuning ⁶Ma et al. (2024)Unifying Multimodal Retrieval via Document Screenshot Embedding
Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. While cost-efficient, this design choice bottlenecks retrieval performance and efficiency: model sizes, attention patterns, image resolutions, and training objectives are designed for generative use cases rather than optimized for retrieval which has been shown in text models to be suboptimal ⁷Lee et al. (2024)NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
Gisserot-Boukhlef et al. (2026)Should We Still Pretrain Encoders with Masked Language Modeling?. Furthermore, scaling trends ⁸Wei et al. (2022)Emergent Abilities of Large Language Models are less pronounced for embedding models; while correlated with model size, strong retrieval performance remains attainable with small models ⁹Clavié (2024)Towards Better Monolingual Japanese Retrievers with Multi-Vector Models.

Recent papers and model releases in the visual retrieval space have claimed performance improvements by scaling the amount of contrastive data and the compute budget ¹⁰Zhang et al. (2025)GME: Improving Universal Multimodal Retrieval by Multimodal LLMs
Xu et al. (2025)Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model, modifying the attention mask ¹¹Chen et al. (2025)MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings, increasing image resolutions ¹²Cohere (2024)Introducing Rerank 3: A New Foundation Model for Efficient Enterprise Search & Retrieval or by introducing more diverse tasks and data sources ¹³Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks.

In this work, we attempt to centralize these efforts and systematically disentangle the impact of core design decisions in visual retriever training.

Through controlled experiments—ranging from language model pretraining to multi-stage, domain-specific fine-tuning, we aim to answer a central question:

Which design choices best boost performance in modern visual document retrievers?

Contribution 1. We revisit core assumptions in visual retriever design, showing that token-level training objectives benefit retrievers by strengthening image–text token alignment—rather than merely producing stronger image embeddings. Our results indicate that causal attention is suboptimal in document retrieval, with bidirectional masking offering clear improvements in multi-vector settings, and that other parameters such as image resolution data mixes should not be overlooked in the training pipeline.

Contribution 2: ModernVBERT. Building on these insights, we release ModernVBERT, a small 250M multimodal encoder that aligns a pretrained language encoder with a vision encoder through Masked Language Modeling (MLM) objective, and ColModernVBERT a variant fine-tuned for document retrieval. Despite its modest size and limited training budget, ColModernVBERT matches models 10x larger on standard visual document retrieval benchmarks, demonstrating the interest of designing a retrieval focused model from the ground up.

We release the model, intermediate checkpoints, and the training code at huggingface.co/ModernVBERT.

Methodology

Our analysis aims at quantifying the impact of design decisions made when training visual retrievers. In opposition to previous work, we begin our analysis as early as language model modality alignment and iteratively study design choices by modifying design choices independently to reduce confounding factors as much as possible ¹⁴Allen-Zhu & Li (2025)Physics of Language Models: Part 1, Learning Hierarchical Language Structures.

Controlled Experimental Setup. A central point of interest is the impact of causal and bidirectional attention masks. While recently studied for textual representation applications ¹⁵Gisserot-Boukhlef et al. (2026)Should We Still Pretrain Encoders with Masked Language Modeling?
Weller et al. (2025)Seq vs Seq: An Open Suite of Paired Encoders and Decoders, we extend the experiment to the vision modality. We use checkpoints released by ¹⁶Gisserot-Boukhlef et al. (2026)Should We Still Pretrain Encoders with Masked Language Modeling? which consist in a series of identical 210M parameter transformer models based on the Llama architecture ¹⁷Touvron et al. (2023)Llama 2: Open Foundation and Fine-Tuned Chat Models trained on 100B tokens that differ only in their attention masking strategy during language model training but that are perfectly identical in terms of training data seen, model size and architecture, learning rate scheduling, etc... The checkpoints we use are enc a bidirectional encoder trained with Masked Language Modeling (MLM), dec, a causal decoder trained with next token prediction, and dec-enc a causal decoder annealed over the end of its textual training by removing the causal mask and switching the training objective to MLM. For the vision tower, we employ the vision component of siglip2-base-16b-512 ¹⁸Tschannen et al. (2025)SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features, a 86M parameter vision transformer contrastively trained on billions of text-image pairs. All ablations thus stem from iso-data controlled setups, and as further described, are further trained on the same data sequence, with the same batch sizes, optimizers, schedulers and on the same hardware.

Model Architecture. Our analyses are not centered around model architectures and to draw broadly applicable insights, we design vision-language models following current standard training practices. In line with most recent work, we employ the early fusion architecture ¹⁹Alayrac et al. (2022)Flamingo: a Visual Language Model for Few-Shot Learning illustrated in Figure 2, in which visual patch embeddings produced by the vision encoder are projected into the language model input embedding space and concatenated with text token embeddings to encourage joint processing ²⁰Li et al. (2022)BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
Alayrac et al. (2022)Flamingo: a Visual Language Model for Few-Shot Learning
Wang et al. (2024)Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
Yang et al. (2025)Qwen2.5-1M Technical Report
Marafioti et al. (2025)SmolVLM: Redefining small and efficient multimodal models. As described in Subsection 5.2.1, we generalize the training loss to function both with causal and masked language modeling objectives. To handle dynamic resolutions, we split large images into 512$\times$512 pixel tiles as expected by the SigLIP encoder²¹Images are downscaled (or upscaled) so that the lengths and widths reach a multiple of 512 pixels to preserve the aspect ratio, padding is used on the smaller side when necessary (i.e. a 1024x1000 px image would be padded to 1024x1024 px).. Following current standard practices, we further process a downscaled version of the full image to improve inter-tile consistency and global visual understanding ²²Lin et al. (2023)SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models
Ye et al. (2023)UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model. The vision tower produces 1024 pixel patch representations for each tile²³The SigLIP tower takes 512x512 px images and processes them by 16x16 px patches (Dosovitskiy et al. 2020). This results in $(512/16)^2=1024$ patches., which we compress to 64 tokens through pixel shuffling ²⁴Shi et al. (2016)Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network with a compression ratio $r=4$, following prior work on models of comparable size ²⁵Marafioti et al. (2025)SmolVLM: Redefining small and efficient multimodal models. We highlight the impact of image resolution and this parameter on the number of visual tokens in Appendix E.3.6.1.

Training Procedure. Our experiments focus on retrieval performance. We employ a standard biphasic training procedure, in which we first run modality alignment to train a pretrained textual language model to understand visual inputs through language modeling objectives ²⁶Liu et al. (2023)Visual Instruction Tuning (Subsection 5.2.1), then rely on a second text-image contrastive learning phase to learn efficient image representations ²⁷Radford et al. (2021)Learning Transferable Visual Models From Natural Language Supervision (Subsection 5.2.2). We further describe the general setup, and detail specific modifications to the default training procedure in the experiment section.

Modality Alignment

We align the vision encoder tower with the language model by training the image embedding projection layer to map visual features into the language model embedding space. The pretrained language model is also fine-tuned with Low-Rank Adapters (LoRA) ²⁸Hu et al. (2021)LoRA: Low-Rank Adaptation of Large Language Models, allowing both image and text models to adapt jointly while reducing the risk of monomodal performance collapse ²⁹Alayrac et al. (2022)Flamingo: a Visual Language Model for Few-Shot Learning
Liu et al. (2023)Visual Instruction Tuning
Laurençon et al. (2024)What matters when building vision-language models?
McKinzie et al. (2024)MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Marafioti et al. (2025)SmolVLM: Redefining small and efficient multimodal models.

Alignment Loss. For decoder‑based models, we train with Causal Language Modeling (CLM) loss on the text tokens, as standardly done in VLM modality alignment:

where $x_{<t}$ denotes all tokens preceding position $t$. We generalize this training scheme to bidirectional encoders models, by using the Masked Language Modeling (MLM) loss on the textual tokens:

where $\mathcal{M}$ is the set of masked token positions and $x_{\backslash\mathcal{M}}$ is the input with those tokens masked out.

Modality Alignment Corpus. Models are modality aligned on a large corpus in large parts derived from The Cauldron 2 ³⁰Laurençon et al. (2024)What matters when building vision-language models? and Docmatix ³¹Laurençon et al. (2024)Building and better understanding vision-language models: insights and future directions.. Our objective being to train document focused retrieval models, we use an adjusted training mixture that upsamples images containing text and documents with varying level of complexities. Our final training corpus consists of approximately 2B text tokens, and includes diverse sources such as web pages, books, and scientific papers. Mixture details are given in Appendix E.1.3.1. We note that controlling the exact data distribution during this phase enables the models we train to specialize early and achieve good document focused downstream performances which many large models struggle with ³²Liu et al. (2023)Improved Baselines with Visual Instruction Tuning.

Parameters. All models are trained using a masking ratio of 0.5 and user-prompt masking to avoid overfitting on chat-template format ³³Huerta-Enochian & Ko (2024)Instruction Fine-Tuning: Does Prompt Loss Matter?
Shi et al. (2024)Instruction Tuning With Loss Over Instructions
Allal et al. (2025)SmolLM2: When Smol Goes Big – Data-Centric Training of a Small Language Model. We employ WSD scheduler ³⁴Hu et al. (2024)MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies with the first 5% of the training as warmup, the last 20% as decay and a maximum learning rate of 1e-4. The ablation models are aligned on 3.5B tokens. We provide additional details on the training setup in Appendix E.1.1.

Contrastive Post-Training

Once the language model has learned to process image tokens jointly with text tokens, we specialize models through a contrastive post-training stage designed to enhance the semantic representation of the output embeddings produced by the model ³⁵Reimers & Gurevych (2019)Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.

Post-training Pairs. The post-training dataset used as starting point in our ablations comprises 118k document-query pairs from the ColPali corpus (Faysse et al. 2025) as well as another 118k of natural image-description pairs from the MSCOCO train set ³⁶Lin et al. (2014)Microsoft COCO: Common Objects in Context.

Contrastive Loss. We employ the InfoNCE loss ³⁷Oord et al. (2018)Representation Learning with Contrastive Predictive Coding, defined as

where $\mathbf{d^+}$ denotes the positive target for the query $\mathbf{q}$, $\mathcal{N}_\mathbf{q}=\mathcal{N}_\mathbf{q}^{\text{in}} \cup \mathcal{N}_\mathbf{q}^{\text{hard}}$ the set of negative targets (in-batch and hard negatives when mentioned), and $\Phi(\mathbf{q},\mathbf{d})$ a similarity function between the token(s) of the query and the documents.³⁸We use the last (EOS) token for causal models, and mean pool all sequence tokens for bidirectional encoders for single-vector models. Alternatively, we use all document and query tokens without pooling for late interaction matching (Faysse et al. 2025). Details in Appendix E.1.2. For general-domain post-training we compute the loss symmetrically ³⁹Radford et al. (2021)Learning Transferable Visual Models From Natural Language Supervision.

Batches Curation. In contrastive learning, batch diversity critically impacts retrieval entropy. Overly heterogeneous batches lead to trivial retrievals, while curated batches yield richer training signals. We employ task-aware batching ⁴⁰Li et al. (2023)Towards General Text Embeddings with Multi-stage Contrastive Learning, grouping documents by source to ensure a homogeneous batch composition.

Ablation Evaluation Setup

The contrastively trained models are evaluated on retrieval and zero-shot classification tasks across multiple domains. Although the main focus remains document retrieval capabilities, evaluated by aggregating scores from the ViDoRe and ViDoRe v2⁴¹We report only the English splits of ViDoRe v2, as our base models are trained on English data only. ⁴²Macé et al. (2026)ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval benchmarks (nDCG@5), we also assess more generalist image retrieval capabilities by selecting tasks from MIEB ⁴³Xiao et al. (2025)MIEB: Massive Image Embedding Benchmark. For natural image retrieval, we aggregate MSCOCO retrieval ⁴⁴Lin et al. (2014)Microsoft COCO: Common Objects in Context and Flickr30k retrieval (nDCG@10) ⁴⁵Plummer et al. (2015)Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models test sets. Finally, following practices in ⁴⁶Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark, we assess both zero-shot and fine-tuning abilities of our models on general classification tasks. Specifically, we measure classification accuracy by fine-tuning a logistic regression head on top of our model's embedding on Stanford Cars ⁴⁷Krause et al. (2013)3D Object Representations for Fine-Grained Categorization and Food101 ⁴⁸Bossard et al. (2014)Food-101 – Mining Discriminative Components with Random Forests, and we evaluate zero-shot performance on FER2013 ⁴⁹Khaireddin & Chen (2021)Facial Emotion Recognition: State of the Art Performance on FER2013 and EuroSAT ⁵⁰Helber et al. (2019)EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification and aggregate the results.

What Makes a Great Visual Retriever?

Vision-language retrievers built upon existing generative VLMs often inherit design choices and weights that may not be well suited for all embedding tasks. Here, we analyze these critical design choices hoping to derive clear insights for developing efficient visual retrievers. Importantly, although we assess design decisions at different stages of the training pipelines, evaluation are always done end-to-end on the final evaluation signal.

Modality Alignment Design

Language modeling Modality Alignment improves document understanding.

According to benchmarks such as MIEB ⁵¹Xiao et al. (2025)MIEB: Massive Image Embedding Benchmark, dual encoder models explicitly trained on contrastive image-text tasks outperform repurposed VLMs in natural image classification tasks.

To assess this, we train an encoder and a decoder vision-language model using the methodology described in Section 5.2 on a mix of natural image and document data (alignment and contrastive training). We compare them with SigLIP2-FT, the 378M dual vision encoder model whose vision component is used by the vision tower of both VLMs, and with the larger SigLIP2-FT Large (881M parameters). Both SigLIP-FT models are finetuned in the same conditions as the VLMs, and initialized from pre-trained weights from scratch on billions of text-image pairs⁵²We report the performance of the untrained off-the-shelf SigLIP in Appendix E.3.1. As shown in Figure 3, the two early fusion VLM variants severely underperform the SigLIP2-FT dual encoders on natural image tasks. In contrast, they achieve significant gains on document retrieval tasks (+6.1 nDCG@5 on ViDoRe and ViDoRe v2 datasets w.r.t. base), even edging out SigLIP2-FT Large that contains 3.5x vision parameters more than both VLMs.

This confirms large-scale contrastive training remains best for high-level image representation tasks (natural images), but sequentially combining a vision model with a pretrained language model facilitates document representation tasks, even with significantly less contrastive post-training. As the rest of this paper shows, steering away from the dual encoder architecture further enables improving performance through many avenues other than text to image contrastive training, for which supervised training samples can be hard to obtain.

Scaling the modality alignment phase for better token representations. Prior work shows that scaling the modality alignment phase of VLMs improves their generative abilities ⁵³Beyer et al. (2024)PaliGemma: A versatile 3B VLM for transfer
McKinzie et al. (2024)MM1: Methods, Analysis &amp; Insights from Multimodal LLM Pre-training
Wang et al. (2024)Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. We test whether similar gains hold in retrieval by contrastively fine-tuning enc checkpoints during MLM modality alignment. Figure 4 illustrates the results of post-trained checkpoints on diverse tasks. Although document retrieval improves consistently with more modality alignment data – largely surpassing the vision tower evaluated in isolation and showing clear scaling benefits – natural image tasks plateau past 1B tokens, far from the standalone dual encoder baseline. This shows that document and natural image retrieval leverage different mechanisms and should not be optimized the same way. Document Retrieval benefits from learning fine-grained interactions between image and text tokens through the language model, while the LM has limited utility for high level natural image tasks.

Bidirectional attention fully unlocks Late Interaction. Inspired by the effectiveness of bidirectional attention in text-only retrieval ⁵⁴Gisserot-Boukhlef et al. (2026)Should We Still Pretrain Encoders with Masked Language Modeling?
Weller et al. (2025)Seq vs Seq: An Open Suite of Paired Encoders and Decoders⁵⁵(Chen et al. 2025) investigate post-hoc removal of the attention mask during visual retrieval fine-tuning., we investigate if it surpasses causal attention in visual document retrieval, particularly when using the multi-vector late interaction matching common in SOTA visual retrievers ⁵⁶Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT
Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models.

Figure 5 reports single vector and late interaction results on the ViDoRe benchmark for various model variants. On top of the standard enc (MLM) and dec (CLM) models, we evaluate the dec-enc and the dec models modality aligned with MLM objectives to determine whether bidirectional attention capabilities can be obtained in later stages of training.

Single-vector embedding results are close between bidirectional and causal attention models for document retrieval, with enc slightly outperforming dec by +1.6 nDCG@5.

Intuitively however, bidirectional attention makes a huge difference when used in late interaction settings, substantially exceeding the causal counterpart by +10.6 nDCG@5.

Causal decoders are incapable of correctly contextualizing image or text token representations seen at the beginning of the sequences. This is a key result as almost all current visual retrievers, including late interaction variants, are causal models, clearly indicating some performance is left on the table.

Removing the causal attention mask during training does not suffice to recover the enc late interaction performance at these data regimes. This indicates converting trained decoders as late interaction retrievers is highly non trivial, and confirms the insights from ⁵⁷Weller et al. (2025)Seq vs Seq: An Open Suite of Paired Encoders and Decoders; when possible, training encoder models from scratch remain better for retrieval tasks.

Contrastive Training Design

The previous subsection established bidirectional encoder models to often be the best option when training visual retrievers. In the following experiments, we assess contrastive training choices and only report results for the encoder model for simplicity.

Image resolution benefits are task-specific. Image resolution plays a critical role in VLM generative capabilities, notably in document-focused tasks, as higher-resolution inputs enables the model to capture finer visual cues ⁵⁸Hu et al. (2024)mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding
Marafioti et al. (2025)SmolVLM: Redefining small and efficient multimodal models.

Modality alignment is done at a fixed image resolution of 1024 pixels (longer side) and we report scores of contrastive training runs with varying settings in Table 1. To vary the resolution, images of the highest quality available are scaled to the desired size (often downscaled) before being fed to the image tokenizer. Our findings confirm that embedding tasks are strongly sensitive to image-resolution. In particular, training with higher resolution inputs substantially improves the results on visual document retrieval benchmarks, consistent with prior work in generative settings (Beyer et al. 2024; McKinzie et al. 2024). Furthermore, adding a cool-down phase by showing higher-resolution images towards the end of the modality alignment phase yields additional gains. This suggests that models can adapt their attention mechanisms to finer details when exposed to increased resolution. Interestingly, these findings do not hold in natural image tasks, where high resolution can even degrade performance.

Increasing the pool of contrastive pairs.

A severe limitation that current visual retrievers face is the lack of large volumes of high quality (document image, query pairs). Previous work ⁵⁹Ma et al. (2024)Unifying Multimodal Retrieval via Document Screenshot Embedding
Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
Zhang et al. (2025)GME: Improving Universal Multimodal Retrieval by Multimodal LLMs has relied on a mix of repurposed existing visual question answering datasets and synthetically generated queries with external LLMs. Even put together however, the field is only a year old, and these datasets remain small in size and often of poor quality.

A central question in our study is whether the abundance of text-only query–document pairs can be exploited to improve visual retrieval via cross-modal capability transfer. To probe this, we run contrastive training under three regimes. Unlike prior work that “warms up” visual retrievers or trains exclusively with text-only pairs ⁶⁰Ma et al. (2024)Unifying Multimodal Retrieval via Document Screenshot Embedding
Jiang et al. (2024)E5-V: Universal Embeddings with Multimodal Large Language Models, we interleave text-only pairs and text–image pairs throughout training at a 1:1 ratio. The dataset sources are detailed in Appendix E.1.3.3

As reported in Table 2, incorporating text-only pairs yields a sizeable improvement on visual document retrieval (+1.7 nDCG@5), indicating clear cross-modal transfer—likely facilitated by the backbone’s jointly learned text–image embedding space. This result suggests that domain-specific training corpora can be assembled irrespective of native modality, reducing duplication of effort and lowering data-collection costs.

We further evaluate training with NatCap, a corpus of natural images paired with synthetic, highly detailed captions (see Appendix E.1.3.2). This scaling step improves downstream performance across the board—most notably on natural-image tasks, and with a smaller but consistent gain on document retrieval (+0.2 nDCG@5). Together, these findings underscore the importance of scaling contrastive learning with high-quality data, but which doesn't need to be exclusively image document focused.

Building a Small yet Mighty Visual Retriever.

Training.

Recipe. Putting together the results from our experiments, we devise a training recipe for a small visual document retriever ModernVBERT. It combines a state-of-the-art 150M text bidirectional encoder ⁶¹Weller et al. (2025)Seq vs Seq: An Open Suite of Paired Encoders and Decoders with the ModernBERT architecture ⁶²Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference and a small vision encoder SigLIP2-16B-512 of 100M parameters ⁶³Tschannen et al. (2025)SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. We modality align both models with a MLM objective for 10B tokens, 3 times longer than during our experiments. To boost document understanding, we augment the input image resolution from 1024px to 2048px during a modality alignment cooldown stage (2B tokens). We call the resulting model ModernVBERT.

Following the findings of Section section, we then scale the contrastive training mix from previous experiments to combine document–query pairs with text-only pairs, and use 1 hard negatives for each document-query pair and 2 for each text-only pairs. We opt for a 2/1 text-to-image ratio following our ablation results introduced in Appendix E.3.3.1. This results in ColModernVBERT, a compact late interaction model. For reference, we also train BiModernVBERT, a single vector variant. More training details are provided in Appendix E.1.1.

Results.

ColModernVBERT. The resulting model, ColModernVBERT, showcases strong performance on visual document retrieval benchmarks, especially relative to its size category (Figure 1). Despite having over 10 times fewer parameters than models such as ColPali released only a year ago, it is only 0.6 nDCG@5 points below on the aggregated ViDoRe benchmark scores (Table 3). It also edges many larger single-vector repurposed VLM models released within the year ⁶⁴Chen et al. (2025)MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings
Jiang et al. (2024)E5-V: Universal Embeddings with Multimodal Large Language Models
Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. It however falls short of top model performance on ViDoRe, which is achieved by larger decoder VLMs pretrained and aligned on billions of tokens of text and image data.

Most sub-1B parameter models evaluated on document retrieval benchmarks are dual encoder models, since early fusion generative models that perform well are not common at this scale. The most related model is a 176M late interaction model, ColFlor ⁶⁵Masry & Hoque (2024)ColFlor: Towards BERT-Size Vision-Language Document Retrieval Models, trained from the Florence2 model ⁶⁶Xiao et al. (2023)Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks. ColFlor is 12.7 nDCG@5 points under ColModernVBERT. ColModernVBERT also largely outperforms off-the-shelf dual encoders, even when those have substantially larger parameter counts. These results highlight the benefits of multi-phase training and early fusion architectures for multimodal document-related tasks, even at smaller parameter counts. We also attribute the strong performance of ColModernVBERT at smaller model sizes to the symbiosis of native bidirectional attention and Late Interaction matching, which largely boosts performance relative to comparable decoder models (Section 5.3.1).

Speed. As noted by ⁶⁷Xiao et al. (2025)MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction, multi-vector visual retrievers are not bottlenecked in their inference speed by the late interaction matching operation, but rather by the latency required to encode queries with the text model. Our model demonstrates that strong performance is not incompatible with speed, even when running inference on consumer CPUs, which is the standard setting in most industrial local deployments of text embedding models. Latencies are computed by averaging query encoding times of all NanoBEIR queries, which are 23.4 words and 147.5 characters long on average, and are run with batch size 1 to replicate online use cases. To prevent RAM bottlenecks, we benchmark on very high RAM (2TB) CPU cloud environments, but note models larger than 3B parameters require more than 12 GB RAM to run optimally.⁶⁸With more standard CPU RAM settings such as those found in low-end servers or Google Colab (12GB RAM), models above 3B parameters must rely on memory offloading to run, which adds up to dozens of seconds of latency per query. (Table 3). ModernVBERT achieves more than a 7x speedup on CPU over models with similar performances on ViDoRe. We further report model latency results on GPU hardware in Appendix E.3.6.2. We notably demonstrate that with batched inference, ModernVBERT based query encoders are able to encode 5000 queries per second on Nvidia H100 GPUs. ModernVBert's small model size also enables efficient batching when encoding documents.

Related Work

Repurposing VLMs for Representation Learning. Motivated by the zero-shot performances of generative VLMs ⁶⁹Alayrac et al. (2022)Flamingo: a Visual Language Model for Few-Shot Learning
Beyer et al. (2024)PaliGemma
Bai et al. (2023)Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond, recent studies have explored repurposing these for multimodal embedding tasks ⁷⁰Ma et al. (2024)Unifying Multimodal Retrieval via Document Screenshot Embedding
Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
Zhang et al. (2025)GME: Improving Universal Multimodal Retrieval by Multimodal LLMs. As backbone generative models improved, retriever performance improved as well showcasing the central impact of language model pretraining and modality alignment ⁷¹Xu et al. (2025)Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model
Nussbaum et al. (2024)Nomic Embed: Training a Reproducible Long Context Text Embedder.

These models remain inherently constrained by their causal attention mechanisms, which have been shown in text settings to limit representational efficiency ⁷²Gisserot-Boukhlef et al. (2026)Should We Still Pretrain Encoders with Masked Language Modeling?
Weller et al. (2025)Seq vs Seq: An Open Suite of Paired Encoders and Decoders. Recent work attempts to address this issue by modifying VLM attention during continual pretraining ⁷³Chen et al. (2025)MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings or contrastive tuning ⁷⁴Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks
Xu et al. (2025)Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model, but no recent work attempts to align natively bidirectional language encoder models with vision encoders. The recent release of long sequence text encoders ⁷⁵Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
Boizard et al. (2025)EuroBERT: Scaling Multilingual Encoders for European Languages makes this possible.

Late Interaction in Visual Document Retrieval To further boost performance, visual document retrievers leverage the late interaction mechanism ⁷⁶Khattab & Zaharia (2020)ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT which matches multiple query embeddings with multiple document embeddings through the MaxSim operation ⁷⁷Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Günther et al. (2025)jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval
Xu et al. (2025)Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model. This enables more granular interactions between image and query tokens, at the cost of additional storage and a slight compute overhead during the matching operation. Efficiency gains have come from improving the storage costs through quantization ⁷⁸Bergum (2025)Scaling ColPali to billions of PDFs with Vespa — blog.vespa.ai, token pruning ⁷⁹Faysse et al. (2025)CroissantLLM: A Truly Bilingual French-English Language Model and more recently the use of Matrioshka losses to compact multi-token representations ⁸⁰Xiao et al. (2025)MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction. Ultimately, the performance bottleneck when running visual retrieval inference with such models now resides mostly in the necessity to rely on costly GPU hardware to encode queries, which sets apart text from vision retrieval. This paper fills this gap by using encoders that run on CPU, of parameter sizes comparable to commonly used local text embedding models ⁸¹Chen et al. (2024)BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
Enevoldsen et al. (2025)MMTEB: Massive Multilingual Text Embedding Benchmark.

Conclusion

In this paper, we question design decisions of current VLM-based retriever models, providing crucial insights into what matters when training early-fusion vision encoders. Our study notably shows that these models generally do not improve retrieval on natural-image tasks compared to dual encoders, yet strong vision-language alignment is essential for document-centric retrieval. We uncover a tight synergy between bidirectional attention and late-interaction retrieval, which underscores a fundamental limitation of repurposing decoder-style generative VLMs for retrieval. To mitigate data scarcity in contrastive learning, we propose augmenting limited image-document/text-query pairs with larger, lower-cost corpora from other modalities. Guided by these insights, we trained ColModernVBERT, a compact yet powerful $250$M-parameter multimodal encoder that matches the performance of models up to $10\times$ larger on visual retrieval benchmarks. We release models and training code to help practitioners reduce cost and latency when deploying visual retrievers in real-world applications, and to encourage research on efficient multimodal embedding models.

Future Work & Limitations.

By design, our analysis targets relatively small models. An important next step is to test whether the observed patterns persist at larger scales—for example, to more rigorously probe the interplay between late interaction and bidirectional attention. Our study also focuses exclusively on English. While we expect the broad trends to generalize and see clear value in releasing multilingual variants, it remains unclear how allocating parameters to additional languages trades off against the understanding of the vision modality, and to what extent this penalizes English retrieval performance as the number of languages is scaled ⁸²Fernandes et al. (2023)Scaling Laws for Multilingual Neural Machine Translation. Finally, although we center on retrieval and sequence-level zero-shot classification, the modality-aligned encoder can be fine-tuned for a range of token-level tasks, including OCR error detection, token-level classification, visual named entity recognition, visually grounded token-level object detection, contextual embeddings ⁸³Conti et al. (2025)Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings. We release our base model to encourage exploration of these directions.