Appendices

Appendix E

ModernVBert Appendix

5,613 words26 min read32 sources cited

Training

Implementation and Resources

**Table 1.** Training details of our final models at each training stage. GPU Hours are on 80GB H100 GPUs.
Model	Batch Size	Learning Rate	Training Steps	Training GPU Hours
Modality Alignment
ModernVBERT-base (Table 2)	4096	1e-4	5500	1920h
Contrastive Learning
Generalist contrastive training (Table 4)	256	2e-4	3917	80h
Document Specialization
Document-focused contrastive training w/ hard negatives (Table 4)	64	2e-4	19602	160h

We list hyperparameters and resource details in Table 1 for the various training stages of our final models. We employ ZeRO stage 1 optimizer ¹Rajbhandari et al. (2020)ZeRO: Memory Optimizations Toward Training Trillion Parameter Models for our modality alignment runs. All ablation models are contrastively trained with gradient checkpointing ²Chen et al. (2016)Training Deep Nets with Sublinear Memory Cost to reduce memory usage. All training runs are performed with FlashAttention 2.0 ³Dao (2023)FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. For LoRA configurations, we consistently use a rank r of $32$ , lora\_alpha of $32$ , and a dropout of $0.1$ . For the implementation, we start from m4⁴SmolVLM trainer, https://github.com/huggingface/smollm and ColPali⁵https://github.com/illuin-tech/colpali codebases for training, and use the MTEB⁶https://github.com/embeddings-benchmark/mteb repository for evaluation.⁷github.com/illuin-tech/modernvbert

Similarity Functions

Single-Vector Similarity. For single-vector models, we apply mean pooling for MLM-aligned encoders and end-of-sequence (EOS) pooling for CLM-based models and compute the cosine similarity of a query $q$ and a document $d$ as

\Phi_\mathrm{CosSim}(\mathbf{q},\mathbf{d}) = \exp(\cos(\mathbf{E}_q,\mathbf{E}_d)/\tau)

Multi-Vector Similarity. For multi-vector models, we adopt the standard late-interaction scoring function defined as:

\Phi_\mathrm{LI}(q,d) = \sum_{i \in \llbracket 1, N_q \rrbracket} \max_{j \in \llbracket 1, N_d \rrbracket} \left\langle \mathbf{E}_q^{(i)} , \mathbf{E}_d^{(j)} \right\rangle ,

where $\mathbf{E}_q^{(i)}$ and $\mathbf{E}_d^{(j)}$ denote token-level embeddings for the query and document, respectively.

Data

Modality Alignment Mixture

For our modality alignment trainings, we rely on The Cauldron dataset ⁸Laurençon et al. (2024)What matters when building vision-language models? and its Docmatix extension ⁹Laurençon et al. (2024)Building and better understanding vision-language models: insights and future directions.. Table 2 provides further details on the constitution of this dataset.

**Table 2.** Aggregated statistics of modality alignment datasets from The Cauldron 2 (Laurençon et al. 2024) and Docmatix (Laurençon et al. 2024), showing image counts, QA pairs, token counts, and the proportional contribution of each subsection to the overall mixture.
Dataset Subsection	# Images	# QA Pairs	# Tokens	% Mix
Captioning	609843	612768	62906011	3.13
Real-world VQA	457360	2125615	23318335	1.16
OCR, Document Understanding	2499258	11415478	426806479	21.21
Chart/Figure Understanding	539743	24444120	30315784	1.51
Table Understanding	163568	229077	21371931	1.06
Reasoning, Logic, Maths	490870	2212629	32450213	1.61
Screenshot to Code	547974	548296	336299551	16.71
Text-only Instructions	0	21482682	1079001075	53.61
Total	5308616	63070665	2012469379	100.00

NatCap

To enrich our contrastive learning data mixture, we construct NatCap (Natural Captions), a large-scale dataset containing around 333000 contextualized image–caption pairs. This dataset is created by generating synthetic captions, along with cross-class and in-class discriminative tags, from existing image classification datasets (see Table 3). For this purpose, we leverage Gemini-flash-2.5¹⁰https://ai.google.dev/gemini-api/docs/models?hl=fr\#gemini-2.5-flash which produces captions conditioned on both the image content and the accompanying dataset metadata, as illustrated in Figure 1. We detail the prompt below.

**Table 3.** *NatCap* Dataset Composition. *NatCap* spans 13 different sources covering various images types. The total dataset is composed of 333k pairs
Dataset	Description	# Items
Caltech101	General objects.	3000
Caltech256	General objects.	30000
Cars	Car model classification.	8000
Country211	Country where the picture is taken.	28000
DTD	Describable textures (texture attributes).	4000
EuroSat	Land use / area zone type.	16000
FER2013	Facial emotion recognition.	28000
FGCVAircraft	Aircraft model recognition.	3000
Food101	Food categories.	75000
OxfordPets	Dog/cat species.	3000
RESISC45	Aerial scene / area zone type.	18000
SUN397	General scenes.	109000
VOC2007	General objects.	8000
TOTAL		333000

**Figure 1.** Example from the NatCap dataset

Contrastive Training Mix

In this subsection, we describe the composition of our data mixes used in the contrastive training stages. Table 4 outlines the datasets included in each mix, including the Document-Focused variant employed for ColModernVBERT.

**Table 4.** **Data mixes for contrastive trainings.** The *Generalist Mix* spans over 1M diverse pairs, while the *Document-Focused Mix* emphasizes document retrieval with extra ColPali epochs.
Source	Description	Pairs	Epochs
Generalist Mix
ColPali (Faysse et al. 2025)	Query–Document images for visual retrieval	118k	1
MSCOCO (Lin et al. 2014)	Natural images with human-written captions	118k	1
NatCap (ours, subsampled)	Diverse images with synthetic captions	118k	1
RLHN (Thakur et al. 2025)	Text–text pairs for complex retrieval	680k	1
TOTAL		1030k
Document-Focused Mix
ColPali (Faysse et al. 2025)	Query–Document images for visual retrieval	118k	3
RLHN (Thakur et al. 2025)	Text–text pairs for complex retrieval	300k	3
TOTAL		1254k

Baselines Details

In this section, we describe the models evaluated in as comparison to our document retriever model.

MoCa-3B ¹¹Chen et al. (2025)MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings. A modality-aware continual pretraining model that transforms a causal vision-language model into a bidirectional multimodal embedding model, using interleaved image-text reconstruction and contrastive alignment to support cross-modal retrieval.

GME-Qwen2 ¹²Zhang et al. (2025)GME: Improving Universal Multimodal Retrieval by Multimodal LLMs. A unified multimodal embedder built on Qwen2-VL ¹³Wang et al. (2024)Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, which produces shared embedding representations across text, image, and fused input modalities, enabling universal multimodal retrieval.

VLM2Vec ¹⁴Jiang et al. (2024)VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. A method that trains a vision-language encoder by converting a VLM through extensive contrastive post-training. Flagship model is based on the model Phi-3.5 ¹⁵Abdin et al. (2024)Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.

E5-V ¹⁶Jiang et al. (2024)E5-V: Universal Embeddings with Multimodal Large Language Models. An adaptation of the E5 embedding approach to multimodal models: it trains only on text pairs yet bridges the modality gap to handle image inputs, reducing cost while achieving universal embeddings.

ColPali ¹⁷Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models. A vision-based document retrieval model that processes document pages as images (no OCR) and produces multi-vector embeddings via a late-interaction mechanism over PaliGemma ¹⁸Beyer et al. (2024)PaliGemma: A versatile 3B VLM for transfer, enabling efficient and accurate retrieval.

ColQwen2.5 ¹⁹Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models. An extension of ColPali ²⁰Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models using Qwen2-VL ²¹Wang et al. (2024)Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution as the backbone, carrying forward the late interaction retrieval paradigm over page image embeddings, capturing layout and textual context without OCR.

Jina-v4 ²²Günther et al. (2025)jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval. A multimodal embedding model combining visual and textual inputs with support for multi-vector (late interaction) embeddings, using adapters over a unified backbone to excel on visually rich document retrieval.

NemoRetriever ²³Xu et al. (2025)Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model. An LI retriever that combines vision-language embeddings with a ColEmbed design, enabling high performance on visual document retrieval with structured patch matching and efficient similarity.

Jina CLIP ²⁴Koukounas et al. (2024)Jina CLIP: Your CLIP Model Is Also Your Text Retriever. A smaller scale vision-language model using CLIP embeddings, applied to document retrieval tasks; although not LI, it offers a lightweight multimodal baseline.

BGE Visualized M3 ²⁵Zhou et al. (2024)VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval. A vision-enhanced version of BGE M3 ²⁶Chen et al. (2024)BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation that supports visual inputs and extends embedding models into multimodal domains.

SigLIP2-L-512/16 ²⁷Tschannen et al. (2025)SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. A multilingual vision-language bi-encoder model, which combines image and text modalities to yield unified embeddings across languages. This configuration handles images of 512x512 pixels and create subpatches of 16x16 pixels.

ColFlor ²⁸Masry & Hoque (2024)ColFlor: Towards BERT-Size Vision-Language Document Retrieval Models. A lightweight OCR-free visual document retriever with only 174M parameters built over Florence-2 and DaViT, delivering strong performance near ColPali with much lower computational cost and much faster encoding.

Additional Ablations

Performance Against Off-the-Shelf Dual Encoder

We study whether the off-the-shelf performance of the standalone vision tower outweighs the burden of adding language parameters and re-training through language modeling, as proposed in our work. Figure 2 shows the results of the various models on the tasks described in Section 5.2. Similarly to Section 5.3.1, we observe that the early fusion model trained with an LM objective significantly outperforms the standalone vision tower on document retrieval tasks (+10.9 nDCG@5). It even surpasses the larger dual encoder (+4.8 nDCG@5) on these latter tasks. We note that the standalone vision tower largely outperforms the early fusion models on the other natural-image tasks, supporting the use of the SigLIP model for these tasks as found in various general benchmarks ²⁹Xiao et al. (2025)MIEB: Massive Image Embedding Benchmark.

Impact of Modality Alignment objective on downstream tasks. Early Fusion of vision and text models boosts document retrieval tasks regardless of the LM objective, but degrades natural image and classification tasks w.r.t. the standalone off — **Figure 2.** **Impact of Modality Alignment objective on downstream tasks.** Early Fusion of vision and text models boosts document retrieval tasks regardless of the LM objective, but degrades natural image and classification tasks w.r.t. the standalone *off-the-shelf* vision model SigLIP. Reported scores are aggregated MIEB scores (nDCG, Accuracy.)

Scaling Dynamics of Attention Masks

We study the different training dynamics of the different training objectives. We compare the enc (MLM) approach with a traditional dec (CLM) objective. Figure 3 presents the performance of the two training objectives across a diverse set of tasks. While starting dec offers an advantage in low-data regimes, enc seems to catch up. In document retrieval tasks, it eventually surpasses dec and scales better.

Attention masks impact on modality alignment phase scaling. The dashed line marks the vision tower baseline. The orange curve shows the model initialized from a decoder LM with a CLM objective, and the blue curve shows the model trained wit — **Figure 3.** **Attention masks impact on modality alignment phase scaling.** The dashed line marks the vision tower baseline. The orange curve shows the model initialized from a decoder LM with a *CLM* objective, and the blue curve shows the model trained with an *MLM* objective from an encoder LM. CLM performs better in low-data regimes, but MLM scales more effectively, surpassing CLM in document retrieval, while captioning and classification remain below the CLIP baseline.

Bridging the Gap with Longer Contrastive Training

We study the impact of additional in-distribution training pairs on embedding tasks by scaling the contrastive training stage. Starting from the final checkpoint of our encoder-based ablation model, we double the contrastive dataset size at each step and train until convergence³⁰To avoid overfitting, we set an early stopping on an eval set. We limit the number of step to one epoch on the full dataset.. This setup tests whether scaling continues to improve performance. Figure 4 shows the scaling behavior. Performance improves overall with more in-distribution data. The vision-tower baseline is quickly surpassed on visual document benchmarks, and scaling narrows the gap on other tasks³¹Note that the models probably won't fully recover baseline vision-tower performance. This highlights the need to choose models according to use case (e.g., lightweight CLIP-like models for image classification).. We note a plateau in captioning and classification, pointing to the need for more diverse data.

**Figure 4.** Contrastive training scaling. Each dot on the blue curve represents one fraction of the baseline contrastive training mix (ColPali + MSCOCO). Performance improves with more in-distribution data, surpassing the baseline on document benchmarks and narrowing the gap on image captioning. There is no clear improvement in image classification, highlighting the need for more diverse pairs.

Optimal Text-To-Image Ratio for Document Retrieval

Our findings in subsection section indicate that incorporating additional text-only pairs boosts document retrieval performance. While our initial experiment employed a 1:1 text-to-image ratio, we further investigate how varying this ratio impacts our broad set of tasks. We start from the best contrastive mix in Table 1, and vary the text-to-image ratio. As shown in Figure 5, increasing the number of text-only pairs for a fixed amount of image pairs consistently enhances retrieval performance. However, for natural image classification tasks, adding more text does not appear to provide benefits.

Optimal text-to-image ratio in contrastive training mix. Increasing the ratio in retrieval tasks consistently improves the performances. — **Figure 5.** **Optimal text-to-image ratio in contrastive training mix.** Increasing the ratio in retrieval tasks consistently improves the performances.

Late Interaction for Non-Documental Retrieval

**Table 5.** **Generalist retrieval performances.** Late interaction benefits extend to non-documental retrieval tasks. Our multi-vector model increases its single-vector counterpart across all tasks, surpassing larger VLM-based retrievers.
		Document Retrieval		Image/Caption Retrieval
	Model Size	ViDoRe(v1)	ViDoRe(v2)	MSCOCO (T $\rightarrow$ I)	Flickr30k (T $\rightarrow$ I)	Average
CLIP Encoders
siglip2-base-patch16-512	376M	36.6	23.4	66.2	86.9	53.3
siglip2-large-patch16-512	882M	43.8	27.0	67.1	88.9	56.7
clip-vit-base-patch16	151M	25.5	20.4	50.3	76.8	43.3
clip-vit-large-patch14	428M	38.0	28.6	52.7	79.3	49.6
VLM-based Encoders
VLM2Vec-Full	4150M	49.8	36.5	59.5	81.8	56.9
e5-v	8360M	62.7	49.4	68.1	89.8	67.5
Early Fusion Encoders
bge-visualized-base	196M	10.3	9.0	50.0	74.1	35.9
bge-visualized-m3	873M	12.4	10.2	39.6	69.0	32.8
*ModernVBERT-embed*	252M	58.4	36.9	56.5	76.0	56.9
ModernVBERT-embed (multi-vector)	252M	76.5	53.9	61.8	81.4	68.4

We want to study if the multi-vector gains transfer to non-documental retrieval. To do so, we contrastively post-train our base model on our generalist post-training mix presented in Table 4. The late interaction generalist exhibits superior performance in retrieval setting, improving its single-vector performance by +20.2% (11.5 points), matching the performance of substantially larger VLM-based retrievers like E5-V (8.3B parameters, 67.5 points) and surpassing dual encoders like SigLIP (882M parameters, 56.7 points). This matches the capabilities observed in Section 5.3.1 for documental settings for models with native bidirectional attention, extending it to natural image tasks. This result extends the prevailing understanding from the document retrieval community, where the superiority of late-interaction is well-documented ((Khattab & Zaharia 2020), (Chaffin 2025), (Faysse et al. 2025)). While this performance gap is widely accepted for document retrieval, its applicability to caption matching tasks has not really been addressed. Our findings provide strong evidence that the fine-grained matching capabilities of late-interaction models are a key driver of performance in this domain too.

Model Merging

Our contrastive learning stage provides direct performance trade-offs on different tasks. Following recent trends, we evaluate how model merging techniques help mitigate performance degradation on specific tasks, while maintaining the performance enabled by the contrastive training ³²Sung et al. (2023)An Empirical Study of Multimodal Model Merging
Dziadzio et al. (2024)How to Merge Your Multimodal Models Over Time?
Li et al. (2024)Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Merging
Zhang et al. (2025)Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. We merge our ablation model after modality alignment with the checkpoint after the full contrastive learning with two methods: SLERP ³³Ilharco et al. (2022)Patching open-vocabulary models by interpolating weights and average merging ³⁴Shoemake (1985)Animating rotation with quaternion curves. For SLERP, we compare three values for the $\lambda$ coefficient ( $0.25$ , $0.5$ , $0.75$ ). Figure 6 displays the trends with the best method (SLERP, $\lambda = 0.75$ ). As we can see, the merged model mitigates the performance drop in Image/Caption Retrieval tasks, while maintaining significant gains on Image Classification tasks. However, merging strongly degrades performance on Document Retrieval, showing that benefits of merging embedding models are task-dependent.

**Figure 6.** Merging model results across tasks. Benefits are task-dependent, with performance degradation w.r.t. both original models in Document Retrieval.

Curriculum For Document Retriever Contrastive Post-Training

**Table 6.** **Performance of *ModernVBERT* Document Specialisation Curriculums.** This table presents the performance of various contrastive training curriculums starting from *ModernVBERT*-base, on the ViDoRe(v1) and ViDoRe(v2) benchmarks. The generalist contrastive learning mix used in the last two models is detailed in Table 4. We see that a preliminary stage of generalist contrastive learning harms the final document retrieval performance, regardless of whether a multi-vector approach is used.
	ViDoRe(v1)	ViDoRe(v2)	Average
Document retrieval contrastive training starting checkpoint
ModernVBERT-base	81.2	56.0	68.6
+ multi-vector generalist CL	80.7	55.4	68.1
+ single-vector generalist CL	80.6	54.0	67.3

We conduct an ablation study to determine the optimal contrastive training curriculum for specializing ModernVBERT in document retrieval. Specifically, we investigate whether a preliminary generalist contrastive training phase, intended to leverage a larger dataset, improves downstream performance. As shown in Table 6, our results demonstrate that this initial generalist phase is detrimental to final performance ( $-0.5\%$ ). The optimal strategy is to specialize the model on the target task directly after its initial Masked Language Modeling (MLM) alignment.

Text-Only Retrieval

**Table 7.** Average NDCG@5 of *ColModernVBERT* on NanoBEIR, a text retrieval benchmark with multiple sub domains. *For multimodal models, we only consider parameters of the text encoder
Model	Params (M)	NDCG@5
Statistical
BM25s	—	0.559
Single Vector
Jina Embeddings v4	3577*	0.623
E5-large-v2	335	0.605
bge-m3 (Bi Encoder)	567	0.590
Qwen3-Embedding-0.6B	600	0.567
Multi Vector
LightOn GTE-ModernColBERT v1	149	0.669
Jina ColBERT v2	137	0.642
bge-m3 (Late Interaction)	567	0.606
ColBERT v2	110	0.593
Colqwen2-v1.0	1580*	0.593
ColModernVBERT	150*	0.589
Colqwen2.5-v0.2	3145*	0.589

The results in Table 7 detail the performance of ColModernVBERT and other baselines on the NanoBEIR text retrieval benchmark. It achieves an average NDCG@5 score competitive with single and multi vector models specialized for text, even without explicit optimization for this modality. This performance is encouraging and indicates a promising direction for training a unified model for both text and image retrieval.

Model Latency

Image Resolution Tradeoffs

Figure 7 presents the pixel shuffling trade-off. Processing larger images creates more visual tokens, leading to very long sequences (around $17'500$ tokens for a 2048x2048 px image with no pixel shuffling). Pixel shuffling allow to compress these sequence by concatenating the embeddings of spatially close patches. This diminishes the number of tokens for longer visual token embeddings. Table 8 presents the latency to process one image of various resolutions on one L4 GPU and CPU.

Image processing parameters impact on visual tokens. Here we assume a square image for simplicity. Scaling the image size introduces naturally more tokens, but having a large enough pixel shuffling ratio ( r ≥ 4 r\geq4 r ≥ 4 ) allows to cou — **Figure 7.** **Image processing parameters impact on visual tokens.** Here we assume a square image for simplicity. Scaling the image size introduces naturally more tokens, but having a large enough pixel shuffling ratio ( $r\geq4$ ) allows to counterbalance by concatenating spatially close patch representations.

**Table 8.** *ModernVBERT* image processing latency. Computing the average time to process a single image on GPU and CPU. The average is computed on 100 images. The values represent the mean latency in milliseconds, with the standard deviation included in parenthesis.
	Num. Visual Tokens	CPU Latency (ms)	GPU Latency (ms)
$512$ px	$128$	$287.2_{(\pm 7.8)}$	$43.6 _{(\pm 1.4)}$
$1024$ px	$320$	$1015.8 _{(\pm 58.1)}$	$150.3 _{(\pm 2.5)}$
$2048$ px	$1088$	$2572.0 _{(\pm 63.9)}$	$363.4 _{(\pm 4.6)}$

Online Query Encoding Latency

We evaluate the query embedding speed of our model on GPU. We use a single Nvidia H100 with 80GB of VRAM. As for Section 5.4.2, latencies are computed in batch size 1 to simulate online situations, and are averaged over all NanoBEIR queries. Only the text parameters are loaded and run, to minimize memory usage. Parameters are cast to bfloat16 and Flash Attention 2 is used. The resulting speeds are often much faster than those obtained by running inference through each model's reference implementation. Results are shown in Table 9). Interestingly in this setup where memory is not a bottleneck, model depth seems to be a large performance driver, sometimes more the parameter count. We finally evaluate batched GPU throughput. We use batches of size 512 by default and iteratively half it when memory is insufficient. We observe that ModernVBERT based models are extremely fast and can process 5000 queries per second. In the table, the reported figures correspond to the inverted throughput (latency per batch divided by the number of queries per batch). These speed and throughput gains are made possible due to a combination of size, and efficient hardware-informed design as well as the support of flash attention and sequence packing other models of the size often lack ³⁵Warner et al. (2024)Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference.

**Table 9.** **Text query encoding latency.** The latency is computed both on high-end CPUs (1TB RAM, 128 cores) and GPU (Nvidia H100, 80GB) (mean ± std). Since only 649 queries are used, standard deviations are not reported in GPU batching mode (batches of 512 queries by default), for which we report the inverse throughput (average latency per batch divided by the batch size).
	Late Interaction	Model Size (B)	CPU Latency (ms)	GPU Latency (ms)	GPU Batching (ms)
$\ge$ 1B Parameters
MoCa-3B		3.75	$158_{(\pm 147)}$	$26_{(\pm 3)}$	$4.54$
VLM2Vec		4.15	$211_{(\pm 253)}$	$21_{(\pm 3)}$	$2.82$
GME-Qwen2-7B		8.29	$412_{(\pm 411)}$	$25_{(\pm 1)}$	$9.07$
E5-V		8.36	$434_{(\pm 379)}$	$22_{(\pm 2)}$	$9.55$
ColPali	✓	2.92	$175_{(\pm 113)}$	$14_{(\pm 1)}$	$3.07$
ColQwen2.5	✓	3.75	$158_{(\pm 147)}$	$26_{(\pm 2)}$	$26$
Jina-v4	✓	3.75	$158_{(\pm 147)}$	$26_{(\pm 2)}$	$4.54$
NemoRetriever-3B	✓	4.40	$155_{(\pm 118)}$	$20_{(\pm 2)}$	$4.59$
$\le$ 1B Parameters
Jina CLIP		.22	$14_{(\pm 7)}$	$6_{(\pm 2)}$	$.69$
BGE Visualized M3		.87	$38_{(\pm 42)}$	$10_{(\pm 2)}$	$.77$
SigLIP2-L-512/16		.88	$25_{(\pm 8)}$	$6_{(\pm 1)}$	$.10$
ColFlor	✓	.17	$17_{(\pm 9)}$	$8_{(\pm 2)}$	$.31$
BiModernVBERT (ours)		.25	$20_{(\pm 11)}$	$14_{(\pm 2)}$	$.20$
ColModernVBERT (ours)	✓	.25	$20_{(\pm 11)}$	$14_{(\pm 2)}$	$.20$

Annotation prompt

You are an image annotator expert.

You will receive an image along with its classification label and the classification task scope, and your task is to provide contextualized metadata about it.

The output should be a JSON object with the following metadata fields:

caption: A descriptive caption of the image accounting for its label. This should be a unique and concise sentence that describes the image in detail.
class_tags: A list of tags that represents the image and can help identify the class. (e.g., for a car image with its model as a class, this could be some specific attribute of the car)
other_tags: A list of tags that represents the image but can help identify the image among others of the same class. (e.g., for a car image with its model as a class, this could be its color or the background of the image)
is_image_class_explicit: Boolean, could the class be inferred from the image alone? (e.g., the class is a country and you cannot necessarily infer it from the image alone, so this would be false)

Please ensure that the output is in valid JSON format.

Example:
You receive an image of what is clearly a car with its model as a class (here Audi TTS coupe 2012) for a car model classification task.
The output could be a JSON object like this:

{
  "caption": "A red Audi TTS coupe 2012 car parked on a sunny street 
                in front of a sport shop.",
  "class_tags": ["sport coupe","four door coupe","17'' alloy wheels"],
  "other_tags": ["sunny street","parked","red","sport shop"],
  "is_image_class_explicit": true
}

——
Classification scope: {task_info}
Image label: {label}
Answer:

References

Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor Sanh (2024). What matters when building vision-language models?. arXiv. Source ↗
Hugo Laurençon, Andrés Marafioti, Victor Sanh, Léo Tronchon (2024). Building and better understanding vision-language models: insights and future directions..
Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, Pierre Colombo (2025). ColPali: Efficient Document Retrieval with Vision Language Models. International Conference on Learning Representations (ICLR 2025). Source ↗
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, Piotr Dollár (2014). Microsoft COCO: Common Objects in Context. arXiv. Source ↗
Nandan Thakur, Crystina Zhang, Xueguang Ma, Jimmy Lin (2025). Fixing Data That Hurts Performance: Cascading LLMs to Relabel Hard Negatives for Robust Information Retrieval. Source ↗
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, Yuxiong He (2020). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. Source ↗
Tianqi Chen, Bing Xu, Chiyuan Zhang, Carlos Guestrin (2016). Training Deep Nets with Sublinear Memory Cost. Source ↗
Tri Dao (2023). FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. Source ↗
Haonan Chen, Hong Liu, Yuping Luo, Liang Wang, Nan Yang, Furu Wei, Zhicheng Dou (2025). MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings. Source ↗
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi Dai, Dingkun Long, Pengjun Xie, Meishan Zhang, Wenjie Li, Min Zhang (2025). GME: Improving Universal Multimodal Retrieval by Multimodal LLMs. Source ↗
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, Junyang Lin (2024). Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. Source ↗
Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen (2024). VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks. Source ↗
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Parul Chopra, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Dan Iter, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell J. Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Chen Liang, Weishung Liu, Eric Lin, Zeqi Lin, Piyush Madan, Arindam Mitra, Hardik Modi, Anh Nguyen, Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sambudha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shital Shah, Ning Shang, Hiteshi Sharma, Xia Song, Masahiro Tanaka, Xin Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Michael Wyatt, Can Xu, Jiahang Xu, Sonali Yadav, Fan Yang, Ziyi Yang, Donghan Yu, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang, Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, Xiren Zhou (2024). Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv. Source ↗
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang (2024). E5-V: Universal Embeddings with Multimodal Large Language Models. Source ↗
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bošnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier Henaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, Xiaohua Zhai (2024). PaliGemma: A versatile 3B VLM for transfer. Source ↗
Michael Günther, Saba Sturua, Mohammad Kalim Akram, Isabelle Mohr, Andrei Ungureanu, Bo Wang, Sedigheh Eslami, Scott Martens, Maximilian Werk, Nan Wang, Han Xiao (2025). jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval. Source ↗
Mengyao Xu, Gabriel Moreira, Ronay Ak, Radek Osmulski, Yauhen Babakhin, Zhiding Yu, Benedikt Schifferer, Even Oldridge (2025). Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model. Source ↗
Andreas Koukounas, Georgios Mastrapas, Michael Günther, Bo Wang, Scott Martens, Isabelle Mohr, Saba Sturua, Mohammad Kalim Akram, Joan Fontanals Martínez, Saahil Ognawala, Susana Guzman, Maximilian Werk, Nan Wang, Han Xiao (2024). Jina CLIP: Your CLIP Model Is Also Your Text Retriever. arXiv. Source ↗
Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, Yongping Xiong (2024). VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval. Source ↗
Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, Zheng Liu (2024). BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv. Source ↗
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, Xiaohua Zhai (2025). SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features. Source ↗
Ahmed Masry, Enamul Hoque (2024). ColFlor: Towards BERT-Size Vision-Language Document Retrieval Models.
Chenghao Xiao, Isaac Chung, Imene Kerboua, Jamie Stirling, Xin Zhang, Márton Kardos, Roman Solomatin, Noura Al Moubayed, Kenneth Enevoldsen, Niklas Muennighoff (2025). MIEB: Massive Image Embedding Benchmark. Source ↗
Omar Khattab, Matei Zaharia (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. Source ↗
Antoine Chaffin (2025). GTE-ModernColBERT. Source ↗
Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, Lijuan Wang (2023). An Empirical Study of Multimodal Model Merging. Source ↗
Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Albanie, Matthias Bethge (2024). How to Merge Your Multimodal Models Over Time?. Source ↗
Mingxin Li, Zhijie Nie, Yanzhao Zhang, Dingkun Long, Richong Zhang, Pengjun Xie (2024). Improving General Text Embedding Model: Tackling Task Conflict and Data Imbalance through Model Merging. Source ↗
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, Jingren Zhou (2025). Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models. Source ↗
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, Ludwig Schmidt (2022). Patching open-vocabulary models by interpolating weights. Source ↗
Ken Shoemake (1985). Animating rotation with quaternion curves. Association for Computing Machinery. Source ↗
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, Iacopo Poli (2024). Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference. Source ↗