ViDoRe Benchmarks: V2 & V3

Introduction

Retrieval-Augmented Generation (RAG) ³Lewis et al. (2020)Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks has become the dominant paradigm for knowledge-intensive NLP tasks ⁴Gao et al. (2023)Retrieval-Augmented Generation for Large Language Models: A Survey
Fan et al. (2024)A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models. Yet practical deployments introduce complexities that academic benchmarks often overlook when focusing on single-document textual retrieval. First, documents encode critical information in visual elements such as tables, charts, and images designed for human interpretation, which text-only pipelines often ignore ⁵Abootorabi et al. (2025)Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Cho et al. (2024)M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding. Second, user queries often require open-ended synthesis, comparison, and reasoning over scattered information, not simple factoid lookup ⁶Tang & Yang (2024)MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
Conti et al. (2025)Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings
Thakur et al. (2025)FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents. Third, trustworthy systems must ground responses to specific source locations (e.g., bounding boxes), to mitigate hallucinations ⁷Gao et al. (2023)Enabling Large Language Models to Generate Text with Citations
Ma et al. (2024)VISA: Retrieval Augmented Generation with Visual Source Attribution.

Existing benchmarks leave these requirements only partially addressed. Early Visual Document Understanding (VDU) benchmarks focus on single-page comprehension, ignoring the complexity of large document corpora ⁸Mathew et al. (2020)DocVQA: A Dataset for VQA on Document Images. Recent retrieval-centric benchmarks do not evaluate generation quality and grounding ⁹Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Günther et al. (2025)jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval. Some multimodal datasets attempt to bridge this gap but rely on extractive, short-answer tasks that fail to exercise complex reasoning ¹⁰Cho et al. (2024)M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding, or lack multilingual diversity and fine-grained visual grounding ¹¹Peng et al. (2025)UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG.

To address these limitations, we introduce ViDoRe V3, a benchmark designed for complex and realistic end-to-end RAG evaluation on visually rich document corpora. Our contributions are:

1. A Human Annotation Methodology for Realistic Queries

We propose an annotation protocol for generating diverse queries and fine-grained query-page annotations. By restricting annotator access to document content during query formulation, we capture authentic search behaviors and mitigate bias toward simple extractive queries. Vision-Language Model (VLM) filtering combined with human expert verification enables efficient, high-quality annotation at scale.

2. The ViDoRe V3 Benchmark

Applying this methodology to 10 industry-relevant document corpora, we build ViDoRe V3, a multilingual RAG benchmark comprising 26,000 pages and 3,099 queries, each available in 6 languages. Two datasets are held out as a private test set to mitigate overfitting. The public datasets are released under a commercially permissive license.

3. Comprehensive Evaluation and Insights

Leveraging our granular annotations, we benchmark state-of-the-art models on (i) retrieval accuracy by modality and language, (ii) answer quality across diverse retrieval pipeline configurations, and (iii) visual grounding fidelity. Our analysis surfaces actionable findings for RAG practitioners.

Related Work

Component-Level Benchmarks (VDU and Retrieval)

VDU has traditionally relied on single-page datasets like DocVQA ¹²Mathew et al. (2020)DocVQA: A Dataset for VQA on Document Images, alongside domain-specialized variants ¹³Mathew et al. (2021)InfographicVQA
Zhu et al. (2022)Towards Complex Document Understanding By Discrete Reasoning
Wang et al. (2024)Charxiv: Charting gaps in realistic chart understanding in multimodal llms. These ignore the multi-page context inherent to RAG. Recent work evaluating bounding-box source grounding ¹⁴Yu et al. (2025)BBox DocVQA: A Large Scale Bounding Box Grounded Dataset for Enhancing Reasoning in Document Visual Question Answer proposes single-page and multi-page tasks but does not address the retrieval component. Conversely, the emergence of late-interaction visual retrievers ¹⁵Ma et al. (2024)Unifying Multimodal Retrieval via Document Screenshot Embedding
Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Yu et al. (2024)VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documents
Xu et al. (2025)Llama Nemoretriever Colembed: Top-Performing Text-Image Retrieval Model spurred the creation of retrieval-centric visual benchmarks like Jina-VDR ¹⁶Günther et al. (2025)jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval and ViDoReV1&V2 ¹⁷Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Macé et al. (2026)ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval, but none of these benchmarks jointly evaluate retrieval and answer generation.

End-to-End Multimodal RAG

While recent textual RAG benchmarks now capture complex user needs like reasoning or summarizing ¹⁸Thakur et al. (2025)FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
Tang & Yang (2024)MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries
Su et al. (2024)Bright: A realistic and challenging benchmark for reasoning-intensive retrieval, multimodal evaluation often remains limited to single page queries ¹⁹Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models. Multi-page datasets like DUDE ²⁰Van Landeghem et al. (2023)Document understanding dataset and evaluation (dude), M3DocRAG ²¹Cho et al. (2024)M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding, ViDoSeek ²²Wang et al. (2025)Vidorag: Visual document retrieval-augmented generation via dynamic iterative reasoning agents or Real-MM-RAG ²³Wasserman et al. (2025)REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark prioritize extractive retrieval, lacking the diversity of queries encountered in realistic settings. UniDocBench ²⁴Peng et al. (2025)UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG represents a concurrent effort that similarly addresses diverse query types and provides comparative evaluation across multiple RAG paradigms. While this benchmark offers valuable contributions, it relies on synthetically generated queries via knowledge-graph traversal, is restricted to English documents, and constrains grounding annotations to parsed document elements. In contrast, our benchmark offers several complementary strengths: fully human-verified annotations, multilingual coverage, free-form bounding box annotations, and a more systematic evaluation of individual visual RAG pipeline components.

Benchmark Creation

We design the benchmark to mirror the diversity of information retrieval situations in large-scale realistic environments. To enable pipeline-agnostic evaluation of the 3 core RAG components (retrieval, generation and grounding), while avoiding limitations of synthetic benchmarks, we employ a rigorous three-stage human-in-the-loop annotation process involving document collection, query generation and grounded query answering (Figure 2).

Document Collection

We curate 10 diverse corpora by manually selecting openly-licensed documents from governmental, educational, and industry sources, focusing on English and French documents (7 and 3 corpora respectively). The corpora span Finance, Computer Science, Energy, Pharmaceuticals, Human Resources, Industrial Maintenance, Telecom, and Physics. Each features domain-specific terminology and document structures representative of realistic retrieval tasks (details in Table 2).

Query Generation

Query Taxonomy

To evaluate document visual retrieval systems across diverse realistic scenarios, we develop a query taxonomy with two orthogonal dimensions: Query Type, defining the user's information need, and Query Format, describing the query's syntactic structure. This dual-axis classification enables more nuanced performance analysis than benchmarks focusing solely on interrogative extractive queries. We define 7 Query Types: open-ended, extractive, numerical, multi-hop, compare-contrast, boolean, and enumerative, and 3 Query Formats: question, keyword, and instruction.

Context Preparation

We further ensure query diversity by pulling summaries from a heterogeneous set of contexts during the generation process.

Two types of input contexts are used: specific document sections that target local information retrieval and cross-section summaries that target multi-document context retrieval. These summaries are produced through a refined process inspired by ViDoRe V2 ²⁵Macé et al. (2026)ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval. First, the text is extracted from PDFs using Docling ²⁶Team (2024)Docling Technical Report along with image descriptions. Then, summaries are generated with Qwen3-235B-Instruct ²⁷Team (2025)Qwen3 Technical Report from each document section. They are clustered to group similar summaries together using Qwen3-Embedding-0.6B ²⁸Zhang et al. (2025)Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models as embedder, UMAP ²⁹McInnes et al. (2020)UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction for dimension reduction and HDBSCAN ³⁰Campello et al. (2013)Density-based clustering based on hierarchical density estimates for clustering. Additionally, cross-section summaries are produced by synthesizing the summaries of 2 to 3 randomly selected sections per cluster. From this pool of summaries, a final subset is curated to maintain a strict balance between single-section and cross-section summaries. The selection also ensures an even distribution across section modalities (text, images, and tables) as defined by the Docling element classification.

Synthetic Query Generation

Queries are generated from the summaries using a first synthetic generation pipeline based on Qwen3-235B. For each summary, a prompt is constructed by sampling a query type and format at random, together with variable attributes such as length and difficulty, in order to promote diversity. The generated queries are subsequently evaluated by the same LLM acting as an automatic judge, which filters outputs according to 4 criteria: information richness, domain relevance, clarity and adherence to query type/format. Finally, 50% of the retained queries are rephrased to further enhance linguistic variance. This pipeline is implemented using NeMo Data Designer ³¹Team (2025)NeMo Data Designer: A framework for generating synthetic data from scratch or based on your own seed data to facilitate generation scaling.

Human Query Writing

Human annotators are provided 2 kinds of contexts: synthetic summaries or specific PDF pages. They are tasked with generating one query following a specific query type and format and one query of their choice that is most adapted to the context provided.

Answer Detection and Generation

Queries are filtered and linked to relevant pages using a hybrid pipeline of VLM pre-filtering and human annotation. It is followed by human answer annotation and visual grounding.

Query-Page Linking

Given the scale of our corpora, manual verification of each page relevance for each query is intractable. We therefore adopt a two-stage annotation pipeline. First, Qwen2.5-VL-32B-Instruct ³²Bai et al. (2025)Qwen2.5-VL Technical Report pre-filters candidate pages by assessing whether each page image is relevant to the query. Queries whose answers span more than 30 flagged pages are discarded. Human annotators then review the remaining query-page pairs, evaluating query quality and rating page relevance on a three-point scale (Not Relevant, Critically Relevant, Fully Relevant).

Relevant Page Selection

To ensure annotation quality, each task is completed by multiple annotators and reviewed by annotation supervisors. Since VLM pre-filtering biases the distribution toward relevant pages, we report Gwet's AC2, as it remains stable under prevalence skew, at 0.760 (see Section D.4 for dataset-level breakdowns). Given this strong but imperfect agreement, we implement a tiered review process: extractive queries require at least one annotator and one reviewer, while more complex non-extractive queries require at least two annotators and one reviewer. A page is retained as relevant if marked by either (i) one annotator and one reviewer, or (ii) at least two annotators.

Answer Generation

For each selected query, annotators were tasked with writing an answer based on the pages they marked as relevant. Given that different annotators might have different answer interpretations and tend not to be exhaustive in their answers, we use Qwen2.5-VL-32B-Instruct to generate a final answer based on the relevant page images marked by the annotators and their answers.

Bounding Boxes and Modality Types

For each relevant page, annotators delineate bounding boxes around content supporting the query and attribute a modality type to each bounding box: Text, Table, Chart, Infographic, Image, Mixed or Other.

Because multiple valid interpretations of bounding boxes can exist, we perform a consistency study to evaluate inter-annotator agreement and establish a human performance upper bound for the task.

We compute inter-annotator agreement on the subset of query-page pairs labeled by two or three annotators. For each annotator, we merge all their bounding boxes into a single zone. We then compare zones across annotators by measuring pixel-level overlap, reporting Intersection over Union (IoU) and F1 score (Dice coefficient). When three annotators label the same sample, we average over all pairwise comparisons.

Across all 10 datasets, we observe an average IoU of 0.50 and F1 of 0.60. These moderate agreement scores reflect the inherent subjectivity of the task: annotators typically agreed on the relevant content but differed in granularity (Appendix appendix), with some marking tight bounds around specific content while others included surrounding context.

Quality Control

The annotation was conducted by a curated pool of 76 domain-qualified experts with native-level language proficiency. Quality control was performed by 13 senior annotators with enhanced domain knowledge and extensive annotation experience. Detailed protocols regarding the annotator pool and training are provided in Appendix D.3.

Final Query Distribution

We conducted a final human review to remove low-quality queries and resolve labeling ambiguities. Figure 3 shows the resulting distribution. Extractive queries predominate due to human annotator preference, followed by open-ended queries from targeted sampling. Multi-hop queries were the hardest to scale, suggesting a need for dedicated pipelines. Figure 4 details page modalities; while text is most prevalent, visual elements like tables, charts, and infographics are well-represented.

Dataset Release and Distribution

We extend the benchmark to rigorously assess cross-lingual retrieval. While source documents are maintained in English and French, we use Qwen3-235B-Instruct to provide translations in 6 languages: English, French, Spanish, German, Italian, and Portuguese. This configuration challenges models to bridge the semantic gap between the query language and the document language, a critical requirement for modern RAG systems.

Finally, to ensure the integrity of evaluation and mitigate data contamination (which was shown to be a major preoccupation for Information Retrieval ³³Liu et al. (2025)Introducing RTEB: A New Standard for Retrieval Evaluation), we adopt a split-release strategy. 8 datasets are made public to facilitate research, while 2 are retained as private hold-out sets. This enables blind evaluation, ensuring that performance metrics reflect true generalization rather than overfitting to public samples.

Experiments and Results

Using our benchmark, we conduct extensive evaluations across all 3 components of RAG pipelines. We assess textual and visual retrievers and rerankers on retrieval performance, evaluate leading VLMs and LLMs on their ability to generate accurate answers from various retrieved contexts, and test VLMs on bounding box generation for visual grounding. From these results, we compile practical insights for RAG practitioners.

Retrieval

We evaluate a large panel of visual and textual retrievers on page-level retrieval ability. Visual retrievers are given page images, while textual retrievers process the Markdown text of each page processed by the NeMo Retriever extraction service³⁴Chunking within pages or providing image descriptions did not improve our results. Thus, we report the results of the simplest pipeline. ³⁵Team (2024)NVIDIA Ingest: An accelerated pipeline for document ingestion. The results reported in Table 1 corroborate findings from existing document retrieval benchmarks ³⁶Faysse et al. (2025)ColPali: Efficient Document Retrieval with Vision Language Models
Günther et al. (2025)jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval: for a given parameter count, visual retrievers outperform textual retrievers, and late interaction methods score higher than dense methods.

We analyze ColEmbed-3B-v2, the best-performing retriever we evaluated across query type, content modality, and query language.

Performance is aligned with query complexity

Figure 5 shows that performance is inversely correlated with query complexity: simple query types such as Boolean and Numerical score significantly higher than Open-ended and Multi-hop queries. Question formulations consistently outperform Instruction and Keyword formats across nearly all categories, underscoring the need for improved handling of these query structures.

Visual Content and multi-page queries are hardest for retrievers

Figure 6 highlights that queries involving visual content like tables or images tend to be more difficult. The Mixed content type scores the lowest, which suggests that integrating information across different modalities within a single page remains a challenge. Additionally, we observe a consistent decline in performance as the number of annotated pages increases (Figure 7), suggesting that retriever effectiveness decreases

when aggregating information from multiple sources is required.

Cross-language queries degrade performance

Retrieval performance is 2–3 points higher in mono-lingual settings (Table 5 and Table 6) than cross-lingual settings (Table 1), showing that models need to better adapt to these settings.

Textual rerankers outperform visual ones

We evaluate the impact of adding a reranker to the textual and visual pipelines of the Jina-v4 retriever. We select zerank-2 ³⁷Entropy (2025)Introducing zerank-2 and jina-reranker-m0 ³⁸AI (2025)jina-reranker-m0: Multilingual Multimodal Document Reranker as two of the leading textual and visual rerankers to date. Results in Table 2 reveal a significant disparity in reranking efficacy between modalities. While the visual retriever initially outperforms the textual base, the textual reranker yields substantial gains (+13.2 NDCG@10), enabling the textual pipeline to achieve the highest overall retrieval performance. In contrast, the visual reranker provides only a marginal average improvement of +0.2 and degrades performance in 4 datasets, underscoring the need for better multilingual visual rerankers.

Final Answer Generation

We evaluate end-to-end answer quality by providing LLMs and VLMs with queries and their corresponding retrieved pages, examining the effects of retrieval pipeline selection, context modality, and generation model choice (Table 3). For this evaluation, we use the best-performing textual and visual retrieval pipelines. We additionally establish an upper bound using an oracle pipeline that supplies the model with ground-truth annotated pages.

In the hybrid configuration, we concatenate the top-5 results from the visual retriever (images) with the top-5 results from the textual retriever (text), without removing duplicates; the retrieval performance is detailed in Table 7. We also consider a hybrid oracle setup, which provides the model with all the ground-truth pages in both modalities.

The correctness of generated answers is assessed against the ground truth final answer by an LLM judge (details in Appendix D.8).

Private datasets are omitted

to maintain their integrity.

Some benchmark queries involve general knowledge manageable by LLMs without retrieval. To prevent memorization from confounding our assessment of the RAG pipeline, we stratify queries by difficulty based on parametric knowledge. A query is categorized as easy if any model in a 6-LLM panel answers it correctly without context; otherwise, it is labeled hard. Overall, 48.6 % of queries are easy (see Table 12 for details).

Visual context helps generation

With a fixed Gemini 3 Pro generator, image-based context outperforms text-based context on the hard subset by 2.4 and 2.8 percentage points for the oracle and ColEmbed-3B-v2 pipelines, respectively (Table 3). This confirms that preserving the visual content of document pages provides better grounding for complex answer generation.

Hybrid retrieval yields the best performance on challenging queries

The hybrid pipeline achieves 54.7 % accuracy on hard queries, surpassing both the strongest textual (52.1 %) and visual (54.5 %) baselines. This complementary effect suggests that text and image representations capture different aspects of document content, and their combination can provide more robust evidence for downstream generation.

Hard queries expose the limits of parametric knowledge in current models

Even with oracle context, performance on hard queries lags behind easy queries by more than 10 percentage points. This gap suggests that the multi-step reasoning and long-context synthesis required for difficult queries remain challenging for current models. While the models we evaluate achieve comparable overall scores, their relative ranking may shift when parametric knowledge is less of an advantage, as shown by GPT 5.2 outperforming Gemini 3 Pro on easy queries but trailing on hard ones.

ViDoRe V3 leaves significant room for future retriever improvements

The 10-point gap between the best non-oracle result (54.7 %) and the image oracle (64.7 %) on hard queries underscores substantial opportunities for improving the retrieval pipeline. Moreover, even with oracle contexts, Gemini 3 Pro performance remains modest, indicating that generation models still struggle to fully exploit the provided information.

Visual Grounding

Beyond generating correct answers, it is highly desirable for RAG pipelines to identify where in the source documents the answer originates, enabling users to verify the grounding of the query answer. We therefore evaluate the ability of LLMs to generate accurate bounding boxes within their final answer. Among the few LLM families with visual grounding capabilities, we select Qwen3-VL-30B-A3B-Instruct and Gemini 3 Pro for evaluation. For each query, we provide the model with the candidate pages shown to the human annotators and prompt it to answer the query while inserting inline bounding boxes in XML format <bboxes image="N"> ... </bboxes> to delimit relevant content (full instructions in Appendix D.7).

We use the bounding boxes produced by the human annotators as our ground truth. Since each query may have 1–3 human annotators, we evaluate VLM predictions independently against each annotator using the same zone-based methodology as the inter-annotator consistency analysis (Section 4.3.3), and report the highest F1 score. This best-match strategy reflects the inherent subjectivity of evidence selection: annotators may legitimately highlight different regions to support the same answer, and a model should not be penalized for matching any valid interpretation.

Visual grounding lags human performance

Inter-annotator agreement on evidence localization reaches an F1 of 0.602, whereas the best-performing models achieve markedly lower scores: 0.089 for Qwen3-VL-30B-A3B-Instruct and 0.065 for Gemini 3 Pro. A page-level analysis (Table 4) reveals that on pages where humans provided bounding boxes, both models annotated the same page only 16–17 % of the time, while 26–27 % of human-annotated pages received no model annotation at all—highlighting recall as the primary bottleneck. Detailed per-domain results and qualitative analysis appear in Appendix D.7 and appendix.

Conclusion

This work introduces ViDoRe V3, a multilingual, human-annotated RAG benchmark that evaluates retrieval, final answer generation, and visual grounding on large industry-relevant document corpora. We design a human-in-the-loop annotation methodology, deployed in a 12,000-hour annotation campaign, that produces diverse realistic queries paired with relevant pages, bounding boxes, and reference answers. Evaluating state-of-the-art RAG pipelines, we find that visual retrievers outperform textual ones, late interaction and textual reranking yield substantial gains, and visual context improves answer generation quality. Looking ahead, ViDoRe V3 highlights several concrete research directions for practical multimodal RAG. Retriever models still struggle on cross-lingual and open-ended queries requiring visual interpretation, while VLMs need improvement in answer generation from multi-page contexts as well as accurate visual grounding. By providing a rigorous framework for evaluating these limitations, ViDoRe V3 serves as a catalyst for the development of more robust, intelligent document understanding models.