Manuel Faysse

PhD Candidate

Lead Research Scientist

Paris, France

Hey! I am Manu, a 2nd year PhD student working on applied NLP and ML Privacy research, but curious about (way too) many other things!

After pretraining at EPFL with a master’s in Robotics and Data Science, and an awesome research stint with the Computational Privacy Group at Imperial College London, I worked as a Research Scientist at Illuin Technology on various NLP use cases, notably deep multimodal models for Document ML and neural information retrieval.

I am now in my academic finetuning phase as a PhD student at CentraleSupélec (Université Paris Saclay), supervised by the distilled knowledge of Pierre Colombo. My research focuses on industrial applications of large language models, with papers on instruction model automatic evaluation, bilingual Large Language Model pretraining (CroissantLLM), multimodal information retrieval (ColPali), as well as model memorization, or confidence estimation techniques for neural information retrieval.

My work has been published in top international conferences (ICML, EMNLP), has been featured in the press (MIT Tech Review, Nature Magazine, Usine Digitale, Usine Nouvelle, etc.), gave way to invited talks (Meta, IBM, Naver, LlamaIndex, etc.) and is currently used in production for various awesome applications.

My PhD is funded through the CIFRE French program in collaboration with Illuin Technology, where I currently hold a Lead Research Scientist position, and spend a minor share of my time advising and accompanying various R&D efforts in the LLM and Vision LLM space.

Don’t hesitate to contact me to discuss, or to inquire about potential collaborations or invited talks !

news

Aug 19, 2024	Gave an invited talk at Unbabel on ColPali and Retrieval in Vision Space.
Jul 26, 2024	Invited at the LlamaIndex webinar to talk about ColPali and Document Retrieval in Vision Space.
Jul 25, 2024	The MIT Technology Review has published a featured article on our work on Copyright Traps in LLMs.
Jun 21, 2024	We release ColPali - Efficient Document Retrieval with Vision Language Models 👀 !
Jun 14, 2024	Gave an invited talk at IBM Research Paris on the topic of CroissantLLM and Large Language Models.

selected publications

2024

ColPali: Efficient Document Retrieval with Vision Language Models

Manuel Faysse, Hugues Sibille, Tony Wu, and 4 more authors

2024

Abs Website

Documents are visually rich structures that convey information through text, as well as tables, figures, page layouts, or fonts. While modern document retrieval systems exhibit strong performance on query-to-text matching, they struggle to exploit visual cues efficiently, hindering their performance on practical document retrieval applications such as Retrieval Augmented Generation. To benchmark current systems on visually rich document retrieval, we introduce the Visual Document Retrieval Benchmark ViDoRe, composed of various page-level retrieving tasks spanning multiple domains, languages, and settings. The inherent shortcomings of modern systems motivate the introduction of a new retrieval model architecture, ColPali, which leverages the document understanding capabilities of recent Vision Language Models to produce high-quality contextualized embeddings solely from images of document pages. Combined with a late interaction matching mechanism, ColPali largely outperforms modern document retrieval pipelines while being drastically faster and end-to-end trainable.
CroissantLLM: A Truly Bilingual French-English Language Model

Manuel Faysse, Patrick Fernandes, Nuno M. Guerreiro, and 13 more authors

2024

Abs Website

We introduce CroissantLLM, a 1.3B language model pretrained on a set of 3T English and French tokens, to bring to the research and industrial community a high-performance, fully open-sourced bilingual model that runs swiftly on consumer-grade local hardware. To that end, we pioneer the approach of training an intrinsically bilingual model with a 1:1 English-to-French pretraining data ratio, a custom tokenizer, and bilingual finetuning datasets. We release the training dataset, notably containing a French split with manually curated, high-quality, and varied data sources. To assess performance outside of English, we craft a novel benchmark, FrenchBench, consisting of an array of classification and generation tasks, covering various orthogonal aspects of model performance in the French Language. Additionally, rooted in transparency and to foster further Large Language Model research, we release codebases, and dozens of checkpoints across various model sizes, training data distributions, and training steps, as well as fine-tuned Chat models, and strong translation models. We evaluate our model through the FMTI framework, and validate 81 % of the transparency criteria, far beyond the scores of even most open initiatives. This work enriches the NLP landscape, breaking away from previous English-centric work in order to strengthen our understanding of multilinguality in language models.

2023

Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applications

Manuel Faysse, Gautier Viaud, Céline Hudelot, and 1 more author

In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Dec 2023

Abs Website

Instruction Fine-Tuning (IFT) is a powerful paradigm that strengthens the zero-shot capabilities of Large Language Models (LLMs), but in doing so induces new evaluation metric requirements. We show LLM-based metrics to be well adapted to these requirements, and leverage them to conduct an investigation of task-specialization strategies, quantifying the trade-offs that emerge in practical industrial settings. Our findings offer practitioners actionable insights for real-world IFT model deployment.