CroissantLLM: A Truly Bilingual French-English Language Model

Introduction

Although a few proprietary models ¹Thakur et al. (2021)BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
Muennighoff et al. (2022)MTEB: Massive Text Embedding Benchmark are still considered to run ahead of the pack ²OpenAI (2023)GPT-4 Technical Report, open weights models such as Llama ³Touvron et al. (2023)Llama: Open and efficient foundation language models
Touvron et al. (2023)Llama 2: Open Foundation and Fine-Tuned Chat Models, Qwen ⁴Bai et al. (2023)Qwen Technical Report or Mistral ⁵Jiang et al. (2023)Mistral 7B
Jiang et al. (2024)Mixtral of Experts are rapidly bridging the performance gap. However, widespread industrial and research adoption of such technology remains challenging for several reasons, including the lack of transparency in the data collection and training processes, the scarcity of existing resources outside of English, and the large-scale and costly nature of existing high-performing models.

Lack of transparency. State-of-the-art models, both proprietary and open-weights are often designed and trained by heavily investor-backed companies, that aim to retain a moat by keeping their training data mix and strategy secret, hindering the rest of the field's ability to fully study and understand these models.

This lack of transparency, ranging from training set composition to lack of evaluation or unclear usage policies, has been characterized by previous works, such as those by ⁶Bommasani et al. (2023)The Foundation Model Transparency Index and ⁷Casper et al. (2024)Black-Box Access is Insufficient for Rigorous AI Audits, pushing for full transparency as a key component for safe model development and use.

The dangers of closed, non-auditable datasets have been exemplified by recent findings showcasing the potential dangers of dataset contamination, whether intentional ⁸Hubinger et al. (2024)Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training or not.⁹https://purl.stanford.edu/kh752sm9123 Furthermore, legal questions arise surrounding data ownership in LLM training corpora ¹⁰NewYorkTimes (2023)The Times sues OpenAI and Microsoft over A.I. use of copyrighted work
Samuelson (2023)Generative AI meets copyright and recent developments in the political landscape, regarding AI (EU AI Act, US senate hearings)¹¹https://www.commerce.senate.gov/2023/9/the-need-for-transparency-in-artificial-intelligence have further emphasized the importance of transparent approaches, both from a legal perspective and to build user trust.

Bias towards English. Although the exact training mix of most well-performing LLMs is not publicly available information, most large models are trained on very English-centric corpora (Touvron et al. 2023). This is the consequence of the important amount of English resources compared to other languages, both in terms of data availability and benchmarks. As could be expected, all publicly available LLMs display a large performance disparity between English and non-English languages when evaluated on downstream tasks ¹²Bandarkar et al. (2023)The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. Moreover, cultural knowledge and biases are mechanically integrated through the training data, leading to models with greater knowledge of American events or biases ¹³Bender et al. (2021)On the dangers of stochastic parrots: Can language models be too big?
Santurkar et al. (2023)Whose Opinions Do Language Models Reflect?. This puts non-English users at a disadvantage when it comes to language model usage and adoption. While the non-English NLP community has produced multilingual datasets ¹⁴Soboleva et al. (2023)SlimPajama: A 627B token cleaned and deduplicated version of RedPajama
Laurençon et al. (2023)The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset and models ¹⁵Scao et al. (2022)Bloom: A 176b-parameter open-access multilingual language model
Shliazhko et al. (2022)mGPT: Few-Shot Learners Go Multilingual in the last few years, the available resources still largely lag behind English ones, hindering industrial adoption in non-English settings.

Challenging to use at scale. Although benefits of scaling models to enormous sizes have been amply demonstrated in terms of performance ¹⁶Hoffmann et al. (2022)Training Compute-Optimal Large Language Models
Wei et al. (2022)Emergent Abilities of Large Language Models
Chowdhery et al. (2022)PaLM: Scaling Language Modeling with Pathways, scale comes at a large cost in terms of hardware requirements and inference speed. Download statistics on HuggingFace show the smallest Llama model (Touvron et al. 2023) to be the most adopted by the community, demonstrating the interest in small but capable models. LLM scaling laws demonstrate the diminishing returns of training a model of a given size past a certain amount of tokens. By continuing pre-training way past the compute-optimal threshold, performance has been shown not to saturate, enabling the training of “inference-optimal'' models of high interest to the industrial and research communities ¹⁷Sardana & Frankle (2023)Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. It is still not fully understood how model performance continues to improve at these later stages of training.

Contributions

In this work, we attempt to bridge the aforementioned limitations and gaps. Our main contributions can be summarized as follows.

Contribution 1: Introduction of a highly-curated, diverse corpus in French. We collect and openly release a 303B token corpus spanning internet data, but also literary work, speech transcripts, legal and administrative documents, scientific articles, business-related documents, etc. The corpus is distributed under permissive licenses, allowing commercial use with no restriction, and is heavily filtered, curated, and deduplicated. To our knowledge, it is the largest multi-source French language corpus released to date of sufficient quality for language modeling purposes¹⁸By this, we imply of sufficient quality to train a language model (little OCR errors, high-quality text) and not exclusively composed of Internet data..

Contribution 2: Training CroissantLLM, a truly bilingual language model. Nowadays, most models display some multilingual capabilities. For example, Bloom has been trained to be massively multilingual ¹⁹Scao et al. (2022)Bloom: A 176b-parameter open-access multilingual language model, Llama contains a minor proportion of non-English data in its training set (Touvron et al. 2023) and Qwen included a significative portion of Chinese data (Bai et al. 2023).²⁰Qwen-VL (Bai et al. 2023) reports 77.3 % & 22.7 % Chinese, but no information is given for the Qwen base model. However, to our knowledge, outside of Chinese with a different alphabet ²¹Zeng et al. (2022)Glm-130b: An open bilingual pre-trained model, no work has studied or attempted to train a multilingual model of significant scale in which English is not the dominant training language.

In this work, we train a model on a 1:1 ratio of English to French with a tokenizer optimized for bilingualism. Our end goal is to have a model less skewed towards English performance or cultural biases.

We motivate this ratio by conducting careful experimentation on smaller model scales to uncover the trade-offs behind language distribution ratios. We opt for a strategy enabling a “best of both worlds'' performance in these two languages, as empirically validated by scaling law experiments. Experimentally, we show the importance of integrating data from another cultural source in the acquisition of cultural knowledge, underlining the importance of the effort.

Contribution 3: FrenchBench: A novel LLM benchmark for the French Language. To evaluate models in French, we introduce a benchmark encompassing various tasks to assess factual knowledge, generative capabilities, language understanding, etc. This benchmark is constructed both from openly available datasets, as well as newly released manually annotated data. We evaluate and report results for our models as well as other models with French-speaking capabilities.²²Code for evaluation is available at https://github.com/EleutherAI/lm-evaluation-harness²³Another complementary initiative has been led for French model evaluation and released concurrently in (Bawden et al. 2024)

Contribution 4: Releasing high-performing, inference-optimal models for the industrial community, together with a wide range of resources for the research community. The models we train are all released under open licenses. Our largest model is trained on a 2300:1 token to parameter ratio (115 times longer than a Chinchilla Optimal 1.3B model) leading to very strong performance in its size category. We show that model performance on downstream tasks continues to dramatically improve with lengthened pre-training runs, although model perplexity does not significantly improve. We release all checkpoints for all model sizes, as well as the exact training data seen at every step for research purposes.²⁴Training data indexing order will be released in a second stage. These models are extremely efficient to run, leading to low-latency, energy-efficient on-edge inference, even on low-resource devices such as phones or personal computers. These releases²⁵Code for dataset collection and filtering is available at https://github.com/ManuelFay/llm-data-hub. Code for model training is hosted at https://github.com/CoderPat/croissant-llm-training. Datasets and model checkpoints are available at https://huggingface.co/CroissantLLM. are motivated by a commitment to transparency to allow research and reassuring users for industrial deployment: our models comply with 81 % of criteria listed by the Foundation Model Transparency Index ²⁶Bommasani et al. (2023)The Foundation Model Transparency Index (see Section section).

The CroissantLLM initiative aims to adress the aforementionned limitations of current models through the release of an inference-optimized, small but capable model that performs well outside of English settings, and that is designed to be as open and transparent as possible. Beyond facilitating industrial LLM adoption and unlocking new generative model use cases, this project is also a proven platform for researching LLMs and kickstarting future pretraining efforts (Martins et al. 2024; Meeus et al. 2024; Shilov et al. 2024; Meeus et al. 2025).

Data

The dominance of the English language in the training data of most current models is undeniable. While multilingual models like Llama leverage some non-English data (Touvron et al. 2023), it corresponds to only a minor part of the corpus, leading to a significant drop in performance across non-English data, with noticeable “American'' bias ²⁷Santurkar et al. (2023)Whose Opinions Do Language Models Reflect?
Navigli et al. (2023)Biases in Large Language Models: Origins, Inventory, and Discussion.

This work aims to offset this trend by using a more balanced bilingual corpus comprising English and French, as well as additional code data. Although both languages belong to the Indo-European language family, they exhibit different morpho-syntactic structures²⁸For example, pronominal objects are placed before (resp. after) the verb in French (resp. English), both languages have different noun phrase constructions (“la commission européenne'' vs. “the European commission''), etc. and French has a richer morphology.²⁹For example, English has 5 verb forms whereas French has 48, French has explicit inflections for grammatical genders, etc. However, note that only English adjectives have morphological constructions for expressing comparison (e.g. easy, easier, easiest). We refer to WALS for more details, e.g. https://wals.info/feature/21B##2/26.7/152.4 We study whether this corpus helps in reducing biases, enabling more varied knowledge sets, and unlocking non-English performance.

A variety of sources are integrated into our corpus, including carefully filtered internet data and high-quality data from a range of sources, all devoid of restrictive licenses ensuring complete openness of the data and the trained model. Data statistics are available in Table 1. As further detailed in 7.3.4, our data corpus contains different amounts of unique English, French, and Code tokens. We obtain our balanced training corpus by upsampling French, Code, and English data with different sampling ratios, such that no performance loss is to be expected ³⁰Muennighoff et al. (2023)Scaling Data-Constrained Language Models.

The scrapping and processing code are available in our code base.³¹https://github.com/ManuelFay/llm-data-hub The license information of all datasets used is given, all allowing for permissive commercial use.

French Data

Table 1 lists the source and some information regarding the French corpus. Details about the data sources are expanded further below.

Web Data. We collect internet data from various web scraps (Oscar ³²Abadji et al. (2022)Towards a Cleaner Document-Oriented Multilingual Crawled Corpus, mC4 ³³Xue et al. (2021)mT5: A massively multilingual pre-trained text-to-text transformer), leveraging the CulturaX corpus ³⁴Nguyen et al. (2023)CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages for heuristic and perplexity filtering, as well as exact and fuzzy deduplication. In total, this represents over 363 million webpages and more than 292 billion tokens, that we split using our custom tokenizer fitted on equal amounts of French and English data.³⁵The mC4 corpus https://huggingface.co/datasets/allenai/c4 is released under the ODC-BY licence https://opendatacommons.org/licenses/by/1-0/ whereas Oscar (Abadji et al. 2022) does not claim data ownership, provides an opt-out strategy for data inclusion, and filtering metadata is released under the Creative Commons CC0 license.https://creativecommons.org/publicdomain/zero/1.0/

We ensure data is of good quality and correctly tagged in French through sampled manual inspection and confirm French-speaking countries are well represented within the dataset. Notably, we include several news sources scraped from Belgium, Switzerland, Canada, and Lebanon, as well as multiple African countries (Senegal, Morocco, Algeria, Cameroon, etc.) ³⁶Contrary to certain languages such as Arabic or Portuguese in which the language differs greatly depending on the country, the written French language in these countries is largely similar with only a minor share of regionally introduced syntax and expressions

Legal and Administrative Data. We introduce 5.3B tokens of data from the French government's open data initiative, ranging from legal tasks to parliamentary discussions and financial transcripts (e.g. legal and administrative jargon). These texts originate from 13 different datasets (the OpenData corpus) and were collected from the French government's open data platform.³⁷Data is released at https://echanges.dila.gouv.fr/OPENDATA/ with the ETALAB open license https://www.etalab.gouv.fr/wp-content/uploads/2017/04/ETALAB-Licence-Ouverte-v2.0.pdf To ensure other French-speaking countries are represented, we add 68M tokens of data from Swiss legislation retrieved from government sources. We perform steps to process, filter, and run exact deduplication on these documents.

Cultural Data. We introduce cultural data from various sources. Notably, we retrieve all Project Gutenberg ³⁸Hart (1971)Project gutenberg books in the French language as of October 2023, corresponding to books released in the public domain (302 million tokens). We also download and aggressively filter manuscripts and documents from the French National Library (Bibliothèque Nationale de France), and filter for documents that belong to the public domain, have undergone an OCR process, and are of high quality.³⁹Metadata is licensed under Open Etalab license https://gallica.bnf.fr/edit/und/conditions-dutilisation-des-contenus-de-gallica To filter out low-quality OCR documents, we implement custom heuristics which we release within our code base. We run all documents through perplexity filters using KenLM 5-grams⁴⁰https://github.com/kpu/kenlm fitted on the French Wikipedia split, and discard documents with perplexity values that are too high (noisy) or too low (repetitive patterns). Thresholds are set through a manual verification process. We deliberately choose to be aggressive in our filtering to ensure only high-quality data is kept and discard the largest portion of the original corpus, keeping about 27M tokens. We choose not to keep any data from the newspaper archives, as the OCR transcription is often too noisy. Additionally, we introduce famous public domain French poems custom scraped from a French poetry website,⁴¹https://www.poesie-francaise.fr/ and run a set of podcasts through a high-quality speech-to-text model to obtain a textual transcription. This process is hard to scale and data splits from these sources are limited in quantity. Data from the OpenSubtitles⁴²https://opus.nlpl.eu/OpenSubtitles2016.php initiative is integrated, corresponding to 41.8 million tokens originating from movie subtitles.⁴³https://www.opensubtitles.org Finally, we add the French data from Wikisource collected as part of the BigScience initiative ⁴⁴Scao et al. (2022)Bloom: A 176b-parameter open-access multilingual language model and obtain 2.7 billion tokens from the process.⁴⁵Licensed under CC BY-SA 4.0, https://en.wikisource.org/wiki/Wikisource:Copyright_policy

Encyclopedia Data. To introduce high-quality factual data to the training corpus, we integrate the French Wikipedia split from November 2023.⁴⁶https://huggingface.co/datasets/wikimedia/wikipedia with a CC-By-SA 3.0 license This corresponds to the latest knowledge cutoff in the training data. In total, more than 2.5 million articles are used, spanning more than 2 billion tokens.

Industrial Data. We scrap high-quality and publicly available data from industrial PDFs via a manually crafted list of websites, from large French and French Canadian (Quebec) companies to government agencies. This business-focused data boosts performance on a series of downstream applications related to industrial NLP. We collect over 36000 PDF multi-page documents and filter them through carefully crafted heuristics, followed by aggressive perplexity filtering.⁴⁷Data is public internet data that we release under MIT license with an opt-out scheme https://huggingface.co/datasets/manu/illuin_layout_dataset_text_only. In total, we obtain over 290000 documents and 191 million tokens.

English Data

Our English data is primarily drawn from the SlimPajama corpus ⁴⁸Soboleva et al. (2023)SlimPajama: A 627B token cleaned and deduplicated version of RedPajama, excluding copyrighted documents. Splits per data source are detailed in Table 2.

Internet Data. Similarly to the French dataset, we rely on carefully filtered content from an assortment of internet sources, including miscellaneous web pages and blogs. The filtering process includes heuristics and perplexity filtering, as well as large-scale deduplication ⁴⁹Soboleva et al. (2023)SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. The SlimPajama corpus includes internet data from the CommonCrawl⁵⁰Common Crawl license https://commoncrawl.org/terms-of-use and C4⁵¹ODC-BY license web scraps, as well as data sourced from Github textual content⁵²Only content under MIT, BSD, and Apache licenses are kept and the StackExchange forums.⁵³CC-By-SA 4.0 license https://archive.org/details/stackexchange

Miscellaneous. Other non-internet-based data sources are included in the SlimPajama dataset, such as scientific articles from Arxiv⁵⁴https://arxiv.org/, with author opt-out options and English documents from Wikipedia.⁵⁵CC-By-SA 3.0 license The SlimPajama dataset also comprises the “Books'' subcorpus, obtained by downloading all book documents from Bibliotik.⁵⁶https://huggingface.co/datasets/the_pile_books3 Some of the documents within this last corpus have been flagged by their owner as proprietary data. We filter out all documents from this subcorpus, and replace them with data from the open-source Project Gutenberg ⁵⁷Hart (1971)Project gutenberg English books under public domains.⁵⁸ From https://huggingface.co/datasets/pg19 with an Apache 2.0 license

Gutenberg Canaries. To assess model memorization to inform about the risks of including private or sensitive data within the training set, we stress test the model by including “canaries'' ⁵⁹Carlini et al. (2019)The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks. These correspond to samples that have been intentionally modified and/or repeated and included within the model training set, and that will enable a posteriori evaluation of the model capacity to memorize data in a “worse than worst-case'' situation.⁶⁰This work is led in parallel to the CroissantLLM project and findings will be independently published. In total the canaries represent 555 million tokens, representing less than 0.04 % of the total tokens seen during training.

Code Data

In line with most recent models (Chowdhery et al. 2022; Scao et al. 2022; Touvron et al. 2023), we integrate code data into our training corpus. Notably, previous work shows that code data benefits natural language tasks and can be particularly useful in data-constrained settings ⁶¹Muennighoff et al. (2023)Scaling Data-Constrained Language Models. Therefore, we include 140B tokens of code data in several common programming languages. Splits and numbers of tokens are detailed in Table 3.

The Stack & StarCoder. We rely on the efforts of the StarCoder project ⁶²Li et al. (2023)StarCoder: may the source be with you!, and use their high-quality filtered code data from The Stack corpus ⁶³Kocetkov et al. (2022)The Stack: 3 TB of permissively licensed source code.⁶⁴Data are drawn from various sources and fall under 193 different permissive licenses. We use the version $1.2$ of the corpus, which has been filtered with respect to data owners opt-out option information. We keep only high-resource programming languages (Java, Javascript, Python, C, C++, SQL) and Jupyter notebooks, as well as a few samples of formatted data (JSON, Tex) and scripting languages (shell scripts, Dockerfiles).

Extra Python code. We extend the corpus with several other sources of Python code due to the popularity of the language in the community. Firstly, we add Pypi packages from recent code dumps, that are filtered to keep only Python and Jupyter files.⁶⁵https://huggingface.co/datasets/vikp/pypi_clean Secondly, in order to promote high-quality problem-solving-centered code, we integrate 1.2B tokens of Python3 data from competitive coding contests ⁶⁶Li et al. (2022)Competition-level code generation with AlphaCode under CC-By-4.0 license. Lastly, following the success of learning from textbooks ⁶⁷Li et al. (2023)Textbooks Are All You Need II: phi-1.5 technical report, we add commented Python code constructed by combining code and text cells from Jupyter Notebooks through the CodeParrot initiative.⁶⁸https://huggingface.co/datasets/codeparrot/github-jupyter-code-to-text Apache 2.0 license

Parallel Data

Following previous work ⁶⁹Anil et al. (2023)PaLM 2 Technical Report, we incorporate vast quantities of parallel data, in our case high-quality English-French translation pairs, in order to improve the multilingual capabilities of the model ⁷⁰Briakou et al. (2023)Searching for Needles in a Haystack: On the Role of Incidental Bilingualism in PaLM's Translation Capability.

Opus. We extract subsets of sentence pairs spanning multiple domains from the OPUS corpus ⁷¹Tiedemann (2012)Parallel Data, Tools and Interfaces in OPUS.⁷²Free license https://opus.nlpl.eu/ Statistics are described in Table 2. In total, we include 400 million parallel sentences and about 36 billion tokens. The data is filtered through a rigorous cleaning pipeline: (1) BiFixer ⁷³Ram\'-Sánchez et al. (2020)Bifixer and Bicleaner: two open-source tools to clean your parallel data⁷⁴https://github.com/bitextor/bifixer is first used to remove duplicate data through fuzzy matching techniques; (2) BiCleaner⁷⁵https://github.com/bitextor/bicleaner is then used to filter data using heuristic and perplexity filters; (3) finally, the state-of-the-art NMT quality estimator CometKiwi ⁷⁶Rei et al. (2022)CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task is used to keep only top quality translation pairs.

Theses. To incorporate versatile academic and scientific language, we augment our dataset with French theses abstracts along with their author-generated English translations. This corresponds to 95000 documents and more than 80 million high-quality tokens.⁷⁷Data licensed under Etalab open license https://www.etalab.gouv.fr/wp-content/uploads/2017/04/ETALAB-Licence-Ouverte-v2.0.pdf

Song Lyrics. Our dataset integrates song lyrics in both French and English, scraped from a specialized community-driven lyrics translation website.⁷⁸https://www.lacoccinelle.net, with opt-out options for data owners and community-contributed translations under free-use license. As such, our model is trained with radically different linguistic styles (e.g. colloquialism), and the wide range of topics can help the model to capture cultural nuances. Lyrics have been translated by hand by the website community. With a total of 70k songs, we have built up a corpus of 53M tokens covering different periods (80s, 90s, 20s, etc.) and musical genres (rap, rock, jazz, etc.) To preserve colloquial expressions and cultural subtleties, we have not filtered song lyrics for explicit content. We validate the original language metadata of the songs through Google's language-detection algorithm.

Training

Our main goal was to train a high-performing, yet resource-friendly bilingual model while optimizing performances across both languages. To focus on the specific challenges of the bilingual paradigm, we rely on previous work to motivate many of our design and hyperparameter choices ⁷⁹Touvron et al. (2023)Llama 2: Open Foundation and Fine-Tuned Chat Models.

Model Architecture

We use the Llama architecture (Touvron et al. 2023), a decoder-based transformer, trained with rotary position encodings ⁸⁰Su et al. (2023)RoFormer: Enhanced Transformer with Rotary Position Embedding and a context length of 2048. We construct 4 different model sizes by jointly scaling the number of attention heads, hidden size, and hidden layers. Table 2 summarizes the sizes of each model in the family.

Tokenizer

Most LLM tokenizers are fitted on English-centric corpora with an information-theoretic optimization objective, for example, Byte-Pair encoding ⁸¹Sennrich et al. (2016)Neural Machine Translation of Rare Words with Subword Units or Unigram ⁸²Kudo (2018)Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates, leading to good fertility⁸³See Appendix for a definition. values (low token per word ratio) on English text, but high fertility in other languages. These phenomena make processing in other languages slower and more costly ⁸⁴Rust et al. (2021)How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Furthermore, subword splits in non-English languages mechanically carry less semantical meaning, potentially being a factor in the degraded performance of models on non-English languages (Rust et al. 2021).

Tokenizer training. We fit our CroissantLLM tokenizer on a corpus of 100B high-quality tokens, with splits of English, French, and code data. We use SentencePiece⁸⁵https://github.com/google/sentencepiece to train a Byte-Pair Encoding tokenizer with a vocabulary size of 32000 tokens, 100 special placeholder tokens, whitespace separation, and byte fallback, inspired by (Touvron et al. 2023; Jiang et al. 2023). The data corpus used to fit the tokenizer is made available,⁸⁶https://huggingface.co/datasets/manu/tok-corpus-shuffled and notably contains large amounts of French data to skew the vocabulary construction process towards optimizing for French as well.

Improved fertility rates. The focus on English, French, and Code enables the CroissantLLM tokenizer to display smaller fertility rates on French texts than the Mistral and Llama models with similar vocabulary sizes, all the while also displaying slightly smaller rates than both in English and Code (Figure 2). This is due to the multilingual support of both Llama and Mistral tokenizers which need to allocate some vocabulary tokens to frequent character patterns from other languages. Roughly, the Llama tokenizer is 17 % less token efficient at encoding French internet data, and up to 40 % less efficient on clean encyclopedia French texts, implying that the 303B unique French tokens in our data training set correspond to more than 360B tokens with the Llama tokenizer. This enables us to pack more data in fewer tokens, leading to improvements in training and inference efficiency.

Selecting an optimal language ratio

A crucial question when training a bilingual model is how to effectively weight data from the two languages to achieve a good trade-off between performance in both. While, intuitively, training on an equal mixture of English and French data may seem to be the obvious solution, differences in data quality available for each language coupled with transfer learning dynamics between both languages could imply that a balanced mix might be sub-optimal. However, training multiple models with different data mixes for comparison is prohibitively expensive.

To offset this computational cost, we leverage recent findings on scaling laws ⁸⁷Kaplan et al. (2020)Scaling Laws for Neural Language Models that show that we can predict the performance of our model by training smaller models on the same dataset. In particular, ⁸⁸Fernandes et al. (2023)Scaling Laws for Multilingual Neural Machine Translation found that, for multilingual models, by training smaller models with varying weights for each language in the data mix, one can fit a multilingual, joint scaling law that can predict the language performance trade-off of larger models, even for novel language weightings not encountered during the fitting of the scaling law.

As such, we fit a joint scaling law as described by (Fernandes et al. 2023) for each language, by training 3 smaller model sizes on 3 different data mixes with varied ratios of English and French data (keeping the amount of Code data fixed). The corpus for these scaling law experiments is a subsampled variant of the larger corpus and is detailed in Appendix G.3.5. We define 3 data mixes by varying the language sampling ratio: (1) equal containing 40 % English data, 40 % French data and 20 % Code data; (2) frplus containing 20 % English data, 60 % French data and 20% Code data; and (3) enplus containing 60 % English data, 20 % French data and 20 % Code data. We then trained a 1.3B model on these subsets of the data for one of the data mixes to validate their predictive power.

Figure 3 shows the performance predicted by jointly-fitted scaling laws as we scale the model and vary the language weightings on the Wiki data validation split.

First, we see that the fitted scaling law is able to predict the performance of the larger model almost perfectly. Secondly, changing the weight of each language in training has a non-symmetrical impact on language performance: by increasing the (relative) weight of French from 50 % to 75 %, we get a marginal performance increase in French, while performance in English drops significantly. This fact is made clear by plotting the effective capacity ratio⁸⁹The relative monolingual model size required to match the performance of the multilingual model on a language. See Appendix for more details of each language as we change the language weight (Figure 4): the “gains'' in parameters from increasing weight of French data are minimal past the 50 % mark.

These findings showcase that multilinguality comes at a price, and training a bilingual model implies accepting a performance loss on a target language compared to an equivalent model trained on a monolingual corpus.

We find equal ratios of English and French data lead to minimized performance hits across both languages (Figure 3) and opt to train our base model in this data configuration.

Final data distribution

Our final dataset is composed of 1.1T unique tokens that originate from sources of various languages, qualities, and quantities. To craft a training set with a language and data distribution that suits our objectives, we upsample some of the sources, notably to balance out French and English data and increase the share of parallel data in our training run. Following work by ⁹⁰Muennighoff et al. (2023)Scaling Data-Constrained Language Models and ⁹¹Luukkonen et al. (2023)FinGPT: Large Generative Models for a Small Language on data-constrained language modeling scaling laws, we upsample French text by a factor of two, and parallel data by a factor of 3. For a 3T token training run, this enables the model to see French data at most 4 times, and English and code data twice, which should have negligible impacts on the performance ⁹²Muennighoff et al. (2022)Crosslingual generalization through multitask finetuning. The final data distribution is shown in Table 1.

All data is provided from the above-listed sources and no synthetic or augmented data is used. Data licenses and copyright information are given for every split to the best of our ability. The data collection and filtering process to construct our final mix from the above-listed sources is entirely done by the authors of this paper, who are employed by the universities or private companies described through their affiliations, under their countries' data protection laws, and compensated at market rates or following the academic salary grid of their institution.

Training framework

We train our models on a modified version of Megatron-Deepspeed,⁹³https://github.com/deep-spin/Megatron-DeepSpeed a training framework built on top of PyTorch. Training is done on a dedicated Nvidia A100 SXM4 80 Gb supercomputer partition

with 30 octo-GPU nodes. We rely on the HuggingFace Transformers and Datasets library for model and data manipulation.

To maximize efficiency, we set the micro-batch size per device to 8 sequences of length 2048, and use 4 gradient accumulation steps, resulting in a total batch size of $8 \times 4 \times 30 \times 8 = 7680$ samples, or $7680*2048 = 15,728,640$ tokens. We achieve a mean efficiency of around 120 TFLOP⁹⁴160 TFLOP per second for our scaling law experiments with one GPU node only per second with activation checkpointing, leading to a total compute estimate of $4.30e22$ FLOPS. Standard Cross-Entropy losses are used on a Causal Language Modeling objective.

Training losses

Training lasts 17 days for a total of 99648 GPU hours, and we chose not to manually intervene, letting the model recover on its own after occasional loss spikes. ⁹⁵Previous literature rarely mentions handling of such issues. Some models have reloaded training by skipping batches (Zhang et al. 2022; Anil et al. 2023), others have seemed to ignore it if loss recovered (Touvron et al. 2023). In our case, we used gradient clipping, weight decay and tuned optimizer values (Zhang et al. 2022; Touvron et al. 2023), to minimize the occurrences of such loss spikes… During training, as spikes were few and far between, and loss seemed to recover quite rapidly every time and without visible consequences, we opted to let the model train with no intervention. We highlight that since our checkpoints and dataloaders are openly available, this enables future experimentations around these loss spikes. We train with a max learning rate of $3e-4$, 1000 warmup steps, and a cosine learning rate with a minimum value of $1e-5$. Curves suggest the model still has not reached a performance plateau after 3T tokens (Figure 5). Checkpoints are stored every 5k steps and released with the rest of the project artifacts.

Environmental impact

The model was exclusively trained on a supercomputer operating on low-carbon nuclear electricity. Between experimental runs, scaling laws, and the final training, 123k A100 hours were used. The Thermal Design Power of the NVIDIA A100 SXM4 80Gb used is 400W corresponding to a total power consumption of 49.2 MWH and considering a grid carbon intensity of 57 gCO2eq/kWh, we estimate a carbon footprint of 2.80 tons of CO2 ⁹⁶Luccioni et al. (2022)Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Model emitted during training.⁹⁷Factoring in an intentionally large data center Power Usage Effectiveness of 1.2 (Luccioni et al. 2022), we estimate an emission of 3.36 tons of CO2.

Interestingly, the model we trained is not “compute-optimal'' according to Chinchilla laws ⁹⁸Hoffmann et al. (2022)Training Compute-Optimal Large Language Models, meaning that less computing could have been used to train a larger model with the same performance. However, our model aims to be used for inference purposes at industrial scales. Our training paradigm is thus to absorb the downstream inference costs, by training a smaller model on a lot more tokens to obtain an inference-optimized model equivalent in performance to a bigger compute-optimal model ⁹⁹Sardana & Frankle (2023)Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws. Each inference of the final model is thus vastly more energy-efficient than a Chinchilla optimal model of equivalent performance ($>$ 3B parameters), and can even run on CPU or mobile devices. Relying on estimates of ¹⁰⁰Kaplan et al. (2020)Scaling Laws for Neural Language Models, at inference, CroissantLLM represents roughly 2.6 GFLOPS per token.

We hope to extend base model evaluation past English benchmarking alone and assess model capabilities in French, aiming for broad coverage across orthogonal capabilities to observe the effect of truly bilingual pre-training. Our evaluation efforts are rooted in transparency, and all results reported in the main technical report are reproducible through code that is open-sourced and public data.

Evaluation Benchmarks

We hope to extend base model evaluation past English benchmarking alone and assess model capabilities in French, aiming for broad coverage across orthogonal capabilities to observe the effect of truly bilingual pre-training. Our evaluation efforts are rooted in transparency, and all results reported in the main technical report are reproducible through code that is open-sourced and public data.¹⁰¹All benchmarks are made available and links will be provided in the non-anonymous version of this work.

English

In English, we evaluate on standard LLM evaluation benchmarks.

HellaSwag. HellaSwag ¹⁰²Zellers et al. (2019)HellaSwag: Can a Machine Really Finish Your Sentence? is a dataset specifically crafted to challenge common-sense reasoning abilities of models by requiring them to predict the endings of sentences in a way that relies on information not present in the preceding context. It focuses on capturing a nuanced and context-dependent understanding of language.

PiQA. PIQA is a dataset for common-sense reasoning and was created to investigate the physical knowledge of existing NLP models ¹⁰³Bisk et al. (2019)PIQA: Reasoning about Physical Commonsense in Natural Language.

SciQ. The SciQ dataset contains 13,679 crowdsourced science exam questions about Physics, Chemistry, and Biology, among others. The questions are in multiple-choice format with 4 answer options each ¹⁰⁴Welbl et al. (2017)Crowdsourcing Multiple Choice Science Questions.

Arc-C. The AI2 reasoning challenge dataset ¹⁰⁵Clark et al. (2018)Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge consists of 7,787 authentic grade-school level, multiple-choice science questions, designed to stimulate research in advanced question-answering. The dataset is divided into a Challenge Set and an Easy Set, with the Challenge Set comprising questions that were answered incorrectly by both a retrieval-based algorithm and a word co-occurrence algorithm. Additionally, the dataset includes a corpus of over 14 million science sentences relevant to the task and provides three neural baseline models.

MT-Bench. MT-Bench ¹⁰⁶Zheng et al. (2023)Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena contains a set of prompts designed to evaluate models on their multi-turn conversation and instruction-following abilities, covering various core model abilities; writing, roleplay, extraction, reasoning, math, coding, knowledge I (STEM), and knowledge II (humanities/social science). MT-Bench performance has been shown to best correlate with human-rated appreciation of a model through the LM-Sys model arena.

French

We aim to evaluate models on their capabilities in French, along several axes including vocabulary, grammar, reading comprehension, factual knowledge, biases, and generative capacities, etc. To this end, we introduce FrenchBench, a novel LLM evaluation benchmark for the French language, testing a large array of model skills in various settings.

FrenchBench comprises several tasks, some included from previous benchmark datasets, others newly released with this work.

FrenchBench Gen

FrenchBench assesses the generative capabilities of LLMs in a few-shot setting. Tasks include title generation, summarization, question generation, and question answering. We detail the tasks and the evaluation metrics used below.

FQuaD. FQuaD ¹⁰⁷d'Hoffschmidt et al. (2020)FQuAD: French Question Answering Dataset is a French Question Answering dataset, containing manually annotated sets of Wikipedia passages, questions, and extractive answer spans in the Squad format. This high-quality dataset is one of the rare human-annotated French datasets and we rely on its public evaluation split for 4 of the FrenchBench tasks.

FQuADGenQ is a question generation task in which passages and answers are given to the model in a few-shot manner, and we compute the ROUGE1 score ¹⁰⁸Lin (2004)ROUGE: A Package for Automatic Evaluation of Summaries with the gold questions.

FquadGenAns is the classic question-answering task, but models generate the answer span themselves, and the ROUGE1 score is computed with the gold extractive answer span.

MultiFQuAD ¹⁰⁹Previously unreleased dataset, evaluation set is released under CC-By-NC SA 4.0 license with this work is a FQuAD variant, with a publicly released evaluation set, in which answers can consist of multiple disjoint spans. We evaluate performance on the concatenation of these gold extractive spans using the ROUGE1 score.

French Trivia. The French Trivia dataset is built from online trivia questions pertaining to French culture. Answers are short and meant to assess latent model knowledge and the impact of pre-training data and cultural references. Intentionally, questions are formulated in English for comparison with monolingual English models.¹¹⁰This is a previously unreleased dataset, released under MIT license with this work.

FrenchBench Multiple Choice

We also assess reasoning, factual knowledge, linguistic capabilities, and model biases through a series of few-shot classification tasks, on which models are given multiple completions (multiple choice), and the answer with the highest likelihood is selected. We experimented with multiple question templates. In the MMLU format, the multiple potential answers are given after the question prefixed by a letter (A, B, C, D) and the model must guess the correct answer by predicting the correct answer's letter. In HellaSwag formatting, the model must complete the sentence and the model chooses the most likely continuation sequence, without prior knowledge of all other options. We find HellaSwag formatting is less abstract, and enables smaller size models to perform better.

French Language Test. The French Language Test is a dataset crafted to assess the grammar and vocabulary capabilities of models through language tests. It provides a structured evaluation of a model's linguistic proficiency, aiming to measure its competency in understanding and generating coherent and grammatically accurate sentences in the French language. It is composed of a fr-grammar and fr-vocabulary multiple choice test.

French Hellaswag and Arc-C. These datasets correspond to machine translations made by GPT3.5 of HellaSwag and Arc-C to French.¹¹¹https://github.com/laiviet/lm-evaluation-harness/tree/main/datasets Manual verification of the translation quality indicates the translations to be far from perfect but sufficient for these datasets to act as a correct performance proxy.

OrangeSum. OrangeSum¹¹²https://huggingface.co/datasets/orange_sum ¹¹³Eddine et al. (2020)BARThez: a Skilled Pretrained French Sequence-to-Sequence Model is a summarization dataset constructed from online News articles. Two standard French summarization tasks span from this dataset; OSum(T) in which the model is tasked with generating the title from the article body, and OSum(A) in which the model must generate the first paragraph of the article aimed to be an abstract of the article. We select the abstract generation task, and measure performance with the ROUGE1 score.

Other Tasks

MT-Bench French. Mt-Bench French¹¹⁴https://huggingface.co/datasets/bofenghuang/mt-bench-french is a translated and adapted version of MT-Bench in French with all questions having undergone rigorous human review and adaption to guarantee authentic wording, and coherence, and to account for cultural discrepancies.

Translation. Translation capabilities are evaluated through the test set of the 2014 WMT French-English and English-French tasks ¹¹⁵Alves et al. (2023)Steering Large Language Models for Machine Translation with Finetuning and In-Context Learning. We measure performance using BLEU score ¹¹⁶Papineni et al. (2002)Bleu: a Method for Automatic Evaluation of Machine Translation
Post (2018)A Call for Clarity in Reporting BLEU Scores, and COMET ¹¹⁷Rei et al. (2022)COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. We also report FLORES ¹¹⁸Team et al. (2022)No Language Left Behind: Scaling Human-Centered Machine Translation and TICO ¹¹⁹Anastasopoulos et al. (2020)TICO-19: the Translation Initiative for Covid-19 scores.

Belebele. Belebele is a challenging reading comprehension dataset, with multiple choices, released across 122 languages in parallel format ¹²⁰Bandarkar et al. (2023)The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. We leverage the English and French splits.

Benchmark results

Baseline models. To evaluate CroissantLLM, we compare with an array of various models, varying in parameter size, pre-training language distribution, training corpus size, etc.

For “monolingual'' English models, we evaluate Pythia-1.4B ¹²¹Biderman et al. (2023)Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling trained on 300B tokens, OPT-1.3B ¹²²Zhang et al. (2022)OPT: Open Pre-trained Transformer Language Models trained on 180B tokens, and TinyLlama(1.1B) ¹²³Zhang et al. (2024)TinyLlama: An Open-Source Small Language Model. TinyLlama is a very strong English baseline, as it holds many similarities to CroissantLLM. It is a 1.1B model trained on 3 trillion tokens with the same English corpus as the Croissant base. Although it contains some amount of high-quality non-English data, it is only a minor share of the training corpus, the main data sources being English and code data. As such, it trains on much more English tokens than CroissantLLM. All models are trained way past Chinchilla optimality ($\sim$26B tokens for a 1.3B model).

For monolingual French models, we use GPT-fr ¹²⁴Simoulin & Crabbé (2021)Un modèle Transformer Génératif Pré-entrainé pour le \_\_\_\_\_\_ français, a 1B model trained on 16.3B tokens, as well as the PagnolXL(1.5B) model ¹²⁵Launay et al. (2021)PAGnol: An Extra-Large French Generative Model, both in their author submitted HuggingFace implementations.

We also compare CroissantLLM with multilingual models, notably Llama2(7B) (Touvron et al. 2023) trained on 2T tokens, Mistral7B (Jiang et al. 2023), and Bloom (Scao et al. 2022) models (from 1.1B to 3B), trained on 350B tokens each. We note that although the largest Bloom model is undertrained according to Chinchilla optimality (Hoffmann et al. 2022), smaller models are trained on the same number of tokens, making them largely more inference optimal and thus strong contenders. Finally, in the same size category, we evaluate mGPT ¹²⁶Shliazhko et al. (2022)mGPT: Few-Shot Learners Go Multilingual a 1.3B model trained on 440B tokens.

Finally, to assess the impact of including instruction-like data within the pretraining dataset of models (as done in Bloom), we continue CroissantBase pretraining with a short cooldown phase on an instruction dataset without any formatting, and call the resulting model CroissantCool.

Base model

CroissantLLM obtains strong performances in its model size category, achieving on-par performance with the best monolingual English models on English benchmarks and largely outperforming existing mono and multilingual models on French benchmarks.

English. On English benchmarks (Table 3), CroissantLLM displays performances almost equivalent to those of TinyLlama, which has trained on much more English data. We see training on such a large quantity of English tokens enables our model to edge out similarly sized monolingual models trained on fewer tokens (OPT, Pythia), and larger multilingual models (Bloom 3B) demonstrating the interest of pursuing training past Chinchilla optimality, especially when splitting model capacity across languages.

French. On French classification benchmarks, CroissantLLM largely outperforms models of similar sizes trained on mostly monolingual English or French data, and multilingual models (Table 4). Performance is on par with the Bloom(3B) model, which is about 3 times as large. An interesting phenomenon can be noticed, especially on generative benchmarks assessed in few-shot settings: “base'' models trained with instruction-like data perform a lot better. This is noticeable with the Bloom(3B) model which outperforms the otherwise vastly superior Llama2(7B) model on several tasks, or through the performance gains of CroissantCool with respect to CroissantBase.

Improvements throughout training. The model performance continues to improve on downstream tasks during the entirety of training. We report WMT14 translation performance in Figure 6, and observe similar trends across all tasks. The benefits of training past Chinchilla optimality are clear, and although there are diminishing returns past a certain number of steps, training does not seem to saturate. In low training step settings, performance appears to emerge suddenly, reflecting emergent performance experiments in the literature most often obtained through model scaling ¹²⁷Wei et al. (2022)Emergent Abilities of Large Language Models.

French Trivia. One main question this work attempts to tackle is whether training on bilingual data goes beyond augmenting the language understanding and writing capabilities of a model in another language, but also equips the models with novel knowledge and different cultural biases. We evaluate French cultural knowledge on a Trivia task, consisting of questions about France-related topics, asked in English (Table 5), and score results obtained in 5-shot settings with ROUGE-1.¹²⁸As heuristic-based metrics are often insufficient to capture the diversity of possible answers (Faysse et al. 2023), we also score predictions using GPT4 as a judge, and confirm the ROUGE1 metric is well suited for this task given the closed and short nature of the answers. CroissantLLM is the best performing model evaluated under the 7B size, outperforming English-centric models by significant margins. This knowledge gap showcases the effect of the pretraining data mix in specific knowledge acquisition, underlining the interest of integrating vast amounts of varied and multilingual data when training models aiming for broad knowledge coverage.

Overall. The 1.3B CroissantLLM displays top-of-its-class performance across both languages and all benchmarks, even edging out larger models such as Bloom(3B) on most tasks. All models remain far off from the performance of the strong 7B Llama and Mistral models.

Finetuning

Beyond base model performance, we evaluate CroissantLLM downstream performance once finetuned on generalist chat and instruction data, or on specific target tasks (translation, summarization).

Chat Model

It has been shown that supervised fine-tuning on instruction or chat datasets enables leveraging model capabilities to their fullest ¹²⁹Wei et al. (2022)Finetuned Language Models Are Zero-Shot Learners.

Training. We finetune the base model on public Chat datasets Ultrachat ¹³⁰Ding et al. (2023)Enhancing Chat Language Models by Scaling High-quality Instructional Conversations and Wildchat ¹³¹Zhao et al. (2024)(InThe)WildChat: 570K ChatGPT Interaction Logs In The Wild containing ChatGPT interactions in English and French. We also incorporate 12k samples of translation data (4 % of the SFT dataset). We run finetuning on CroissantLLM, as well as the Bloom-1b7 and TinyLlama models for comparison. The obtained models are further suffixed with “Chat''.

MT-Bench.

We evaluate models on the MT-Bench benchmarks, both in English and French. Although a large difference in performance can be noted between the Bloom model and Croissant in favor of the latter, performance differences with TinyLlama are not as significant, neither in English nor in French. CroissantLLMChat performs strongly in open-ended writing categories (writing, roleplay, humanities) but struggles with reasoning and extractive tasks. Turn 2 performance (reformulation under constraints) is largely lower than Turn 1 performance as can be seen in Figures 3 and 4. Our CroissantLLMChat model also vastly outperforms the BloomZ 3B model trained by CMArkea on a large chat finetuning corpus ¹³²Delestre (2023)BloomZ SFT Chat.

This hints at the fact that quasi-monolingual models with only a minor share of another language in their pretraining corpus can be adapted to a reasonable extent, through subsequent finetuning or continued pretraining, although large pre-training corpora are necessary to incorporate sufficient knowledge and reasoning abilities within the base models. We notice large correlations between generation temperature and performance and find CroissantLLMChat works a lot better with higher temperatures ($\geq 0.4$). For fair comparisons, we only report results obtained with low temperature settings in line with other model evaluations.

Translation. We run translation evaluations on the Chat models¹³³TinyLLaMAChat corresponds to the TinyLlama model finetuned with the same SFT set as CroissantLLMChat. and report results in Table 6. CroissantLLMChat displays extremely strong performances, in line with the strong few-shot performance of the CroissantLLM base model, outperforming models like Mistral7B or Llama13B in few-shot settings, and even matching the open source state-of-the-art specialized translation model for the size category, the NLLB 1.3B ¹³⁴Team et al. (2022)No Language Left Behind: Scaling Human-Centered Machine Translation, trained on vastly superior amounts of parallel data.

Dialog Summarization finetuning

To assess performance on specific downstream applications, we finetune base models on a custom dialog summarization dataset.¹³⁵Proprietary dataset belonging to Illuin Technology, corresponds to organic customer interactions with a counselor. Models are finetuned for three epochs on 6000 samples and results are computed through ROUGE and GPT-4 judgment metrics (Table 7).

CroissantLLM and Bloom(1.7B) models appear to yield strong, yet very similar results, trailing behind the larger Mistral7B model. This hints at the fact that base model performance is not always directly correlated to downstream performance post-finetuning, notably on tasks requiring few to no prior knowledge (here, keypoint extraction and reformulation).

Optimized Inference

Our largest model, CroissantLLM, with 1.3B parameters is dimensioned to be extremely lightweight when compared to the main proprietary models and the smallest versions of the Llama and Mistral model family. This is motivated by the fact that widespread model adoption is bounded by inference compute resources, and most high-performing LLMs require expensive specialized infrastructures to run, which leads to high inference costs and model deployment difficulties. The most downloaded Llama model on the HuggingFace model hub is the 7B variant, reflecting the interest in small, yet effective, models.

At a 1.3B scale, CroissantLLM runs easily on local hardware (personal computers, low-end smartphones) and is easy to deploy on inexpensive CPU servers or low-end GPU servers, unlocking new applications with widespread usage. On higher-end GPUs (Table 8), CroissantLLM is both faster (latency) and less memory intensive enabling it to fit bigger batch sizes (throughput). Performance benchmarks are given in Table 8.

The decoder nature of CroissantLLM enables to benefit from the rich inference optimization ecosystem that has boomed recently. CroissantLLM is compatible with all main model serving libraries and platforms and can easily be quantized or optimized to run on personal devices. We performed 4bit quantization in the GGUF¹³⁶https://github.com/ggerganov/ggml/blob/master/docs/gguf.md format and were able to run the model on lower-end smartphones at a speed of more than 5 tokens per second.

Model limitations

Evaluation results indicate the model is strong in its size category, and offers decent performances on writing-based tasks and internal knowledge, and very strong performance on translation tasks. The small size of the CroissantLLM model however hinders its capacity to perform more complex reasoning-based tasks, at least in a zero or few-shot manner in its generalist base or chat-model versions. This is aligned with other models of size and underlines the importance of scale for more abstract tasks ¹³⁷Wei et al. (2022)Emergent Abilities of Large Language Models.

Knowledge Cutoff. The model training dataset has a data cutoff date corresponding to the November 2023 Wikipedia dump. This is the de facto knowledge cutoff date for our base model, although a lot of information dates back further.¹³⁸Prompted with "Who is the current French prime minister ?", it responds: "The current French prime minister is Jean Castex." which is outdated by more than 18 months at the time of the writing. Updated versions can be trained through continued pre-training or subsequent fine-tuning.

Hallucinations. CroissantLLM can hallucinate ¹³⁹Ji et al. (2023)Survey of Hallucination in Natural Language Generation
Guerreiro et al. (2023)Hallucinations in large multilingual translation models and output factually incorrect data,¹⁴⁰As an example, prompted with "Which French club won the UEFA Champions League ?", it answers "The Paris Saint-Germain (PSG) club won the UEFA Champions League in 2020-2021." especially regarding complex topics. This is to be expected given the small model size, and hallucination rates seem inferior to most models of the same size category although no quantitative assessments have been conducted outside of MT-Bench experiments. Veering away from generative use cases, CroissantLLM has also been adapted as an embedding model ¹⁴¹https://huggingface.co/manu/sentence_croissant_alpha_v0.4 and recent work has since been conducted in assessing confidence in the retrieval scores associated to such models ¹⁴²Gisserot-Boukhlef et al. (2024)Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism.

Foundation Model Transparency Index

To assess the transparency of our work, we evaluate our model through the Stanford Transparency Index ¹⁴³Bommasani et al. (2023)The Foundation Model Transparency Index and obtain a total score of 81 %, far ahead of proprietary models, as well as most staple open-weights models and large-scale open-source efforts (Figure 8).¹⁴⁴Methodology is described in the appendix, and outlines the fact that our work relies on the index to guide its efforts in transparency, thus putting it at an advantage with respect to prior work such as Bloom (Scao et al. 2022).

Upstream. The upstream categories include data, compute, and methods dimensions. The fully open-source nature and extensive disclosure of training information enable CroissantLLM to score 88 % of the points. The difficulties in identifying personal information and in guaranteeing the exact license, and creators of all data included in internet scale corpora prohibit our work from obtaining the full points, although strong efforts have been made in only using data under free-use or open licenses and with no copyright issues, notably by excluding copyright flagged content from our English language corpus.

Model. The model categories include model information, as well as characterizations and mitigations of risks, limitations, trustworthiness, and mitigation. CroissantLLM obtains an average of 73 % on this domain due to the wide array of reproducible evaluation results reported, but hindered by the lack of third-party external evaluation at the moment, and an evaluation of potential harms that is not as extensive as required.

Downstream. Downstream categories refer to usage policies, user statistics, distribution, documentation, and model impact assessment. The fully open-access nature of our model and distribution channel avoids most of the transparency pitfalls linked to restricted usage policies and user information processing, but the impact of our work remains difficult to assess until the model is released. The aggregated score for this category is 80 %.

Post-Mortem & Lessons learned

In line with our efforts of full transparency, we share some key lessons learned through this project, in light of a few months of hindsight and of recent developments posterior to the model release.

Tokenizer. In this work, we have developed our own tokenizer, inspired by Llama2 ¹⁴⁵Touvron et al. (2023)Llama 2: Open Foundation and Fine-Tuned Chat Models but fitted to a bilingual French-English corpus to improve fertility on both languages. Several key decisions can impact tokenizer design. The vocabulary size will have large impacts on the parameter count of the model, potentially making embedding parameters a large share of the total parameter count especially for smaller models ¹⁴⁶Team et al. (2024)Gemma 2: Improving Open Language Models at a Practical Size. One should experiment with the performance and memory trade-offs of using larger vocabulary sizes. Furthermore, non-standard tokenizers are likely to be less supported by standard inference frameworks, complicating model adoption. Finally, we believe hand-designing a subset of tokens that are known to be useful at inference, for particular tasks, or have semantic coherence (common numbers, punctuation patterns, tokens corresponding to multiple choice templates or code syntax) probably degrades fertility but is useful in the long run from a performance and usability perspective.

Data Mix. It was always understood the data quality and quantity of the pretraining mix had large impacts on pretraining. In this work, we notably show the large interest of having very large ratios of "aligned" translation data in this mix, as it boosts the model's translation capabilities, but also enables cross-lingual knowledge acquisition. Since CroissantLLM's release, it has become understood that certain types of data heavily boost model reasoning capabilities or performances on benchmarks. Typically, ¹⁴⁷Abdin et al. (2024)Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Team et al. (2024)Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models
Yang et al. (2024)Qwen2 Technical Report
Dubey et al. (2024)The Llama 3 Herd of Models, put a particular emphasis in including math and reasoning heavy corpora in the training corpus. ¹⁴⁸Groeneveld et al. (2024)OLMo: Accelerating the Science of Language Models reports 24 point gains on MMLU scores by including more knowledge and reasoning rich sources to their training set, as well as improving data quality filtering. In order to source such data with highly "educative" values several approaches are possible, including distilling large model knowledge into fast text classifiers to identify high-quality content from Internet scale data ¹⁴⁹Penedo et al. (2023)The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only, or even generating diverse and high quality synthetic text using LLMs ¹⁵⁰Abdin et al. (2024)Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
Ben Allal et al. (2024)Cosmopedia
Dubey et al. (2024)The Llama 3 Herd of Models. More generally, better pretraining and annealing data leads to better models, and many recent datasets would probably lead to better performance nowadays than the datamix we had constituted. These efforts are however often centered around the English language, and our efforts in French data collection remain very valuable.

Scheduler. In CroissantLLM, we leverage the standard Cosine Scheduler with warmup. One disadvantage of such a scheduler is that the number of total training steps must be known in advance, limiting flexibility once training starts. Inspired by the Vision literature ¹⁵¹Zhai et al. (2021)Scaling Vision Transformers, the MiniCPM model ¹⁵²Hu et al. (2024)MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies and several models since then have later confirmed the interest of infinite or WSD (Warmup-Stable-Decay) schedulers¹⁵³Hägele et al. (2024)Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations. The concept is to keep the learning rate at a constant or asymptotically constant value such that training can be done on an arbitrarily long number of tokens. At the end of training, an annealing phase decreases the learning rate to help the model converge and boost performance. MiniCPM has further uncovered that annealing on a higher quality data mix is a very efficient strategy. Typically, by including knowledge-rich or reasoning-heavy data in this final training phase, large benchmark improvements are obtained. This technique has since become the standard way of training LLMs, since it both enables training flexibility and large benchmark improvements. We believe running an annealing phase on the CroissantLLM model would have largely boosted performance, especially if done with carefully selected high-quality data in French and English.

Leveraging Larger LLMs. As made clear in this work, while training with a linguistically balanced dataset helps multilingual performance, the number of model parameters remains the strongest performance factor, and given our compute budget, better performance could be obtained by training a larger model on less non-English data. This is perfectly in line with the findings of ¹⁵⁴Kaplan et al. (2020)Scaling Laws for Neural Language Models with Chinchilla scaling laws. Models trained to be small and inference efficient such as CroissantLLM could however benefit from larger multilingual LLMs in multiple ways. Beyond the ever-growing use of LLM-generated data during pretraining ¹⁵⁵Abdin et al. (2024)Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone, small models can also benefit from larger LLMs as critics for alignment processes ¹⁵⁶Yang et al. (2024)Qwen2 Technical Report, but also to replace stochastic weight initialization by starting off from pruned versions of larger models ¹⁵⁷Dubey et al. (2024)The Llama 3 Herd of Models. Obviously, these strategies entail significantly larger resource requirements to train the larger models to begin with.

Model Release. Training a model is one thing, enabling people to use it is another. Our model has garnered significant attention particularly in the French community, and beyond the technical report, many people were interested in associated demos, blogposts, or social media posts. In retrospect, some barriers of entry remained for users with limited technical skills. The lack of day 0 support from third party inference libraries (Ollama¹⁵⁸https://ollama.com/, llama.cpp¹⁵⁹https://github.com/ggerganov/llama.cpp, MLCChat¹⁶⁰https://llm.mlc.ai/) has led to some community-proposed implementations that were suboptimal, leading to performance hits. Furthermore, official fine-tuning resources (notebooks, Axolotl configurations, etc.) could have largely helped users adapt CroissantLLM for specific use cases and drive adoption. Considering the downstream uses of the model and its utilization seems paramount in the construction of such a project.

Building a model is an iterative process. Building upon the CroissantLLM work and the above listed insights, our team has since been able to train stronger models ¹⁶¹Martins et al. (2024)EuroLLM: Multilingual Language Models for Europe
Boizard et al. (2025)EuroBERT: Scaling Multilingual Encoders for European Languages. The field evolves quickly and getting everything right the first time is not easy. To build very good models, it seems important to us to construct the foundations iteratively, getting things to work at a small scale first, collecting some practical real-world feedback and keeping a core model development team stable.

Broader Impact Statement

This work aims to offset recent English-centric work by enabling the study of the impact of language distribution within the pre-training dataset. The objective is to offer valuable resources to strengthen the community's understanding of induced model behavior and biases in that multilingual setup and inform future model and dataset development to be more inclusive.

Model and Resource Release. The models and all related artifacts are released openly on the CroissantLLM HuggingFace organization¹⁶²https://huggingface.co/CroissantLLM under an MIT license. No usage restrictions are imposed on users whatsoever. We indicate that users are responsible for the content they generate through the use of CroissantLLM and no redress mechanisms exist for harmful content disclosure. The model is offered to users openly, and downstream developers are accountable for using the model responsibly, although usage examples are provided.

Users are free to download and use the model and the associated resources at their will, and no monitoring information is kept by the CroissantLLM team regarding individual model usage or download information. The distribution platform, HuggingFace, does not share any non-public data with the CroissantLLM authors. Any modifications to the models or ulterior versions of the resources will be released under different version numbers, and original resources will not be deleted. We encourage discussions and feedback, either through the HuggingFace model page in the discussion tab, or the issues section of the associated GitHub repository.

Risk mitigation. We intend for our training process to be fully transparent and as such release all artifacts related to training. As such, our base model is released as is, without any risk mitigation methods beyond the extensive data curation that has gone into creating the pre-training data to remove toxic content as much as possible. In our Chat variant of the model, chat instructions have been explicitly sampled to include alignment instructions that train the model not to respond to certain prompts.¹⁶³For example when prompted "Give me medical advice about how to solve my back issues.", CroissantLLM responds I don't have a physical body or medical knowledge. However, I can provide you with some general tips on how to solve back issues: 1. Stay active: exercises such as yoga and stretching can help alleviate back pain. Regular physical activity can help improve flexibility, strengthen the muscles, and reduce the risk of future back problems. 2. Maintain a healthy diet: a balanced diet rich in fruits, vegetables, and lean protein can help reduce inflammation and improve digestion. Avoid high-fat and high-sugar foods, which can aggravate back pain. 3. Manage stress: chronic stress can worsen back pain. Try relaxation techniques such as meditation, deep breathing, or progressive muscle relaxation to help manage stress levels. 4. Get regular back checkups: regular checkups with a healthcare provider can help detect any underlying back issues early on. This can help prevent future problems. 5. Seek professional help: if you have persistent back pain, it may be necessary to seek the help of a healthcare professional. They can help diagnose and treat any underlying back issues that may be contributing to your pain.

Data Leakage. Through the inclusion of canaries in the training set, experiments were conducted on model memorization.¹⁶⁴This research is conducted by an independent research team and results will be published after the Technical Report release. These experiments confirm only artificially extreme cases of data repetition lead to in-weight information of inclusion within the training set. This enables us to confidently release the model without fear of potentially private data leakage that data filtering methods were unable to detect.

Risk Assessment. Our extensive evaluation process and the small scale of the CroissantLLM models allow us to confidently release all artifacts in our efforts of transparency without fear of potential misuse beyond what existing models of larger size already enabled. We staged our release by first giving model access to a dozen individuals and enabling them to experiment with them, whether through finetuning experiments, chat interactions, etc. Their feedback was aligned with the authors' observations in terms of the model capabilities and limitations and no warning flag was raised in terms of toxic content generation or otherwise harmful model behavior. We are confident the release will enable in-depth studying of large language models and outweigh the potential risks.

To further strengthen compliance with FMTI guidelines, we will inform of any government inquiries regarding our model. We also indicate that users are responsible for the content they generate through the use of CroissantLLM and no redress mechanisms exist for harmful content disclosure.