Why you should care about your tokenizer - Part III

How we built a sustainable, efficient, and multilingual tokenizer, thanks to Jérôme Louradour

Building a custom tokenizer for French and 4 other languages (German, Spanish, Italian and English)

In parts I and II of this series, we showed that tokenizers are a crucial component of efficient multilingual LLMs.

So, what choices did we make to create a tokenizer for LUCIE, our 7-Billion parameter model (currently in training)?

• First, we aimed to keep our tokenizer’s fertility close to 1 on non-English languages. Fertility is the average number of tokens per word, measuring the granularity as well as the compression of a tokenizer.

• Second, we aim to keep the size of our vocabulary manageable for our system’s memory and computing capacity. The size of vocabulary (the dictionnary of individual tokens) being inversely correlated with fertility, i.e. vocabulary increases when fertility decreases.

• Third we aim to reach a parity between LUCIE’s five languages close to 1. Parity ensures all languages are tokenized fairly. It represents the ratio between the number of tokens needed to encode the same information in different languages. For instance, in our example of the Victor Hugo sentence (in Part II of the series), parity would be 13/19 = 0.68, showing that French and English are not treated equally by ChatGPT’s tokenizer.

A remark on why we do not aim for fertility lower than 1 :

• Lack of granularity: if you take tokens that are longer than words your model will have less diverse output. You will lose on precision and finesse. An example of that is that if you cut text into groups of words instead of subwords your model will not be able to come with words that would not exist in its dataset. For instance your model would not be able to come up with the - invented - word “robotity” if it does not have granular tokens like ro, bot, and ity as basic bricks to build upon.

• Memory : The smaller the fertility, the larger the vocabulary. When the vocabulary increases, the system needs more memory to store it.

• Compute: even though a low fertility can optimize the energetic consumption of your model, it is only true to some extent. Indeed, the increase in vocabulary size that results from a low fertility also has a significant effect on the computing power needed to generate text.

Symmetrically, we do not aim for a fertility that is too high because it would be very demanding in terms of computation and energy and affect the downstream performance, speed and cost of our model.

So in fact, we try to minimize fertility while putting a constraint on the size of our vocabulary (i.e., the number of unique tokens).

Having made these trade-offs, we finally trained our custom tokenizer, and here is a comparaison of its performance on 5 european languages with that of other tokenizers, including ones specifically fit for the French language.

Fertility of different tokenizers including ours (in red) over several corpora in 5 languages. The lower the fertility, the better the compression of the tokenizer

As we can see all tokenizers perform well on English but there is a greater discrepancy on other languages. Our tokenizer (the red curves) performs fairly well with regard to consistency across languages as well as in compactness (keeping fertility close to 1).

As an example, our tokenizer outperforms Mistral’s tokenizer by 15% on fertility in French for the same vocabulary size (32k tokens).

Thus, running a model in French based on OpenLLM’s tokenizer would save energy compared to running the same model but based on Mistral’s, which is not specifically optimized for French and non-English languages.

Thanks to OpenLLM’s tokenizer, French speakers will now be able to interact with LLMs at a lower cost, greater speed, with a reduced environmental impact. (Cocorico 🐓)

Consider following us if you want to know more about our initiatives in building sustainable, inclusive and multilingual AI systems.