New Language Model VEXMLM Boosts AI Performance for Ethiopian and Eritrean Languages

Researchers have developed VEXMLM, a new variant of the multilingual language model XLM-R, specifically designed to improve natural language processing for low-resource Ge'ez-script languages like Amharic (Ethiopia) and Tigrinya (Ethiopia and Eritrea). Traditional large language models often struggle with these languages because their Latin-script-centric tokenizers inefficiently break down Ge'ez words into many subwords, hindering performance.

VEXMLM addresses this by incorporating language-specific SentencePiece tokenizers and expanding XLM-R's vocabulary with 30,000 Ge'ez-script subwords. This vocabulary expansion significantly reduces tokenizer 'fertility' (the number of subword tokens per word), making the processing of Amharic and Tigrinya text much more efficient. The model then undergoes a two-stage training process, including continued masked language modeling and fine-tuning on tasks like question answering, named entity recognition, and sentiment analysis.

While VEXMLM shows modest improvements in named entity recognition and comparable performance in sentiment analysis, its primary breakthrough lies in the efficiency of Ge'ez-script tokenization. The study highlights that while vocabulary expansion is key for efficiency, continued pretraining is essential for the expanded model to surpass baseline performance on downstream tasks. This research provides valuable open-source resources, including a GitHub repository and Hugging Face model, fostering further development in African language AI.