Enhancing AI Language Models for Ge'ez-Script and Low-Resource African Languages

Multilingual pre-trained language models (PLMs) often struggle with languages that use non-Latin scripts and have fewer digital resources. This performance degradation is largely due to their tokenizers, which are typically optimized for Latin scripts, leading to high out-of-vocabulary rates and excessive subword fragmentation in languages like Amharic and Tigrinya. This research introduces VEXMLM, a specialized variant of XLM-R designed to overcome these challenges for Ge'ez-script languages.

VEXMLM's development involved a unique two-stage training process. First, a language-specific SentencePiece tokenizer was created using curated Amharic and Tigrinya corpora. This allowed for the extension of XLM-R's vocabulary with 30,000 Ge'ez-script subwords, whose embeddings were initialized by averaging existing XLM-R subword embeddings. The model then underwent continued masked language modeling on these extended vocabularies, followed by supervised fine-tuning on critical natural language processing tasks such as question answering (QA), named entity recognition (NER), and sentiment analysis (SA).

The results demonstrate VEXMLM's significant superiority over existing models like XLM-R and Glot500. On Amharic and Tigrinya QA tasks, VEXMLM achieved substantially higher Exact Match (EM) and F1 scores. Similarly, for sentiment analysis, it outperformed both comparative models in accuracy. Crucially, VEXMLM also dramatically improved the accuracy of named entity recognition for out-of-vocabulary tokens, raising it from 81.4% to 94.3% across 11 of the 19 evaluated African languages.

This research offers several key contributions, including a tailored vocabulary-extension and embedding-initialization procedure for Ge'ez script, and a two-stage training strategy. The improvements observed on Amharic and Tigrinya were successfully transferred to 17 other typologically related, unaugmented African languages, highlighting the broad applicability of this approach.

For Africa, this development is profoundly significant. By enhancing AI's capability to understand and process Ge'ez-script and other low-resource African languages, VEXMLM paves the way for more inclusive and effective AI applications across the continent. This advancement can support digital literacy, improve access to information, and facilitate the development of AI tools that genuinely serve the diverse linguistic needs of African populations, reducing the digital divide for many communities.