New Nsanku Benchmark Exposes LLM Limitations in Translating Ghanaian Languages

A new research paper introduces "Nsanku," a comprehensive benchmark designed to assess the zero-shot machine translation performance of large language models (LLMs) for Ghanaian languages. This initiative addresses a significant gap in AI research, as the capabilities of LLMs for low-resource African languages remain largely unexplored, despite their impressive multilingual skills in more widely supported languages. Understanding these capabilities is crucial for fostering inclusive AI development on the continent.

Nsanku systematically evaluates 19 different open-weight and proprietary LLMs across 43 distinct Ghanaian languages, pairing them with English. The evaluation process utilized 300 sentence pairs per language, meticulously sourced from the YouVersion Bible platform. Performance was measured using established automatic metrics like Bilingual Evaluation Understudy (BLEU) and Character n-gram F-Score (chrF), complemented by an average accuracy score and a cross-language consistency analysis.

The findings reveal that while some commercial models like Gemini-2.5-Flash, Claude-Sonnet-4-5, and GPT-4.1 showed the highest overall average scores, their performance was still relatively modest, with Gemini leading at an average of 26.88. Among open-weight models, Kimi-K2-Instruct-0905 performed best. These scores indicate that even the most advanced LLMs struggle with accurate translation for these specific languages.

A critical insight from the consistency analysis is that no LLM, nor any specific Ghanaian language, achieved simultaneously high performance and high consistency. This suggests that current LLMs are not yet reliably equipped for large-scale, dependable translation of Ghanaian languages. Performance varied significantly, with Siwu achieving the highest per-language average score and Nkonya scoring the lowest, highlighting uneven capabilities across different low-resource languages.

The Nsanku benchmark not only provides a vital evaluation but also establishes a publicly available and extensible infrastructure for future African language NLP research. This resource is essential for researchers and developers aiming to improve AI's utility and accessibility for diverse linguistic communities across Africa, ensuring that AI advancements are relevant and beneficial to all.