New Benchmark Reveals Large Knowledge Gaps in LLMs Regarding African Facts

A new research paper introduces "NaijaFacts," a pioneering benchmark designed to assess large language models' (LLMs) knowledge of Nigeria-specific facts and statistics, rather than just their linguistic competence in African languages. Developed by native speakers and grounded in official Nigerian statistics, the benchmark includes questions in English, Nigerian Pidgin, Yoruba, Igbo, and Hausa. This approach allows researchers to differentiate between a model's inability to understand a low-resource language (language effect) and its fundamental lack of knowledge about a specific fact (knowledge effect).

The study tested eight prominent LLMs, including frontier models and those adapted for Nigeria. The findings are stark: a significant knowledge gap exists concerning Nigeria. While models achieved 49-50% accuracy on similar questions about the US and UK when queried in English, their accuracy dropped to a mere 20% for Nigeria-related facts. The language effect, though present, was smaller, with accuracy degrading from English to Yoruba, largely correlating with web representation of these languages.

Crucially, the research demonstrates that increasing model scale does not resolve this issue. Even the most powerful frontier models, which scored 66-79% on UK and US statistics, failed to surpass 31% on Nigerian statistics. This deficit is not unique to Nigeria; extending the analysis to all 54 African countries using the best-performing model revealed a consistent pattern: high accuracy for general cultural and political facts (96-98%) but a dramatic drop to 26% for statistics across the continent.

The implications for Africa are profound. The study highlights that current LLMs, despite their advanced linguistic capabilities, possess a severe deficiency in factual knowledge about African nations. This means that AI applications intended for use in Africa, from education to policy-making, will likely provide inaccurate or incomplete information when dealing with local contexts and data, underscoring an urgent need for targeted data collection and model training focused on African realities.