New 10-Billion-Token African Academic Corpus Addresses LLM Bias and Advances AI Sovereignty

Researchers have introduced Africa-Papers, a monumental 10-billion-token multi-domain academic corpus designed to address the critical underrepresentation of African scholarship in large language model (LLM) pre-training datasets. Existing foundational datasets contain less than 0.5% African scientific, medical, legal, and agricultural content, leading to significant performance failures in LLMs when dealing with African-specific contexts like tropical epidemiology, smallholder farming, regional governance, and indigenous languages.

Africa-Papers is the largest curated Pan-African academic pre-training corpus to date, comprising over 844,807 full-text papers. It aggregates content from diverse sources, including peer-reviewed journals across 31 African nations, the CGIAR CGSpace agricultural repository, African-authored STEM/AI preprints on arXiv, and open-access international publications by scholars affiliated with African institutions.

This initiative is crucial for advancing African intellectual sovereignty in AI. By providing a rich, domain-specific dataset, Africa-Papers aims to enable the development of more accurate, culturally relevant, and context-aware foundation models. The corpus is meticulously processed into a clean, layout-aware Markdown format, preserving complex structures like mathematical equations and tabular data, and is accompanied by a standardized 8-domain taxonomy.

The release of Africa-Papers openly on the Hugging Face Hub under the AfriSpeech organization signifies a major step towards fostering global research and development in AI with a strong African focus. It will facilitate continual pre-training and domain evaluation for foundation models worldwide, ensuring that future AI systems are better equipped to understand and serve African populations and contexts.