TukaBench: Enhancing AI Safety Evaluation for African Languages and Cultures

The field of Large Language Model (LLM) safety evaluation is predominantly focused on English, leaving a significant gap for low-resource languages, especially those spoken across Africa. This creates a critical oversight, as LLMs deployed in African contexts may exhibit different safety behaviors and vulnerabilities when interacting in local languages and cultural nuances.

To address this imbalance, researchers have introduced TukaBench, a novel jailbreak benchmark specifically designed for seven African languages. TukaBench extends existing evaluation methods by incorporating four distinct prompt settings: direct human translation of English prompts, English prompts adapted to African cultural contexts then translated, human-curated prompts validated with advanced LLMs, and code-switched prompts combining English and African languages. This comprehensive approach allows for a granular analysis of how language, cultural grounding, and prompt structure influence model safety.

The evaluation results from TukaBench reveal important insights: prompting LLMs in African languages generally leads to a reduction in refusal rates compared to English prompts, with culturally adapted prompts showing the lowest refusal. Furthermore, the study identifies two key structural limitations in LLMs when interacting with low-resource languages: failures in model comprehension and a decrease in the reliability of LLM-as-a-judge evaluation methods. To better capture these issues, TukaBench introduces 'Deflection' as a new metric alongside 'Refused' and 'Jailbroken', and validates judge outputs with human annotations, highlighting reduced agreement in lower-resource languages.

For Africa, TukaBench represents a crucial step towards developing more equitable and safer AI systems. By uncovering language-specific vulnerabilities and biases, it emphasizes the necessity of localized AI safety research and development. This work is vital for ensuring that AI technologies are not only accessible but also culturally appropriate and safe for diverse African populations, preventing potential harms that could arise from inadequately evaluated models.

The findings underscore the importance of moving beyond English-centric paradigms in AI development and evaluation, advocating for robust, culturally sensitive benchmarks that reflect the linguistic diversity of the continent. Such efforts are essential for fostering trust in AI and ensuring its responsible deployment across Africa.