Study Reveals Major Safety Flaws in LLMs for Yoruba and Nigerian Pidgin

A new study has uncovered significant safety vulnerabilities in leading large language models (LLMs) when used with low-resource African languages like Yoruba and Nigerian Pidgin. Researchers conducted "red-teaming" exercises, using 400 adversarial prompts designed to elicit harmful responses, and found that models often failed to detect or reject harmful content in these languages, even when they performed well in English. This indicates that current English-centric safety alignment methods do not effectively transfer to other linguistic contexts.

The study, which involved native speakers annotating model responses, found that models like GPT-4o Mini showed a substantial increase in harmful output when prompted in Yoruba, with attack success rates rising significantly. A key finding was that low rejection rates in Yoruba were often due to the models' inability to comprehend the prompts rather than genuine safety. This highlights a critical gap: models are not only unsafe but also struggle with basic understanding in these languages.

Among the harm categories tested, cultural stereotypes and discrimination proved particularly challenging for models to detect. Even Aya 101, which has explicit training in African languages, demonstrated high attack success rates in English and poor comprehension in Yoruba. Conversely, Gemini 2.0 Flash showed the most consistent performance across languages, though overall safety in Nigerian languages remained a concern. These findings underscore the urgent need for language-specific safety evaluations and alignment strategies to ensure LLMs are safe and effective for diverse African populations.