South African News Sites Struggle to Block AI Content Scraping, Exposing Local Language Journalism

A new report reveals that nearly 70% of South African news websites are not taking action to prevent AI crawlers from scraping their content. The study, conducted by the Media Leadership Think Tank at the University of Pretoria's Gordon Institute of Business Science, the Journalism Relay Project, and the International Fund for Public Interest Media, analyzed the robots.txt files of 263 South African news sites.

The findings indicate that while most sites have a robots.txt file, only about 30% use it to block AI crawlers. This capacity to restrict AI access is heavily concentrated among larger, well-resourced publishers, leaving smaller, community, and vernacular outlets largely exposed. The report highlights that implementing such blocks requires technical capacity, which many smaller newsrooms publishing in African languages like Sesotho or isiXhosa often lack.

The consequence of this 'protocol gap' is significant: valuable African-language journalism, crucial for training AI models in low-resource languages, is being absorbed without consent, attribution, or payment. This provides raw material for AI systems that often fail to adequately serve these same language communities. The report argues that in the absence of clear legal frameworks, robots.txt has become a de facto governance tool, but one that only financially stable publishers can effectively utilize.

This issue unfolds within an active regulatory landscape, with South Africa's Competition Commission addressing publisher controls and recommending compensation from Google. Internationally, similar inquiries are underway in Brazil and Nigeria, underscoring a growing global concern over the unauthorized use of media content by generative AI platforms. The study aims to provide Global South publishers with comparative data to better understand their position in this evolving digital environment.