Temporal Annotation Proximity Boosts Quality for African Language AI Datasets

This research investigates the critical challenge of maintaining high annotation quality in sentiment datasets, particularly when annotation efforts extend over long periods with limited annotator pools. The study introduces a new Setswana sentiment dataset comprising 3,565 tweets, meticulously annotated by three native speakers across eight distinct batches. A key finding reveals that while overall inter-annotator agreement (IAA) was excellent, per-batch agreement significantly declined over time, highlighting a crucial issue in data collection methodology.

Through a series of targeted analyses, the researchers identified that label confusion frequently occurs at the negative/neutral boundary, and some annotators exhibited "autopilot labeling" drift. Crucially, the dominant predictor of annotation quality was found to be temporal simultaneity: tweets annotated within a minute of each other achieved near-perfect agreement, whereas those annotated more than a day apart showed significantly lower agreement. Interestingly, annotation speed and tweet-level linguistic features did not correlate meaningfully with agreement levels.

These findings have profound implications for the development of high-quality artificial intelligence models, especially for under-resourced languages like many spoken across Africa. The quality of training data directly impacts the performance and reliability of AI systems. By identifying temporal simultaneity as a key factor, this research provides actionable insights for optimizing annotation campaigns, ensuring more consistent and accurate datasets. This is vital for building robust AI applications that can effectively understand and process the nuances of African languages.

The study also benchmarked several multilingual encoders, including proprietary models like GPT-5 and Gemini, on the Setswana sentiment classification task. Fine-tuning these models on the newly created dataset resulted in substantial performance gains, demonstrating the value of high-quality, language-specific data. The researchers have generously released the Setswana dataset, along with per-annotation timestamps and analysis code, to foster reproducible quality auditing and support the broader development of future African language NLP resources. This contribution is instrumental in advancing AI capabilities for the continent's diverse linguistic landscape.