UrduQA: A Contamination-Resistant Benchmark for Question Answering in Urdu

Abstract

We introduce UrduQA, a 4,200-question benchmark built from post-2025 sources with paraphrase-based contamination checks. We evaluate fourteen open and proprietary models, finding a 19-point gap between English and Urdu performance that narrows substantially with vocabulary extension and modest fine-tuning.