UrduQA: why we built a benchmark that models cannot have memorised
Static benchmarks decay the moment they enter training data. Here is how we built one that stays honest, and what it tells us about the state of Urdu language technology.
Dr. Bilal Ahmed
Associate Professor · Language & Reasoning
When we started evaluating language models on Urdu two years ago, the numbers looked encouraging. Then we noticed something: the best-performing models did suspiciously well on questions drawn from Wikipedia and suspiciously badly on questions drawn from last month’s newspapers.
The explanation was mundane. The benchmarks everyone used were on the internet, and so were the models’ training sets.
Building for contamination resistance
UrduQA has three properties that keep it honest:
- Post-cutoff sources. Every question is drawn from documents published after the training cutoff of the models we evaluate, and we refresh a portion each quarter.
- Paraphrase checks. Each question has a paraphrased twin. A large gap between twin scores signals memorisation of surface form.
- Held-out release. Half the benchmark is public; half is run by us on request. Rankings on the two halves should agree.
What we found
Across fourteen models, the gap between English and Urdu performance is nineteen points on average. Two interventions close most of it: extending the tokeniser vocabulary with Urdu subwords (which also cuts token cost by 40%) and a modest amount of instruction fine-tuning on Urdu data.
The full paper appears at ACL 2026. The public half of the benchmark and the evaluation harness are on the lab’s GitHub.
- language
- evaluation
- urdu
Get the next note by email
One considered letter a month on research, admissions and what we are building.
More in Research
Six months in the field with the smallholder vision kit
What we learned deploying crop-disease detection on sub-$50 hardware across forty-one farms in Punjab — including everything that broke.