Measuring Reasoning Without Leaking the Test: Procedural Generation for LLM Evaluation

Abstract

Static reasoning benchmarks decay as their contents enter training corpora. We propose procedurally generated evaluation families with controllable difficulty and show that model rankings remain stable across generations while absolute scores reveal contamination in three widely used benchmarks.