FindStatBench: Evaluating Large Language Models on Combinatorial Code Synthesis
New benchmark reveals that even the most advanced LLMs struggle with combinatorial logic and brittle rule induction.
We introduce FindStatBench, an execution benchmark for evaluating large language models on combinatorial code synthesis. Built from FindStat, it contains 2,329 tasks across 24 collections and 5.52M hidden instances, covering statistic synthesis, which maps objects to integers, an...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
FindStatBench highlights a critical plateau in LLM reasoning, where even top-tier models struggle with the symbolic complexity of combinatorial mathematics. The findings suggest that current prompting techniques are insufficient for complex rule induction and that existing model architectures are hitting performance limits on specialized, non-trivial algorithmic tasks.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.