Research paper
Mathador-LM: A Dynamic Benchmark for Mathematical Reasoning on Large Language Models
We introduce Mathador-LM, a new benchmark for evaluating the mathematical reasoning on large language models (LLMs), combining ruleset interpretation, planning, and problem-solving. This benchmark is inspired by the Mathador game, where the objective is to reach a target number using basic arithmetic operations on a given set of base numbers, following a simple set of rules. We show that, across leading LLMs, we obtain stable average performance while generating benchmark instances \emph{dynamically}, following a target difficulty level. Thus, our benchmark alleviates concerns about test-set leakage into training data, an issue that often und
Kurtic, Eldar, Moeini, Amir, Alistarh, Dan
arXiv.org perpetual, non-exclusive licenseView resource