Evaluating Interactive Reasoning in Large Language Models: A Hierarchical Benchmark with Executable Games
A new hierarchical benchmark tests LLM reasoning through interactive game-based environments and goal-oriented logic.
We introduce a multi-turn interactive framework for reasoning evaluation that treats reasoning as active evidence acquisition and belief updating. Wherein, LLMs receive only the task rules, must issue targeted queries to a hidden environment, integrate partial observations over t...
ArXiv CS.AI
Source
Generated insight by OnAirToday AI
This research shifts evaluation from static text processing to active, environment-driven decision-making, providing a more granular look at how LLMs handle uncertainty. By testing metacognitive abilities through counterfactuals, the study reveals significant performance gaps in the models' capacity to reason under dynamic constraints.
Comments
Keep comments relevant and respectful. No links, advertising, self-promotion, or spam.