SunTzu: A Long-Horizon Decision Making Benchmark for Large Language Models

To evaluate the long-horizon decision-making ability of large language models, we propose SunTzu, a benchmark designed for evaluating LLMs in StarCraft II. We further design a hierarchical decision-making baseline agent framework that self-corrects based on expert knowledge and self-improves via supervised fine-tuning (SFT), substantially improving LLM strategic reasoning and execution.

Links: arXiv · Code

Paper | Code