Overall, the paper lacks clarity and depth in describing both the technical implementation and practical contributions.
### **Major comments**
1. Unclear contribution: The paper does not effectively justify why this benchmark must exist as a standalone contribution rather than an addition to existing Starcraft II resources. The contribution seems limited to a collection of scripts and metrics, which could likely be integrated into the existing environment without creating a separate benchmark.
2. Lack of implementation details: Key technical aspects of the implementation are insufficiently described, making it hard to understand the benchmark's novelty and how it's technically realized. Several things are not clear, such as:
- Integration: How are LLM agents integrated with StarCraft II? How can users use the benchmark? Does the benchmark use a custom API or an interface for this?
- Decision Tracking: How is decision-making tracked and analyzed? While Table 3 provides a decision trajectory, details of how this is analyzed and used are missing.
- Computational Requirements: What hardware/software is necessary to run this benchmark effectively? This information is critical for usability but is absent.
- Opponents: Are the LLMs evaluated with built-in agents or newly introduced opponents? The fact that agents are evaluated against built-in agents in Starcraft II is mentioned as a limitation, but it is unclear whether the authors change this in their benchmark.
3. Incomplete metric information: The metrics lack context. For instance, while Appendix A.1 outlines the metrics, there are no defined ranges, leaving the reader unsure of how to interpret scores. For example, how should a Real-Time Decision score of 21.12 versus 37.51 in Table 4 be interpreted? Similarly, terms such as “effective” actions in EPM or “collected vespene” are not unexplained, reducing the metrics’ interpretability (how do we know that these are the right metrics to assess decision-making and planning?).
4. Missing benchmark discussion and limitations: A discussion about future development and limitations of the benchmark is missing, which limits the reader's understanding of the benchmark's intended scope and future extensions.
5. Figure 2 indicates a large variance. Why are there no error bars in the tables?
6. It's important to have the prompt included in the appendix or supplement. Was it possibly in a supplement that I cannot access?
### **Minor comments** (These did not affect my score)
- Abstract: Lines 016-019 are a bit difficult to understand; consider rephrasing
- Figure 2: It’s unclear what this Figure is meant to convey, and the Figure lacks labeled y-axes.
- In Section 4.3, line 367 states "Definitions and methods for these metrics will be further detailed in the figure 4.3." This seems to refer to a table, possibly Table 3, rather than a figure.
- In Table 3, "OBSERVERtgreater" should probably be "OBSERVER."
- Lines 323 + 350 state that screenshots illustrating decision traces will be provided in the appendix, but these are not included
- I don't understand what is meant when the authors state that civilization and the other games are not "strategic and tactical" in Table 1. Additionally, Werewolf is clearly an imperfect information game. The authors should reconsider this table because I believe many of the entries are inaccurate.
- Why is the score in Table 4 unnormalized? It's an incomprehensible number as it stands.
6. It's important to have the prompt included in the appendix or supplement. Was it possibly in a supplement that I cannot access? ### **Minor comments** (These did not affect my score) - Abstract: Lines 016-019 are a bit difficult to understand; consider rephrasing - Figure 2: It’s unclear what this Figure is meant to convey, and the Figure lacks labeled y-axes.