Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play (SS4.4), suggesting that our curriculum training methodology is improving these models in human-like steps.

The review comment to be evaluated is:

The authors offer a quite extensive evaluation of the proposed approach as well as the basic models used. This evaluation includes a consideration of differences between the used positions and the source (teacher) of the proposed move. Therefore, the evaluation is beyond what is usually expected. However, the used definition of the "human similarity" metric is problematic. For one, expecting an exact matching of ground truth moves is very strict and a similarity metric should be preferred. Although, this is likely not a trivial task, because similarity to human play can potentially be only evaluated by human experts. However, the phrase "behavioral data used to train the original model" suggested, that in most experiments, only moves from the Maia-1200 set are considered. (Also see clarity) In case this understanding is correct, the metric only evaluates deviation from novice play - not from human play. Therefore, the correlation between increase in playing strength and decrease of similarity becomes obvious. A potential resolution could be comparing to any human play (independent of the playing strength or up to the teachers playing strength). This should be an approximate if any human would play like the engine.
Another problem is, that it seems, that learning approach has not converged, after using all batches once (Fig 2,3). Without convergence, the used learning rate becomes an interesting question, because it implicitly defines the trade-off between initial weights and new data.
Furthermore, Section 3.3. is a bit lacking - considering it is the introduction of the main approach. (See clarity)

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

Notably, we significantly optimized the preprocessing pipeline and trained a broader set of models, going up to Maia-2600.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

However, in the case of chess, we also have access to human data at a broad range of skill levels, so we also present results to evaluate our updated models against this additional data (SS4.4). To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model. This win percentage is then used as a proxy for measuring the model strength.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4).

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage). For each design from \(R=1400\) to \(3000\), we fine-tune the student for up to 10,000 batches of data drawn from \((X_{R},Y_{R})\), where a batch is a set of 1024 data points.

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes.

Overall, these results suggest that the strength of a teacher matters significantly when training models.

Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions.

Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

The results are displayed in Figure 3.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

* Mistakes identified by a _higher-rated human-like model_. The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

While it is true that both human students and Maia models tend to make mistakes in the opening phase of the game, it is likely that these decisions are not game deciding, so it may be less critical to look at these positions. On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

While this sacrifices the prediction accuracy slightly (by 3%), it aligns better with our goal of designing a curriculum, since a curriculum may span positions from various games (at varying points in each game), which may not preserve the context of a game's history. We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "strengthened models achieve human similarity on datasets corresponding to their strengthened level of play" appears to be generally supported but somewhat overstated. While Figure 5 shows that models trained with higher-rated teachers tend to achieve better move-matching accuracy with higher-rated human datasets, the relationship isn't perfectly linear or consistent. The evidence shows varying patterns of similarity across different rating levels, and the improvement in human similarity isn't as clear-cut as the claim suggests. The review comment also raises valid concerns about the definition and measurement of "human similarity."

<score>3</score>

Justification:
1. Figure 5 shows some correlation between teacher rating and similarity to higher-rated datasets, but the patterns are complex and not uniformly strong
2. The review points out legitimate concerns about the human similarity metric being too strict and potentially only evaluating deviation from novice play
3. The evidence demonstrates a relationship between model strength and human similarity, but the claim overstates the certainty and consistency of this relationship
4. The paper's own discussion acknowledges some limitations and variability in the results that aren't reflected in the claim's strong wording


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play (SS4.4), suggesting that our curriculum training methodology is improving these models in human-like steps.

The review comment to be evaluated is:

## Strengths
The writing of the paper is exemplary, making it relatively easy to understand. I particularly liked the sec. 1 motivation behind the need for human-like models, even though they may have a worse performance than the models not taking this into account.

## Weaknesses
I find the evaluation of the human likeness of the models to be performed on the original datasets (of the original training, not the finetuning) unacceptable. Doing this makes the metric to count both "unhuman" and "human, but of higher-rating" as unhuman behavior, which doesn't follow the motivation of improving the models while keeping them human. While the sec. 4.4 shows partial results on the target rating, I find presentation of the original rating results as confusing, and the sec. 4.4 to not be detailed enough to override this assessment.

If the proper train/test split was not used, the results may suffer from overfitting (ie. the reported human likeness would not constitute a good approximation of a real move-matching accuracy on new positions).

The presented results provide no new insights for me, and are predictable:

1. As human likeness measures accuracy of reproducing the original data, it drops when finetuning on different data [4]. The higher the rating gap (=> bigger domain shift), the bigger the catastrophic forgetting.
2. As we train on the data with humans of a better rating, the performance (assuming enough data) improves, as the model gets to reproduce the better moves.
3. The initial speed of improvement is lower for datasets that are more different (ie. higher rating) from the original, due to less common features being reusable in the new domain. This implies that, for a fixed sample budget, to get the best final performance there will be a tradeoff between low rating (=> the domain is close and there are a lot of reusable features), and high rating (=> overall better moves to reproduce). Authors also provide a correct intuition in the paper behind this: "After a model undergoes curriculum training, [it] will naturally degrade in its initial performance, because we are teaching it to play differently".

I don't find the results in Experiment 3 very informative, as it's hard to understand the difference between the handcrafted data distributions (is the score lower because the data is of less variance, or coming from different-rating source, or the chosen positions don't cover the space of positions well enough to improve the model, etc.).

The benefits of introducing the curriculum training framework are not clear to me: the concrete method for updating the network (finetuning) is prior work, as is generating the actions from previous trajectories or other models. The paper makes no claims about any of the instantiation of the framework (using trajectories or other models' actions, or changing the states only) as being empirically better then the others, nor does it compare to a natural baseline of training on the new dataset from scratch.

## Actionable feedback

1. Present the results in the experimental section on the target domain (ie. it's human if it's close to what the human of a given strength would play)
2. Clarify whether a train/test split was used in the paper and if not, provide the results where the evaluation data is separate from the training one.
3. Provide a clearer value for an ICLR reader: what it is actually supposed to learn from the paper? The reader would be aware of the techniques for improving the models from the literature, is one of the methods better than the others, in what situations? Which of the claims are only applicable in the domain of human-like models or domain of chess? Is there anything else to take home?

[4]: https://en.wikipedia.org/wiki/Catastrophic_interference

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

Notably, we significantly optimized the preprocessing pipeline and trained a broader set of models, going up to Maia-2600.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

However, in the case of chess, we also have access to human data at a broad range of skill levels, so we also present results to evaluate our updated models against this additional data (SS4.4). To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model. This win percentage is then used as a proxy for measuring the model strength.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4).

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage). For each design from \(R=1400\) to \(3000\), we fine-tune the student for up to 10,000 batches of data drawn from \((X_{R},Y_{R})\), where a batch is a set of 1024 data points.

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes.

Overall, these results suggest that the strength of a teacher matters significantly when training models.

Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions.

Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

The results are displayed in Figure 3.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

* Mistakes identified by a _higher-rated human-like model_. The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

While it is true that both human students and Maia models tend to make mistakes in the opening phase of the game, it is likely that these decisions are not game deciding, so it may be less critical to look at these positions. On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

While this sacrifices the prediction accuracy slightly (by 3%), it aligns better with our goal of designing a curriculum, since a curriculum may span positions from various games (at varying points in each game), which may not preserve the context of a game's history. We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I note several key points regarding the claim that "strengthened models achieve human similarity on datasets corresponding to their strengthened level of play":

The evidence from Figure 5 shows that models trained with different teachers tend to achieve highest similarity scores with datasets matching their teacher's rating level. However, the review raises important concerns about:
1. The evaluation methodology potentially confounding "unhuman" and "human but higher-rated" behaviors
2. Possible lack of proper train/test split raising overfitting concerns
3. Insufficient detail in section 4.4 to fully support the claim

While there is some evidence supporting the claim through matching patterns in Figure 5, the methodological concerns and lack of rigorous validation suggest the claim is somewhat overstated.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play (SS4.4), suggesting that our curriculum training methodology is improving these models in human-like steps.

The review comment to be evaluated is:

Strength:
This paper studies curriculum learning and human-like models in the widely-studied domain of Chess. They provide detailed experiments to find the relation between strength and similarity.

Weaknesses:
However, there are no new techniques in the methodology, and the experiment results are not surprising. To be more specific, they used transferring learning to fine-tune a 1200m model with datasets of different ratings. It is very obvious that the playing strengths will increase and the human similarity will decrease when comparing the fine-tuned model with the original one. Besides, in subsection 4-4, the author said “the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset”. This conclusion does not give any new findings. Overall, there are no new findings and the novelty is poor.

Other comments:
* Fig. 1 and Fig. 5 only provide the decimal numbers, while the readers might be interested in the actual difference in smaller grains.
* In experiment 4.1, the author said “at least 49% … the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches … with a 55% win probability”, and “a 55% win probability by training on a 1800-rated curriculum for 1000 batches but at the cost of a much lower move-matching accuracy.” But according to Fig. 2, the move-matching accuracy is about 48.6% for 1800-rated curriculum of 1000 batches, which is not “much lower” as stated in the context. The results seem to have not much difference when comparing different settings.
* The author should explain how a move is generated by the trained model, is it by MCTS or just by the move with the maximum policy?
* The author claims to propose a framework for curriculum learning, but according to the contents of the paper, this seems exaggerated.
* In subsection 4.3, the author should provide the total number of positions for the three categories of mistakes. Especially for human experts, the 80 simulated games seem to be much smaller than the other two categories which can be generated by programs.
* In Fig. 1 and Fig. 5, what does the color mean?
* On page 5, “50% .” -> “50%.”
* In Data Pipeline section of the appendix, “from 2021-06 to 2021-03”, is it typo for “2021-06 to 2022-03”?

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

Notably, we significantly optimized the preprocessing pipeline and trained a broader set of models, going up to Maia-2600.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

However, in the case of chess, we also have access to human data at a broad range of skill levels, so we also present results to evaluate our updated models against this additional data (SS4.4). To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model. This win percentage is then used as a proxy for measuring the model strength.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4).

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage). For each design from \(R=1400\) to \(3000\), we fine-tune the student for up to 10,000 batches of data drawn from \((X_{R},Y_{R})\), where a batch is a set of 1024 data points.

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes.

Overall, these results suggest that the strength of a teacher matters significantly when training models.

Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions.

Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

The results are displayed in Figure 3.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

* Mistakes identified by a _higher-rated human-like model_. The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

While it is true that both human students and Maia models tend to make mistakes in the opening phase of the game, it is likely that these decisions are not game deciding, so it may be less critical to look at these positions. On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

While this sacrifices the prediction accuracy slightly (by 3%), it aligns better with our goal of designing a curriculum, since a curriculum may span positions from various games (at varying points in each game), which may not preserve the context of a game's history. We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence carefully, I find that the claim stating "models achieve human similarity on datasets corresponding to their strengthened level of play" is somewhat overstated. While Figure 5 shows some correlation between teacher rating and similarity to corresponding human datasets, the pattern isn't as definitive as claimed. The move-matching accuracies show relatively small variations (45-51%) across different rating levels. The review comment correctly points out that this result isn't surprising and represents an expected outcome of the training process rather than a novel finding.

<score>3</score>

Justification:
1. The evidence shows a correlation between teacher rating and dataset similarity, but the differences are subtle (variations of 2-3 percentage points in many cases).
2. The reviewer points out that this is an expected outcome of the training methodology rather than a novel insight.
3. The claim could be more precisely worded to acknowledge that the similarities are modest and the pattern, while present, isn't as strong as suggested.
4. The phrase "human-like steps" implies a stronger parallel to human learning than the evidence fully supports.
5. The evidence in Figure 5 shows that models often maintain similar accuracy levels across multiple rating bands, rather than showing sharp distinctions at specific rating levels.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play (SS4.4), suggesting that our curriculum training methodology is improving these models in human-like steps.

The review comment to be evaluated is:

Strengths

- **************Clarity**************: The paper is well written and easy to understand, and the sections naturally follow from one to the other.
- **********Novelty:********** The paper is relatively novel in its approach towards understanding curriculum learning and building human-like models. While the zone of proximal development (ZPD: problems with medium difficulty are the most useful) have been studied by measuring problem difficulty, the paper looks at teacher/annotator proficiency.
- The analysis in the paper is thorough, and the graphs and methods are easy to understand.
- The insight of accounting for labeller/ teacher proficiency is quite powerful.

Weakness

- The interpretation of the result (most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere in the middle) in Experiment 2 is not well supported. The authors themselves suggest the alternate hypothesis that seems more likely: Maia- 1200 cannot account for the stylistic differences between a human-like model and an alpha-beta model like Stockfish. The alternate hypothesis seems likelier as the most similar stylistic teachers with the highest ratings are the best teachers. Without an experiment to test this, one of the central claims of the paper is not on firm grounds.
- I am unsure how the results in this paper might extend to a domain outside of chess: either for teaching people or for building stronger human-like models. The discussion section is lacking here, with just one application domain in the paper! For example, the claim that “that the teacher is much more important than the curriculum.” might not extend to other domains.
- Experiments 2,3 and Figures 2 and 3 show human similarity based on the games of 1200 rated players. When improving the strength of a model, one might want to see a resemblance closer to the target rating rather than the base rating. So, if I wish to have a certain similarity with a 1600-rated human, would it be better to train using a 1600 agent or a 1400 agent or an 1800 agent? Using matching accuracy from 1200-rated games does not seem right (this is also seen in experiment 4).

Minor

- The caption or the graph in figure 5 does not mention the metric shown.

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

Notably, we significantly optimized the preprocessing pipeline and trained a broader set of models, going up to Maia-2600.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

However, in the case of chess, we also have access to human data at a broad range of skill levels, so we also present results to evaluate our updated models against this additional data (SS4.4). To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model. This win percentage is then used as a proxy for measuring the model strength.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4).

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage). For each design from \(R=1400\) to \(3000\), we fine-tune the student for up to 10,000 batches of data drawn from \((X_{R},Y_{R})\), where a batch is a set of 1024 data points.

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes.

Overall, these results suggest that the strength of a teacher matters significantly when training models.

Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions.

Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

The results are displayed in Figure 3.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

* Mistakes identified by a _higher-rated human-like model_. The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

While it is true that both human students and Maia models tend to make mistakes in the opening phase of the game, it is likely that these decisions are not game deciding, so it may be less critical to look at these positions. On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

While this sacrifices the prediction accuracy slightly (by 3%), it aligns better with our goal of designing a curriculum, since a curriculum may span positions from various games (at varying points in each game), which may not preserve the context of a game's history. We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that it is somewhat overstated. While Figure 5 shows some correlation between teacher ratings and similarity to corresponding human datasets, the claim that models "achieve human similarity on datasets corresponding to their strengthened level of play" oversimplifies the complex patterns shown. The data demonstrates varying degrees of similarity across different rating levels, rather than a perfect correspondence. Additionally, one of the reviewer's weaknesses points out that experiments mainly show human similarity based on 1200-rated games, questioning whether this truly represents achievement of human similarity at higher rating levels.

<score>3</score>

Justification:
1. The evidence shows some correlation between teacher ratings and human similarity metrics, but not as definitive as claimed
2. The reviewer raises valid concerns about measuring similarity primarily against 1200-rated games
3. Figure 5 shows varying patterns of similarity across rating levels, not clear achievement of similarity at specific strengthened levels
4. The evidence suggests improvements in human-like aspects but doesn't fully support the strong claim of "achieving" human similarity at corresponding levels

The claim would be better stated with more nuanced language acknowledging the varying degrees of similarity observed and the limitations in how human similarity was measured.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To achieve this goal, we extend the concept of curriculum learning to settings with multiple labeling strategies, allowing us to vary both the _curriculum_ (dataset) and the _teacher_ (labeling strategy).

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model on a particular curriculum and teacher, we adopt the fine-tuning methodology of McIlroy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al., 2019; Yosinski et al., 2014; Kornblith et al., 2019) While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy).

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

Note that our curriculum training notion extends the standard curriculum learning notion developed by Bengio et al. (2009) to settings with multiple teachers (labeling strategies).

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

We adopt the Maia-1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum \(X\) and teacher \(Y\), varying both \(X\) and \(Y\) (SS4.1), varying only \(Y\) (SS4.2), and varying only \(X\) (SS4.3).

We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum.

Note that in this experiment, we have simultaneously updated the curriculum \(X\) and the teacher \(Y\). In order to separately examine the effects of curricula and the effects of teachers on the fine-tuning process, we run two additional experiments to investigate these factors separately.

Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model. We now hold the teacher \(Y\) constant and vary \(X\), which board positions we teach from, to examine the effect of curricula. From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

One possible class of curricula we could analyze is the distributions of board positions encountered at different levels.

Instead, we use specific domain knowledge in chess to develop our own sets of curricula based on fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze. * Mistakes identified by a _superhuman chess engine_.

* Mistakes identified by a _human expert_.

* Mistakes identified by a _higher-rated human-like model_. The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

If our objective is to maximize strength gains with a constraint on the data usage, both the blunder curriculum and the 1400-inflection curriculum are more efficient at teaching the base model than a random curriculum, while the rooks curriculum underperforms in this regard.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

To investigate this, we took the 11 models trained in Experiments 2 and 3 after learning the full 10,000-batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.

Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

At a high level, we find that the choice of teacher is critical for maximizing the efficiency of strength gains. In many cases, the best teacher is _not_ the strongest teacher. Notably, Stockfish (the world's strongest chess engine) is a worse teacher than even a low-rated human-like model. Our results also suggest that the choice of curriculum is less important than the choice of teacher.

While this sacrifices the prediction accuracy slightly (by 3%), it aligns better with our goal of designing a curriculum, since a curriculum may span positions from various games (at varying points in each game), which may not preserve the context of a game's history. We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 4: The results of fixing Y to Maia-2400 while varying X to positions identified by a superhuman chess engine (blunder), a human expert (rooks), and a human-like AI (1400-inflection).

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about extending curriculum learning to settings with multiple labeling strategies appears well-stated. The paper thoroughly demonstrates this extension through:
1. Clear formal definition of curriculum training with distinct curriculum (X) and teacher (Y) components
2. Comprehensive experiments varying both X and Y independently and together
3. Detailed results showing different impacts of curricula and teachers (Figures 2-5)
4. Evidence of systematic investigation of different teacher types (human-rated players, Maia models, Stockfish) and curriculum types (rating-based, mistake-based)

The claim accurately reflects what was implemented and studied without exaggeration.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To achieve this goal, we extend the concept of curriculum learning to settings with multiple labeling strategies, allowing us to vary both the _curriculum_ (dataset) and the _teacher_ (labeling strategy).

The review comment to be evaluated is:

The authors offer a quite extensive evaluation of the proposed approach as well as the basic models used. This evaluation includes a consideration of differences between the used positions and the source (teacher) of the proposed move. Therefore, the evaluation is beyond what is usually expected. However, the used definition of the "human similarity" metric is problematic. For one, expecting an exact matching of ground truth moves is very strict and a similarity metric should be preferred. Although, this is likely not a trivial task, because similarity to human play can potentially be only evaluated by human experts. However, the phrase "behavioral data used to train the original model" suggested, that in most experiments, only moves from the Maia-1200 set are considered. (Also see clarity) In case this understanding is correct, the metric only evaluates deviation from novice play - not from human play. Therefore, the correlation between increase in playing strength and decrease of similarity becomes obvious. A potential resolution could be comparing to any human play (independent of the playing strength or up to the teachers playing strength). This should be an approximate if any human would play like the engine.
Another problem is, that it seems, that learning approach has not converged, after using all batches once (Fig 2,3). Without convergence, the used learning rate becomes an interesting question, because it implicitly defines the trade-off between initial weights and new data.
Furthermore, Section 3.3. is a bit lacking - considering it is the introduction of the main approach. (See clarity)

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model on a particular curriculum and teacher, we adopt the fine-tuning methodology of McIlroy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al., 2019; Yosinski et al., 2014; Kornblith et al., 2019) While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy).

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

Note that our curriculum training notion extends the standard curriculum learning notion developed by Bengio et al. (2009) to settings with multiple teachers (labeling strategies).

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

We adopt the Maia-1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum \(X\) and teacher \(Y\), varying both \(X\) and \(Y\) (SS4.1), varying only \(Y\) (SS4.2), and varying only \(X\) (SS4.3).

We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum.

Note that in this experiment, we have simultaneously updated the curriculum \(X\) and the teacher \(Y\). In order to separately examine the effects of curricula and the effects of teachers on the fine-tuning process, we run two additional experiments to investigate these factors separately.

Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model. We now hold the teacher \(Y\) constant and vary \(X\), which board positions we teach from, to examine the effect of curricula. From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

One possible class of curricula we could analyze is the distributions of board positions encountered at different levels.

Instead, we use specific domain knowledge in chess to develop our own sets of curricula based on fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze. * Mistakes identified by a _superhuman chess engine_.

* Mistakes identified by a _human expert_.

* Mistakes identified by a _higher-rated human-like model_. The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

If our objective is to maximize strength gains with a constraint on the data usage, both the blunder curriculum and the 1400-inflection curriculum are more efficient at teaching the base model than a random curriculum, while the rooks curriculum underperforms in this regard.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

To investigate this, we took the 11 models trained in Experiments 2 and 3 after learning the full 10,000-batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.

Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

At a high level, we find that the choice of teacher is critical for maximizing the efficiency of strength gains. In many cases, the best teacher is _not_ the strongest teacher. Notably, Stockfish (the world's strongest chess engine) is a worse teacher than even a low-rated human-like model. Our results also suggest that the choice of curriculum is less important than the choice of teacher.

While this sacrifices the prediction accuracy slightly (by 3%), it aligns better with our goal of designing a curriculum, since a curriculum may span positions from various games (at varying points in each game), which may not preserve the context of a game's history. We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 4: The results of fixing Y to Maia-2400 while varying X to positions identified by a superhuman chess engine (blunder), a human expert (rooks), and a human-like AI (1400-inflection).

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the extension of curriculum learning to multiple labeling strategies is well-supported by the extensive experimental evidence presented. The paper clearly demonstrates how they vary both the curriculum (dataset) and teacher (labeling strategy) through systematic experiments shown in Figures 2-5. The results validate different combinations of curricula and teachers, showing their effects on model performance and human similarity. The claim aligns well with what was actually implemented and tested, without exaggerating the findings.

<score>1</score>

Justification:
1. The evidence directly supports the claim through:
- Formal definition of curriculum training that explicitly incorporates both curriculum X and teacher Y
- Comprehensive experiments varying both curriculum and teacher (Fig 2)
- Controlled experiments isolating effects of teacher (Fig 3) and curriculum (Fig 4)
- Results showing distinct impacts of curriculum and teacher choices
- Clear methodology extending Bengio et al.'s curriculum learning

2. The reviewer comments, while critical of some aspects like human similarity metrics and convergence, do not dispute the fundamental extension of curriculum learning claimed.

3. The claim is precisely worded and matches what was demonstrated:
- Uses measured language ("extend the concept")
- Accurately describes what was varied ("both the curriculum and teacher")
- Makes no exaggerated claims about performance or superiority

4. Any limitations are properly acknowledged in the paper's discussion.

The claim is well-stated (score of 1) as it accurately reflects what was demonstrated without overreaching beyond the evidence presented.