Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play, suggesting that our curriculum training methodology is improving them in human-like steps.

The evidence to be evaluated is:Recently, McIlroy-Young et al. (2020) released the Maia Chess engine, which repurposes the AlphaZero architecture to use supervised learning on human games to mimic human play directly, instead of trying to learn optimal play. Maia was trained on a massive public human chess dataset provided by the website Lichess.org, and showed the ability to mimic human behavior at a variety of different skill levels in chess. Further work by McIlroy-Young et al. (2022b) expanded the capability of Maia to mimic individual chess players as well as distinguish between individual playing styles (McIlroy-Young et al., 2021).

In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model on a particular curriculum and teacher, we adopt the fine-tuning methodology of McIlroy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al., 2019; Yosinski et al., 2014; Kornblith et al., 2019) While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate. There is strong empirical evidence supporting the notion that human chess training can build knowledge to improve future skill (Vaci et al., 2019).

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess. We adopt the Maia-1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum \(X\) and teacher \(Y\), varying both \(X\) and \(Y\) (SS4.1), varying only \(Y\) (SS4.2), and varying only \(X\) (SS4.3). We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4). We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\).

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability.

As an approximate solution, we use Maia models that we have trained to serve as the teacher \(Y\). Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available. Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

In the previous set of experiments, we evaluated human similarity by using the 1200-rated human dataset. Because the student model was originally trained and evaluated on this dataset, it makes sense to use this same dataset as a way to measure how much the updated model has deviated from the original model.

To investigate this, we took the 11 models trained in Experiments 2 and 3 after learning the full 10,000-batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

Moreover, the presence of such models could introduce new forms of human-AI interactions, raising new questions about how humans perceive, trust, and evaluate these AI systems.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "strengthened models achieve human similarity on datasets corresponding to their strengthened level of play" appears to be well-supported. Figure 5 shows clear correlations between teacher model ratings and similarity to corresponding human datasets - for example, models taught by Maia-1600 are most similar to 1600-rated datasets, while those taught by Maia-2400 align best with 2400-rated datasets. The evidence demonstrates that as models are strengthened through curriculum training, they maintain human-like characteristics at their new skill level, matching the claim's scope without overstatement.

<score>1</score>

Justification:
1. The claim is directly supported by empirical evidence shown in Figure 5's heatmap, which demonstrates clear relationships between model training and human similarity at corresponding skill levels.
2. The text explicitly states "there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model."
3. The claim is carefully worded to focus specifically on human similarity at corresponding skill levels, without making broader unsupported generalizations.
4. The methodology and evaluation metrics are well-documented and appropriately used to support the specific claim being made.
5. The authors acknowledge limitations and maintain appropriate scope in their conclusions.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play, suggesting that our curriculum training methodology is improving them in human-like steps.

The review comment to be evaluated is:

The authors offer a quite extensive evaluation of the proposed approach as well as the basic models used. This evaluation includes a consideration of differences between the used positions and the source (teacher) of the proposed move. Therefore, the evaluation is beyond what is usually expected. However, the used definition of the "human similarity" metric is problematic. For one, expecting an exact matching of ground truth moves is very strict and a similarity metric should be preferred. Although, this is likely not a trivial task, because similarity to human play can potentially be only evaluated by human experts. However, the phrase "behavioral data used to train the original model" suggested, that in most experiments, only moves from the Maia-1200 set are considered. (Also see clarity) In case this understanding is correct, the metric only evaluates deviation from novice play - not from human play. Therefore, the correlation between increase in playing strength and decrease of similarity becomes obvious. A potential resolution could be comparing to any human play (independent of the playing strength or up to the teachers playing strength). This should be an approximate if any human would play like the engine.
Another problem is, that it seems, that learning approach has not converged, after using all batches once (Fig 2,3). Without convergence, the used learning rate becomes an interesting question, because it implicitly defines the trade-off between initial weights and new data.
Furthermore, Section 3.3. is a bit lacking - considering it is the introduction of the main approach. (See clarity)

The evidence to be evaluated is:Recently, McIlroy-Young et al. (2020) released the Maia Chess engine, which repurposes the AlphaZero architecture to use supervised learning on human games to mimic human play directly, instead of trying to learn optimal play. Maia was trained on a massive public human chess dataset provided by the website Lichess.org, and showed the ability to mimic human behavior at a variety of different skill levels in chess. Further work by McIlroy-Young et al. (2022b) expanded the capability of Maia to mimic individual chess players as well as distinguish between individual playing styles (McIlroy-Young et al., 2021).

In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model on a particular curriculum and teacher, we adopt the fine-tuning methodology of McIlroy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al., 2019; Yosinski et al., 2014; Kornblith et al., 2019) While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate. There is strong empirical evidence supporting the notion that human chess training can build knowledge to improve future skill (Vaci et al., 2019).

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess. We adopt the Maia-1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum \(X\) and teacher \(Y\), varying both \(X\) and \(Y\) (SS4.1), varying only \(Y\) (SS4.2), and varying only \(X\) (SS4.3). We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4). We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\).

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability.

As an approximate solution, we use Maia models that we have trained to serve as the teacher \(Y\). Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available. Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

In the previous set of experiments, we evaluated human similarity by using the 1200-rated human dataset. Because the student model was originally trained and evaluated on this dataset, it makes sense to use this same dataset as a way to measure how much the updated model has deviated from the original model.

To investigate this, we took the 11 models trained in Experiments 2 and 3 after learning the full 10,000-batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

Moreover, the presence of such models could introduce new forms of human-AI interactions, raising new questions about how humans perceive, trust, and evaluate these AI systems.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim that "strengthened models achieve human similarity on datasets corresponding to their strengthened level of play" is moderately overstated. While Figure 5 shows some correlation between teacher model ratings and similarity to higher-rated human datasets, the reviewer correctly points out limitations in how "human similarity" is defined and measured. The metric only evaluates move-matching accuracy against specific rating bands, which may not fully capture human-like play. Additionally, the evidence shows varying degrees of similarity across different rating levels rather than clear one-to-one correspondence.

<score>4</score>

Justification:
1. The claim oversimplifies complex patterns shown in Figure 5, where models show varying degrees of similarity across different rating levels rather than precise matching to specific levels.

2. The reviewer highlights valid concerns about the definition and measurement of "human similarity" being too narrow and potentially problematic.

3. The paper's own evidence suggests a more nuanced relationship between model strengthening and human similarity than the claim implies.

4. The data suggests trends but does not conclusively demonstrate that strengthened models achieve true human-like play at corresponding skill levels.

The claim would be better stated with more qualification about the specific metrics used and acknowledgment of the limitations in measuring human similarity.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play, suggesting that our curriculum training methodology is improving them in human-like steps.

The review comment to be evaluated is:

## Strengths
The writing of the paper is exemplary, making it relatively easy to understand. I particularly liked the sec. 1 motivation behind the need for human-like models, even though they may have a worse performance than the models not taking this into account.

## Weaknesses
I find the evaluation of the human likeness of the models to be performed on the original datasets (of the original training, not the finetuning) unacceptable. Doing this makes the metric to count both "unhuman" and "human, but of higher-rating" as unhuman behavior, which doesn't follow the motivation of improving the models while keeping them human. While the sec. 4.4 shows partial results on the target rating, I find presentation of the original rating results as confusing, and the sec. 4.4 to not be detailed enough to override this assessment.

If the proper train/test split was not used, the results may suffer from overfitting (ie. the reported human likeness would not constitute a good approximation of a real move-matching accuracy on new positions).

The presented results provide no new insights for me, and are predictable:

1. As human likeness measures accuracy of reproducing the original data, it drops when finetuning on different data [4]. The higher the rating gap (=> bigger domain shift), the bigger the catastrophic forgetting.
2. As we train on the data with humans of a better rating, the performance (assuming enough data) improves, as the model gets to reproduce the better moves.
3. The initial speed of improvement is lower for datasets that are more different (ie. higher rating) from the original, due to less common features being reusable in the new domain. This implies that, for a fixed sample budget, to get the best final performance there will be a tradeoff between low rating (=> the domain is close and there are a lot of reusable features), and high rating (=> overall better moves to reproduce). Authors also provide a correct intuition in the paper behind this: "After a model undergoes curriculum training, [it] will naturally degrade in its initial performance, because we are teaching it to play differently".

I don't find the results in Experiment 3 very informative, as it's hard to understand the difference between the handcrafted data distributions (is the score lower because the data is of less variance, or coming from different-rating source, or the chosen positions don't cover the space of positions well enough to improve the model, etc.).

The benefits of introducing the curriculum training framework are not clear to me: the concrete method for updating the network (finetuning) is prior work, as is generating the actions from previous trajectories or other models. The paper makes no claims about any of the instantiation of the framework (using trajectories or other models' actions, or changing the states only) as being empirically better then the others, nor does it compare to a natural baseline of training on the new dataset from scratch.

## Actionable feedback

1. Present the results in the experimental section on the target domain (ie. it's human if it's close to what the human of a given strength would play)
2. Clarify whether a train/test split was used in the paper and if not, provide the results where the evaluation data is separate from the training one.
3. Provide a clearer value for an ICLR reader: what it is actually supposed to learn from the paper? The reader would be aware of the techniques for improving the models from the literature, is one of the methods better than the others, in what situations? Which of the claims are only applicable in the domain of human-like models or domain of chess? Is there anything else to take home?

[4]: https://en.wikipedia.org/wiki/Catastrophic_interference

The evidence to be evaluated is:Recently, McIlroy-Young et al. (2020) released the Maia Chess engine, which repurposes the AlphaZero architecture to use supervised learning on human games to mimic human play directly, instead of trying to learn optimal play. Maia was trained on a massive public human chess dataset provided by the website Lichess.org, and showed the ability to mimic human behavior at a variety of different skill levels in chess. Further work by McIlroy-Young et al. (2022b) expanded the capability of Maia to mimic individual chess players as well as distinguish between individual playing styles (McIlroy-Young et al., 2021).

In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model on a particular curriculum and teacher, we adopt the fine-tuning methodology of McIlroy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al., 2019; Yosinski et al., 2014; Kornblith et al., 2019) While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate. There is strong empirical evidence supporting the notion that human chess training can build knowledge to improve future skill (Vaci et al., 2019).

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess. We adopt the Maia-1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum \(X\) and teacher \(Y\), varying both \(X\) and \(Y\) (SS4.1), varying only \(Y\) (SS4.2), and varying only \(X\) (SS4.3). We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4). We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\).

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability.

As an approximate solution, we use Maia models that we have trained to serve as the teacher \(Y\). Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available. Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

In the previous set of experiments, we evaluated human similarity by using the 1200-rated human dataset. Because the student model was originally trained and evaluated on this dataset, it makes sense to use this same dataset as a way to measure how much the updated model has deviated from the original model.

To investigate this, we took the 11 models trained in Experiments 2 and 3 after learning the full 10,000-batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

Moreover, the presence of such models could introduce new forms of human-AI interactions, raising new questions about how humans perceive, trust, and evaluate these AI systems.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find it partially overstated. While Figure 5 shows that curriculum-trained models achieve higher move-matching accuracy with higher-rated datasets corresponding to their training level (e.g., models taught by Maia-1600 match better with 1600-rated datasets), the claim of achieving "human similarity" is too strong. The reviewer rightfully points out that the evaluation methodology has limitations - it doesn't properly separate training/test data and may confound "unhuman" behavior with "human but higher-rated" behavior. The methodology needs more rigorous validation before making such definitive claims about human-likeness.

<score>4</score>

Justification:
1. The evidence shows correlation between teacher rating and move-matching accuracy, but doesn't conclusively prove "human similarity"
2. The reviewer raises valid concerns about:
   - Lack of proper train/test split
   - Potentially flawed evaluation metrics
   - Limited analysis of what constitutes human-like behavior
3. The claim states conclusions too definitively given these methodological limitations
4. The phrase "achieve human similarity" overstates what the data actually demonstrates


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play, suggesting that our curriculum training methodology is improving them in human-like steps.

The review comment to be evaluated is:

Strength:
This paper studies curriculum learning and human-like models in the widely-studied domain of Chess. They provide detailed experiments to find the relation between strength and similarity.

Weaknesses:
However, there are no new techniques in the methodology, and the experiment results are not surprising. To be more specific, they used transferring learning to fine-tune a 1200m model with datasets of different ratings. It is very obvious that the playing strengths will increase and the human similarity will decrease when comparing the fine-tuned model with the original one. Besides, in subsection 4-4, the author said “the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset”. This conclusion does not give any new findings. Overall, there are no new findings and the novelty is poor.

Other comments:
* Fig. 1 and Fig. 5 only provide the decimal numbers, while the readers might be interested in the actual difference in smaller grains.
* In experiment 4.1, the author said “at least 49% … the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches … with a 55% win probability”, and “a 55% win probability by training on a 1800-rated curriculum for 1000 batches but at the cost of a much lower move-matching accuracy.” But according to Fig. 2, the move-matching accuracy is about 48.6% for 1800-rated curriculum of 1000 batches, which is not “much lower” as stated in the context. The results seem to have not much difference when comparing different settings.
* The author should explain how a move is generated by the trained model, is it by MCTS or just by the move with the maximum policy?
* The author claims to propose a framework for curriculum learning, but according to the contents of the paper, this seems exaggerated.
* In subsection 4.3, the author should provide the total number of positions for the three categories of mistakes. Especially for human experts, the 80 simulated games seem to be much smaller than the other two categories which can be generated by programs.
* In Fig. 1 and Fig. 5, what does the color mean?
* On page 5, “50% .” -> “50%.”
* In Data Pipeline section of the appendix, “from 2021-06 to 2021-03”, is it typo for “2021-06 to 2022-03”?

The evidence to be evaluated is:Recently, McIlroy-Young et al. (2020) released the Maia Chess engine, which repurposes the AlphaZero architecture to use supervised learning on human games to mimic human play directly, instead of trying to learn optimal play. Maia was trained on a massive public human chess dataset provided by the website Lichess.org, and showed the ability to mimic human behavior at a variety of different skill levels in chess. Further work by McIlroy-Young et al. (2022b) expanded the capability of Maia to mimic individual chess players as well as distinguish between individual playing styles (McIlroy-Young et al., 2021).

In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model on a particular curriculum and teacher, we adopt the fine-tuning methodology of McIlroy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al., 2019; Yosinski et al., 2014; Kornblith et al., 2019) While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate. There is strong empirical evidence supporting the notion that human chess training can build knowledge to improve future skill (Vaci et al., 2019).

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess. We adopt the Maia-1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum \(X\) and teacher \(Y\), varying both \(X\) and \(Y\) (SS4.1), varying only \(Y\) (SS4.2), and varying only \(X\) (SS4.3). We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4). We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\).

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability.

As an approximate solution, we use Maia models that we have trained to serve as the teacher \(Y\). Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available. Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

In the previous set of experiments, we evaluated human similarity by using the 1200-rated human dataset. Because the student model was originally trained and evaluated on this dataset, it makes sense to use this same dataset as a way to measure how much the updated model has deviated from the original model.

To investigate this, we took the 11 models trained in Experiments 2 and 3 after learning the full 10,000-batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

Moreover, the presence of such models could introduce new forms of human-AI interactions, raising new questions about how humans perceive, trust, and evaluate these AI systems.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find the claim "our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play" is partially supported but somewhat overstated. While Figure 5 shows that models trained with higher-rated teachers tend to match better with higher-rated human datasets, the evidence doesn't definitively prove they achieve human similarity in a comprehensive way. The reviewer also points out that this finding is somewhat obvious and not novel. The correlation exists but the strength of "achievement" is overstated.

<score>3</score>

Justification:
1. Supporting evidence:
- Figure 5 shows correlation between teacher rating and similarity to corresponding human dataset
- The text states "the model taught by Maia-1600 is most similar to the 1600-rated dataset"

2. Limitations/Overstatement:
- The term "achieve human similarity" implies a stronger relationship than just correlation
- Reviewer notes this is an expected/obvious outcome
- Limited metrics for measuring "human similarity" (mainly move-matching accuracy)
- No clear threshold defined for what constitutes "achieving" similarity
- No comprehensive comparison of how the models differ from human play beyond move matching

The claim would be better stated by saying the models show increased similarity to higher-rated human play patterns rather than definitively "achieving" human similarity.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Finally, we show that our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play, suggesting that our curriculum training methodology is improving them in human-like steps.

The review comment to be evaluated is:

Strengths

- **************Clarity**************: The paper is well written and easy to understand, and the sections naturally follow from one to the other.
- **********Novelty:********** The paper is relatively novel in its approach towards understanding curriculum learning and building human-like models. While the zone of proximal development (ZPD: problems with medium difficulty are the most useful) have been studied by measuring problem difficulty, the paper looks at teacher/annotator proficiency.
- The analysis in the paper is thorough, and the graphs and methods are easy to understand.
- The insight of accounting for labeller/ teacher proficiency is quite powerful.

Weakness

- The interpretation of the result (most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere in the middle) in Experiment 2 is not well supported. The authors themselves suggest the alternate hypothesis that seems more likely: Maia- 1200 cannot account for the stylistic differences between a human-like model and an alpha-beta model like Stockfish. The alternate hypothesis seems likelier as the most similar stylistic teachers with the highest ratings are the best teachers. Without an experiment to test this, one of the central claims of the paper is not on firm grounds.
- I am unsure how the results in this paper might extend to a domain outside of chess: either for teaching people or for building stronger human-like models. The discussion section is lacking here, with just one application domain in the paper! For example, the claim that “that the teacher is much more important than the curriculum.” might not extend to other domains.
- Experiments 2,3 and Figures 2 and 3 show human similarity based on the games of 1200 rated players. When improving the strength of a model, one might want to see a resemblance closer to the target rating rather than the base rating. So, if I wish to have a certain similarity with a 1600-rated human, would it be better to train using a 1600 agent or a 1400 agent or an 1800 agent? Using matching accuracy from 1200-rated games does not seem right (this is also seen in experiment 4).

Minor

- The caption or the graph in figure 5 does not mention the metric shown.

The evidence to be evaluated is:Recently, McIlroy-Young et al. (2020) released the Maia Chess engine, which repurposes the AlphaZero architecture to use supervised learning on human games to mimic human play directly, instead of trying to learn optimal play. Maia was trained on a massive public human chess dataset provided by the website Lichess.org, and showed the ability to mimic human behavior at a variety of different skill levels in chess. Further work by McIlroy-Young et al. (2022b) expanded the capability of Maia to mimic individual chess players as well as distinguish between individual playing styles (McIlroy-Young et al., 2021).

In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher). When training an existing human-like model on a particular curriculum and teacher, we adopt the fine-tuning methodology of McIlroy-Young et al. (2022b), which itself draws on recent methods from the transfer learning literature (Sun et al., 2019; Yosinski et al., 2014; Kornblith et al., 2019) While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate. There is strong empirical evidence supporting the notion that human chess training can build knowledge to improve future skill (Vaci et al., 2019).

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy). To develop human-like models, we replicate and extend the Maia framework developed by McIlroy-Young et al. (2020). We acquire a database of human chess moves from the online platform Lichess.org, where each move is labeled according to the player's Elo (skill) rating. We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

To evaluate whether the model maintains its human-like playing style, we evaluate how well the updated model can predict human behavioral data.

To provide more insight into this metric, we trained a set of human-like models (following the procedure in SS3.1) from Maia-1200 to Maia-2600, and measure their move-matching accuracy on the datasets with player ratings from 1200 to 2600.

This result replicates the findings of McIlroy-Young et al. (2020) and provides evidence that move-matching accuracy is a reasonable measure to evaluate whether the updated model is still human-like.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model.

We can see that as we increase the distance between two models, the win rate of the higher-rated model increases.

Our results support this claim, by showing that the difference between (for example) a Maia-1800 and Maia-1600 is smaller than the difference between an 1800-rated human and a 1600-rated human. We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2. In this work, we use the domain of chess to evaluate the effectiveness of curriculum training. We pre-trained a set of Maia models to serve as the initial human-like model \(M\) and the potential teachers \(Y\). We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert. In this section, we examine the effectiveness of curriculum training in the domain of chess. We adopt the Maia-1200 model as our student (i.e., the initial human-like model we wish to improve), and update it using different designs of curriculum \(X\) and teacher \(Y\), varying both \(X\) and \(Y\) (SS4.1), varying only \(Y\) (SS4.2), and varying only \(X\) (SS4.3). We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength. We then extend our evaluation of human similarity to cover datasets of higher-rated players, to measure the progress of the student with respect to higher-rated human-like models (SS4.4). We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\).

If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability.

As an approximate solution, we use Maia models that we have trained to serve as the teacher \(Y\). Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available. Using this curriculum design, we ran another experiment with the same training procedure and evaluation methods as Experiment 1.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

From the previous experiment, we see that Maia-2200, Maia-2400, and Maia-2600 are the three most efficient teachers for retaining human similarity and improving playing strength, and are all very similar to each other overall.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

The final category is positions where higher-rated Maia models disagree with the move that the lower-level Maia-student would make. While this is not conclusive evidence of a mistake, it is a possible indication that the lower-level Maia is playing suboptimally. We selected positions where Maia-1200 and Maia-1400 differ, which we denote as 1400-inflection positions. Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

In the previous set of experiments, we evaluated human similarity by using the 1200-rated human dataset. Because the student model was originally trained and evaluated on this dataset, it makes sense to use this same dataset as a way to measure how much the updated model has deviated from the original model.

To investigate this, we took the 11 models trained in Experiments 2 and 3 after learning the full 10,000-batch curriculum, and analyzed their move-matching accuracy against higher-rated datasets, to see if the updated models play like higher-rated humans.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset. Interestingly, each of the models taught using targeted curricula and Maia-2400 teacher all perform most similar to the 2400-dataset.

In this paper, we propose a generalized framework for improving a human-like model while maintaining its human similarity, using a novel extension of curriculum learning which considers both the curriculum data to train on, and the teacher used to label this data.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

Moreover, the presence of such models could introduce new forms of human-AI interactions, raising new questions about how humans perceive, trust, and evaluate these AI systems.

In this work, we train human-like models using a publicly available dataset from Lichess.org.

We also expand the models to incorporate higher-rated styles. The prior work trained models rated between 1100 and 1900, while we added additional higher-rating models until the rating of 2600 to facilitate a richer discussion of curriculum design and model improvement.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the claim and evidence, I find the claim "our strengthened models achieve human similarity on datasets corresponding to their strengthened level of play" is moderately well-supported but contains some overstatement. Figure 5 shows models trained with different teachers do achieve higher similarity scores with datasets matching their teacher's rating level. However, the claim implies this is conclusively proven for all strengthened models, when the evidence only demonstrates this pattern for a subset of curriculum-updated models from Experiments 2 and 3. The reviewers also note uncertainty about how well similarity measures based on 1200-rated games represent true human-likeness at higher skill levels.

<score>2</score>

Justification:
1. Supporting evidence:
- Figure 5 shows correlation between teacher rating and which human dataset the model matches best with (e.g., "model taught by Maia-1600 is most similar to 1600-rated dataset")
- Results in section 4.4 demonstrate updated models show increased similarity to higher-rated datasets

2. Limitations/Overstatement:
- The claim makes a broad statement about "our strengthened models" but evidence only covers specific experimental conditions
- Reviewer points out potential issues with using 1200-rated games to measure similarity at higher ratings
- Authors acknowledge this is a "suggestion" rather than definitive proof in discussion
- Limited testing of human similarity beyond move-matching accuracy
- No direct comparison to how human players progress in similarity metrics during skill development

The score of 2 reflects that while there is meaningful supporting evidence for the core observation, the claim would be better stated with more qualification about the specific conditions tested and limitations of the similarity measures used.