Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find that the choice of teacher has a strong impact on both playing strength and human similarity; for example, a teacher that is too strong can be less effective at improving playing strength and degrade human similarity more rapidly.

The review comment to be evaluated is:

Strength:
This paper studies curriculum learning and human-like models in the widely-studied domain of Chess. They provide detailed experiments to find the relation between strength and similarity.

Weaknesses:
However, there are no new techniques in the methodology, and the experiment results are not surprising. To be more specific, they used transferring learning to fine-tune a 1200m model with datasets of different ratings. It is very obvious that the playing strengths will increase and the human similarity will decrease when comparing the fine-tuned model with the original one. Besides, in subsection 4-4, the author said “the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset”. This conclusion does not give any new findings. Overall, there are no new findings and the novelty is poor.

Other comments:
* Fig. 1 and Fig. 5 only provide the decimal numbers, while the readers might be interested in the actual difference in smaller grains.
* In experiment 4.1, the author said “at least 49% … the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches … with a 55% win probability”, and “a 55% win probability by training on a 1800-rated curriculum for 1000 batches but at the cost of a much lower move-matching accuracy.” But according to Fig. 2, the move-matching accuracy is about 48.6% for 1800-rated curriculum of 1000 batches, which is not “much lower” as stated in the context. The results seem to have not much difference when comparing different settings.
* The author should explain how a move is generated by the trained model, is it by MCTS or just by the move with the maximum policy?
* The author claims to propose a framework for curriculum learning, but according to the contents of the paper, this seems exaggerated.
* In subsection 4.3, the author should provide the total number of positions for the three categories of mistakes. Especially for human experts, the 80 simulated games seem to be much smaller than the other two categories which can be generated by programs.
* In Fig. 1 and Fig. 5, what does the color mean?
* On page 5, “50% .” -> “50%.”
* In Data Pipeline section of the appendix, “from 2021-06 to 2021-03”, is it typo for “2021-06 to 2022-03”?

The evidence to be evaluated is:In general, optimizing AI towards human similarity will necessarily lead to reduced task performance, especially in scenarios where (non-human-like) AI achieves superhuman performance. In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy).

We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

In particular, it is not the case that higher-rated models are better at predicting lower-rated datasets.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model. This win percentage is then used as a proxy for measuring the model strength. To provide more insight into this metric, we have the models from Maia-1200 to Maia-2600 play against each other. The match win percentages (based on 5120 games played between each pair of models) are shown in Figure 0(b). We can see that as we increase the distance between two models, the win rate of the higher-rated model increases. We also see that as ratings go up, the win rates tend to be closer to 50%.

Jacob et al. (2022) showed that higher-rated Maia models trained using standard imitation learning were suboptimal, and claimed that this would lead to weaker play than expected.

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model.

Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2.

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum. Overall, these results suggest that the strength of a teacher matters significantly when training models.

As an approximate solution, we use Maia models that we have trained to serve as the teacher \(Y\). Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available.

We also see the same non-trivial result that the most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere in the middle. In particular, the most efficient teacher is Maia-2200, Maia-2400, or Maia-2600, and these three points tend to be clustered closely together.

This could purely be because Stockfish's strength outclasses all humans, so much of the move advice Stockfish teaches is out of reach for the Maia-1200 student.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

One hypothesis is that the rooks curriculum (human expert recommendation) is too specific in the positions it tries to teach.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

This provides more credence to our earlier observation that the teacher is much more important than the curriculum.

At a high level, we find that the choice of teacher is critical for maximizing the efficiency of strength gains. In many cases, the best teacher is _not_ the strongest teacher. Notably, Stockfish (the world's strongest chess engine) is a worse teacher than even a low-rated human-like model.

Although we did find evidence that efficient curriculum design is possible, it remains an open question how best to select a curriculum for training human-like models in chess.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I've carefully reviewed the claim "We find that the choice of teacher has a strong impact on both playing strength and human similarity; for example, a teacher that is too strong can be less effective at improving playing strength and degrade human similarity more rapidly."

This claim appears well-stated and accurately reflects the evidence presented. The evidence strongly supports this through:

1. Figure 3 clearly shows Stockfish (strongest teacher) performing worse than mid-rated teachers
2. The data shows mid-rated teachers (Maia-2200 to 2600) being most efficient
3. Figure 5 demonstrates how different teachers influence human similarity at different skill levels
4. Multiple experimental results confirm the teacher's significant impact on both metrics

The reviewer's comments don't dispute this specific claim, and their criticisms focus more on novelty rather than the validity of these specific findings.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We find that the choice of teacher has a strong impact on both playing strength and human similarity; for example, a teacher that is too strong can be less effective at improving playing strength and degrade human similarity more rapidly.

The review comment to be evaluated is:

Strengths

- **************Clarity**************: The paper is well written and easy to understand, and the sections naturally follow from one to the other.
- **********Novelty:********** The paper is relatively novel in its approach towards understanding curriculum learning and building human-like models. While the zone of proximal development (ZPD: problems with medium difficulty are the most useful) have been studied by measuring problem difficulty, the paper looks at teacher/annotator proficiency.
- The analysis in the paper is thorough, and the graphs and methods are easy to understand.
- The insight of accounting for labeller/ teacher proficiency is quite powerful.

Weakness

- The interpretation of the result (most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere in the middle) in Experiment 2 is not well supported. The authors themselves suggest the alternate hypothesis that seems more likely: Maia- 1200 cannot account for the stylistic differences between a human-like model and an alpha-beta model like Stockfish. The alternate hypothesis seems likelier as the most similar stylistic teachers with the highest ratings are the best teachers. Without an experiment to test this, one of the central claims of the paper is not on firm grounds.
- I am unsure how the results in this paper might extend to a domain outside of chess: either for teaching people or for building stronger human-like models. The discussion section is lacking here, with just one application domain in the paper! For example, the claim that “that the teacher is much more important than the curriculum.” might not extend to other domains.
- Experiments 2,3 and Figures 2 and 3 show human similarity based on the games of 1200 rated players. When improving the strength of a model, one might want to see a resemblance closer to the target rating rather than the base rating. So, if I wish to have a certain similarity with a 1600-rated human, would it be better to train using a 1600 agent or a 1400 agent or an 1800 agent? Using matching accuracy from 1200-rated games does not seem right (this is also seen in experiment 4).

Minor

- The caption or the graph in figure 5 does not mention the metric shown.

The evidence to be evaluated is:In general, optimizing AI towards human similarity will necessarily lead to reduced task performance, especially in scenarios where (non-human-like) AI achieves superhuman performance. In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

Finally, we introduce our curriculum training approach, which extends the concept of curriculum learning to multiple teachers, enabling us to examine the impact of varying both the curriculum (dataset) and the teaching advice (label strategy).

We then group the moves by Elo rating, and use supervised learning to train a neural network to predict what move a human would play in a given chess position. For example, we train a model on moves made by players rated between 1150 to 1249, and we denote this model as Maia-1200.

In particular, it is not the case that higher-rated models are better at predicting lower-rated datasets.

To evaluate the strength of the updated model, we have the updated model play against the original model in a large number of full-length chess games (until checkmate or a draw is reached), and measure the match win percentage of the updated model. This win percentage is then used as a proxy for measuring the model strength. To provide more insight into this metric, we have the models from Maia-1200 to Maia-2600 play against each other. The match win percentages (based on 5120 games played between each pair of models) are shown in Figure 0(b). We can see that as we increase the distance between two models, the win rate of the higher-rated model increases. We also see that as ratings go up, the win rates tend to be closer to 50%.

Jacob et al. (2022) showed that higher-rated Maia models trained using standard imitation learning were suboptimal, and claimed that this would lead to weaker play than expected.

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model.

Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

To evaluate the performance of curriculum training, we vary the amount of data used to update the model \(M\) and measure the strength (match win percentage) and human similarity (move-matching accuracy) of the updated model, as per SS3.2.

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student.

For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum. Overall, these results suggest that the strength of a teacher matters significantly when training models.

As an approximate solution, we use Maia models that we have trained to serve as the teacher \(Y\). Thus, we teach Maia-1200 using a curriculum where \(X\) is the set of positions encountered by 1200-rated humans, and \(Y\) are the moves that Maia-1400 would make in these positions. We also introduce an additional possible teacher, Stockfish, the strongest chess engine in the world. This allows us to approximate much higher rated humans, since no models higher than Maia-2600 are available.

We also see the same non-trivial result that the most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere in the middle. In particular, the most efficient teacher is Maia-2200, Maia-2400, or Maia-2600, and these three points tend to be clustered closely together.

This could purely be because Stockfish's strength outclasses all humans, so much of the move advice Stockfish teaches is out of reach for the Maia-1200 student.

Overall, these results confirm our observation in Experiment 1 that the teacher has a significant effect in fine-tuning a human-like model.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

One hypothesis is that the rooks curriculum (human expert recommendation) is too specific in the positions it tries to teach.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis. If our objective is to increase the model strength while ensuring the model maintains human similarity above a selected threshold, the blunder curriculum outperforms random while the 1400-inflection and rooks curriculum do not.

Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

This provides more credence to our earlier observation that the teacher is much more important than the curriculum.

At a high level, we find that the choice of teacher is critical for maximizing the efficiency of strength gains. In many cases, the best teacher is _not_ the strongest teacher. Notably, Stockfish (the world's strongest chess engine) is a worse teacher than even a low-rated human-like model.

Although we did find evidence that efficient curriculum design is possible, it remains an open question how best to select a curriculum for training human-like models in chess.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

This work develops a methodology for improving the strength of a given human-like model, while maintaining its human-like property.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 3: The results of varying the teacher Y to Stockfish (a superhuman chess engine) and Maia models of different ratings, while fixing the input distribution X to the student’s positions.

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "the choice of teacher has a strong impact on both playing strength and human similarity" appears to be well-supported by multiple experiments and data visualizations. Figures 2 and 3 clearly demonstrate how different teacher ratings affect both metrics. The second part of the claim about "a teacher that is too strong can be less effective" is also substantiated by the evidence showing Stockfish (strongest teacher) performing worse than lower-rated models, and mid-rated teachers (2200-2600) being most efficient. The claim accurately reflects the findings without exaggeration.

<score>1</score>

Key supporting evidence:
1. Figures 2 and 3 show clear relationships between teacher rating and model performance
2. Results showing Stockfish being a worse teacher than human-like models
3. Data showing optimal teachers are often mid-rated (2200-2600) rather than highest-rated
4. Strong correlation between teacher rating and resulting model similarity (Figure 5)
5. Reviewer comments acknowledge the thorough analysis while noting reasonable limitations

The claim stays firmly grounded in the evidence without overstating conclusions.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also find that the choice of curriculum can impact these metrics, but to a smaller extent; for example, training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum.

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength.

We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student. To investigate the efficiency of this design, we gradually increase the amount of data used to update the student model. For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum.

Instead, we use specific domain knowledge in chess to develop our own sets of curricula based on fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze. * Mistakes identified by a _superhuman chess engine_. We choose the position of blunders, which is defined as a move which lowers expected win probability by over 15% as predicted by Stockfish.

* Mistakes identified by a _human expert_. We have collaborated with an International Master (IM) on this work.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis.

However, the difference in performance is small across all curricula. Overall, our results demonstrate that, among a general set of curriculum we have examined, altering curriculum only brings marginal benefits compared with using random curriculum.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

Our results also suggest that the choice of curriculum is less important than the choice of teacher. Although we did find evidence that efficient curriculum design is possible, it remains an open question how best to select a curriculum for training human-like models in chess.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 4: The results of fixing Y to Maia-2400 while varying X to positions identified by a superhuman chess engine (blunder), a human expert (rooks), and a human-like AI (1400-inflection).

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "curriculum choice impacts metrics to a smaller extent than other factors, with human mistake curriculum providing only marginal benefit over random curriculum" appears well-stated. Figure 4 directly shows the similar performance curves between different curricula (blunder, random, rocks), with only small differences between them. The text explicitly states "the difference in performance is small across all curricula" and that "altering curriculum only brings marginal benefits compared with using random curriculum." The evidence strongly supports this modest characterization of curriculum effects.

<score>1</score>

Justification:
1. Figure 4 visually demonstrates the small differences between curriculum types
2. The paper directly acknowledges that "curriculum choice is less important than the choice of teacher"
3. The authors' conclusions about marginal benefits are supported by their experimental results
4. The claim uses appropriately measured language ("smaller extent," "marginal benefit") that accurately reflects the scale of effects shown in the data
5. No exaggeration of significance or overstatement of findings is present in the claim


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also find that the choice of curriculum can impact these metrics, but to a smaller extent; for example, training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum.

The review comment to be evaluated is:

The authors offer a quite extensive evaluation of the proposed approach as well as the basic models used. This evaluation includes a consideration of differences between the used positions and the source (teacher) of the proposed move. Therefore, the evaluation is beyond what is usually expected. However, the used definition of the "human similarity" metric is problematic. For one, expecting an exact matching of ground truth moves is very strict and a similarity metric should be preferred. Although, this is likely not a trivial task, because similarity to human play can potentially be only evaluated by human experts. However, the phrase "behavioral data used to train the original model" suggested, that in most experiments, only moves from the Maia-1200 set are considered. (Also see clarity) In case this understanding is correct, the metric only evaluates deviation from novice play - not from human play. Therefore, the correlation between increase in playing strength and decrease of similarity becomes obvious. A potential resolution could be comparing to any human play (independent of the playing strength or up to the teachers playing strength). This should be an approximate if any human would play like the engine.
Another problem is, that it seems, that learning approach has not converged, after using all batches once (Fig 2,3). Without convergence, the used learning rate becomes an interesting question, because it implicitly defines the trade-off between initial weights and new data.
Furthermore, Section 3.3. is a bit lacking - considering it is the introduction of the main approach. (See clarity)

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength.

We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student. To investigate the efficiency of this design, we gradually increase the amount of data used to update the student model. For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum.

Instead, we use specific domain knowledge in chess to develop our own sets of curricula based on fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze. * Mistakes identified by a _superhuman chess engine_. We choose the position of blunders, which is defined as a move which lowers expected win probability by over 15% as predicted by Stockfish.

* Mistakes identified by a _human expert_. We have collaborated with an International Master (IM) on this work.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis.

However, the difference in performance is small across all curricula. Overall, our results demonstrate that, among a general set of curriculum we have examined, altering curriculum only brings marginal benefits compared with using random curriculum.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

Our results also suggest that the choice of curriculum is less important than the choice of teacher. Although we did find evidence that efficient curriculum design is possible, it remains an open question how best to select a curriculum for training human-like models in chess.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 4: The results of fixing Y to Maia-2400 while varying X to positions identified by a superhuman chess engine (blunder), a human expert (rooks), and a human-like AI (1400-inflection).

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the statement about curriculum choice having only marginal benefits compared to random curriculum is generally well-supported by the evidence, but slightly understated. Figure 4 shows small but noticeable differences between curricula, with the blunder curriculum performing slightly better than random across most training durations. The text explicitly states "the difference in performance is small across all curricula" and "altering curriculum only brings marginal benefits compared with using random curriculum." While the claim is accurate in describing the relative impact as smaller than other factors, characterizing it as "only marginal" slightly understates the consistent, though modest, improvements shown in the data.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also find that the choice of curriculum can impact these metrics, but to a smaller extent; for example, training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum.

The review comment to be evaluated is:

## Strengths
The writing of the paper is exemplary, making it relatively easy to understand. I particularly liked the sec. 1 motivation behind the need for human-like models, even though they may have a worse performance than the models not taking this into account.

## Weaknesses
I find the evaluation of the human likeness of the models to be performed on the original datasets (of the original training, not the finetuning) unacceptable. Doing this makes the metric to count both "unhuman" and "human, but of higher-rating" as unhuman behavior, which doesn't follow the motivation of improving the models while keeping them human. While the sec. 4.4 shows partial results on the target rating, I find presentation of the original rating results as confusing, and the sec. 4.4 to not be detailed enough to override this assessment.

If the proper train/test split was not used, the results may suffer from overfitting (ie. the reported human likeness would not constitute a good approximation of a real move-matching accuracy on new positions).

The presented results provide no new insights for me, and are predictable:

1. As human likeness measures accuracy of reproducing the original data, it drops when finetuning on different data [4]. The higher the rating gap (=> bigger domain shift), the bigger the catastrophic forgetting.
2. As we train on the data with humans of a better rating, the performance (assuming enough data) improves, as the model gets to reproduce the better moves.
3. The initial speed of improvement is lower for datasets that are more different (ie. higher rating) from the original, due to less common features being reusable in the new domain. This implies that, for a fixed sample budget, to get the best final performance there will be a tradeoff between low rating (=> the domain is close and there are a lot of reusable features), and high rating (=> overall better moves to reproduce). Authors also provide a correct intuition in the paper behind this: "After a model undergoes curriculum training, [it] will naturally degrade in its initial performance, because we are teaching it to play differently".

I don't find the results in Experiment 3 very informative, as it's hard to understand the difference between the handcrafted data distributions (is the score lower because the data is of less variance, or coming from different-rating source, or the chosen positions don't cover the space of positions well enough to improve the model, etc.).

The benefits of introducing the curriculum training framework are not clear to me: the concrete method for updating the network (finetuning) is prior work, as is generating the actions from previous trajectories or other models. The paper makes no claims about any of the instantiation of the framework (using trajectories or other models' actions, or changing the states only) as being empirically better then the others, nor does it compare to a natural baseline of training on the new dataset from scratch.

## Actionable feedback

1. Present the results in the experimental section on the target domain (ie. it's human if it's close to what the human of a given strength would play)
2. Clarify whether a train/test split was used in the paper and if not, provide the results where the evaluation data is separate from the training one.
3. Provide a clearer value for an ICLR reader: what it is actually supposed to learn from the paper? The reader would be aware of the techniques for improving the models from the literature, is one of the methods better than the others, in what situations? Which of the claims are only applicable in the domain of human-like models or domain of chess? Is there anything else to take home?

[4]: https://en.wikipedia.org/wiki/Catastrophic_interference

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength.

We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student. To investigate the efficiency of this design, we gradually increase the amount of data used to update the student model. For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum.

Instead, we use specific domain knowledge in chess to develop our own sets of curricula based on fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze. * Mistakes identified by a _superhuman chess engine_. We choose the position of blunders, which is defined as a move which lowers expected win probability by over 15% as predicted by Stockfish.

* Mistakes identified by a _human expert_. We have collaborated with an International Master (IM) on this work.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis.

However, the difference in performance is small across all curricula. Overall, our results demonstrate that, among a general set of curriculum we have examined, altering curriculum only brings marginal benefits compared with using random curriculum.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

Our results also suggest that the choice of curriculum is less important than the choice of teacher. Although we did find evidence that efficient curriculum design is possible, it remains an open question how best to select a curriculum for training human-like models in chess.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 4: The results of fixing Y to Maia-2400 while varying X to positions identified by a superhuman chess engine (blunder), a human expert (rooks), and a human-like AI (1400-inflection).

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll evaluate the claim "training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum."

Based on the evidence, particularly Figure 4 and the text discussing Experiment 3, this claim appears well-stated (score of 1). The evidence shows very similar performance curves between different curricula (blunder, random, rocks, 1400i), with only small differences in outcomes. The paper explicitly states "the difference in performance is small across all curricula" and "altering curriculum only brings marginal benefits compared with using random curriculum." The review comment doesn't contradict this specific claim about curriculum benefits.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also find that the choice of curriculum can impact these metrics, but to a smaller extent; for example, training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum.

The review comment to be evaluated is:

Strength:
This paper studies curriculum learning and human-like models in the widely-studied domain of Chess. They provide detailed experiments to find the relation between strength and similarity.

Weaknesses:
However, there are no new techniques in the methodology, and the experiment results are not surprising. To be more specific, they used transferring learning to fine-tune a 1200m model with datasets of different ratings. It is very obvious that the playing strengths will increase and the human similarity will decrease when comparing the fine-tuned model with the original one. Besides, in subsection 4-4, the author said “the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset”. This conclusion does not give any new findings. Overall, there are no new findings and the novelty is poor.

Other comments:
* Fig. 1 and Fig. 5 only provide the decimal numbers, while the readers might be interested in the actual difference in smaller grains.
* In experiment 4.1, the author said “at least 49% … the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches … with a 55% win probability”, and “a 55% win probability by training on a 1800-rated curriculum for 1000 batches but at the cost of a much lower move-matching accuracy.” But according to Fig. 2, the move-matching accuracy is about 48.6% for 1800-rated curriculum of 1000 batches, which is not “much lower” as stated in the context. The results seem to have not much difference when comparing different settings.
* The author should explain how a move is generated by the trained model, is it by MCTS or just by the move with the maximum policy?
* The author claims to propose a framework for curriculum learning, but according to the contents of the paper, this seems exaggerated.
* In subsection 4.3, the author should provide the total number of positions for the three categories of mistakes. Especially for human experts, the 80 simulated games seem to be much smaller than the other two categories which can be generated by programs.
* In Fig. 1 and Fig. 5, what does the color mean?
* On page 5, “50% .” -> “50%.”
* In Data Pipeline section of the appendix, “from 2021-06 to 2021-03”, is it typo for “2021-06 to 2022-03”?

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength.

We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student. To investigate the efficiency of this design, we gradually increase the amount of data used to update the student model. For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum.

Instead, we use specific domain knowledge in chess to develop our own sets of curricula based on fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze. * Mistakes identified by a _superhuman chess engine_. We choose the position of blunders, which is defined as a move which lowers expected win probability by over 15% as predicted by Stockfish.

* Mistakes identified by a _human expert_. We have collaborated with an International Master (IM) on this work.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis.

However, the difference in performance is small across all curricula. Overall, our results demonstrate that, among a general set of curriculum we have examined, altering curriculum only brings marginal benefits compared with using random curriculum.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

Our results also suggest that the choice of curriculum is less important than the choice of teacher. Although we did find evidence that efficient curriculum design is possible, it remains an open question how best to select a curriculum for training human-like models in chess.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 4: The results of fixing Y to Maia-2400 while varying X to positions identified by a superhuman chess engine (blunder), a human expert (rooks), and a human-like AI (1400-inflection).

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "training on a curriculum of human mistakes provides only a marginal benefit over training on random curriculum" appears to be well-stated. Figure 4 directly shows the comparison between different curricula (blunder, rocks, random, 1400i), and the performance differences are indeed small. The paper explicitly states "the difference in performance is small across all curricula" and concludes that "altering curriculum only brings marginal benefits compared with using random curriculum." The reviewer's comments also support this, noting the results have "not much difference when comparing different settings." The evidence consistently supports this conservative characterization of the curriculum's impact.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We also find that the choice of curriculum can impact these metrics, but to a smaller extent; for example, training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum.

The review comment to be evaluated is:

Strengths

- **************Clarity**************: The paper is well written and easy to understand, and the sections naturally follow from one to the other.
- **********Novelty:********** The paper is relatively novel in its approach towards understanding curriculum learning and building human-like models. While the zone of proximal development (ZPD: problems with medium difficulty are the most useful) have been studied by measuring problem difficulty, the paper looks at teacher/annotator proficiency.
- The analysis in the paper is thorough, and the graphs and methods are easy to understand.
- The insight of accounting for labeller/ teacher proficiency is quite powerful.

Weakness

- The interpretation of the result (most data-efficient curriculum is often not the highest or lowest-rated teacher, but somewhere in the middle) in Experiment 2 is not well supported. The authors themselves suggest the alternate hypothesis that seems more likely: Maia- 1200 cannot account for the stylistic differences between a human-like model and an alpha-beta model like Stockfish. The alternate hypothesis seems likelier as the most similar stylistic teachers with the highest ratings are the best teachers. Without an experiment to test this, one of the central claims of the paper is not on firm grounds.
- I am unsure how the results in this paper might extend to a domain outside of chess: either for teaching people or for building stronger human-like models. The discussion section is lacking here, with just one application domain in the paper! For example, the claim that “that the teacher is much more important than the curriculum.” might not extend to other domains.
- Experiments 2,3 and Figures 2 and 3 show human similarity based on the games of 1200 rated players. When improving the strength of a model, one might want to see a resemblance closer to the target rating rather than the base rating. So, if I wish to have a certain similarity with a 1600-rated human, would it be better to train using a 1600 agent or a 1400 agent or an 1800 agent? Using matching accuracy from 1200-rated games does not seem right (this is also seen in experiment 4).

Minor

- The caption or the graph in figure 5 does not mention the metric shown.

The evidence to be evaluated is:In this paper, we choose to handle this trade-off by extending the Curriculum Learning concept of Bengio et al. (2009) to support varying both the curriculum and the labeling strategy (teacher).

While we intend curriculum training to be a domain-agnostic framework for efficiently developing strong and human-like AI systems, our choice of chess allows us to draw from existing chess pedagogical literature to inform our choices of specific curricula to investigate.

The goal of this work is to investigate data-efficient approaches for improving the strength of a given human-like model, while maintaining its human-like property.

We formally introduce the notion of _curriculum training_ as the data-efficient fine-tuning of an existing human-like model, with the dual objectives of improving the model's strength and preserving the human similarity of the original model. In curriculum training, we start with a human-like model \(M\), which we call the _student_. Our goal is to choose a pair of _curriculum_\(X\) and _teacher_\(Y\) to update the given model. A curriculum is specified by the input distribution \(X\). For example, \(X\) could be a distribution of chess positions that players of a certain rating have encountered in their games. A teacher is specified by the labeling strategy \(Y\), which specifies the labels for each input \(x\) drawn from \(X\).

We vary the choice of teachers \(Y\) to be different ratings of human players or human-like models, or a superhuman chess engine, and examine their impact. We also examine the impact of choosing curriculum \(X\) based on the potential mistakes that the student might make, using the assistance of a superhuman chess engine, stronger-rated Maia models, or a human expert.

We evaluate the effectiveness of curriculum training on our three performance measures: data-efficiency, human similarity, and model strength.

We first examine the most natural design of the curriculum and teacher pair: updating the initial human-like model Maia-1200 using data from higher-rated human players. Specifically, let \(X_{R}\) be the distribution of positions encountered by \(R\)-rated players, and \(Y_{R}\) be the moves made by \(R\)-rated players in these positions.

In this experiment, we set our student \(M\) as Maia-1200, and we vary the rating \(R\) from \(1400\) to \(3000\) and use \((X_{R},Y_{R})\) as the (curriculum, teacher) to update the student. To investigate the efficiency of this design, we gradually increase the amount of data used to update the student model. For each resulting model (updated using a different amount of data and different \((X_{R},Y_{R})\)), we measure its human similarity (i.e., move matching accuracy) and strength (i.e., match win percentage).

The results, as displayed in Figure 2, demonstrate the trade-offs of our three performance metrics for different pairs of curriculum and teacher \((X_{R},Y_{R})\). First of all, if our goal is purely to increase strength without concerns for human similarity or data efficiency, training using the highest rated curriculum leads to the strongest model. If our objective is to increase the model's strength while maintaining the model's human similarity above a selected threshold (without concern for data efficiency), using the closest-rated data for fine-tuning leads to the optimal outcome. For instance, if we want to maintain a move-matching accuracy of at least 49%, the optimal strategy is to train on 1400-rated data curriculum for 10,000 batches, which gives us a model with a 55% win probability. We could more efficiently achieve a 55% win probability by training on a 1800-rated curriculum for 1000 batches, but at the cost of a much lower move-matching accuracy.

If our objective is to maximize strength gains with a constraint on data usage, we see an interesting and non-trivial result: at many points during fine-tuning, the optimal curriculum is not the highest-rated or lowest-rated one, but at a point in between. For example, at batch-30, the 2000-rated curriculum is optimal, while at batch-100 and batch-300, the 2400-rated curriculum is optimal. Past this point, it appears that the 2600-rated curriculum is optimal at all batch sizes. However, it would not be surprising if the trend continues: if we had access to the 3000-rated curriculum at higher batch sizes, the high-rated model might perform worse than a 2600-rated or 2800-rated curriculum.

Instead, we use specific domain knowledge in chess to develop our own sets of curricula based on fixing mistakes made by the student. We list three categories of mistakes, and choose one representative curriculum for each category to analyze. * Mistakes identified by a _superhuman chess engine_. We choose the position of blunders, which is defined as a move which lowers expected win probability by over 15% as predicted by Stockfish.

* Mistakes identified by a _human expert_. We have collaborated with an International Master (IM) on this work.

As this is also a common problem for human chess players, we selected this as our representative curriculum.

Using these curricula and the Maia-2400 teacher, we ran another experiment with the same training procedure and evaluation methods as Experiments 1 and 2. For context, we also present a baseline curriculum which is just a random set of positions encountered at the student's rating level.

On the other hand, the blunder and 1400-inflection curricula (positions identified by superhuman and human-like AIs) look at mistakes that the model makes across the entire distribution of general board states, but the emphasis on mistakes leads to better outcomes than no emphasis.

However, the difference in performance is small across all curricula. Overall, our results demonstrate that, among a general set of curriculum we have examined, altering curriculum only brings marginal benefits compared with using random curriculum.

Overall, we find that the updated models are more similar to higher-rated datasets than the original dataset. Looking at the models from Experiment 2, where we varied the model's teacher, there is a very strong correlation between the rating of the teacher and the most similar dataset of the updated model. For instance, the model taught by Maia-1600 is most similar to the 1600-rated dataset, while the model taught by Maia-2400 is most similar to the 2400-rated dataset.

Our results also suggest that the choice of curriculum is less important than the choice of teacher. Although we did find evidence that efficient curriculum design is possible, it remains an open question how best to select a curriculum for training human-like models in chess.

Nevertheless, the results in SS4.4 suggest that our curriculum training methodology is able to improve human-like models in a way that ensures human similarity at their improved level of play, on a path similar to human learning.

Input: Figure 2: The results of using higher-rated player data to fine-tune a lower-rated student model. Each curve represents the change in human similarity and strength when fine-tuning the Maia-1200 model on increasing amounts of data from players of a particular rating. For each curve, we label the points that are trained using different numbers of batches, from 10 batches to 100,000 batches.

Input: Figure 4: The results of fixing Y to Maia-2400 while varying X to positions identified by a superhuman chess engine (blunder), a human expert (rooks), and a human-like AI (1400-inflection).

Input: Figure 5: The results of analyzing how similar curriculum-updated models are to human datasets at a range of skill levels.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "training on a curriculum of human mistakes provides only a marginal benefit over training on a random curriculum" appears well-supported. Figure 4 directly compares different curricula (blunder, random, rooks) against each other while using Maia-2400 as the teacher, and shows very similar performance curves with only slight variations. The text explicitly states "the difference in performance is small across all curricula" and "altering curriculum only brings marginal benefits compared with using random curriculum." The evidence demonstrates that while there are small differences between curricula, they are indeed marginal compared to the impact of choosing different teachers.

<score>1</score>