Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To reduce variance from estimating information-gain intrinsic rewards in complex domains, we simplify and derive an intrinsic reward based on prediction errors to achieve superior stability and scalability.

The evidence to be evaluated is:The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning. We will discuss in detail the design of intrinsic reward \(r^{\prime}_{intr}\) in the next section.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form:

The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Further derivation (see Appendix A.1 for details) shows that this intrinsic reward can be estimated by the difference of two prediction errors: \[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\]

Empirically, we find that this intrinsic reward suffers form high variance, sometimes causing instability in training. Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance:

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

In Figure 2, MetaCURE-PE outperforms MetaCURE-IG, as its intrinsic reward is of lower variance. However, the prediction errors may lead to bad exploration behaviors under certain scenarios.

While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

**Prediction loss as intrinsic reward** Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore. Oh et al. (2015) directly predicts the image observation, and Stadie et al. (2015) utilize prediction loss in the latent space, extracting useful features from observations. To avoid trivial solutions in learning the latent space, Pathak et al. (2017) introduces an inverse model to guide the learning of latent space, only predicting things that the agent can control.

By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about simplifying information-gain intrinsic rewards to achieve superior stability and scalability appears to be supported but slightly overstated. The evidence shows that MetaCURE-PE (simplified prediction error reward) generally performs better than MetaCURE-IG (information gain reward) in most scenarios, demonstrating improved stability. However, there are important caveats - MetaCURE-IG performs better in noisy environments (Fig 5) and the prediction error approach can lead to suboptimal exploration in certain cases. The performance differences between the two approaches are often modest rather than dramatically superior.

<score>3</score>

Justification:
1. The evidence supports that simplifying to prediction errors reduces variance:
- "Empirically, we find that this intrinsic reward suffers from high variance, sometimes causing instability in training"
- "MetaCURE-PE outperforms MetaCURE-IG, as its intrinsic reward is of lower variance"

2. However, the claim of "superior" stability/scalability is overstated:
- Performance differences between PE and IG variants are often small (Fig 2, Fig 6)
- PE approach has documented limitations: "prediction errors may lead to bad exploration behaviors under certain scenarios"
- IG performs better in noisy environments: "MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics"

3. The key limitation acknowledgments in the paper counter the strong wording of "superior":
- Each approach has pros and cons for different scenarios
- Neither variant consistently dominates the other across all conditions

The evidence supports that the simplification improves stability but doesn't fully justify characterizing it as "superior" without important qualifying statements about the tradeoffs involved.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To reduce variance from estimating information-gain intrinsic rewards in complex domains, we simplify and derive an intrinsic reward based on prediction errors to achieve superior stability and scalability.

The review comment to be evaluated is:

This paper introduces the use of intrinsically motivated exploration in meta-training and adaptation phase in meta Reinforcement Learning. The proposed framework consists of using task-inference, already used in other works like PEARL and VariBAD, and two deep neural networks, trained with an arbitrary off-policy algorithm, respectively for exploitation and exploration phases.

I think the paper is well-written and discusses the related literature sufficiently well. The empirical results are significantly better than the considered baselines.
Despite these good qualities, I think the major weakness of this paper is the lack of contribution. It looks to me that the major contribution discussed by the authors is the use of intrinsically motivated exploration in meta-training and adaptation, while the use of task-inference has been already proposed in other works and the use of different policy networks for exploitation and exploration is trivial. Considering the intrinsically motivated exploration, I miss the importance of the contribution, since it is already well-known that this technique can be beneficial, especially in sparse MDPs. The intrinsic reward makes sense, but it is not novel, as similar intrinsic rewards have been proposed in VIME and Pathak et al. (2017).
Experimental results are good, but not well described. For example, I don't find a clear explanation of the tasks used in meta-training. The random seeds are only three, which makes the results less reliable.
Considering this, I don't consider this paper significant enough to be accepted.

Minor issues:
* Pag. 1, typo: "tasks" -> tasks
* Pag. 1, "Based this objective" -> Based on this objective
* the R function in (1) is not formally defined.

Pros
------
* Well-written;
* Good discussion of related literature;
* Good experimental results.

Cons
-------
* Lack of contribution;
* Few, but important, missing details about the experimental setting, and too few seeds.

Post-rebuttal feedback
-------------------------------
I thank the author for their reply and I encourage them to make the suggested improvements to the paper.

The evidence to be evaluated is:The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning. We will discuss in detail the design of intrinsic reward \(r^{\prime}_{intr}\) in the next section.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form:

The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Further derivation (see Appendix A.1 for details) shows that this intrinsic reward can be estimated by the difference of two prediction errors: \[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\]

Empirically, we find that this intrinsic reward suffers form high variance, sometimes causing instability in training. Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance:

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

In Figure 2, MetaCURE-PE outperforms MetaCURE-IG, as its intrinsic reward is of lower variance. However, the prediction errors may lead to bad exploration behaviors under certain scenarios.

While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

**Prediction loss as intrinsic reward** Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore. Oh et al. (2015) directly predicts the image observation, and Stadie et al. (2015) utilize prediction loss in the latent space, extracting useful features from observations. To avoid trivial solutions in learning the latent space, Pathak et al. (2017) introduces an inverse model to guide the learning of latent space, only predicting things that the agent can control.

By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim: "To reduce variance from estimating information-gain intrinsic rewards in complex domains, we simplify and derive an intrinsic reward based on prediction errors to achieve superior stability and scalability."

This claim appears to be well-stated (1) for several reasons:
- The evidence clearly demonstrates that the information-gain (IG) intrinsic reward suffered from high variance, as stated in the text
- Experimental results confirm that MetaCURE-PE (prediction error) slightly outperforms MetaCURE-IG in most tasks (Figure 2)
- The paper acknowledges limitations, noting that while PE is more stable, it may lead to suboptimal exploration in certain scenarios (shown in Figure 5)
- The derivation and simplification process is thoroughly documented with mathematical details

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To reduce variance from estimating information-gain intrinsic rewards in complex domains, we simplify and derive an intrinsic reward based on prediction errors to achieve superior stability and scalability.

The review comment to be evaluated is:

This paper proposes a novel meta-learning method, aimed at solving the following problem: at test time, the agent has N episodes to gather information [exploration phase], and we care about its return in the N+1th episode [exploitation phase]. To this end, the authors propose to learn a separate exploration and exploitation policy. The core of the algorithm is to use an exploration bonus for the exploration policy that rewards finding novel trajectories. This should help it to collect valuable information during meta training and meta testing. The exploration policy is only used in the N+1th episode, and  is conditioned on a context vector which is computed from the exploration trajectories.

Pro:
- Very relevant subject.
- The authors bring up an interesting point, that metaRL methods should consider both exploration during meta training and meta testing.
- I think it makes sense to separate exploration and exploitation policy explicitly.
- The authors propose to sparsify common metaRL benchmarks. Evaluating your algorithm on environments that are proposed in the same paper can be tricky (and often means that the empirical results are evaluated more critically), but I think the suggestion of looking at sparse metaRL tasks is useful for the field and should be considered by other researchers as well.
- The authors provide illustrations of the behaviours of the learned policy at meta-test time, which I think is really nice!

Concerns:
- I fee like there's a mismatch between the reward that the explorer gets, and the problem setting that the authors consider. The way I understand it, the two terms in the explorer's reward (Eq (2)) can be helpful during meta-training but I don't see how that translates to meta-test time. Let me try to explain:
 - [First term] The objective of the agent is to maximise the return in the *last* episode (the exploitation episode). Then why is the explorer not maximising the return of the exploitation phase, but instead gets a reward that corresponds to the exploration episodes? Doesn't that contradict the premise that we're not interested at all in the reward that is collected during exploration? Also, what is n (you write "training iterations" - is that just the number of environment steps per task? number of policy updates over the entirety of meta-training? Does this mean this is annealed to zero over training? Maybe then it shouldn't be called $\gamma$ so that it's not confused with the per-episode discount usually used in RL.).
 - [Second term] The reward bonus can be seen as a type of novelty bonus for finding new trajectories. Can the authors clarify why this would help (a) during meta-training and (b) during meta-testing? I can see how it helps during training, but I don't understand why it also helps at test time. At the end of Sec 3.2, you write "the intrinsic reward is estimated on multiple tasks and does not diminish during meta-training". Why not? In Eq 5, does $c_{-1:i-1}$ come from multiple tasks or just one? Why can't this diminish over time, shouldn't our model $p(r,s|x,a,z)$ (dropping the subscripts for brevity) get better over time? Or do you just mean that the weight $\lambda$ is not annealed? And doesn't it have all the information to, in principle, predict r and s exactly (except for aleatoric noise) so then this term reduces to the noise in the environment? Or should the terms in the subscript not match the ones within the expectation? It's also a bit unclear what the expectation is over, the subscript just shows some tuples. What's the probability distribution? Where do the terms in the subscript come from? Are $q$ and $p^\mathcal{T}$ mixed in here? It would help my understanding if these things could be made more explicit.
 - [Missing term?] I understand the above two terms might help during training, but they are only auxiliary tasks in a way. And if the first term is annealed to zero (if that's indeed what's happening?), and the second term reduces to aleatoric noise (is that's indeed what's happening?), what is the exploration policy even doing at the end of training? Should there be a third term, which is the return in the *exploitation* phase, which is the only thing that the exploration policy should try to maximise at meta-test time?
- I think the mathematical formulation has to be fleshed out. The paper is sometimes hard to follow because the notation is unclear, and not all terms are explained. I have a few open questions because of this:
 - What are the expectations in (3), (4), (5) over?
 - Not all terms in Figure 1 are defined or mentioned in the text. Are $L_\pi$ and $L_{Q_\pi}$ loss terms from SAC, so they correspond to equation (1)? And $L_{\pi_e}$ corresponds to equation (2)? And what is $L_{pred}$? All these should be explicitly defined in the text with an equation.
 - Sec 2, directly after Eq (1): $c^\mathcal{T}_{\pi_e}$ is not defined.
- In Figure 2, PEARL (Walker-Vel-Sparse) and VariBAD (Point-Robot-Sparse) seem to first learn and then diverge. Do the authors have an explanation as to why this is? It seems to me that maybe the hyperparameters just weren't set right, and makes me a bit sceptical of the results. When PEARL/VariBAD *do* learn, their (end-) performance is much closer to that of MetaCURE.
- The authors propose two versions of MetaCURE (IG and PE). On the continuous control tasks PE is better, but then on the noisy point robot PE completely fails and IG succeeds. I think the authors are not doing themselves a favour by setting up the presentation of their methods like this. It would be better to propose *one* method, and then fully focus on this. The way it is now, there are two methods where neither of them is clearly better. I think that's confusing to the reader, and less convincing. Shouldn't there be a way to develop an exploration bonus that can do well across all environments?
- The "noisy" environment is a POMDP (the agent gets noisy state observations). So doesn't that mean that there is simply a mismatch between the problem setting and the way the policy is set up? I'm not sure that this is a good experiment to add, because the solution (using the PE explorer) does not directly address the problem (partial observability). Maybe a noisy reward function would fit better here?

Other questions:
- Can you explain why the same exploration mechanism useful during meta-training and meta-testing? Aren't we mixing across-task and within-task information gain, and if so, why is that okay?
- What is the difference between row 1 and row 2 in Figure 2? This is not explained in the text. Is that separation just because everything in one figure would've been too much? And how exactly were the baselines evaluated, always after the same number of episodes?
- Why is the loss of the explorer not backpropagated through the context encoder? How does that fit together with "our intrinsic reward supports end-to-end learning and collect online data for effective task inference"?
- Why does the exploiter get a sample $z$ and not the entire distribution $q$? Is that because we assume that all information is there and $q$ is concentrated anyway?
- You mention that the intrinsic reward of PE is of lower variance. Could you explain why? Is that a hypothesis (if so, say so explicit) or do you have any empirical/theoretical evidence for this (again, if so, make it explicit).

Suggestions:
- You write that "efficient exploration of meta-RL needs to consider both training and adaptation phases simultaneously". This is a good point and I agree with it, and I think the paper can be improved by fleshing this argument out more. Help the reader understand why this is the case, and what the (different) challenges in exploration are during training and testing. One open question I still have for example (see above) is why the *same* exploration mechanism is useful during meta-training and meta-testing. Why not separate them? This should be explained in the paper. In the related work section, you write that prior methods "utilise policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards". This point can be made about almost all prior methods, and should come in the introduction. Then explain intuitively *why* this fails, maybe which an example. Another question I have is, how does the Ishii et al. 2002 paper back up this point that this distinction exists?
- The experiments section relies too much on the appendix, and it's quite unnatural to read it like this. Either I have to jump back and forth between the appendix and the main text, or I have to take your word for what is shown in the results in the appendix. You introduce five continuous control tasks at the start, but then actually only show 3 and the other 2 are in the appendix. I think this section can be improved by focusing on the main important points in the main text, explain this well and mention that more results can be found in the appendix.
- Mention in the main text that the sparse setting for Meta-World means you use the success signal only. This is a very difficult setting and is worth highlighting instead of burying it in the appendix.
- I think a discussion about what an good/optimal exploration strategy would look like can help the reader understand the problem setting, and what you are trying to achieve. Right now, in Sec 4.3 you write for example that PEARL "explored less effectively" because it keeps the belief about the current task throughout an entire episode. That's not a fundamental problem with Thompson-sampling like methods though, but is quite problem-specific. If the number of adaptation episodes that are available corresponds to the number of possible tasks, then Thompson sampling always does optimal exploration. But, if the number of possible tasks exceeds the number of exploration episodes Thompson sampling fails. Adding discussions like these to understand what type exploration *is* necessary and why your method can achieve it, could strengthen the paper.

Typos:
- Intro, Paragraph 3: taskss -> tasks; Based -> Based on
- Background, Paragraph 2: Full stop missing at end
- 3.2, second paragraph + last paragraph: starts with free space
- Figure 2, caption, writes "five" tasks but there are only three (other two in appendix)
- Sec 4 (intro): 3. What's -> What are; 4. Is MetaCURE's -> Are MetaCURE's

------------------------------------------------------------------------------------
UPDATE

Thanks for your detailed response!

A1: I see, that makes more sense to me now. I'm still not 100% convinced that it might not be better to let the policy entirely meta-learn what a good exploration policy is, and only use the inductive bias (terms 1 and 2) for how to do so for meta-training and anneal those terms over time. This would mean that at test time the only thing that the exploration policy should do is maximise the return in the *exploitation* phase - and basically figure out what the best way to do so is (so you'd need the "missing term" I describe above). But that's just a hunch, not sure if that would actually work better.

A2-A8: Thanks a lot for clarifying.

I'll keep my current score given all reviewers agree that the work is unpolished in its current form, and because the authors plan to propose a new version of MetaCURE soon. I think this is very interesting and promising work and look forward to reading an updated version in the future!

The evidence to be evaluated is:The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning. We will discuss in detail the design of intrinsic reward \(r^{\prime}_{intr}\) in the next section.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form:

The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Further derivation (see Appendix A.1 for details) shows that this intrinsic reward can be estimated by the difference of two prediction errors: \[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\]

Empirically, we find that this intrinsic reward suffers form high variance, sometimes causing instability in training. Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance:

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

In Figure 2, MetaCURE-PE outperforms MetaCURE-IG, as its intrinsic reward is of lower variance. However, the prediction errors may lead to bad exploration behaviors under certain scenarios.

While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

**Prediction loss as intrinsic reward** Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore. Oh et al. (2015) directly predicts the image observation, and Stadie et al. (2015) utilize prediction loss in the latent space, extracting useful features from observations. To avoid trivial solutions in learning the latent space, Pathak et al. (2017) introduces an inverse model to guide the learning of latent space, only predicting things that the agent can control.

By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence, the claim about simplifying and deriving prediction error rewards for superior stability/scalability appears partially overstated.

While the evidence shows MetaCURE-PE (prediction error) performs well on some tasks and reduces variance compared to MetaCURE-IG (information gain), it fails completely on noisy environments where MetaCURE-IG succeeds. The review also points out that having two methods where "neither of them is clearly better" weakens the presentation. The evidence indicates tradeoffs between the approaches rather than clear superiority of the simplified prediction error reward.

The claim would be more accurate if it acknowledged these limitations and tradeoffs rather than suggesting universal benefits from the simplification.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To reduce variance from estimating information-gain intrinsic rewards in complex domains, we simplify and derive an intrinsic reward based on prediction errors to achieve superior stability and scalability.

The review comment to be evaluated is:

Summary: This paper aims to facilitate meta-reinforcement learning by improving sample efficiency during meta-training as well as adaptation. Towards this goal, the paper proposes to use an intrinsic reward (two versions) and to learn a separate exploration policy to collect data for faster task inference.

Strengths -

Approaches to improve sample efficiency in meta-RL would be of interest to the community and are potentially useful to RL practitioners.

The proposed technique does not utilise dense rewards during meta-training. The use of intrinsic rewards in this approach possibly helps to overcome an important weakness of some previous methods (like PEARL or MAESN), which find it difficult to use sparse rewards during meta-training and assume access to dense reward in the meta-training phase.

The approach outperforms other meta-RL techniques within the given experimental setup. Additionally, the visualisations of adaptation behaviour (like in Figures 3 and 4) were quite helpful in understanding the method. However, I will discuss my concerns regarding the experiments in the subsequent sections.

Weaknesses and concerns -

The writing and clarity of the paper can be improved. It could be useful to provide additional details to motivate the problem and choice of experimental setting. In the introduction, providing further comments (or citations) could justify this claim: “However, these exploration mechanisms are still insufficient in either meta-training or adaptation and underperform in complex sparse-reward tasks.” It would be helpful to provide more details about the proposed method beyond Figure 1. Is the encoder a recurrent network? And the meta-predictor is another feedforward neural network?

Regarding the experiments, I would like to better understand the motivation behind the choice of restricting the adaptation phase to 2-4 episodes and then evaluating on the last episode. Is this choice realistic in applications? It could be worth considering that such a choice does not take into account potentially catastrophic behaviour by the exploratory policy and is closer to best arm identification (rather than regret minimization) in the multi-armed bandit setting.

It appears that a crucial reason for the better performance of the proposed approach is that it can explore within an episode. The off-policy competitor, PEARL, which uses posterior sampling, keeps its belief fixed for one episode and takes actions accordingly. While the experiments illustrate this weakness of the specifics of PEARL, in principle, there is no requirement in posterior sampling to update and sample from the posterior only once per episode. Since using an intrinsic reward introduces new hyperparameters, it could also be possible to introduce a hyperparameter for the number of steps (say T) between which PEARL updates and samples from the posterior. Tuning such a hyperparameter will also allow PEARL to explore within an episode and could make it much harder to beat. It could be informative to see such a comparison.

Another main concern is regarding the novelty of the proposed approach since the major off-policy meta-RL component appears similar to PEARL. And as the authors mention in the related work section, similar intrinsic rewards and maintaining a separate exploration policy (Gurumurthy et al., 2019) have also been studied earlier.

Related work -

The related work section is missing the original references. For example, the paper refers to recent work related to the use of prediction error as an intrinsic reward: Oh et al. (2015), Stadie et al. (2015) and Pathak et al. (2017). However, prediction error-based intrinsic rewards were introduced in 1990 [b]. Using information gain as a reward for RL agents to better explore the environment was introduced in 1995 [a]. Meta-RL was introduced in 1994 [c]. Intrinsically guided exploration in Meta-RL was introduced in 1997-2002 [d]. Please correct and describe what's really novel in your approach.

[a] J. Storck et al. "Reinforcement driven information acquisition in non-deterministic environments." Proceedings of ICANN'95, Paris. Vol. 2. 1995.

[b] J. Schmidhuber. "A possibility for implementing curiosity and boredom in model-building neural controllers." Proc. of SAB 1991.

[c]          ---.          "On learning how to learn learning strategies." TR FKI-198-94, TUM, 1994. http://people.idsia.ch/~juergen/FKI-198-94ocr.pdf

[d]          ---.          "What's interesting?" TR IDSIA-35-97, IDSIA, July 1997. Also in: Exploring the Predictable. In Ghosh, S. Tsutsui, eds., Advances in Evolutionary Computing, p. 579-612, Springer, 2002. ftp://ftp.idsia.ch/pub/juergen/explorepredictable.pdf

Other minor comments -

- At first glance, it was slightly unclear what the curves in Figure 2 represent. I would suggest moving something like the statement on Page 4, “meta-testing performance as a function of number of experiences collected during meta-training” to the caption of Figure 2.

Some typos -

- Page 1, paragraph 3 in the Introduction:

taskss -> tasks
Based this objective -> Based on

- Page 2, section 3.1, 1st paragraph

Explorer policy to learn an behavior -> a behavior

- Page 3, last paragraph

KL distance -> divergence

- Page 7, Figure 5 caption

nosier -> noisier

The evidence to be evaluated is:The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning. We will discuss in detail the design of intrinsic reward \(r^{\prime}_{intr}\) in the next section.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form:

The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Further derivation (see Appendix A.1 for details) shows that this intrinsic reward can be estimated by the difference of two prediction errors: \[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\]

Empirically, we find that this intrinsic reward suffers form high variance, sometimes causing instability in training. Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance:

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

In Figure 2, MetaCURE-PE outperforms MetaCURE-IG, as its intrinsic reward is of lower variance. However, the prediction errors may lead to bad exploration behaviors under certain scenarios.

While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

**Prediction loss as intrinsic reward** Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore. Oh et al. (2015) directly predicts the image observation, and Stadie et al. (2015) utilize prediction loss in the latent space, extracting useful features from observations. To avoid trivial solutions in learning the latent space, Pathak et al. (2017) introduces an inverse model to guide the learning of latent space, only predicting things that the agent can control.

By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about simplifying information-gain rewards to achieve superior stability and scalability is partially supported but somewhat overstated. While the evidence shows that MetaCURE-PE (with simplified prediction error rewards) performs slightly better than MetaCURE-IG in most cases, there are important nuances. For example, Fig. 5 demonstrates that MetaCURE-PE fails in noisy environments where MetaCURE-IG succeeds. The evidence suggests the simplified approach trades off some exploration capabilities for increased stability in certain scenarios, rather than achieving universally "superior" performance as claimed.

<score>3</score>

Justification:
1. The paper demonstrates that prediction error rewards can improve stability in many cases, supported by performance curves in Fig. 2 showing MetaCURE-PE slightly outperforming MetaCURE-IG.

2. However, the claim overstates the universality of the benefits:
- Fig. 5 shows MetaCURE-PE failing in noisy environments (average return 7.3 vs 20.2 for MetaCURE-IG)
- The text acknowledges "prediction errors may lead to bad exploration behaviors under certain scenarios"
- The reviewer points out that similar approaches have been studied before, suggesting the novelty and superiority claims may be overstated

3. A more nuanced claim acknowledging these tradeoffs would be better supported by the evidence.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To reduce variance from estimating information-gain intrinsic rewards in complex domains, we simplify and derive an intrinsic reward based on prediction errors to achieve superior stability and scalability.

The review comment to be evaluated is:

*SUMMARY*

The paper presents a method for efficient task identification to improve adaptation in a meta RL setting. The approach is based on learning an exploration policy to quickly discriminate the task at hand, so that to leverage a task-specific policy for exploitation. To do so, it employs an intrinsic reward proportional to the information gain (or prediction error) over both the transition and reward models. Finally, the algorithm is evaluated over a set of continuous control domains with sparse rewards.

*EVALUATION*

The idea of focusing the exploration on identifying the current task, instead of collecting good samples for policy improvements, is quite interesting, albeit not completely novel. Unfortunately, the lacking presentation does not help assessing the real quality of the contributions and the robustness of the experimental analysis, and it make me leaning towards a negative evaluation. However, I believe that a partial overhaul of the paper structure, along with a sharper experimental analysis, would make for a nice contribution to the meta RL community.

*DETAILED COMMENTS*

C1) As noted by the authors, the idea of maximizing information gain over the transition model in a single-environment setting has been presented in VIME (Houthooft et al. 2016), while Zhou et al. (2018) employs information gain to fast identification of the transition model in multi-environment setting. Thus, the main contribution of this work seems to account also for the reward model in the information gain computation. However, it is not clear to me how this can lead to a significant improvement, especially in sparse-reward settings, where rewards are zero almost everywhere.

C2) An information theoretic approach to leverage the reward structure in a multi-goal scenario has been previously proposed in (Goyal et al. InfoBot: Transfer and exploration via the information bottleneck. ICLR 2019). Though the setting is not equivalent, I am  wondering how their concept of *decision state* relates to the idea of fast identification of the reward model.

C3) The paper propose two alternatives reward function, one that essentially normalizes the prediction error with the predictability of the transition (IG), and the other that solely consider the prediction error (PE). While the experimental analysis shows a slight edge for the latter, I would say that this might be negligible with respect to the ability of the former to deal with noisy environments (as explained in Section 4.4).

*QUESTIONS*

- Could the authors address the comments above in their response?
- The last line of Section 3 says that the proposed intrinsic reward does not vanish over time: Can the authors explain why?
- The experimental setup describes a very short adaptation phase (hundreds of samples) but the subsequent plots are functions of millions of samples. Could the authors clarify the experimental setup, especially the number of samples employed in the meta-training and meta-testing phases?

*ADDITIONAL FEEDBACK*

- I would suggest to revise the structure of the paper by focusing on the core ideas first, and to the implementation details later.
- I did not find the notation crystal clear, especially regarding the role of the context c, and the indiscriminate use of the words trajectory, episode, experience. To me it would be better to refer as sample every (s,a,r,s) tuple and trajectory every sequence of samples from the initial state to the end of an episode.
- In my opinion the experimental section could be sharper. I would suggest to focus on a smaller set of domains (such as one with changing dynamics, one with changing rewards and one with both for the Mujoco set, in addition to the two Meta-World) and a smaller set of comparisons, and to seek for a clearer illustrations of the benefits of MetaCURE with respect to other methods.
- I believe the ablation study to be central in the evaluation of MetaCURE. Especially, I would have preferred to see a convincing assessment of the benefit of computing the information gain over both the reward and transition models, with respect to accounting just for the former or the latter.
- I would not claim the performance of Meta-World Push (Table 1) to be significantly higher than PEARL, since the confidence intervals are overlapping. Averaging over more trials might help.
- I think the main text relies too much on the Appendix: It is fine to have proofs and derivations out of the paper, but I would not directly comment in the experimental section figures that only appear in the Appendix.

#############

AFTER RESPONSE

I would like to thank the authors for the detailed response. I encourage them to keep working on MetaCURE: With some improvements I believe it will provide a valuable contribution to the meta-RL field.

The evidence to be evaluated is:The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning. We will discuss in detail the design of intrinsic reward \(r^{\prime}_{intr}\) in the next section.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form:

The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Further derivation (see Appendix A.1 for details) shows that this intrinsic reward can be estimated by the difference of two prediction errors: \[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\]

Empirically, we find that this intrinsic reward suffers form high variance, sometimes causing instability in training. Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance:

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

In Figure 2, MetaCURE-PE outperforms MetaCURE-IG, as its intrinsic reward is of lower variance. However, the prediction errors may lead to bad exploration behaviors under certain scenarios.

While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

**Prediction loss as intrinsic reward** Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore. Oh et al. (2015) directly predicts the image observation, and Stadie et al. (2015) utilize prediction loss in the latent space, extracting useful features from observations. To avoid trivial solutions in learning the latent space, Pathak et al. (2017) introduces an inverse model to guide the learning of latent space, only predicting things that the agent can control.

By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, I find that the claim is somewhat overstated based on both empirical results and reviewer comments. While the paper does show that using prediction errors as intrinsic rewards (MetaCURE-PE) achieves good performance, the simplification from information gain to prediction errors involves tradeoffs - MetaCURE-PE performs slightly better in some cases but worse in noisy environments compared to MetaCURE-IG. The reviewer also notes that the benefits of the simplification are "negligible" compared to MetaCURE-IG's ability to handle noisy environments. The claim implies universal superiority without acknowledging these important limitations and context-dependent tradeoffs.

<score>4</score>

Justification:
1. The claim states "superior stability and scalability" but Figures 5 and 10 show MetaCURE-PE failing in noisy environments while MetaCURE-IG remains stable
2. Reviewer comment C3 indicates the performance difference is "negligible"
3. The authors' own results in Figure 2 show only "slightly better" performance
4. The paper acknowledges MetaCURE-PE "may lead to bad exploration behaviors under certain scenarios"

The evidence suggests more measured language acknowledging the tradeoffs would better reflect the actual findings.