Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We develop a novel off-policy meta-RL algorithm, Meta-RL with effiCient Uncertainty Reduction Exploration (MetaCURE), that incorporates our intrinsic rewards and separates exploration and exploitation policies.

The evidence to be evaluated is:During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During meta-training, MetaCURE iteratively infers the posterior task belief with contexts and performs both exploration and exploitation policies to collect data.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance. To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation. The exploiter \(\pi\) is expected to utilize collected experiences and optimize exploitation behavior.

Finally, we utilize SAC (Haarnoja et al., 2018), an off-policy RL algorithm to train the policies.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance: \[r^{\prime}_{PE}(c_{-1:i-1},a_{i})=\mathbb{E}_{(z,r_{i},s_{i+1})(c_{-1:i-1},a_{ i})}[-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)] \tag{5}\] Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

1. Can MetaCURE achieve good final adaptation performance in sparse-reward tasks that require efficient exploration in both meta-training and adaptation? 2. Do the explorer and exploiter obtain desirable behaviors? 3. What's the pros and cons of the two proposed intrinsic rewards? 4. Is MetaCURE's components vital for efficient exploration?

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration. Also, we do not provide dense rewards while meta-training, which is different from the setting of PEARL.

We implement two variants of MetaCURE with PyTorch (Paszke et al., 2019): MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

We test MetaCURE-PE as well as baselines on two Meta-World task sets: Push and Reach.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

MetaCURE outperforms these two baselines by performing task inference and separating exploration and exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

These methods utilize policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards.

These methods also fail to meta-train policies with sparse rewards.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

\[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\] In this subsection we provide pseudo-codes for MetaCURE's meta-training and adaptation, as shown in Algorithm 1 and Algorithm 2.

```

```**Algorithm 1** MetaCURE: Meta-training Phase``` Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor ```**Algorithm 2** MetaCURE: Adaptation Phase We show additional visualization results on Cheetah-Vel-Sparse, as shown in Figure 7.

\(\beta\) is a hyper-parameter controlling the weight of KL divergence in the encoder's loss, similar to Rakelly et al. (2019).

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-supported. The paper presents a comprehensive description of MetaCURE's architecture, intrinsic reward mechanisms, and experimental validation. The evidence shows:

1. Clear theoretical framework separating exploration/exploitation policies
2. Detailed implementation of two intrinsic reward variants
3. Strong empirical results demonstrating superior performance over baselines across multiple sparse-reward environments
4. Extensive ablation studies validating the importance of each component
5. Visualizations confirming effective exploration behaviors

The claim about developing a novel algorithm with separate policies and intrinsic rewards is accurately stated and matches the depth of evidence provided, without overreaching beyond what is demonstrated.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We develop a novel off-policy meta-RL algorithm, Meta-RL with effiCient Uncertainty Reduction Exploration (MetaCURE), that incorporates our intrinsic rewards and separates exploration and exploitation policies.

The review comment to be evaluated is:

This paper introduces the use of intrinsically motivated exploration in meta-training and adaptation phase in meta Reinforcement Learning. The proposed framework consists of using task-inference, already used in other works like PEARL and VariBAD, and two deep neural networks, trained with an arbitrary off-policy algorithm, respectively for exploitation and exploration phases.

I think the paper is well-written and discusses the related literature sufficiently well. The empirical results are significantly better than the considered baselines.
Despite these good qualities, I think the major weakness of this paper is the lack of contribution. It looks to me that the major contribution discussed by the authors is the use of intrinsically motivated exploration in meta-training and adaptation, while the use of task-inference has been already proposed in other works and the use of different policy networks for exploitation and exploration is trivial. Considering the intrinsically motivated exploration, I miss the importance of the contribution, since it is already well-known that this technique can be beneficial, especially in sparse MDPs. The intrinsic reward makes sense, but it is not novel, as similar intrinsic rewards have been proposed in VIME and Pathak et al. (2017).
Experimental results are good, but not well described. For example, I don't find a clear explanation of the tasks used in meta-training. The random seeds are only three, which makes the results less reliable.
Considering this, I don't consider this paper significant enough to be accepted.

Minor issues:
* Pag. 1, typo: "tasks" -> tasks
* Pag. 1, "Based this objective" -> Based on this objective
* the R function in (1) is not formally defined.

Pros
------
* Well-written;
* Good discussion of related literature;
* Good experimental results.

Cons
-------
* Lack of contribution;
* Few, but important, missing details about the experimental setting, and too few seeds.

Post-rebuttal feedback
-------------------------------
I thank the author for their reply and I encourage them to make the suggested improvements to the paper.

The evidence to be evaluated is:During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During meta-training, MetaCURE iteratively infers the posterior task belief with contexts and performs both exploration and exploitation policies to collect data.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance. To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation. The exploiter \(\pi\) is expected to utilize collected experiences and optimize exploitation behavior.

Finally, we utilize SAC (Haarnoja et al., 2018), an off-policy RL algorithm to train the policies.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance: \[r^{\prime}_{PE}(c_{-1:i-1},a_{i})=\mathbb{E}_{(z,r_{i},s_{i+1})(c_{-1:i-1},a_{ i})}[-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)] \tag{5}\] Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

1. Can MetaCURE achieve good final adaptation performance in sparse-reward tasks that require efficient exploration in both meta-training and adaptation? 2. Do the explorer and exploiter obtain desirable behaviors? 3. What's the pros and cons of the two proposed intrinsic rewards? 4. Is MetaCURE's components vital for efficient exploration?

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration. Also, we do not provide dense rewards while meta-training, which is different from the setting of PEARL.

We implement two variants of MetaCURE with PyTorch (Paszke et al., 2019): MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

We test MetaCURE-PE as well as baselines on two Meta-World task sets: Push and Reach.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

MetaCURE outperforms these two baselines by performing task inference and separating exploration and exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

These methods utilize policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards.

These methods also fail to meta-train policies with sparse rewards.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

\[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\] In this subsection we provide pseudo-codes for MetaCURE's meta-training and adaptation, as shown in Algorithm 1 and Algorithm 2.

```

```**Algorithm 1** MetaCURE: Meta-training Phase``` Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor ```**Algorithm 2** MetaCURE: Adaptation Phase We show additional visualization results on Cheetah-Vel-Sparse, as shown in Figure 7.

\(\beta\) is a hyper-parameter controlling the weight of KL divergence in the encoder's loss, similar to Rakelly et al. (2019).

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate the claim: "We develop a novel off-policy meta-RL algorithm, Meta-RL with effiCient Uncertainty Reduction Exploration (MetaCURE), that incorporates our intrinsic rewards and separates exploration and exploitation policies."

The reviewer's concerns about novelty are partially valid - some components like task inference and separate policies aren't entirely new. However, the evidence shows that MetaCURE makes meaningful contributions through:
1. Effective combination of intrinsic rewards for both meta-training and adaptation
2. Strong empirical results across sparse-reward tasks
3. Clear visualization of improved exploration behaviors compared to baselines
4. Novel information-theoretic intrinsic reward formulation

While not entirely groundbreaking, the claim is generally well-stated as it accurately reflects the key aspects of the work without major exaggeration.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We develop a novel off-policy meta-RL algorithm, Meta-RL with effiCient Uncertainty Reduction Exploration (MetaCURE), that incorporates our intrinsic rewards and separates exploration and exploitation policies.

The review comment to be evaluated is:

This paper proposes a novel meta-learning method, aimed at solving the following problem: at test time, the agent has N episodes to gather information [exploration phase], and we care about its return in the N+1th episode [exploitation phase]. To this end, the authors propose to learn a separate exploration and exploitation policy. The core of the algorithm is to use an exploration bonus for the exploration policy that rewards finding novel trajectories. This should help it to collect valuable information during meta training and meta testing. The exploration policy is only used in the N+1th episode, and  is conditioned on a context vector which is computed from the exploration trajectories.

Pro:
- Very relevant subject.
- The authors bring up an interesting point, that metaRL methods should consider both exploration during meta training and meta testing.
- I think it makes sense to separate exploration and exploitation policy explicitly.
- The authors propose to sparsify common metaRL benchmarks. Evaluating your algorithm on environments that are proposed in the same paper can be tricky (and often means that the empirical results are evaluated more critically), but I think the suggestion of looking at sparse metaRL tasks is useful for the field and should be considered by other researchers as well.
- The authors provide illustrations of the behaviours of the learned policy at meta-test time, which I think is really nice!

Concerns:
- I fee like there's a mismatch between the reward that the explorer gets, and the problem setting that the authors consider. The way I understand it, the two terms in the explorer's reward (Eq (2)) can be helpful during meta-training but I don't see how that translates to meta-test time. Let me try to explain:
 - [First term] The objective of the agent is to maximise the return in the *last* episode (the exploitation episode). Then why is the explorer not maximising the return of the exploitation phase, but instead gets a reward that corresponds to the exploration episodes? Doesn't that contradict the premise that we're not interested at all in the reward that is collected during exploration? Also, what is n (you write "training iterations" - is that just the number of environment steps per task? number of policy updates over the entirety of meta-training? Does this mean this is annealed to zero over training? Maybe then it shouldn't be called $\gamma$ so that it's not confused with the per-episode discount usually used in RL.).
 - [Second term] The reward bonus can be seen as a type of novelty bonus for finding new trajectories. Can the authors clarify why this would help (a) during meta-training and (b) during meta-testing? I can see how it helps during training, but I don't understand why it also helps at test time. At the end of Sec 3.2, you write "the intrinsic reward is estimated on multiple tasks and does not diminish during meta-training". Why not? In Eq 5, does $c_{-1:i-1}$ come from multiple tasks or just one? Why can't this diminish over time, shouldn't our model $p(r,s|x,a,z)$ (dropping the subscripts for brevity) get better over time? Or do you just mean that the weight $\lambda$ is not annealed? And doesn't it have all the information to, in principle, predict r and s exactly (except for aleatoric noise) so then this term reduces to the noise in the environment? Or should the terms in the subscript not match the ones within the expectation? It's also a bit unclear what the expectation is over, the subscript just shows some tuples. What's the probability distribution? Where do the terms in the subscript come from? Are $q$ and $p^\mathcal{T}$ mixed in here? It would help my understanding if these things could be made more explicit.
 - [Missing term?] I understand the above two terms might help during training, but they are only auxiliary tasks in a way. And if the first term is annealed to zero (if that's indeed what's happening?), and the second term reduces to aleatoric noise (is that's indeed what's happening?), what is the exploration policy even doing at the end of training? Should there be a third term, which is the return in the *exploitation* phase, which is the only thing that the exploration policy should try to maximise at meta-test time?
- I think the mathematical formulation has to be fleshed out. The paper is sometimes hard to follow because the notation is unclear, and not all terms are explained. I have a few open questions because of this:
 - What are the expectations in (3), (4), (5) over?
 - Not all terms in Figure 1 are defined or mentioned in the text. Are $L_\pi$ and $L_{Q_\pi}$ loss terms from SAC, so they correspond to equation (1)? And $L_{\pi_e}$ corresponds to equation (2)? And what is $L_{pred}$? All these should be explicitly defined in the text with an equation.
 - Sec 2, directly after Eq (1): $c^\mathcal{T}_{\pi_e}$ is not defined.
- In Figure 2, PEARL (Walker-Vel-Sparse) and VariBAD (Point-Robot-Sparse) seem to first learn and then diverge. Do the authors have an explanation as to why this is? It seems to me that maybe the hyperparameters just weren't set right, and makes me a bit sceptical of the results. When PEARL/VariBAD *do* learn, their (end-) performance is much closer to that of MetaCURE.
- The authors propose two versions of MetaCURE (IG and PE). On the continuous control tasks PE is better, but then on the noisy point robot PE completely fails and IG succeeds. I think the authors are not doing themselves a favour by setting up the presentation of their methods like this. It would be better to propose *one* method, and then fully focus on this. The way it is now, there are two methods where neither of them is clearly better. I think that's confusing to the reader, and less convincing. Shouldn't there be a way to develop an exploration bonus that can do well across all environments?
- The "noisy" environment is a POMDP (the agent gets noisy state observations). So doesn't that mean that there is simply a mismatch between the problem setting and the way the policy is set up? I'm not sure that this is a good experiment to add, because the solution (using the PE explorer) does not directly address the problem (partial observability). Maybe a noisy reward function would fit better here?

Other questions:
- Can you explain why the same exploration mechanism useful during meta-training and meta-testing? Aren't we mixing across-task and within-task information gain, and if so, why is that okay?
- What is the difference between row 1 and row 2 in Figure 2? This is not explained in the text. Is that separation just because everything in one figure would've been too much? And how exactly were the baselines evaluated, always after the same number of episodes?
- Why is the loss of the explorer not backpropagated through the context encoder? How does that fit together with "our intrinsic reward supports end-to-end learning and collect online data for effective task inference"?
- Why does the exploiter get a sample $z$ and not the entire distribution $q$? Is that because we assume that all information is there and $q$ is concentrated anyway?
- You mention that the intrinsic reward of PE is of lower variance. Could you explain why? Is that a hypothesis (if so, say so explicit) or do you have any empirical/theoretical evidence for this (again, if so, make it explicit).

Suggestions:
- You write that "efficient exploration of meta-RL needs to consider both training and adaptation phases simultaneously". This is a good point and I agree with it, and I think the paper can be improved by fleshing this argument out more. Help the reader understand why this is the case, and what the (different) challenges in exploration are during training and testing. One open question I still have for example (see above) is why the *same* exploration mechanism is useful during meta-training and meta-testing. Why not separate them? This should be explained in the paper. In the related work section, you write that prior methods "utilise policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards". This point can be made about almost all prior methods, and should come in the introduction. Then explain intuitively *why* this fails, maybe which an example. Another question I have is, how does the Ishii et al. 2002 paper back up this point that this distinction exists?
- The experiments section relies too much on the appendix, and it's quite unnatural to read it like this. Either I have to jump back and forth between the appendix and the main text, or I have to take your word for what is shown in the results in the appendix. You introduce five continuous control tasks at the start, but then actually only show 3 and the other 2 are in the appendix. I think this section can be improved by focusing on the main important points in the main text, explain this well and mention that more results can be found in the appendix.
- Mention in the main text that the sparse setting for Meta-World means you use the success signal only. This is a very difficult setting and is worth highlighting instead of burying it in the appendix.
- I think a discussion about what an good/optimal exploration strategy would look like can help the reader understand the problem setting, and what you are trying to achieve. Right now, in Sec 4.3 you write for example that PEARL "explored less effectively" because it keeps the belief about the current task throughout an entire episode. That's not a fundamental problem with Thompson-sampling like methods though, but is quite problem-specific. If the number of adaptation episodes that are available corresponds to the number of possible tasks, then Thompson sampling always does optimal exploration. But, if the number of possible tasks exceeds the number of exploration episodes Thompson sampling fails. Adding discussions like these to understand what type exploration *is* necessary and why your method can achieve it, could strengthen the paper.

Typos:
- Intro, Paragraph 3: taskss -> tasks; Based -> Based on
- Background, Paragraph 2: Full stop missing at end
- 3.2, second paragraph + last paragraph: starts with free space
- Figure 2, caption, writes "five" tasks but there are only three (other two in appendix)
- Sec 4 (intro): 3. What's -> What are; 4. Is MetaCURE's -> Are MetaCURE's

------------------------------------------------------------------------------------
UPDATE

Thanks for your detailed response!

A1: I see, that makes more sense to me now. I'm still not 100% convinced that it might not be better to let the policy entirely meta-learn what a good exploration policy is, and only use the inductive bias (terms 1 and 2) for how to do so for meta-training and anneal those terms over time. This would mean that at test time the only thing that the exploration policy should do is maximise the return in the *exploitation* phase - and basically figure out what the best way to do so is (so you'd need the "missing term" I describe above). But that's just a hunch, not sure if that would actually work better.

A2-A8: Thanks a lot for clarifying.

I'll keep my current score given all reviewers agree that the work is unpolished in its current form, and because the authors plan to propose a new version of MetaCURE soon. I think this is very interesting and promising work and look forward to reading an updated version in the future!

The evidence to be evaluated is:During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During meta-training, MetaCURE iteratively infers the posterior task belief with contexts and performs both exploration and exploitation policies to collect data.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance. To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation. The exploiter \(\pi\) is expected to utilize collected experiences and optimize exploitation behavior.

Finally, we utilize SAC (Haarnoja et al., 2018), an off-policy RL algorithm to train the policies.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance: \[r^{\prime}_{PE}(c_{-1:i-1},a_{i})=\mathbb{E}_{(z,r_{i},s_{i+1})(c_{-1:i-1},a_{ i})}[-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)] \tag{5}\] Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

1. Can MetaCURE achieve good final adaptation performance in sparse-reward tasks that require efficient exploration in both meta-training and adaptation? 2. Do the explorer and exploiter obtain desirable behaviors? 3. What's the pros and cons of the two proposed intrinsic rewards? 4. Is MetaCURE's components vital for efficient exploration?

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration. Also, we do not provide dense rewards while meta-training, which is different from the setting of PEARL.

We implement two variants of MetaCURE with PyTorch (Paszke et al., 2019): MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

We test MetaCURE-PE as well as baselines on two Meta-World task sets: Push and Reach.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

MetaCURE outperforms these two baselines by performing task inference and separating exploration and exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

These methods utilize policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards.

These methods also fail to meta-train policies with sparse rewards.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

\[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\] In this subsection we provide pseudo-codes for MetaCURE's meta-training and adaptation, as shown in Algorithm 1 and Algorithm 2.

```

```**Algorithm 1** MetaCURE: Meta-training Phase``` Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor ```**Algorithm 2** MetaCURE: Adaptation Phase We show additional visualization results on Cheetah-Vel-Sparse, as shown in Figure 7.

\(\beta\) is a hyper-parameter controlling the weight of KL divergence in the encoder's loss, similar to Rakelly et al. (2019).

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the extensive evidence, this claim appears to be moderately overstated. While MetaCURE does demonstrate a novel off-policy meta-RL approach with separate exploration and exploitation policies using intrinsic rewards, several key issues emerge:

1. The reviewer raises valid concerns about mismatches between the exploration rewards and the problem setting, particularly at meta-test time.

2. The evidence shows mixed results between the two variants (IG and PE) - neither consistently outperforms the other across all environments.

3. The visualization results, while promising, don't fully validate that the separation of policies and intrinsic rewards are both necessary and sufficient for the claimed efficient exploration.

4. Some experimental results show potential instability (e.g., PEARL/VariBAD learning then diverging), suggesting the comparisons may not be entirely fair.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We develop a novel off-policy meta-RL algorithm, Meta-RL with effiCient Uncertainty Reduction Exploration (MetaCURE), that incorporates our intrinsic rewards and separates exploration and exploitation policies.

The review comment to be evaluated is:

Summary: This paper aims to facilitate meta-reinforcement learning by improving sample efficiency during meta-training as well as adaptation. Towards this goal, the paper proposes to use an intrinsic reward (two versions) and to learn a separate exploration policy to collect data for faster task inference.

Strengths -

Approaches to improve sample efficiency in meta-RL would be of interest to the community and are potentially useful to RL practitioners.

The proposed technique does not utilise dense rewards during meta-training. The use of intrinsic rewards in this approach possibly helps to overcome an important weakness of some previous methods (like PEARL or MAESN), which find it difficult to use sparse rewards during meta-training and assume access to dense reward in the meta-training phase.

The approach outperforms other meta-RL techniques within the given experimental setup. Additionally, the visualisations of adaptation behaviour (like in Figures 3 and 4) were quite helpful in understanding the method. However, I will discuss my concerns regarding the experiments in the subsequent sections.

Weaknesses and concerns -

The writing and clarity of the paper can be improved. It could be useful to provide additional details to motivate the problem and choice of experimental setting. In the introduction, providing further comments (or citations) could justify this claim: “However, these exploration mechanisms are still insufficient in either meta-training or adaptation and underperform in complex sparse-reward tasks.” It would be helpful to provide more details about the proposed method beyond Figure 1. Is the encoder a recurrent network? And the meta-predictor is another feedforward neural network?

Regarding the experiments, I would like to better understand the motivation behind the choice of restricting the adaptation phase to 2-4 episodes and then evaluating on the last episode. Is this choice realistic in applications? It could be worth considering that such a choice does not take into account potentially catastrophic behaviour by the exploratory policy and is closer to best arm identification (rather than regret minimization) in the multi-armed bandit setting.

It appears that a crucial reason for the better performance of the proposed approach is that it can explore within an episode. The off-policy competitor, PEARL, which uses posterior sampling, keeps its belief fixed for one episode and takes actions accordingly. While the experiments illustrate this weakness of the specifics of PEARL, in principle, there is no requirement in posterior sampling to update and sample from the posterior only once per episode. Since using an intrinsic reward introduces new hyperparameters, it could also be possible to introduce a hyperparameter for the number of steps (say T) between which PEARL updates and samples from the posterior. Tuning such a hyperparameter will also allow PEARL to explore within an episode and could make it much harder to beat. It could be informative to see such a comparison.

Another main concern is regarding the novelty of the proposed approach since the major off-policy meta-RL component appears similar to PEARL. And as the authors mention in the related work section, similar intrinsic rewards and maintaining a separate exploration policy (Gurumurthy et al., 2019) have also been studied earlier.

Related work -

The related work section is missing the original references. For example, the paper refers to recent work related to the use of prediction error as an intrinsic reward: Oh et al. (2015), Stadie et al. (2015) and Pathak et al. (2017). However, prediction error-based intrinsic rewards were introduced in 1990 [b]. Using information gain as a reward for RL agents to better explore the environment was introduced in 1995 [a]. Meta-RL was introduced in 1994 [c]. Intrinsically guided exploration in Meta-RL was introduced in 1997-2002 [d]. Please correct and describe what's really novel in your approach.

[a] J. Storck et al. "Reinforcement driven information acquisition in non-deterministic environments." Proceedings of ICANN'95, Paris. Vol. 2. 1995.

[b] J. Schmidhuber. "A possibility for implementing curiosity and boredom in model-building neural controllers." Proc. of SAB 1991.

[c]          ---.          "On learning how to learn learning strategies." TR FKI-198-94, TUM, 1994. http://people.idsia.ch/~juergen/FKI-198-94ocr.pdf

[d]          ---.          "What's interesting?" TR IDSIA-35-97, IDSIA, July 1997. Also in: Exploring the Predictable. In Ghosh, S. Tsutsui, eds., Advances in Evolutionary Computing, p. 579-612, Springer, 2002. ftp://ftp.idsia.ch/pub/juergen/explorepredictable.pdf

Other minor comments -

- At first glance, it was slightly unclear what the curves in Figure 2 represent. I would suggest moving something like the statement on Page 4, “meta-testing performance as a function of number of experiences collected during meta-training” to the caption of Figure 2.

Some typos -

- Page 1, paragraph 3 in the Introduction:

taskss -> tasks
Based this objective -> Based on

- Page 2, section 3.1, 1st paragraph

Explorer policy to learn an behavior -> a behavior

- Page 3, last paragraph

KL distance -> divergence

- Page 7, Figure 5 caption

nosier -> noisier

The evidence to be evaluated is:During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During meta-training, MetaCURE iteratively infers the posterior task belief with contexts and performs both exploration and exploitation policies to collect data.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance. To learn efficient exploration in sparse-reward tasks, the reward signal for the explorer is defined as follows: \[r_{\text{explorer}}(c_{-1:i-1},a_{i})=\gamma^{n}r(s_{i},a_{i})+\lambda r^{ \prime}_{intr}(c_{-1:i-1},a_{i}) \tag{2}\] where \(r\) is the extrinsic reward, \(r^{\prime}_{intr}\) is an intrinsic reward, \(\lambda\)\(>\)\(0\) is a hyper-parameter and \(\gamma\in[0,1]\) is a discount factor, and \(n\) is the number of training iterations. The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information. This modification reuses learned knowledge from raw experiences and accelerates learning.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation. The exploiter \(\pi\) is expected to utilize collected experiences and optimize exploitation behavior.

Finally, we utilize SAC (Haarnoja et al., 2018), an off-policy RL algorithm to train the policies.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Inspired by curiosity-driven exploration in traditional RL (Still and Precup, 2012; Pathak et al., 2017; Burda et al., 2018), we remove the first term in \(r^{\prime}_{IG}\) and use prediction errors as the intrinsic reward to reduce the estimation variance: \[r^{\prime}_{PE}(c_{-1:i-1},a_{i})=\mathbb{E}_{(z,r_{i},s_{i+1})(c_{-1:i-1},a_{ i})}[-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)] \tag{5}\] Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

1. Can MetaCURE achieve good final adaptation performance in sparse-reward tasks that require efficient exploration in both meta-training and adaptation? 2. Do the explorer and exploiter obtain desirable behaviors? 3. What's the pros and cons of the two proposed intrinsic rewards? 4. Is MetaCURE's components vital for efficient exploration?

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration. Also, we do not provide dense rewards while meta-training, which is different from the setting of PEARL.

We implement two variants of MetaCURE with PyTorch (Paszke et al., 2019): MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

MetaCURE is the only algorithm that manages to acquire good performance on all five task sets, and MetaCURE with the prediction error intrinsic performs slightly better.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

We test MetaCURE-PE as well as baselines on two Meta-World task sets: Push and Reach.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

MetaCURE outperforms these two baselines by performing task inference and separating exploration and exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

These methods utilize policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards.

These methods also fail to meta-train policies with sparse rewards.

EPI (Zhou et al., 2018) learns exploration policies guided with intrinsic rewards based on prediction improvement in dynamics.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

\[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\] In this subsection we provide pseudo-codes for MetaCURE's meta-training and adaptation, as shown in Algorithm 1 and Algorithm 2.

```

```**Algorithm 1** MetaCURE: Meta-training Phase``` Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor ```**Algorithm 2** MetaCURE: Adaptation Phase We show additional visualization results on Cheetah-Vel-Sparse, as shown in Figure 7.

\(\beta\) is a hyper-parameter controlling the weight of KL divergence in the encoder's loss, similar to Rakelly et al. (2019).

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance. To show how intrinsic rewards benefit adaptation exploration, we design an environment similar to Point-Robot-Sparse and has a sub-optimal goal that keeps unchanged across all tasks, as shown in Figure 10. While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, including detailed methodology, experimental results, and visualizations, I find this claim to be generally well-stated but slightly overstated. While the algorithm does demonstrate novel aspects by combining off-policy meta-RL with intrinsic rewards and separate exploration/exploitation policies, the reviewer correctly points out that some key components are similar to existing work (like PEARL) and similar intrinsic rewards have been studied before. The experimental results show strong performance but the paper would benefit from acknowledging these limitations more explicitly.

The evidence shows that MetaCURE achieves superior performance on sparse-reward tasks compared to baselines, supporting the novelty and effectiveness of the overall approach. However, rather than being entirely "novel," it builds upon and combines existing concepts in a useful but incremental way.

<score>2</score>