Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address this challenge of meta-RL, we introduce information-theoretic intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation.

The review comment to be evaluated is:

Summary: This paper aims to facilitate meta-reinforcement learning by improving sample efficiency during meta-training as well as adaptation. Towards this goal, the paper proposes to use an intrinsic reward (two versions) and to learn a separate exploration policy to collect data for faster task inference.

Strengths -

Approaches to improve sample efficiency in meta-RL would be of interest to the community and are potentially useful to RL practitioners.

The proposed technique does not utilise dense rewards during meta-training. The use of intrinsic rewards in this approach possibly helps to overcome an important weakness of some previous methods (like PEARL or MAESN), which find it difficult to use sparse rewards during meta-training and assume access to dense reward in the meta-training phase.

The approach outperforms other meta-RL techniques within the given experimental setup. Additionally, the visualisations of adaptation behaviour (like in Figures 3 and 4) were quite helpful in understanding the method. However, I will discuss my concerns regarding the experiments in the subsequent sections.

Weaknesses and concerns -

The writing and clarity of the paper can be improved. It could be useful to provide additional details to motivate the problem and choice of experimental setting. In the introduction, providing further comments (or citations) could justify this claim: “However, these exploration mechanisms are still insufficient in either meta-training or adaptation and underperform in complex sparse-reward tasks.” It would be helpful to provide more details about the proposed method beyond Figure 1. Is the encoder a recurrent network? And the meta-predictor is another feedforward neural network?

Regarding the experiments, I would like to better understand the motivation behind the choice of restricting the adaptation phase to 2-4 episodes and then evaluating on the last episode. Is this choice realistic in applications? It could be worth considering that such a choice does not take into account potentially catastrophic behaviour by the exploratory policy and is closer to best arm identification (rather than regret minimization) in the multi-armed bandit setting.

It appears that a crucial reason for the better performance of the proposed approach is that it can explore within an episode. The off-policy competitor, PEARL, which uses posterior sampling, keeps its belief fixed for one episode and takes actions accordingly. While the experiments illustrate this weakness of the specifics of PEARL, in principle, there is no requirement in posterior sampling to update and sample from the posterior only once per episode. Since using an intrinsic reward introduces new hyperparameters, it could also be possible to introduce a hyperparameter for the number of steps (say T) between which PEARL updates and samples from the posterior. Tuning such a hyperparameter will also allow PEARL to explore within an episode and could make it much harder to beat. It could be informative to see such a comparison.

Another main concern is regarding the novelty of the proposed approach since the major off-policy meta-RL component appears similar to PEARL. And as the authors mention in the related work section, similar intrinsic rewards and maintaining a separate exploration policy (Gurumurthy et al., 2019) have also been studied earlier.

Related work -

The related work section is missing the original references. For example, the paper refers to recent work related to the use of prediction error as an intrinsic reward: Oh et al. (2015), Stadie et al. (2015) and Pathak et al. (2017). However, prediction error-based intrinsic rewards were introduced in 1990 [b]. Using information gain as a reward for RL agents to better explore the environment was introduced in 1995 [a]. Meta-RL was introduced in 1994 [c]. Intrinsically guided exploration in Meta-RL was introduced in 1997-2002 [d]. Please correct and describe what's really novel in your approach.

[a] J. Storck et al. "Reinforcement driven information acquisition in non-deterministic environments." Proceedings of ICANN'95, Paris. Vol. 2. 1995.

[b] J. Schmidhuber. "A possibility for implementing curiosity and boredom in model-building neural controllers." Proc. of SAB 1991.

[c]          ---.          "On learning how to learn learning strategies." TR FKI-198-94, TUM, 1994. http://people.idsia.ch/~juergen/FKI-198-94ocr.pdf

[d]          ---.          "What's interesting?" TR IDSIA-35-97, IDSIA, July 1997. Also in: Exploring the Predictable. In Ghosh, S. Tsutsui, eds., Advances in Evolutionary Computing, p. 579-612, Springer, 2002. ftp://ftp.idsia.ch/pub/juergen/explorepredictable.pdf

Other minor comments -

- At first glance, it was slightly unclear what the curves in Figure 2 represent. I would suggest moving something like the statement on Page 4, “meta-testing performance as a function of number of experiences collected during meta-training” to the caption of Figure 2.

Some typos -

- Page 1, paragraph 3 in the Introduction:

taskss -> tasks
Based this objective -> Based on

- Page 2, section 3.1, 1st paragraph

Explorer policy to learn an behavior -> a behavior

- Page 3, last paragraph

KL distance -> divergence

- Page 7, Figure 5 caption

nosier -> noisier

The evidence to be evaluated is:In meta-RL, we consider a distribution of tasks \(p(\mathcal{T})\), with each task \(\mathcal{T}\) modelled as a Markov Decision Process (MDP), which consists of a state space, an action space, a transition function and a reward function. In common meta-RL settings (Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019), tasks differ in the transition and/or reward function, and we can describe a task \(\mathcal{T}\) with a tuple \(\langle p_{0}^{T}(s_{0}),p^{T}(s^{\prime}|s,a),r^{\mathcal{T}}(s,a)\rangle\), with \(p_{0}^{T}(s_{0})\) the initial state distribution, \(p^{T}(s^{\prime}|s,a)\) the transition probability and \(r^{\mathcal{T}}(s,a)\) the reward function.

A common objective for meta-RL is to optimize final performance after few-shot adaptation (Finn et al., 2017; Gupta et al., 2018; Stadie et al., 2018; Rothfuss et al., 2019). During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.

The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

\[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\] Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

Besides, we are also interested in its sample efficiency in meta-training and adaptation.

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

PEARL fails to effectively explore within short adaptation, as posterior sampling (Osband and Van Roy, 2017) is only optimal in asymptotic performance. RL\({}^{2}\) achieves good performance in tasks with simple dynamics like Point-Robot-Sparse and Cheetah-Vel-Sparse, but fails in more complex tasks. MAML acquires good performance in Walker-Params-Sparse but fails to learn tasks with different reward functions.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

Measuring information gain is an important way of designing intrinsic rewards for exploration. VIME (Houthooft et al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim et al., 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space.

Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Input: A set of meta-training tasks \(\{\mathcal{T}^{i}\}_{i=1,2,...,T}\) drawn from \(p(\mathcal{T})\)  Initialize replay buffers \(\mathcal{B}^{i},\mathcal{B}^{i}_{enc}\) for each task \(\mathcal{T}^{i}\) while not done do for each task \(\mathcal{T}^{i}\)do  Initialize experience context \(c_{i}=\{\}\) for episodes=1,...,\(K_{1}\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c_{i}))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c_{i}\), \(\mathcal{B}^{i}\) and \(\mathcal{B}^{i}_{enc}\) endfor for episodes=1,...,\(K_{2}\)do  Sample \(z\sim q_{\phi}(z|c_{i})\)  Collect a trajectory with \(\pi(a|s,z)\), add collected experience to \(\mathcal{B}^{i}\) endfor for for steps in training steps do for each \(\mathcal{T}^{i}\)do  Sample encoder batch \(b^{i}_{enc}\) and policy training batch \(b^{i}\)  Sample \(z\sim q_{\phi}(z|b^{i}_{enc})\), compute \(r^{e}\) with Equation 5  Train encoder \(q_{\phi}\), \(\pi\) and \(\pi^{e}\) using SAC, \(b^{i}\) and \(r^{e}\), train the predictor to minimize prediction loss endfor endfor endwhile

Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.

While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about introducing information-theoretic intrinsic motivations is moderately overstated. While the paper demonstrates improved exploration using intrinsic rewards and shows positive experimental results, several key limitations exist:

1. The reviewer points out that similar intrinsic rewards and exploration policies have been studied before, with some key references from the 1990s not properly acknowledged.

2. The experimental comparisons could be stronger - the reviewer notes that competitor methods like PEARL could potentially perform better with minor modifications.

3. The novelty is questioned since major components appear similar to existing work.

The evidence shows the approach works but oversells its novelty and contribution.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address this challenge of meta-RL, we introduce information-theoretic intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation.

The review comment to be evaluated is:

*SUMMARY*

The paper presents a method for efficient task identification to improve adaptation in a meta RL setting. The approach is based on learning an exploration policy to quickly discriminate the task at hand, so that to leverage a task-specific policy for exploitation. To do so, it employs an intrinsic reward proportional to the information gain (or prediction error) over both the transition and reward models. Finally, the algorithm is evaluated over a set of continuous control domains with sparse rewards.

*EVALUATION*

The idea of focusing the exploration on identifying the current task, instead of collecting good samples for policy improvements, is quite interesting, albeit not completely novel. Unfortunately, the lacking presentation does not help assessing the real quality of the contributions and the robustness of the experimental analysis, and it make me leaning towards a negative evaluation. However, I believe that a partial overhaul of the paper structure, along with a sharper experimental analysis, would make for a nice contribution to the meta RL community.

*DETAILED COMMENTS*

C1) As noted by the authors, the idea of maximizing information gain over the transition model in a single-environment setting has been presented in VIME (Houthooft et al. 2016), while Zhou et al. (2018) employs information gain to fast identification of the transition model in multi-environment setting. Thus, the main contribution of this work seems to account also for the reward model in the information gain computation. However, it is not clear to me how this can lead to a significant improvement, especially in sparse-reward settings, where rewards are zero almost everywhere.

C2) An information theoretic approach to leverage the reward structure in a multi-goal scenario has been previously proposed in (Goyal et al. InfoBot: Transfer and exploration via the information bottleneck. ICLR 2019). Though the setting is not equivalent, I am  wondering how their concept of *decision state* relates to the idea of fast identification of the reward model.

C3) The paper propose two alternatives reward function, one that essentially normalizes the prediction error with the predictability of the transition (IG), and the other that solely consider the prediction error (PE). While the experimental analysis shows a slight edge for the latter, I would say that this might be negligible with respect to the ability of the former to deal with noisy environments (as explained in Section 4.4).

*QUESTIONS*

- Could the authors address the comments above in their response?
- The last line of Section 3 says that the proposed intrinsic reward does not vanish over time: Can the authors explain why?
- The experimental setup describes a very short adaptation phase (hundreds of samples) but the subsequent plots are functions of millions of samples. Could the authors clarify the experimental setup, especially the number of samples employed in the meta-training and meta-testing phases?

*ADDITIONAL FEEDBACK*

- I would suggest to revise the structure of the paper by focusing on the core ideas first, and to the implementation details later.
- I did not find the notation crystal clear, especially regarding the role of the context c, and the indiscriminate use of the words trajectory, episode, experience. To me it would be better to refer as sample every (s,a,r,s) tuple and trajectory every sequence of samples from the initial state to the end of an episode.
- In my opinion the experimental section could be sharper. I would suggest to focus on a smaller set of domains (such as one with changing dynamics, one with changing rewards and one with both for the Mujoco set, in addition to the two Meta-World) and a smaller set of comparisons, and to seek for a clearer illustrations of the benefits of MetaCURE with respect to other methods.
- I believe the ablation study to be central in the evaluation of MetaCURE. Especially, I would have preferred to see a convincing assessment of the benefit of computing the information gain over both the reward and transition models, with respect to accounting just for the former or the latter.
- I would not claim the performance of Meta-World Push (Table 1) to be significantly higher than PEARL, since the confidence intervals are overlapping. Averaging over more trials might help.
- I think the main text relies too much on the Appendix: It is fine to have proofs and derivations out of the paper, but I would not directly comment in the experimental section figures that only appear in the Appendix.

#############

AFTER RESPONSE

I would like to thank the authors for the detailed response. I encourage them to keep working on MetaCURE: With some improvements I believe it will provide a valuable contribution to the meta-RL field.

The evidence to be evaluated is:In meta-RL, we consider a distribution of tasks \(p(\mathcal{T})\), with each task \(\mathcal{T}\) modelled as a Markov Decision Process (MDP), which consists of a state space, an action space, a transition function and a reward function. In common meta-RL settings (Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019), tasks differ in the transition and/or reward function, and we can describe a task \(\mathcal{T}\) with a tuple \(\langle p_{0}^{T}(s_{0}),p^{T}(s^{\prime}|s,a),r^{\mathcal{T}}(s,a)\rangle\), with \(p_{0}^{T}(s_{0})\) the initial state distribution, \(p^{T}(s^{\prime}|s,a)\) the transition probability and \(r^{\mathcal{T}}(s,a)\) the reward function.

A common objective for meta-RL is to optimize final performance after few-shot adaptation (Finn et al., 2017; Gupta et al., 2018; Stadie et al., 2018; Rothfuss et al., 2019). During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.

The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

\[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\] Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

Besides, we are also interested in its sample efficiency in meta-training and adaptation.

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

PEARL fails to effectively explore within short adaptation, as posterior sampling (Osband and Van Roy, 2017) is only optimal in asymptotic performance. RL\({}^{2}\) achieves good performance in tasks with simple dynamics like Point-Robot-Sparse and Cheetah-Vel-Sparse, but fails in more complex tasks. MAML acquires good performance in Walker-Params-Sparse but fails to learn tasks with different reward functions.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

Measuring information gain is an important way of designing intrinsic rewards for exploration. VIME (Houthooft et al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim et al., 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space.

Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Input: A set of meta-training tasks \(\{\mathcal{T}^{i}\}_{i=1,2,...,T}\) drawn from \(p(\mathcal{T})\)  Initialize replay buffers \(\mathcal{B}^{i},\mathcal{B}^{i}_{enc}\) for each task \(\mathcal{T}^{i}\) while not done do for each task \(\mathcal{T}^{i}\)do  Initialize experience context \(c_{i}=\{\}\) for episodes=1,...,\(K_{1}\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c_{i}))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c_{i}\), \(\mathcal{B}^{i}\) and \(\mathcal{B}^{i}_{enc}\) endfor for episodes=1,...,\(K_{2}\)do  Sample \(z\sim q_{\phi}(z|c_{i})\)  Collect a trajectory with \(\pi(a|s,z)\), add collected experience to \(\mathcal{B}^{i}\) endfor for for steps in training steps do for each \(\mathcal{T}^{i}\)do  Sample encoder batch \(b^{i}_{enc}\) and policy training batch \(b^{i}\)  Sample \(z\sim q_{\phi}(z|b^{i}_{enc})\), compute \(r^{e}\) with Equation 5  Train encoder \(q_{\phi}\), \(\pi\) and \(\pi^{e}\) using SAC, \(b^{i}\) and \(r^{e}\), train the predictor to minimize prediction loss endfor endfor endwhile

Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.

While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, this claim is partially supported but somewhat overstated. While the paper presents an approach using information-theoretic intrinsic rewards for meta-RL exploration, the results show mixed performance compared to baselines. The evidence demonstrates that MetaCURE can enable efficient exploration in some scenarios, particularly for sparse reward settings. However, the claim's broad statement about "enabling efficient exploration in both meta-training and adaptation" overstates the consistency and generality of the results, as performance varies across different tasks and the benefits are most pronounced in specific scenarios like sparse rewards.

<score>3</score>

Justification:
- The results show MetaCURE can outperform baselines in sparse reward settings
- Performance benefits are inconsistent across different tasks
- Limited comparison to other intrinsic motivation approaches
- Results suggest benefits are more task-specific than generally applicable
- Review comments highlight that some key claimed benefits need stronger empirical support


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Inspired by the common task-inference component in context-based meta-RL algorithms (such as PEARL (Rakelly et al., 2019) and VariBAD (Zintgraf et al., 2019)), we leverage an insight that exploration behaviors should collect trajectories that contain rich information gain about the current task, and design an exploration objective to maximize the information gain for inferring tasks.

The evidence to be evaluated is:In common meta-RL settings (Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019), tasks differ in the transition and/or reward function, and we can describe a task \(\mathcal{T}\) with a tuple \(\langle p_{0}^{T}(s_{0}),p^{T}(s^{\prime}|s,a),r^{\mathcal{T}}(s,a)\rangle\), with \(p_{0}^{T}(s_{0})\) the initial state distribution, \(p^{T}(s^{\prime}|s,a)\) the transition probability and \(r^{\mathcal{T}}(s,a)\) the reward function.

During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

In order to learn effective task encodings, its decoder is designed to recover the action value function of the exploiter, which captures rich and temporally-extended information about the current task.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.

The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

These tasks (except for Point-Robot-Sparse) are simulated via MuJoCo (Todorov et al., 2012) and are benchmarks commonly used by current meta-learning algorithms (Mishra et al., 2017; Finn et al., 2017; Rothfuss et al., 2019; Rakelly et al., 2019). Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration. Also, we do not provide dense rewards while meta-training, which is different from the setting of PEARL.

We compare our algorithm against several representative meta-RL algorithms, including RL2(Duan et al., 2016), MAML (Finn et al., 2017), E-RL2(Stadie et al., 2018), ProMP (Rothfuss et al., 2019), MAME (Gurumurthy et al., 2019), PEARL (Rakelly et al., 2019) and VariBAD (Zintgraf et al., 2019).

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5.

PEARL fails to effectively explore within short adaptation, as posterior sampling (Osband and Van Roy, 2017) is only optimal in asymptotic performance.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

We test MetaCURE-PE as well as baselines on two Meta-World task sets: Push and Reach. Although these task sets are not very distinct, they are still very difficult to learn due to their complexity in dynamics and long time-horizon. To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4. We compare against PEARL, which explores via posterior sampling. While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

MetaCURE outperforms these two baselines by performing task inference and separating exploration and exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information.

Most context-based methods automatically balances between exploration and exploitation during adaptation by maximizing average adaptation performance (Duan et al., 2016; Zintgraf et al., 2019), and E-RL\({}^{2}\)(Stadie et al., 2018) directly optimizes for final performance.

PEARL (Rakelly et al., 2019), a recently proposed approach, utilizes posterior sampling (Osband and Van Roy, 2017) to explore, and is less efficient with short adaptation, as posterior sampling is only optimal is asymptotic performance.

Measuring information gain is an important way of designing intrinsic rewards for exploration. VIME (Houthooft et al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim et al., 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors.

In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

```

Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor

\(\beta\) is a hyper-parameter controlling the weight of KL divergence in the encoder's loss, similar to Rakelly et al. (2019).

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.

While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find that the inspiration from PEARL and VariBAD for designing the information gain exploration objective is reasonably well-supported. The evidence shows:

1. Clear demonstration of information gain maximization through theoretical formulation and empirical results
2. Direct comparisons with PEARL and VariBAD showing improved exploration effectiveness
3. Ablation studies validating the benefits of intrinsic rewards for exploration
4. Visualizations confirming more efficient exploration behaviors compared to posterior sampling

The claim is carefully worded with appropriate scope and acknowledgment of inspiration sources, while being grounded in detailed experimental validation. This indicates a well-stated claim without exaggeration.

<score>1</score>

Justification:
1. The evidence demonstrates clear theoretical and empirical connections to PEARL/VariBAD's task inference components
2. Results show improved exploration over baseline methods
3. Ablation studies verify the benefits of the proposed approach
4. The claim avoids overstating novelty by acknowledging inspiration from prior work
5. Visualization evidence directly supports the claimed exploration benefits


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Inspired by the common task-inference component in context-based meta-RL algorithms (such as PEARL (Rakelly et al., 2019) and VariBAD (Zintgraf et al., 2019)), we leverage an insight that exploration behaviors should collect trajectories that contain rich information gain about the current task, and design an exploration objective to maximize the information gain for inferring tasks.

The review comment to be evaluated is:

This paper introduces the use of intrinsically motivated exploration in meta-training and adaptation phase in meta Reinforcement Learning. The proposed framework consists of using task-inference, already used in other works like PEARL and VariBAD, and two deep neural networks, trained with an arbitrary off-policy algorithm, respectively for exploitation and exploration phases.

I think the paper is well-written and discusses the related literature sufficiently well. The empirical results are significantly better than the considered baselines.
Despite these good qualities, I think the major weakness of this paper is the lack of contribution. It looks to me that the major contribution discussed by the authors is the use of intrinsically motivated exploration in meta-training and adaptation, while the use of task-inference has been already proposed in other works and the use of different policy networks for exploitation and exploration is trivial. Considering the intrinsically motivated exploration, I miss the importance of the contribution, since it is already well-known that this technique can be beneficial, especially in sparse MDPs. The intrinsic reward makes sense, but it is not novel, as similar intrinsic rewards have been proposed in VIME and Pathak et al. (2017).
Experimental results are good, but not well described. For example, I don't find a clear explanation of the tasks used in meta-training. The random seeds are only three, which makes the results less reliable.
Considering this, I don't consider this paper significant enough to be accepted.

Minor issues:
* Pag. 1, typo: "tasks" -> tasks
* Pag. 1, "Based this objective" -> Based on this objective
* the R function in (1) is not formally defined.

Pros
------
* Well-written;
* Good discussion of related literature;
* Good experimental results.

Cons
-------
* Lack of contribution;
* Few, but important, missing details about the experimental setting, and too few seeds.

Post-rebuttal feedback
-------------------------------
I thank the author for their reply and I encourage them to make the suggested improvements to the paper.

The evidence to be evaluated is:In common meta-RL settings (Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019), tasks differ in the transition and/or reward function, and we can describe a task \(\mathcal{T}\) with a tuple \(\langle p_{0}^{T}(s_{0}),p^{T}(s^{\prime}|s,a),r^{\mathcal{T}}(s,a)\rangle\), with \(p_{0}^{T}(s_{0})\) the initial state distribution, \(p^{T}(s^{\prime}|s,a)\) the transition probability and \(r^{\mathcal{T}}(s,a)\) the reward function.

During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

In order to learn effective task encodings, its decoder is designed to recover the action value function of the exploiter, which captures rich and temporally-extended information about the current task.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.

The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

These tasks (except for Point-Robot-Sparse) are simulated via MuJoCo (Todorov et al., 2012) and are benchmarks commonly used by current meta-learning algorithms (Mishra et al., 2017; Finn et al., 2017; Rothfuss et al., 2019; Rakelly et al., 2019). Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration. Also, we do not provide dense rewards while meta-training, which is different from the setting of PEARL.

We compare our algorithm against several representative meta-RL algorithms, including RL2(Duan et al., 2016), MAML (Finn et al., 2017), E-RL2(Stadie et al., 2018), ProMP (Rothfuss et al., 2019), MAME (Gurumurthy et al., 2019), PEARL (Rakelly et al., 2019) and VariBAD (Zintgraf et al., 2019).

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5.

PEARL fails to effectively explore within short adaptation, as posterior sampling (Osband and Van Roy, 2017) is only optimal in asymptotic performance.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

We test MetaCURE-PE as well as baselines on two Meta-World task sets: Push and Reach. Although these task sets are not very distinct, they are still very difficult to learn due to their complexity in dynamics and long time-horizon. To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4. We compare against PEARL, which explores via posterior sampling. While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

MetaCURE outperforms these two baselines by performing task inference and separating exploration and exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information.

Most context-based methods automatically balances between exploration and exploitation during adaptation by maximizing average adaptation performance (Duan et al., 2016; Zintgraf et al., 2019), and E-RL\({}^{2}\)(Stadie et al., 2018) directly optimizes for final performance.

PEARL (Rakelly et al., 2019), a recently proposed approach, utilizes posterior sampling (Osband and Van Roy, 2017) to explore, and is less efficient with short adaptation, as posterior sampling is only optimal is asymptotic performance.

Measuring information gain is an important way of designing intrinsic rewards for exploration. VIME (Houthooft et al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim et al., 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors.

In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

```

Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor

\(\beta\) is a hyper-parameter controlling the weight of KL divergence in the encoder's loss, similar to Rakelly et al. (2019).

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.

While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about leveraging insight from task-inference for information gain exploration appears well-supported. The paper demonstrates through theoretical framework, implementation details, and experimental results (especially Figures 3, 4, 5, and 10) how the exploration policy effectively collects informative trajectories for task inference. The evidence shows clear comparisons between MetaCURE and baselines like PEARL, validating that the approach leads to more efficient exploration. The reviewer's criticism focuses mainly on novelty rather than the validity of this core claim.

<score>1</score>

Justification:
1. The theoretical framework clearly establishes how the information gain objective guides exploration behavior
2. Multiple experiments and visualizations demonstrate the exploration policy collecting task-relevant information
3. Comparative results against PEARL and ablation studies validate the effectiveness of the proposed approach
4. The reviewer's concerns about novelty do not invalidate the accuracy of the specific claim being assessed
5. The evidence comprehensively supports the claim's scope without exaggeration