Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

The subset selection model is trained by meta-learning with a distribution of sets.

The review comment to be evaluated is:

### Summary

The authors present a stochastic algorithm for selecting a subset of a large dataset, while trying to preserve statistics from the original dataset. The algorithm is a two-step process, first selecting a set of "candidates" based on individual features, then filtering to a final subset which may account for interactions between candidate items.

### Strengths

The paper provides empirical experimental results which suggest the superiority of the proposed method against two baselines: Learning to Sample (LTS) and random selection (RS).

Very few grammatical mistakes.

### Concerns and Issues that Need Clarification

1. A significant amount of active learning literature deals with subset selection. However, this paper fails to discuss or even cite any results from the active learning literature. It would be helpful for the authors to compare/contrast their proposed method with existing active learning results. For example, consider
   - [*Active Learning for Convolutional Neural Networks: A Core-Set Approach* by Sener and Savarese (2018)](https://openreview.net/forum?id=H1aIuk-RW)
   - [*Selection via Proxy: Efficient Data Selection for Deep Learning* by Coleman et al. (2019)](https://openreview.net/forum?id=HJg2b0VYDr)
   - [*Submodularity in Data Subset Selection and Active Learning* by Wei et al. (2015)](http://proceedings.mlr.press/v37/wei15.html)
   - [*On Statistical Bias In Active Learning: How and When to Fix It* (2020)](https://openreview.net/forum?id=JiYq3eqTKY) - this paper is a submission to ICLR 2020, so obviously there is no expectation that the authors would be familiar with this work. However, I bring this paper up because it discusses a similar topic of trying to select data points from a larger dataset that provide a reasonable estimate of a loss function on the "population" vs. a "sample."

2. The algorithm, as described, states that individual elements of a dataset $D$ are "possibly represented" as a pair $(x_i, y_i)$. What does "possibly represented" mean? Assuming that the dataset elements are given as a $(x_i, y_i)$ pair, the stochastic subset selection algorithm seems to describe instance selection. However, the paper also claims that the algorithm works for feature selection. What are $D$, $x_i$, and $y_i$ in the feature selection tasks? The paper makes an attempt at clarifying this by stating, "Here, x is 2-dimensional and y is 3-dimensional for RGB images." But this still leaves the reader clueless - why is $x$ only 2-dimensional? What are those 2 dimensions?

3. The end of the introduction claims that the proposed subset selection method "learns to sample a subset from a larger set with linear time complexity," yet Section 3.4 claims that the inference complexity is $O(n) + O(k^2 d/q)$, which is nonlinear if either $k$ or $d$ is chosen to be dependent on $n$. Could the authors please clarify?

4. The authors claim that their method uses meta-learning, but no mention of meta learning is made beyond the "Preliminaries" subsection of the "Approach" section. How is this method a example of meta-learning?

5. The paper provides experimental evidence that the proposed method outperforms other subset selection baselines, such as Learning to Select (LTS) and random selection (RS). However, the paper provides little (if any) intuition or formal justification for why the proposed method performs better. The authors need to explain the motivation behind the proposed algorithm.

6. The writing and figures lack clarity and are often difficult to follow. Some examples:
   - The context for the mathematical expression in (4) is very unclear. Is (4) an objective function? If so, is this expression supposed to be minimized or maximized? And what is the random variable $Z$ supposed to represent? And why is there a period in the middle of the expression?
   - Figure 2 is poorly explained and difficult to understand. I'm assuming that the blue vertical vector is supposed to represent $D_c$. Is that correct? And are the dark shaded boxes supposed to represent the "candidates" and the lightly shaded boxes supposed to represent the items that were not selected? As a reader, I should not have to guess what the colors mean. They should be explained clearly in the caption, or at the very least, in the paper text. Furthermore, it seems to me (again it's never made explicit) that Figure 2b is depicting the Autoregressive Subset Selection ("AutoSelect") routine. In this case, why is there a white row vector of $D = d_1, \dotsc, d_n$? In Algorithm 1, it seems as though AutoSelect only takes as input $D_s$ and $D_c$, which are subsets of $D$ and not $D$ itself.

### Original Rating and Confidence

**Rating** - 4: Ok but not good enough - rejection

**Confidence** - 4: The reviewer is confident but not absolutely certain that the evaluation is correct

### Updated Review after Author Response

Thank you for the thoughtful responses to my questions, and apologies for the delay in updating my review. In general the responses have addressed my questions, and I have updated my rating accordingly. However, there are still several issues in my mind:

1. **Connection to active learning** - The authors correctly point out that active learning does not assume access to labels, whereas the proposed method does. However, I believe the comparison against selection algorithms such as k-center greedy should be included in the paper for completeness sake, and not just left as a comment in this forum.

2. **Dataset Distillation: Instance Selection** - Upon re-reading the paper carefully, I found this subsection in Section 3.6 to be confusing. Why are there multiple $D_i$? The section seems to only talk about one $D_i$.

3. **Set vs. Tensor**: In the new "representation learning" subsection of Section 2, the paper stresses that the model learns a representation of a set, instead of an individual data point. However, many tasks (e.g., image reconstruction) do impose some sort of ordering. The paper lacks clarity about where permutation-invariance is used vs. not used.

4. **Other places that need clarification**:
  - Equation (2): presumably you threshold the output of the sigmoid function, right? Because in Algorithm 1, it seems like $z_i$ are supposed to be boolean-valued.
  - Appendix: there are several missing closing parentheses

The evidence to be evaluated is:Furthermore, our method can be applied both at the instance and dataset level.

However, while VAE learns a compact representation of a _data point_, our model learns a compact representation of a _set_.

In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_. The overall objective is then to minimize the loss function with respect to the subset \(D_{s}\), \(\mathbb{E}_{p(D,|D)}[\ell(.,D_{s})]\). When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\).

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017).

The key to avoiding samples with redundant information in \(D_{s}^{(k)}\) lies in the fact that for each element added to \(D_{s}\), it's selection is conditioned on both \(D_{c}\) and all elements in \(D_{s}^{(i-1)}\).

In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

\[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]+\beta\text{KL}[p(Z|D) ||r(Z)]] \tag{4}\]where \(r(Z)\) is a sparse prior. In our experiments, we set the parameter of the Bernoulli spares prior \(r(Z)\) to either \(0.1\) or \(0.01\) for different levels of sparsity and \(\beta\) is set to \(1.0\) or \(0.001\).

Given \(D_{s}\) and a network \(p_{\theta}(Y|X,D_{s})\) parameterized by \(\theta\), the task is to reconstruct \(D=(X,Y)\). The objective function for this task is given as: \[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(Y|X,D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{5}\] Minimizing this objective ensures that we learn a compact subset(\(D_{s}\)) most representative of \(D\) and \(D_{s}\) can then be used for other tasks.

The objective function for this task is given as: \[\mathbb{E}_{p(D)}[\mathbb{E}_{p(D_{s}|D)}[-\log p_{\theta}(y_{D}|D_{s})]+\beta \text{KL}[p(Z|D)||r(Z)]] \tag{6}\]

For this task, we are given a dataset \(\mathcal{D}=\{D_{1},\dots,D_{n}\}\) where each \(D_{i}\) is a set of data points sampled from the entire dataset. Using CelebA as in illustrative example, some \(D_{i}\) may consist of \(|D_{i}|\) randomly sampled faces from the whole dataset.

We describe a model capable of taking as input \(D_{i}\in\mathcal{D}\) to perform a task such as the reconstruction of all elements in the given dataset. For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

In addition to Equation 7 and the informativity loss, the model is optimized together with the subset selection model already described. When the model is fully optimized, it is applied to the instance selection task on the given dataset. In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances.

We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements.

This is our stochastic subset selection model trained for reconstruction, but then later used for classification. Our model outperforms this variant, showing the effectiveness of training with the target task.

On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection.

The idea of the greedy training algorithm is to train the auto-regressive model to select the best next \(q\) elements from the candidate set to minimize the target loss on the selected samples.

3:\(p(D)\) (distribution of sets)

9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\) 10:\(D^{(j)}_{\theta}\sim p(D^{(j)}_{\theta}|D^{(j)})\) for \(j=1\ldots m\)

12:\(I^{(j)}\sim\) random \(i\)-element subset of \(D^{(j)}_{\theta}\) for \(j=1\ldots m\) 13:\(Q^{(j)}\sim\) select a \(q\)-element subset from \(D^{(j)}_{\theta}\setminus I^{(j)}\) (with the auto-regressive model) 14:\(\theta\leftarrow\theta-\alpha\nabla_{\theta}\frac{1}{m}\sum_{j=1}^{m}\ell( \cdot,I^{(j)}\cup Q^{(j)}),\phi\leftarrow\phi-\alpha\nabla_{\phi}\frac{1}{m} \sum_{j=1}^{m}\ell(\cdot,I^{(j)}\cup Q^{(j)})\)

SSS consists of \(r(D)\), \(\rho(d_{i},r(D))\) and \(f(d,D_{c},D_{s}^{(i-1)}\). We describe the models in this section. For all experiments, \(r(D)\) is implemented as DeepSets. This means that we take the mean of all the samples in a set to obtain the set representation. \(\rho(d_{i},r(D))\) is implemented as a neural network with the following specifications: there are 3 Linear layers each followed by ReLU activation. Also, all inputs are projected into feature space using 3 Linear layers, each followed by ReLU activation.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "The subset selection model is trained by meta-learning with a distribution of sets" appears to be well-stated. The evidence strongly supports this through:

1. Explicit mathematical formulation showing training with distribution p(D) of sets (Equations 4-6)
2. Algorithm details showing minibatch sampling from p(D) (steps 9-14)
3. Clear framework explanation that when "D itself follows a distribution of sets as in the meta-learning framework" the objective becomes E_D[E_p(Ds|D)[ℓ(.,Ds)]]

The claim accurately reflects what's described in the methodology without exaggerating the scope or implications.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address this challenge of meta-RL, we introduce information-theoretic intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation.

The evidence to be evaluated is:In meta-RL, we consider a distribution of tasks \(p(\mathcal{T})\), with each task \(\mathcal{T}\) modelled as a Markov Decision Process (MDP), which consists of a state space, an action space, a transition function and a reward function. In common meta-RL settings (Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019), tasks differ in the transition and/or reward function, and we can describe a task \(\mathcal{T}\) with a tuple \(\langle p_{0}^{T}(s_{0}),p^{T}(s^{\prime}|s,a),r^{\mathcal{T}}(s,a)\rangle\), with \(p_{0}^{T}(s_{0})\) the initial state distribution, \(p^{T}(s^{\prime}|s,a)\) the transition probability and \(r^{\mathcal{T}}(s,a)\) the reward function.

A common objective for meta-RL is to optimize final performance after few-shot adaptation (Finn et al., 2017; Gupta et al., 2018; Stadie et al., 2018; Rothfuss et al., 2019). During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.

The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

\[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\] Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

Besides, we are also interested in its sample efficiency in meta-training and adaptation.

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

PEARL fails to effectively explore within short adaptation, as posterior sampling (Osband and Van Roy, 2017) is only optimal in asymptotic performance. RL\({}^{2}\) achieves good performance in tasks with simple dynamics like Point-Robot-Sparse and Cheetah-Vel-Sparse, but fails in more complex tasks. MAML acquires good performance in Walker-Params-Sparse but fails to learn tasks with different reward functions.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

Measuring information gain is an important way of designing intrinsic rewards for exploration. VIME (Houthooft et al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim et al., 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space.

Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Input: A set of meta-training tasks \(\{\mathcal{T}^{i}\}_{i=1,2,...,T}\) drawn from \(p(\mathcal{T})\)  Initialize replay buffers \(\mathcal{B}^{i},\mathcal{B}^{i}_{enc}\) for each task \(\mathcal{T}^{i}\) while not done do for each task \(\mathcal{T}^{i}\)do  Initialize experience context \(c_{i}=\{\}\) for episodes=1,...,\(K_{1}\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c_{i}))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c_{i}\), \(\mathcal{B}^{i}\) and \(\mathcal{B}^{i}_{enc}\) endfor for episodes=1,...,\(K_{2}\)do  Sample \(z\sim q_{\phi}(z|c_{i})\)  Collect a trajectory with \(\pi(a|s,z)\), add collected experience to \(\mathcal{B}^{i}\) endfor for for steps in training steps do for each \(\mathcal{T}^{i}\)do  Sample encoder batch \(b^{i}_{enc}\) and policy training batch \(b^{i}\)  Sample \(z\sim q_{\phi}(z|b^{i}_{enc})\), compute \(r^{e}\) with Equation 5  Train encoder \(q_{\phi}\), \(\pi\) and \(\pi^{e}\) using SAC, \(b^{i}\) and \(r^{e}\), train the predictor to minimize prediction loss endfor endfor endwhile

Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.

While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on my review of the evidence, the claim about introducing information-theoretic intrinsic motivations appears well-supported. The paper presents detailed theoretical frameworks, equations, and extensive experimental results across multiple tasks showing that their approach enables efficient exploration in both meta-training and adaptation phases. The empirical evidence demonstrates superior performance compared to baselines, particularly in sparse reward settings. The visualizations effectively illustrate how the intrinsic motivations lead to better exploration behaviors. While the claim is broad, the comprehensive evidence justifies its scope and magnitude.

<score>1</score>

Justification:
The claim is well-stated and accurately reflects the evidence without exaggeration because:
- Provides clear mathematical formulation of information-theoretic intrinsic rewards (Equations 3-5)
- Shows empirical results demonstrating improved exploration efficiency and performance
- Includes detailed ablation studies validating the importance of intrinsic motivations
- Features visualizations that directly illustrate the claimed exploration benefits
- Acknowledges limitations in implementation approximations


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address this challenge of meta-RL, we introduce information-theoretic intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation.

The review comment to be evaluated is:

This paper introduces the use of intrinsically motivated exploration in meta-training and adaptation phase in meta Reinforcement Learning. The proposed framework consists of using task-inference, already used in other works like PEARL and VariBAD, and two deep neural networks, trained with an arbitrary off-policy algorithm, respectively for exploitation and exploration phases.

I think the paper is well-written and discusses the related literature sufficiently well. The empirical results are significantly better than the considered baselines.
Despite these good qualities, I think the major weakness of this paper is the lack of contribution. It looks to me that the major contribution discussed by the authors is the use of intrinsically motivated exploration in meta-training and adaptation, while the use of task-inference has been already proposed in other works and the use of different policy networks for exploitation and exploration is trivial. Considering the intrinsically motivated exploration, I miss the importance of the contribution, since it is already well-known that this technique can be beneficial, especially in sparse MDPs. The intrinsic reward makes sense, but it is not novel, as similar intrinsic rewards have been proposed in VIME and Pathak et al. (2017).
Experimental results are good, but not well described. For example, I don't find a clear explanation of the tasks used in meta-training. The random seeds are only three, which makes the results less reliable.
Considering this, I don't consider this paper significant enough to be accepted.

Minor issues:
* Pag. 1, typo: "tasks" -> tasks
* Pag. 1, "Based this objective" -> Based on this objective
* the R function in (1) is not formally defined.

Pros
------
* Well-written;
* Good discussion of related literature;
* Good experimental results.

Cons
-------
* Lack of contribution;
* Few, but important, missing details about the experimental setting, and too few seeds.

Post-rebuttal feedback
-------------------------------
I thank the author for their reply and I encourage them to make the suggested improvements to the paper.

The evidence to be evaluated is:In meta-RL, we consider a distribution of tasks \(p(\mathcal{T})\), with each task \(\mathcal{T}\) modelled as a Markov Decision Process (MDP), which consists of a state space, an action space, a transition function and a reward function. In common meta-RL settings (Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019), tasks differ in the transition and/or reward function, and we can describe a task \(\mathcal{T}\) with a tuple \(\langle p_{0}^{T}(s_{0}),p^{T}(s^{\prime}|s,a),r^{\mathcal{T}}(s,a)\rangle\), with \(p_{0}^{T}(s_{0})\) the initial state distribution, \(p^{T}(s^{\prime}|s,a)\) the transition probability and \(r^{\mathcal{T}}(s,a)\) the reward function.

A common objective for meta-RL is to optimize final performance after few-shot adaptation (Finn et al., 2017; Gupta et al., 2018; Stadie et al., 2018; Rothfuss et al., 2019). During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.

The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

\[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\] Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

Besides, we are also interested in its sample efficiency in meta-training and adaptation.

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

PEARL fails to effectively explore within short adaptation, as posterior sampling (Osband and Van Roy, 2017) is only optimal in asymptotic performance. RL\({}^{2}\) achieves good performance in tasks with simple dynamics like Point-Robot-Sparse and Cheetah-Vel-Sparse, but fails in more complex tasks. MAML acquires good performance in Walker-Params-Sparse but fails to learn tasks with different reward functions.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

Measuring information gain is an important way of designing intrinsic rewards for exploration. VIME (Houthooft et al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim et al., 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space.

Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Input: A set of meta-training tasks \(\{\mathcal{T}^{i}\}_{i=1,2,...,T}\) drawn from \(p(\mathcal{T})\)  Initialize replay buffers \(\mathcal{B}^{i},\mathcal{B}^{i}_{enc}\) for each task \(\mathcal{T}^{i}\) while not done do for each task \(\mathcal{T}^{i}\)do  Initialize experience context \(c_{i}=\{\}\) for episodes=1,...,\(K_{1}\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c_{i}))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c_{i}\), \(\mathcal{B}^{i}\) and \(\mathcal{B}^{i}_{enc}\) endfor for episodes=1,...,\(K_{2}\)do  Sample \(z\sim q_{\phi}(z|c_{i})\)  Collect a trajectory with \(\pi(a|s,z)\), add collected experience to \(\mathcal{B}^{i}\) endfor for for steps in training steps do for each \(\mathcal{T}^{i}\)do  Sample encoder batch \(b^{i}_{enc}\) and policy training batch \(b^{i}\)  Sample \(z\sim q_{\phi}(z|b^{i}_{enc})\), compute \(r^{e}\) with Equation 5  Train encoder \(q_{\phi}\), \(\pi\) and \(\pi^{e}\) using SAC, \(b^{i}\) and \(r^{e}\), train the predictor to minimize prediction loss endfor endfor endwhile

Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.

While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim that "we introduce information-theoretic intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation" appears to be well-supported. The evidence demonstrates:

1. Clear implementation of information-theoretic intrinsic rewards (Equations 3-4)
2. Empirical results showing improved exploration compared to baselines (Figures 2-5)
3. Ablation studies confirming the importance of intrinsic rewards (Figure 9-10)
4. Visualizations demonstrating effective exploration behaviors (Figures 3-4)

While similar intrinsic rewards existed in prior work (e.g., VIME, EPI), the paper's novel application to meta-RL and adaptation is well-substantiated. The claim accurately reflects the evidence without overstatement.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address this challenge of meta-RL, we introduce information-theoretic intrinsic motivations for learning to collect informative trajectories and enable efficient exploration in both meta-training and adaptation.

The review comment to be evaluated is:

This paper proposes a novel meta-learning method, aimed at solving the following problem: at test time, the agent has N episodes to gather information [exploration phase], and we care about its return in the N+1th episode [exploitation phase]. To this end, the authors propose to learn a separate exploration and exploitation policy. The core of the algorithm is to use an exploration bonus for the exploration policy that rewards finding novel trajectories. This should help it to collect valuable information during meta training and meta testing. The exploration policy is only used in the N+1th episode, and  is conditioned on a context vector which is computed from the exploration trajectories.

Pro:
- Very relevant subject.
- The authors bring up an interesting point, that metaRL methods should consider both exploration during meta training and meta testing.
- I think it makes sense to separate exploration and exploitation policy explicitly.
- The authors propose to sparsify common metaRL benchmarks. Evaluating your algorithm on environments that are proposed in the same paper can be tricky (and often means that the empirical results are evaluated more critically), but I think the suggestion of looking at sparse metaRL tasks is useful for the field and should be considered by other researchers as well.
- The authors provide illustrations of the behaviours of the learned policy at meta-test time, which I think is really nice!

Concerns:
- I fee like there's a mismatch between the reward that the explorer gets, and the problem setting that the authors consider. The way I understand it, the two terms in the explorer's reward (Eq (2)) can be helpful during meta-training but I don't see how that translates to meta-test time. Let me try to explain:
 - [First term] The objective of the agent is to maximise the return in the *last* episode (the exploitation episode). Then why is the explorer not maximising the return of the exploitation phase, but instead gets a reward that corresponds to the exploration episodes? Doesn't that contradict the premise that we're not interested at all in the reward that is collected during exploration? Also, what is n (you write "training iterations" - is that just the number of environment steps per task? number of policy updates over the entirety of meta-training? Does this mean this is annealed to zero over training? Maybe then it shouldn't be called $\gamma$ so that it's not confused with the per-episode discount usually used in RL.).
 - [Second term] The reward bonus can be seen as a type of novelty bonus for finding new trajectories. Can the authors clarify why this would help (a) during meta-training and (b) during meta-testing? I can see how it helps during training, but I don't understand why it also helps at test time. At the end of Sec 3.2, you write "the intrinsic reward is estimated on multiple tasks and does not diminish during meta-training". Why not? In Eq 5, does $c_{-1:i-1}$ come from multiple tasks or just one? Why can't this diminish over time, shouldn't our model $p(r,s|x,a,z)$ (dropping the subscripts for brevity) get better over time? Or do you just mean that the weight $\lambda$ is not annealed? And doesn't it have all the information to, in principle, predict r and s exactly (except for aleatoric noise) so then this term reduces to the noise in the environment? Or should the terms in the subscript not match the ones within the expectation? It's also a bit unclear what the expectation is over, the subscript just shows some tuples. What's the probability distribution? Where do the terms in the subscript come from? Are $q$ and $p^\mathcal{T}$ mixed in here? It would help my understanding if these things could be made more explicit.
 - [Missing term?] I understand the above two terms might help during training, but they are only auxiliary tasks in a way. And if the first term is annealed to zero (if that's indeed what's happening?), and the second term reduces to aleatoric noise (is that's indeed what's happening?), what is the exploration policy even doing at the end of training? Should there be a third term, which is the return in the *exploitation* phase, which is the only thing that the exploration policy should try to maximise at meta-test time?
- I think the mathematical formulation has to be fleshed out. The paper is sometimes hard to follow because the notation is unclear, and not all terms are explained. I have a few open questions because of this:
 - What are the expectations in (3), (4), (5) over?
 - Not all terms in Figure 1 are defined or mentioned in the text. Are $L_\pi$ and $L_{Q_\pi}$ loss terms from SAC, so they correspond to equation (1)? And $L_{\pi_e}$ corresponds to equation (2)? And what is $L_{pred}$? All these should be explicitly defined in the text with an equation.
 - Sec 2, directly after Eq (1): $c^\mathcal{T}_{\pi_e}$ is not defined.
- In Figure 2, PEARL (Walker-Vel-Sparse) and VariBAD (Point-Robot-Sparse) seem to first learn and then diverge. Do the authors have an explanation as to why this is? It seems to me that maybe the hyperparameters just weren't set right, and makes me a bit sceptical of the results. When PEARL/VariBAD *do* learn, their (end-) performance is much closer to that of MetaCURE.
- The authors propose two versions of MetaCURE (IG and PE). On the continuous control tasks PE is better, but then on the noisy point robot PE completely fails and IG succeeds. I think the authors are not doing themselves a favour by setting up the presentation of their methods like this. It would be better to propose *one* method, and then fully focus on this. The way it is now, there are two methods where neither of them is clearly better. I think that's confusing to the reader, and less convincing. Shouldn't there be a way to develop an exploration bonus that can do well across all environments?
- The "noisy" environment is a POMDP (the agent gets noisy state observations). So doesn't that mean that there is simply a mismatch between the problem setting and the way the policy is set up? I'm not sure that this is a good experiment to add, because the solution (using the PE explorer) does not directly address the problem (partial observability). Maybe a noisy reward function would fit better here?

Other questions:
- Can you explain why the same exploration mechanism useful during meta-training and meta-testing? Aren't we mixing across-task and within-task information gain, and if so, why is that okay?
- What is the difference between row 1 and row 2 in Figure 2? This is not explained in the text. Is that separation just because everything in one figure would've been too much? And how exactly were the baselines evaluated, always after the same number of episodes?
- Why is the loss of the explorer not backpropagated through the context encoder? How does that fit together with "our intrinsic reward supports end-to-end learning and collect online data for effective task inference"?
- Why does the exploiter get a sample $z$ and not the entire distribution $q$? Is that because we assume that all information is there and $q$ is concentrated anyway?
- You mention that the intrinsic reward of PE is of lower variance. Could you explain why? Is that a hypothesis (if so, say so explicit) or do you have any empirical/theoretical evidence for this (again, if so, make it explicit).

Suggestions:
- You write that "efficient exploration of meta-RL needs to consider both training and adaptation phases simultaneously". This is a good point and I agree with it, and I think the paper can be improved by fleshing this argument out more. Help the reader understand why this is the case, and what the (different) challenges in exploration are during training and testing. One open question I still have for example (see above) is why the *same* exploration mechanism is useful during meta-training and meta-testing. Why not separate them? This should be explained in the paper. In the related work section, you write that prior methods "utilise policy injected with random noise for meta-training exploration, which is ineffective and empirically fail with sparse rewards". This point can be made about almost all prior methods, and should come in the introduction. Then explain intuitively *why* this fails, maybe which an example. Another question I have is, how does the Ishii et al. 2002 paper back up this point that this distinction exists?
- The experiments section relies too much on the appendix, and it's quite unnatural to read it like this. Either I have to jump back and forth between the appendix and the main text, or I have to take your word for what is shown in the results in the appendix. You introduce five continuous control tasks at the start, but then actually only show 3 and the other 2 are in the appendix. I think this section can be improved by focusing on the main important points in the main text, explain this well and mention that more results can be found in the appendix.
- Mention in the main text that the sparse setting for Meta-World means you use the success signal only. This is a very difficult setting and is worth highlighting instead of burying it in the appendix.
- I think a discussion about what an good/optimal exploration strategy would look like can help the reader understand the problem setting, and what you are trying to achieve. Right now, in Sec 4.3 you write for example that PEARL "explored less effectively" because it keeps the belief about the current task throughout an entire episode. That's not a fundamental problem with Thompson-sampling like methods though, but is quite problem-specific. If the number of adaptation episodes that are available corresponds to the number of possible tasks, then Thompson sampling always does optimal exploration. But, if the number of possible tasks exceeds the number of exploration episodes Thompson sampling fails. Adding discussions like these to understand what type exploration *is* necessary and why your method can achieve it, could strengthen the paper.

Typos:
- Intro, Paragraph 3: taskss -> tasks; Based -> Based on
- Background, Paragraph 2: Full stop missing at end
- 3.2, second paragraph + last paragraph: starts with free space
- Figure 2, caption, writes "five" tasks but there are only three (other two in appendix)
- Sec 4 (intro): 3. What's -> What are; 4. Is MetaCURE's -> Are MetaCURE's

------------------------------------------------------------------------------------
UPDATE

Thanks for your detailed response!

A1: I see, that makes more sense to me now. I'm still not 100% convinced that it might not be better to let the policy entirely meta-learn what a good exploration policy is, and only use the inductive bias (terms 1 and 2) for how to do so for meta-training and anneal those terms over time. This would mean that at test time the only thing that the exploration policy should do is maximise the return in the *exploitation* phase - and basically figure out what the best way to do so is (so you'd need the "missing term" I describe above). But that's just a hunch, not sure if that would actually work better.

A2-A8: Thanks a lot for clarifying.

I'll keep my current score given all reviewers agree that the work is unpolished in its current form, and because the authors plan to propose a new version of MetaCURE soon. I think this is very interesting and promising work and look forward to reading an updated version in the future!

The evidence to be evaluated is:In meta-RL, we consider a distribution of tasks \(p(\mathcal{T})\), with each task \(\mathcal{T}\) modelled as a Markov Decision Process (MDP), which consists of a state space, an action space, a transition function and a reward function. In common meta-RL settings (Duan et al., 2016; Finn et al., 2017; Zintgraf et al., 2019; Rakelly et al., 2019), tasks differ in the transition and/or reward function, and we can describe a task \(\mathcal{T}\) with a tuple \(\langle p_{0}^{T}(s_{0}),p^{T}(s^{\prime}|s,a),r^{\mathcal{T}}(s,a)\rangle\), with \(p_{0}^{T}(s_{0})\) the initial state distribution, \(p^{T}(s^{\prime}|s,a)\) the transition probability and \(r^{\mathcal{T}}(s,a)\) the reward function.

A common objective for meta-RL is to optimize final performance after few-shot adaptation (Finn et al., 2017; Gupta et al., 2018; Stadie et al., 2018; Rothfuss et al., 2019). During adaptation, an agent first utilizes some exploration policy \(\pi^{e}\) to explore for a few episodes, and then updates an exploitation policy \(\pi\) to maximize expected return.

To support efficient exploration in both meta-training and adaptation, we propose MetaCURE, a novel off-policy meta-RL algorithm, that learns separate exploration and exploitation policies. The exploration policy aims to collect trajectories that maximize the agent's information gain to reduce uncertainty of task inference.

In this section, we first present the MetaCURE framework and then discuss its intrinsic reward design for learning an efficient exploration policy for both meta-training and adaptation. As shown in Figure 1, MetaCURE is composed of three main components: (i) a task encoder \(q_{\phi}(z|c)\) that extracts information from context \(c\) and estimates the posterior of the task belief \(z\), (ii) an _Explorer_ to learn an behavior or exploration policy, and (iii) an _Exploiter_ to learn the target or exploitation policy.

During the adaptation phase, only the explorer is used to collect informative trajectories for inferring the posterior task belief, and the exploiter utilizes this posterior belief to maximize final performance.

The intrinsic reward is conditioned on past trajectories \(c\), which direct the explorer to collect relevant information for task inference. Empirically, we utilize the task belief \(q_{\phi}(z|c)\) instead of context \(c\) as explorer's input, because it extracts useful task-specific information.

As training proceeds, the extrinsic reward gradually diminishes, and the explorer converges to behaviors that collects useful experiences for adaptation.

As shown in Figure 1, to support efficient exploration in both meta-training and adaptation, we maximize agent's information gain during exploration, and introduce an information-theoretic intrinsic reward in the following form: \[r^{\prime}_{IG}(c_{-1:i-1},a_{i})=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i })}[D_{KL}[p(z|c_{-1:i-1})||p(z|c_{-1:i})]] \tag{3}\] The left-hand term \(p(z|c_{-1:i-1})\) in the KL distance is the agent's task belief before observing \(c_{i}\), while the right-hand term is the posterior task belief. This term reflects how much the agent's belief has changed after collecting an experience.

\[\begin{split}& r^{\prime}_{IG}(c_{-1:i-1},a_{i})\\ &=\mathbb{E}_{(r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[D_{KL}[p(z|c_{ -1:i-1})||p(z|c_{-1:i})]]\\ &=\mathbb{E}_{(z,r_{i},s_{i+1})|(c_{-1:i-1},a_{i})}[\log p(r_{i },s_{i+1}|c_{-1:i-1},a_{i})-\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)]\end{split} \tag{4}\] Although sharing some similarities with EPI (Zhou et al., 2018), our intrinsic reward possesses several key differences: first, it maximizes both information gains in reward and dynamics prediction, while EPI ignores reward signals that are often critical for task inference in meta-RL; secondly, our intrinsic reward supports end-to-end learning and collect online data for effective task inference, while EPI requires a fixed dataset to compute its intrinsic reward and lacks an effective mechanism for adaptive exploration.

Unlike the prediction-based curiosity in traditional RL, this intrinsic reward is estimated on multiple tasks, and does not diminish during meta-training.

Besides, we are also interested in its sample efficiency in meta-training and adaptation.

We evaluate our algorithm on various continuous control task sets with sparse rewards, in which exploration is vital for good performance.

Unlike previous evaluation settings, we limit the length of adaptation phase to address the importance of efficient adaptation exploration.

MetaCURE-IG utilizes the information-gain intrinsic in Equation 4, and MetaCURE-PE utilizes the prediction error intrinsic in Equation 5. For both variants, the exploitation policy is only performed in the last adaptation episode, and the exploration policy is used for other adaptation episodes.

PEARL fails to effectively explore within short adaptation, as posterior sampling (Osband and Van Roy, 2017) is only optimal in asymptotic performance. RL\({}^{2}\) achieves good performance in tasks with simple dynamics like Point-Robot-Sparse and Cheetah-Vel-Sparse, but fails in more complex tasks. MAML acquires good performance in Walker-Params-Sparse but fails to learn tasks with different reward functions.

As for sample efficiency, MetaCURE and PEARL enjoy higher sample efficiency as they are off-policy trained, needing far fewer samples to converge.

To address the importance of exploration, we make the rewards sparse, which makes the tasks even harder.

MetaCURE achieves significantly higher success rates than baselines by achieving efficient exploration. To prove that MetaCURE indeed learns efficient exploration and exploitation strategies, we visualize MetaCURE-PE's adaptation phase in Point-Robot-Sparse and Walker-Vel-Sparse, as shown in Figure 3 and Figure 4.

While MetaCURE effectively explores before carrying out good exploitation behaviors, PEARL keeps its belief about current task throughout an entire episode, and explores less effectively.

We test MetaCURE with two intrinsic rewards on this task set, and performance and visualizations are shown in Figure 5. While MetaCURE-PE is interested in distinct regions where the prediction loss is large and fails to explore possible goals, MetaCURE-IG manages to effectively collect information about task and ignores irrelevant noises in dynamics.

The intrinsic rewards are vital for exploration in both meta-training and adaptation. MetaCURE without intrinsic rewards cannot explore effectively during meta-training, as shown in Appendix A.9.1.

As shown in Appendix A.9.2, while MetaCURE with intrinsic rewards is able to perform good exploration behaviors that benefit final adaptation performance, MetaCURE without intrinsic rewards is attracted by the sub-optimal goal and cannot acquire necessary information for final exploitation.

Compared to meta supervised learning, in meta-RL the agent is not given a task-specific dataset to adapt to, and it must explore the environment to collect useful information. This exploration part is vital for both meta-training and adaptation (Finn and Levine, 2019).

Measuring information gain is an important way of designing intrinsic rewards for exploration. VIME (Houthooft et al., 2016) measures the mutual information between trajectories and the transition function, while EMI (Kim et al., 2019) measures the mutual information between state and action with the next state in consideration, as well as the mutual information between state and next state with the action in consideration, both in the latent space.

Prediction loss can serve as a kind of curiosity-driven intrinsic reward to encourage exploration. Intuitively, high prediction loss implies that the situation has not been fully explored, and visiting these situations helps the agent to explore.

In this paper, we propose a novel off-policy meta-RL algorithm to facilitate meta-training and adaptation exploration, which becomes critical for sparse reward signals. By introducing information-theoretic intrinsic motivations and separating exploration and exploitation policies, our method is able to acquire sophisticated exploration behaviors. Our algorithms of two intrinsic rewards both achieve the state-of-the-art performance on various sparse-reward MuJoCo locomotion task sets and more difficult Meta-World tasks.

The intrinsic rewards proposed in Equation 4 and Equation 5 requires estimating density functions \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i},z)\) and \(\log p(r_{i},s_{i+1}|c_{-1:i-1},a_{i})\), which is difficult in continuous state and action spaces. In implementation, we estimate them with prediction errors, which is commonly used in density estimation (Chung et al., 2015; Babaeizadeh et al., 2017; Ratzlaff et al., 2019): \[\begin{split}& r_{IG}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}\\ &-|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i}),s_{pred}(c_{-1:i-1},a_{i}))|_{2}]\\ & r_{PE}^{\prime}(c_{-1:i-1},a_{i})\\ &\approx\mathbb{E}_{(z,r_{i},s_{i+1})(|c_{-1:i-1},a_{i})}[|(r_{i},s_{i+1})-(r_{pred}(c_{-1:i-1},a_{i},z),s_{pred}(c_{-1:i-1},a_{i},z))|_{2}]\\ \end{split} \tag{10}\]

Input: A set of meta-training tasks \(\{\mathcal{T}^{i}\}_{i=1,2,...,T}\) drawn from \(p(\mathcal{T})\)  Initialize replay buffers \(\mathcal{B}^{i},\mathcal{B}^{i}_{enc}\) for each task \(\mathcal{T}^{i}\) while not done do for each task \(\mathcal{T}^{i}\)do  Initialize experience context \(c_{i}=\{\}\) for episodes=1,...,\(K_{1}\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c_{i}))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c_{i}\), \(\mathcal{B}^{i}\) and \(\mathcal{B}^{i}_{enc}\) endfor for episodes=1,...,\(K_{2}\)do  Sample \(z\sim q_{\phi}(z|c_{i})\)  Collect a trajectory with \(\pi(a|s,z)\), add collected experience to \(\mathcal{B}^{i}\) endfor for for steps in training steps do for each \(\mathcal{T}^{i}\)do  Sample encoder batch \(b^{i}_{enc}\) and policy training batch \(b^{i}\)  Sample \(z\sim q_{\phi}(z|b^{i}_{enc})\), compute \(r^{e}\) with Equation 5  Train encoder \(q_{\phi}\), \(\pi\) and \(\pi^{e}\) using SAC, \(b^{i}\) and \(r^{e}\), train the predictor to minimize prediction loss endfor endfor endwhile

Input: Meta-test task \(\mathcal{T}\) drawn from \(p(\mathcal{T})\)  Initialize experience context \(c=\{\}\) for episodes=1,...,\(N-1\)do for steps=1,...,\(H\)do  Take action according to \(\pi^{e}(a|s,q_{\phi}(z|c))\)  Add collected experience \((s,a,r,s^{\prime})\) to \(c\) endfor  Sample \(z\sim q_{\phi}(z|c)\) for steps=1,2,...,\(H\)do  Take action according to \(\pi(a|s,z)\) endfor

Without intrinsic rewards, MetaCURE cannot perform good meta-training exploration and leads to poor performance.

While the explorer without intrinsic rewards keeps going to the sub-optimal goal to maximize average return and fails to collect useful information, the intrinsically motivated explorer effectively explores possible goals and manages to find the real goal, thus attaining higher final performance.

Input: Figure 3: Adaptation visualization of MetaCURE-PE and PEARL on Point-Robot-Sparse. The agent is given four episodes to perform adaptation. Trajectories in dark purple are trajectories of the first adaptation episode, while the light yellow trajectories are trajectories of the last episode. Dark blue circles indicate rewarding regions for current task, while light blue circles indicate rewarding regions for other meta-testing tasks. PEARL explores by rolling out exploitation policies. MetaCURE-PE is intrinsically motivated to gather task-specific knowledge, efficiently exploring possible goals and achieving good exploration and exploitation behaviors.

Input: Figure 4: Visualization of MetaCURE-PE and PEARL on the Walker-Vel-Sparse tasks. The solid red line is the target velocity, and the region bounded by red dash lines represents velocities that get informative rewards. While PEARL keeps its belief unchanged during an entire episode, MetaCUREPE first efficiently explores the goal velocity before performing good exploitation policies.

Input: Figure 5: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-Sparse-Noise. (b) Adaptation visualization of MetaCURE-IG. It is motivated to gain task information, and is not disturbed by noises in dynamics. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as distinct regions obtain nosier dynamics and larger prediction losses.

Input: Figure 1: MetaCURE’s meta-training pipeline (with prediction error intrinsic). Sharing buffers for two policies achieves high sample efficiency as both policies can be off-policy trained with history experiences. Batches from B and Benc are sampled from the same task. πe’s Q function is omitted for brevity.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input: Figure 8: (a) Learning curves of MetaCURE-IG and MetaCURE-PE on Point-Robot-SparseRandom. (b) Adaptation visualization of MetaCURE-IG. The agent gets transmitted when it touches the green circle. (c) Adaptation visualization of MetaCURE-PE. It fails to obtain good exploration behaviors, as it is interested in the randomness of being transmitted.

Input: Figure 6: Evaluation of MetaCURE and several meta-RL baselines on Reacher-Goal-Sparse and Walker-Rand-Params.

Input: Figure 7: Visualization of MetaCURE and PEARL on the Cheetah-Vel-Sparse tasks.

Input: Figure 2: Evaluation of MetaCURE and several meta-RL baselines on various sparse-reward continuous control task sets. MetaCURE achieves superior final performance as well as high sample efficiency, while none of the baseline algorithms manage to solve all five tasks. MetaCURE-PE slightly outperforms MetaCURE-IG.

Input: Figure 9: MetaCURE without intrinsic rewards cannot explore effectively during meta-training and performs poorly.

Input: Figure 10: (a) Point-Robot-Sparse with a sub-optimal goal. The sub-optimal goal is illustrated in green, which requires no adaptation exploration but offers lower returns. The cross indicates the agent’s initial position. Blue circles indicate optimal goals. (b) Learning curves of MetaCURE w/o intrinsic rewards on the illustrated task set. (c) Adaptation visualization of MetaCURE with intrinsic rewards (curiosity intrinsic). MetaCURE finds the optimal goal, and leads to higher final performance. (d) Adaptation visualization of MetaCURE without intrinsic rewards. Explorer without intrinsic rewards maximizes average extrinsic return, leading to inefficient exploration.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I carefully evaluated the claim regarding information-theoretic intrinsic motivations for meta-RL exploration against the extensive evidence provided.

The evidence strongly supports that the authors developed two variants of intrinsic rewards (IG and PE) that enable effective exploration in both meta-training and adaptation phases, as demonstrated through:
- Clear empirical results showing superior performance on sparse-reward tasks
- Visualizations illustrating targeted exploration behaviors
- Ablation studies confirming the necessity of intrinsic rewards
- Theoretical grounding in information theory
- Detailed mathematical formulation and implementation

The claim appears well-stated and accurately reflects the key contribution without exaggerating its scope or impact. The authors acknowledge limitations and provide thorough experimental validation.

<score>1</score>