Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We here demonstrate the surprising power of a simple approach: Continual Learning with Experience And Replay (CLEAR).

The review comment to be evaluated is:

The authors propose an approach to augment experience replay buffers with properties that can alleviate issues with catastrophic forgetting. The buffers are augmented by storing both new and historical experiences, along with the desired historical policy & value distribution. The AC learning now couples two additional losses that ensures the new policy does not drift away from old actor distribution (via KL) and new value does not drift away from old critic distribution (via L2 loss).

The authors provided clear experimental evidence that shows how an RL agent that does not use CLEAR will observe catastrophic when we sequentially train different tasks (and it is not due to destructive interference using the simultaneous and separate training/evaluation experiments). Author also showed how different replay make ups can change the result of CLEAR (and it's a matter of empirical tuning).

The formulation of CLEAR also is simple while delivering interesting results. It would have been nice to see how this is used in a practical setting as all these are synthetic environments / tasks. The discussion on relationship with biological mechanism also seems unnecessary as it's unclear whether the mechanism proposed is actually what's in the CLS.

The evidence to be evaluated is:Here, in CLEAR, we demonstrate that a mixture of replay data and fresh experience protects against catastrophic forgetting while also permitting fast learning, and performs better than either pure on-policy learning or pure off-policy learning from replay. We provide a thorough investigation and robust algorithm in CLEAR, conducting a wide variety of tests on both limited-size and unbounded buffers with complex RL tasks using state-of-the-art methods, and improve the stability of simple replay with the addition of behavioral cloning. CLEAR uses actor-critic training on a mixture of new and replayed experiences. In the case of replay experiences, two additional loss terms are added to induce behavioral cloning between the network and its past self. The motivation for behavioral cloning is to prevent network output on replayed tasks from drifting while learning new tasks. We penalize (1) the KL divergence between the historical policy distribution and the present policy distribution, (2) the L2 norm of the difference between the historical and present value functions.

We apply CLEAR in a distributed training context based on the Importance Weighted Actor-Learner Architecture (Espeholt et al., 2018). A single learning network is fed experiences (both novel and \[v_{s}:=V(h_{s})+\sum_{t=s}^{s+n-1}\gamma^{t-s}\left(\prod_{i=s}^{t-1}c_{i} \right)\delta_{t}V,\] where \(\delta_{t}V:=\rho_{t}\left(r_{t}+\gamma V(h_{t+1})-V(h_{t})\right)\), \(c_{i}:=\min(\bar{c},\frac{\pi_{\theta}(a_{i}|h_{i})}{\mu(a_{i}|h_{i})})\), and \(\rho_{t}=\min(\bar{\rho},\frac{\pi_{\theta}(a_{i}|h_{i})}{\mu(a_{i}|h_{i})})\), for constants \(\bar{c}\) and \(\bar{\rho}\).

The loss functions \(L_{\text{value}}\), \(L_{\text{policy-gradient}}\), and \(L_{\text{entropy}}\) are applied both for new and replay experiences. In addition, we add \(L_{\text{policy-cloning}}\) and \(L_{\text{value-cloning}}\) for replay experiences only. In general, our experiments use a 50-50 mixture of novel and replay experiences, though performance does not appear to be very sensitive to this ratio.

Catastrophic forgetting occurs when a task's performance goes down not as a result of incompatibility with another task but as a result of the second task overwriting it within the model. As we aim to illustrate, the two are independent phenomena, and while interference may happen, forgetting is ubiquitous.

By contrast, there is a large difference between either of the above modes of training and sequential training, where performance on a task decays immediately when training switches to another task - that is, catastrophic forgetting.

We here demonstrate the efficacy of CLEAR for diminishing catastrophic forgetting (Figure 2).

Our method effectively eliminates forgetting on all three tasks, while preserving overall training performance (see "Sequential" training in Figure 1 for reference). When the task switches, there is little, if any, dropoff in performance when using CLEAR, and the network picks up immediately where it left off once a task returns later in training. Without behavioral cloning, the mixture of new experience and replay still reduces catastrophic forgetting, though the effect is reduced.

We believe that 50-50 new-replay represents a good tradeoff, combining significantly reduced catastrophic forgetting

Just as Figure 2 shows the importance of behavioral cloning for maintaining past performance on a task, so this experiment shows that off-policy learning can actually increase performance from replay alone. Both ingredients are necessary for the success of CLEAR.

We find that all buffers perform well and conclude that it is possible to learn and reduce catastrophic forgetting even with a replay buffer that is significantly smaller than the total number of experiences.

In this experiment (Figure 5), we find that this is not a problem for CLEAR, relying as it does on a mixture of off- and on-policy learning. Specifically, we find the performance attained on a task is largely independent of the amount of data stored in the buffer and on the identities of the preceding tasks.

Notably, CLEAR succeeds in greatly reducing catastrophic forgetting for all tasks, and the effect on the probe task does not diminish as the probe task is introduced later on in the training sequence.

Finally, we compare our method to Progress & Compress (P&C) (Schwarz et al., 2018) and Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), state-of-the-art methods for reducing catastrophic forgetting that, unlike replay, assume that the boundaries between different tasks are known (Figure 6).

In this case, we use CLEAR with a 75-25 balance of new-replay experience. We find that we obtain comparable performance to P&C and better performance than EWC, despite CLEAR being significantly simpler and agnostic to the boundaries between tasks.

We have described a simple but powerful approach for preventing catastrophic forgetting in continual learning settings. CLEAR uses on-policy learning on fresh experiences to adapt rapidly to new tasks, while using off-policy learning with behavioral cloning on replay experience to maintain and modestly enhance performance on past tasks.

Our method is simple, scalable, and practical; it takes advantage of the general abundance of memory and storage in modern computers and computing facilities. We believe that the broad applicability and simplicity of the approach make CLEAR a candidate "first line of defense" against catastrophic forgetting in many RL contexts.

A training episode (_unroll_) is generated and inserted into the actor's buffer. Reservoir sampling is used (see further details below) if the buffer has reached its maximum capacity. The actor then samples another unroll from the buffer. The new unroll and replay unroll are both fed into a queue of examples that are read by the learner. The actor waits until its last example in the queue has been read before creating another.

Each element of a batch is a pair (new unroll, replay unroll) from the queue provided by actors. Thus, the number of new unrolls and the number of replay unrolls both equal the entire batch size. Depending on the buffer utilization hyperparameter (see Figure 3), the learner uses a balance of new and replay examples, taking either the new unroll or the replay unroll from each pair.

Our replay buffer stores all information necessary for the V-Trace algorithm, namely the input presented by the environment, the output logits of the network, the value function output by the network, the action taken, and the reward obtained. Leveraging the distributed setup, the buffer is split among all actors equally, so that, for example, if the total buffer size were one million across a hundred actors, then each actor would have buffer capacity of ten thousand. All buffer sizes are measured in environment frames (not in numbers of unrolls), in keeping with the \(x\)-axis of our training plots. For baseline experiments, no buffer was used, while all other parameters and the network remained constant. Unless otherwise specified, the replay buffer was capped at half the number of environment frames on which the network is trained. This is by design - to show that even past the buffer capacity, replay continues to prevent catastrophic forgetting. When the buffer fills up, then new unrolls are added by reservoir sampling, so that the buffer at any given point contains a uniformly random sample of all unrolls up until the present time. Reservoir sampling is implemented as in Isele & Cosgun (2018) by having each unroll associated with a random number between 0 and 1. A threshold is initialized to 0 and rises with time so that the number of unrolls above the threshold is fixed at the capacity of the buffer. Each unroll is either stored or abandoned in its entirety; no unroll is partially stored, as this would preclude training.

We observe that, unlike in the original experiment (Figure 5), the performance obtained on the probe task natlab_varying_map_randomized deteriorates markedly as it appears later in the sequence of tasks. For later positions in the sequence, the probe task comprises a smaller percentage of replay experience, thereby impeding purely off-policy learning. This result underlines why CLEAR uses new experience, as well as replay, to allow rapid learning of new tasks.

Input: Figure 5: The DMLab task natlab varying map randomize (brown line) is presented at different positions within a cyclically repeating sequence of three other DMLab tasks. We find that (1) performance on the probe task is independent of its position in the sequence, (2) the effectiveness of CLEAR does not degrade as more experiences and tasks are introduced into the buffer.

Input: Figure 14: Alternative plot of the experiments shown in Figure 6, showing that CLEAR attains comparable or better performance than the more complicated methods Progress & Compress (P&C) and Elastic Weight Consolidation (EWC), which also require information about task boundaries, unlike CLEAR.

Input: Figure 13: Quantitative comparison of the final cumulative performance between baseline (standard training), CLEAR, Elastic Weight Consolidation (EWC), and Progress & Compress (P&C) (see Figure 14 below). Overall, CLEAR performs comparably to or better than P&C, and significantly better than EWC and baseline.

Input: Figure 1: Separate, simultaneous, and sequential training: the x-axis denotes environment steps summed across all tasks and the y-axis episode score. In “Sequential”, thick line segments are used to denote the task currently being trained, while thin segments are plotted by evaluating performance without learning. In simultaneous training, performance on explore object locations small is higher than in separate training, an example of modest constructive interference. In sequential training, tasks that are not currently being learned exhibit very dramatic catastrophic forgetting. (See Appendix C for a different plot of these data.)

Input: Figure 12: Alternative plot of the experiments shown in Figure 4, showing that reduced-size buffers still allow CLEAR to achieve essentially the same performance.

Input: Figure 11: Alternative plot of the experiments shown in Figure 3, comparing performance between using CLEAR with 75-25 new-replay experience, 50-50 new-replay experience, and 100% replay experience. An equal balance of new and replay experience seems to represent a good tradeoff between stability and plasticity, while 100% replay reduces forgetting but lowers performance overall.

Input: Figure 10: Alternative plot of the experiments shown in Figure 2, showing how applying CLEAR when training on sequentially presented tasks gives almost the same results as training on all tasks simultaneously (compare to sequential and simultaneous training in Figure 9 above). Applying CLEAR without behavioral cloning also yields decent results.

Input: Figure 2: Demonstration of CLEAR on three DMLab tasks, which are trained cyclically in sequence. CLEAR reduces catastrophic forgetting so significantly that sequential tasks train almost as well as simultaneous tasks (see Figure 1). When the behavioral cloning loss terms are ablated, there is still reduced forgetting from off-policy replay alone. As above, thicker line segments are used to denote the task that is currently being trained. (See Appendix C for a different plot of these data.)

Input: Figure 6: Comparison of CLEAR to Progress & Compress (P&C) and Elastic Weight Consolidation (EWC). We find that CLEAR demonstrates comparable or greater performance than these methods, despite being significantly simpler and not requiring any knowledge of boundaries between tasks. (See Appendix C for a different plot of these data.)

Input: Figure 3: Comparison between different proportions of new and replay examples while cycling training among three tasks. We observe that with a 75-25 new-replay split, CLEAR eliminates most, but not all, catastrophic forgetting. At the opposite extreme, 100% replay prevents forgetting at the expense of reduced overall performance (with an especially noticeable reduction in early performance on the task rooms keys doors puzzle). A 50-50 split represents a good tradeoff between these extremes. (See Appendix C for a different plot of these data.)

Input: Figure 14: Alternative plot of the experiments shown in Figure 6, showing that CLEAR attains comparable or better performance than the more complicated methods Progress & Compress (P&C) and Elastic Weight Consolidation (EWC), which also require information about task boundaries, unlike CLEAR.

Input: Figure 13: Quantitative comparison of the final cumulative performance between baseline (standard training), CLEAR, Elastic Weight Consolidation (EWC), and Progress & Compress (P&C) (see Figure 14 below). Overall, CLEAR performs comparably to or better than P&C, and significantly better than EWC and baseline.

Input: Figure 12: Alternative plot of the experiments shown in Figure 4, showing that reduced-size buffers still allow CLEAR to achieve essentially the same performance.

Input: Figure 11: Alternative plot of the experiments shown in Figure 3, comparing performance between using CLEAR with 75-25 new-replay experience, 50-50 new-replay experience, and 100% replay experience. An equal balance of new and replay experience seems to represent a good tradeoff between stability and plasticity, while 100% replay reduces forgetting but lowers performance overall.

Input: Figure 10: Alternative plot of the experiments shown in Figure 2, showing how applying CLEAR when training on sequentially presented tasks gives almost the same results as training on all tasks simultaneously (compare to sequential and simultaneous training in Figure 9 above). Applying CLEAR without behavioral cloning also yields decent results.

Input: Figure 8: Quantitative comparison of the final cumulative performance between standard training (“Sequential (no CLEAR)”) and various versions of CLEAR (see Figures 9, 10, 11, and 12 below) on a cyclically repeating sequence of DMLab tasks. We also include the results of training on each individual task with a separate network (“Separate”) and on all tasks simultaneously (“Simultaneous”) instead of sequentially. As described in Section 4.1, these situations represent no-forgetting scenarios and thus present upper bounds on the performance expected in a continual learning setting, where tasks are presented sequentially. Remarkably, CLEAR achieves performance comparable to “Separate” and “Simultaneous”, demonstrating that forgetting is virtually eliminated.

Input: Figure 4: Comparison of the performance of training with different buffer sizes, using reservoir sampling to ensure that each buffer stores a uniform sample from all past experience. We observe only minimal difference in performance, with the smallest buffer (storing 1 in 200 experiences) demonstrating some catastrophic forgetting. (See Appendix C for a different plot of these data.)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim "We here demonstrate the surprising power of a simple approach: Continual Learning with Experience And Replay (CLEAR)" is slightly overstated. While CLEAR shows strong performance in reducing catastrophic forgetting and achieves comparable or better results than more complex methods like P&C and EWC, characterizing it as "surprisingly powerful" overstates its novelty and impact. The evidence demonstrates CLEAR is effective and practical, but it builds on existing concepts of experience replay and behavioral cloning. The method performs well but delivers expected rather than surprising results given its design. Additionally, the review notes that all experiments were conducted in synthetic environments, limiting claims about broader applicability.

<score>3</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

We here demonstrate the surprising power of a simple approach: Continual Learning with Experience And Replay (CLEAR).

The review comment to be evaluated is:

The paper proposes a novel trial to alleviate the catastrophic forgetting for continual learning which is kind a mixture model of on and off-policy. The core concept of the method is utilizing experience replay buffer for all past events with new experience. They mainly worked on their method in the setting of reinforcement learning. In the experiments, they show that the model successfully mitigate the catastrophic forgetting with this behavioral cloning, and has the performance comparable to recent continual learning approaches.

The paper is easy to follow, and the methodology is quite intuitive and straight forward. In this paper, I have several questions.

Q1. I wonder the reason that every tasks are trained cyclically in sequence. And is there any trial to learn each task just once and observe the catastrophic forgetting of them when they have to detain the learned knowledge in a long time without training them again, as does most of visual domain experiments of the other continual learning research.

Q2. In figure 5, I wonder why the natlab_varying_map_ramdomize(probe task) can perform well even they didn’t learn yet. The score of brown line increases nearly 60~70% of final score(after trained) during training the first task. Because the tasks are deeply correlated? or it is just common property of probe task?

Q3. Using reservoir(buffer) to prevent catastrophic forgetting is natural and reasonable. Is there some of quantitative comparison in the sense of memory requirement and runtime? I feel that 5 or 50 million experiences at each task are huge enough to memorize and manage.

Additionally, in the experiment of figure 5, I think it could be much clear with a verification that the probe task  is semantically independent (no interference) over all the other tasks.

Also, it is quite hard to compare the performance of the models just with plots. I expect that it could be much better to show some of quantitative  results(as number).

The evidence to be evaluated is:Here, in CLEAR, we demonstrate that a mixture of replay data and fresh experience protects against catastrophic forgetting while also permitting fast learning, and performs better than either pure on-policy learning or pure off-policy learning from replay. We provide a thorough investigation and robust algorithm in CLEAR, conducting a wide variety of tests on both limited-size and unbounded buffers with complex RL tasks using state-of-the-art methods, and improve the stability of simple replay with the addition of behavioral cloning. CLEAR uses actor-critic training on a mixture of new and replayed experiences. In the case of replay experiences, two additional loss terms are added to induce behavioral cloning between the network and its past self. The motivation for behavioral cloning is to prevent network output on replayed tasks from drifting while learning new tasks. We penalize (1) the KL divergence between the historical policy distribution and the present policy distribution, (2) the L2 norm of the difference between the historical and present value functions.

We apply CLEAR in a distributed training context based on the Importance Weighted Actor-Learner Architecture (Espeholt et al., 2018). A single learning network is fed experiences (both novel and \[v_{s}:=V(h_{s})+\sum_{t=s}^{s+n-1}\gamma^{t-s}\left(\prod_{i=s}^{t-1}c_{i} \right)\delta_{t}V,\] where \(\delta_{t}V:=\rho_{t}\left(r_{t}+\gamma V(h_{t+1})-V(h_{t})\right)\), \(c_{i}:=\min(\bar{c},\frac{\pi_{\theta}(a_{i}|h_{i})}{\mu(a_{i}|h_{i})})\), and \(\rho_{t}=\min(\bar{\rho},\frac{\pi_{\theta}(a_{i}|h_{i})}{\mu(a_{i}|h_{i})})\), for constants \(\bar{c}\) and \(\bar{\rho}\).

The loss functions \(L_{\text{value}}\), \(L_{\text{policy-gradient}}\), and \(L_{\text{entropy}}\) are applied both for new and replay experiences. In addition, we add \(L_{\text{policy-cloning}}\) and \(L_{\text{value-cloning}}\) for replay experiences only. In general, our experiments use a 50-50 mixture of novel and replay experiences, though performance does not appear to be very sensitive to this ratio.

Catastrophic forgetting occurs when a task's performance goes down not as a result of incompatibility with another task but as a result of the second task overwriting it within the model. As we aim to illustrate, the two are independent phenomena, and while interference may happen, forgetting is ubiquitous.

By contrast, there is a large difference between either of the above modes of training and sequential training, where performance on a task decays immediately when training switches to another task - that is, catastrophic forgetting.

We here demonstrate the efficacy of CLEAR for diminishing catastrophic forgetting (Figure 2).

Our method effectively eliminates forgetting on all three tasks, while preserving overall training performance (see "Sequential" training in Figure 1 for reference). When the task switches, there is little, if any, dropoff in performance when using CLEAR, and the network picks up immediately where it left off once a task returns later in training. Without behavioral cloning, the mixture of new experience and replay still reduces catastrophic forgetting, though the effect is reduced.

We believe that 50-50 new-replay represents a good tradeoff, combining significantly reduced catastrophic forgetting

Just as Figure 2 shows the importance of behavioral cloning for maintaining past performance on a task, so this experiment shows that off-policy learning can actually increase performance from replay alone. Both ingredients are necessary for the success of CLEAR.

We find that all buffers perform well and conclude that it is possible to learn and reduce catastrophic forgetting even with a replay buffer that is significantly smaller than the total number of experiences.

In this experiment (Figure 5), we find that this is not a problem for CLEAR, relying as it does on a mixture of off- and on-policy learning. Specifically, we find the performance attained on a task is largely independent of the amount of data stored in the buffer and on the identities of the preceding tasks.

Notably, CLEAR succeeds in greatly reducing catastrophic forgetting for all tasks, and the effect on the probe task does not diminish as the probe task is introduced later on in the training sequence.

Finally, we compare our method to Progress & Compress (P&C) (Schwarz et al., 2018) and Elastic Weight Consolidation (EWC) (Kirkpatrick et al., 2017), state-of-the-art methods for reducing catastrophic forgetting that, unlike replay, assume that the boundaries between different tasks are known (Figure 6).

In this case, we use CLEAR with a 75-25 balance of new-replay experience. We find that we obtain comparable performance to P&C and better performance than EWC, despite CLEAR being significantly simpler and agnostic to the boundaries between tasks.

We have described a simple but powerful approach for preventing catastrophic forgetting in continual learning settings. CLEAR uses on-policy learning on fresh experiences to adapt rapidly to new tasks, while using off-policy learning with behavioral cloning on replay experience to maintain and modestly enhance performance on past tasks.

Our method is simple, scalable, and practical; it takes advantage of the general abundance of memory and storage in modern computers and computing facilities. We believe that the broad applicability and simplicity of the approach make CLEAR a candidate "first line of defense" against catastrophic forgetting in many RL contexts.

A training episode (_unroll_) is generated and inserted into the actor's buffer. Reservoir sampling is used (see further details below) if the buffer has reached its maximum capacity. The actor then samples another unroll from the buffer. The new unroll and replay unroll are both fed into a queue of examples that are read by the learner. The actor waits until its last example in the queue has been read before creating another.

Each element of a batch is a pair (new unroll, replay unroll) from the queue provided by actors. Thus, the number of new unrolls and the number of replay unrolls both equal the entire batch size. Depending on the buffer utilization hyperparameter (see Figure 3), the learner uses a balance of new and replay examples, taking either the new unroll or the replay unroll from each pair.

Our replay buffer stores all information necessary for the V-Trace algorithm, namely the input presented by the environment, the output logits of the network, the value function output by the network, the action taken, and the reward obtained. Leveraging the distributed setup, the buffer is split among all actors equally, so that, for example, if the total buffer size were one million across a hundred actors, then each actor would have buffer capacity of ten thousand. All buffer sizes are measured in environment frames (not in numbers of unrolls), in keeping with the \(x\)-axis of our training plots. For baseline experiments, no buffer was used, while all other parameters and the network remained constant. Unless otherwise specified, the replay buffer was capped at half the number of environment frames on which the network is trained. This is by design - to show that even past the buffer capacity, replay continues to prevent catastrophic forgetting. When the buffer fills up, then new unrolls are added by reservoir sampling, so that the buffer at any given point contains a uniformly random sample of all unrolls up until the present time. Reservoir sampling is implemented as in Isele & Cosgun (2018) by having each unroll associated with a random number between 0 and 1. A threshold is initialized to 0 and rises with time so that the number of unrolls above the threshold is fixed at the capacity of the buffer. Each unroll is either stored or abandoned in its entirety; no unroll is partially stored, as this would preclude training.

We observe that, unlike in the original experiment (Figure 5), the performance obtained on the probe task natlab_varying_map_randomized deteriorates markedly as it appears later in the sequence of tasks. For later positions in the sequence, the probe task comprises a smaller percentage of replay experience, thereby impeding purely off-policy learning. This result underlines why CLEAR uses new experience, as well as replay, to allow rapid learning of new tasks.

Input: Figure 5: The DMLab task natlab varying map randomize (brown line) is presented at different positions within a cyclically repeating sequence of three other DMLab tasks. We find that (1) performance on the probe task is independent of its position in the sequence, (2) the effectiveness of CLEAR does not degrade as more experiences and tasks are introduced into the buffer.

Input: Figure 14: Alternative plot of the experiments shown in Figure 6, showing that CLEAR attains comparable or better performance than the more complicated methods Progress & Compress (P&C) and Elastic Weight Consolidation (EWC), which also require information about task boundaries, unlike CLEAR.

Input: Figure 13: Quantitative comparison of the final cumulative performance between baseline (standard training), CLEAR, Elastic Weight Consolidation (EWC), and Progress & Compress (P&C) (see Figure 14 below). Overall, CLEAR performs comparably to or better than P&C, and significantly better than EWC and baseline.

Input: Figure 1: Separate, simultaneous, and sequential training: the x-axis denotes environment steps summed across all tasks and the y-axis episode score. In “Sequential”, thick line segments are used to denote the task currently being trained, while thin segments are plotted by evaluating performance without learning. In simultaneous training, performance on explore object locations small is higher than in separate training, an example of modest constructive interference. In sequential training, tasks that are not currently being learned exhibit very dramatic catastrophic forgetting. (See Appendix C for a different plot of these data.)

Input: Figure 12: Alternative plot of the experiments shown in Figure 4, showing that reduced-size buffers still allow CLEAR to achieve essentially the same performance.

Input: Figure 11: Alternative plot of the experiments shown in Figure 3, comparing performance between using CLEAR with 75-25 new-replay experience, 50-50 new-replay experience, and 100% replay experience. An equal balance of new and replay experience seems to represent a good tradeoff between stability and plasticity, while 100% replay reduces forgetting but lowers performance overall.

Input: Figure 10: Alternative plot of the experiments shown in Figure 2, showing how applying CLEAR when training on sequentially presented tasks gives almost the same results as training on all tasks simultaneously (compare to sequential and simultaneous training in Figure 9 above). Applying CLEAR without behavioral cloning also yields decent results.

Input: Figure 2: Demonstration of CLEAR on three DMLab tasks, which are trained cyclically in sequence. CLEAR reduces catastrophic forgetting so significantly that sequential tasks train almost as well as simultaneous tasks (see Figure 1). When the behavioral cloning loss terms are ablated, there is still reduced forgetting from off-policy replay alone. As above, thicker line segments are used to denote the task that is currently being trained. (See Appendix C for a different plot of these data.)

Input: Figure 6: Comparison of CLEAR to Progress & Compress (P&C) and Elastic Weight Consolidation (EWC). We find that CLEAR demonstrates comparable or greater performance than these methods, despite being significantly simpler and not requiring any knowledge of boundaries between tasks. (See Appendix C for a different plot of these data.)

Input: Figure 3: Comparison between different proportions of new and replay examples while cycling training among three tasks. We observe that with a 75-25 new-replay split, CLEAR eliminates most, but not all, catastrophic forgetting. At the opposite extreme, 100% replay prevents forgetting at the expense of reduced overall performance (with an especially noticeable reduction in early performance on the task rooms keys doors puzzle). A 50-50 split represents a good tradeoff between these extremes. (See Appendix C for a different plot of these data.)

Input: Figure 14: Alternative plot of the experiments shown in Figure 6, showing that CLEAR attains comparable or better performance than the more complicated methods Progress & Compress (P&C) and Elastic Weight Consolidation (EWC), which also require information about task boundaries, unlike CLEAR.

Input: Figure 13: Quantitative comparison of the final cumulative performance between baseline (standard training), CLEAR, Elastic Weight Consolidation (EWC), and Progress & Compress (P&C) (see Figure 14 below). Overall, CLEAR performs comparably to or better than P&C, and significantly better than EWC and baseline.

Input: Figure 12: Alternative plot of the experiments shown in Figure 4, showing that reduced-size buffers still allow CLEAR to achieve essentially the same performance.

Input: Figure 11: Alternative plot of the experiments shown in Figure 3, comparing performance between using CLEAR with 75-25 new-replay experience, 50-50 new-replay experience, and 100% replay experience. An equal balance of new and replay experience seems to represent a good tradeoff between stability and plasticity, while 100% replay reduces forgetting but lowers performance overall.

Input: Figure 10: Alternative plot of the experiments shown in Figure 2, showing how applying CLEAR when training on sequentially presented tasks gives almost the same results as training on all tasks simultaneously (compare to sequential and simultaneous training in Figure 9 above). Applying CLEAR without behavioral cloning also yields decent results.

Input: Figure 8: Quantitative comparison of the final cumulative performance between standard training (“Sequential (no CLEAR)”) and various versions of CLEAR (see Figures 9, 10, 11, and 12 below) on a cyclically repeating sequence of DMLab tasks. We also include the results of training on each individual task with a separate network (“Separate”) and on all tasks simultaneously (“Simultaneous”) instead of sequentially. As described in Section 4.1, these situations represent no-forgetting scenarios and thus present upper bounds on the performance expected in a continual learning setting, where tasks are presented sequentially. Remarkably, CLEAR achieves performance comparable to “Separate” and “Simultaneous”, demonstrating that forgetting is virtually eliminated.

Input: Figure 4: Comparison of the performance of training with different buffer sizes, using reservoir sampling to ensure that each buffer stores a uniform sample from all past experience. We observe only minimal difference in performance, with the smallest buffer (storing 1 in 200 experiences) demonstrating some catastrophic forgetting. (See Appendix C for a different plot of these data.)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I've carefully reviewed the claim "We here demonstrate the surprising power of a simple approach: Continual Learning with Experience And Replay (CLEAR)" against the evidence provided.

Summary: The evidence strongly supports this claim through extensive experimental results showing CLEAR's effectiveness at preventing catastrophic forgetting while maintaining good performance across multiple tasks. The experiments demonstrate that CLEAR performs comparably or better than more complex methods like P&C and EWC, despite being simpler and not requiring task boundary information. The method works well even with reduced buffer sizes and different ratios of new/replay experiences. While the word "surprising" might seem strong, the evidence shows that this simple approach achieves impressive results, making the claim well-stated and supported by comprehensive empirical evidence.

<score>1</score>