Continual Diffuser (CoD): Mastering Continual Offline Reinforcement Learning with Experience Rehearsal
Jifeng Hu Li Shen Sili Huang Zhejian Yang Hechang Chen Lichao Sun Yi Chang Dacheng Tao
 


















^1,3,4,5,7School of Artificial Intelligence, Jilin University, Changchun, China 

^2JD Explore Academy, Beijing, China 

^6Lehigh University, Bethlehem, Pennsylvania, USA 

^8College of Computing and Data Science, NTU, Singapore 


<https://github.com/JF-Hu/Continual_Diffuser>



§ ABSTRACT 

Artificial neural networks, especially recent diffusion-based models, have shown remarkable superiority in gaming, control, and QA systems, where the training tasks' datasets are usually static.
However, in real-world applications, such as robotic control of reinforcement learning (RL), the tasks are changing, and new tasks arise in a sequential order.
This situation poses the new challenge of plasticity-stability trade-off for training an agent who can adapt to task changes and retain acquired knowledge.
In view of this, we propose a rehearsal-based continual diffusion model, called Continual Diffuser (), to endow the diffuser with the capabilities of quick adaptation (plasticity) and lasting retention (stability).
Specifically, we first construct an offline benchmark that contains 90 tasks from multiple domains.
Then, we train the  on each task with sequential modeling and conditional generation for making decisions.
Next, we preserve a small portion of previous datasets as the rehearsal buffer and replay it to retain the acquired knowledge. 
Extensive experiments on a series of tasks show  can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based methods and other representative baselines on most tasks.
Source code is available at https://github.com/JF-Hu/Continual_Diffuserhere.









§ INTRODUCTION

Artificial neural networks, such as diffusion models, have made impressive successes in decision-making scenarios, e.g., game playing <cit.>, robotics manipulation <cit.>, and autonomous driving <cit.>.
However, in most situations, a new challenge of difficult adaption to changing data arises when we adopt the general strategy of learning during the training phase and evaluating with fixed neural network weights <cit.>.
Changes are prevalent in real-world applications when performing learning in games, logistics, and control systems.
A crucial step towards achieving Artificial General Intelligence (AGI) is mastering the human-like ability to continuously learn and quickly adapt to new scenarios over the duration of their lifetime <cit.>.
Unfortunately, it is usually ineffective for current methods to simply continue learning on new scenarios when new datasets arrive.
They will show a dilemma between storing historical knowledge (stability) in their brains and adapting to environmental changes (plasticity) <cit.>.

Recently, we have noticed that diffusion probabilistic models (DPMs) have emerged as an expressive structure for tackling complex decision-making tasks such as robotics manipulation by formulating deep reinforcement learning (RL) as a sequential modeling problem <cit.>. 
Although recent DPMs have shown impressive performance in robotics manipulation, they, however, usually focus on a narrow setting, where the environment is well-defined and remains static all the time <cit.>, just like we introduce above.  
In contrast, in real-world applications, the environment changes dynamically in chronological order, forming a continuous stream of data encompassing various tasks.
In this situation, it is challenging for the agents to contain historical knowledge (stability) in their brains and adapt to environmental changes (plasticity) quickly based on already acquired knowledge <cit.>.
Thus, a natural question arises: 




Can we incorporate DPMs' merit of high expression and concurrently endow DPMs the ability towards better plasticity and stability in continual offline RL?





Facing the long-standing challenge of plasticity-stability dilemma in continual RL, current studies of continual learning can be roughly classified into three categories.
Structure-based methods <cit.> propose the use of a base model for pertaining and sub-modules for each task so as to store separated knowledge and reduce catastrophic forgetting. 
Regularization-based methods <cit.> propose using auxiliary regularization loss such as L_2 penalty, KL divergence, and weight importance to contain policy optimization and avoid catastrophic forgetting during training.
Rehearsal-based methods <cit.> are considered simple yet effective in alleviating catastrophic forgetting as rehearsal mimics the memory consolidation mechanism of hippocampus replay inside biological systems.
There are many strategies to perform rehearsal. For instance, a typical method is gradient projection <cit.>, which contains the gradients from new data loss as close as to previous tasks, furthest preventing performance decrease.

Although these methods are effective for continual learning, they present limited improvement in continual offline RL because of extra challenges such as distribution shift and value uncertain estimation.
Recently, diffusion-based methods, such as DD and Diffuser <cit.>, propose to resolve the above two extra challenges from sequential modeling and have shown impressive results in many offline RL tasks.
However, they concentrate solely on training a diffuser that can only solve one task, thus showing limitations in real-world applications where training datasets or tasks usually arrive sequentially.
Though recent works, such as MTDIFF <cit.>, consider diffusers as planner or data generators for multi-task RL, the problem setting of their work is orthogonal to ours.

In this view, we take one step forward to investigate diffusers with arriving datasets and find that recent state-of-the-art diffusion-based models suffer from catastrophic forgetting when new tasks arrive sequentially (See Section <ref> for more details.).
To address this issue, we propose “Continual Diffuser” (), which endows the diffuser with the capabilities of quickly adapting to new tasks (plasticity) meanwhile retaining the historical knowledge (stability) with experience rehearsal.
First of all, to take advantage of the potential of diffusion models, we construct an offline RL benchmark that consists of 90 tasks from multiple domains, such as Continual World (CW) and Gym-MuJoCo.
These continual datasets will be released to all researchers soon at the present stage, and we will actively maintain and progressively incorporate more datasets into our benchmark.
Based on the benchmark, we train our method on each task with sequential modeling of trajectories and make decisions with conditional generation in evaluation.
Then, a small portion of each previous task dataset is reserved as the rehearsal buffer to replay periodically to our model.
Finally, extensive experiments on a series of tasks show that  can achieve a promising plasticity-stability trade-off and outperform existing diffusion-based models and other representative continual RL methods on most tasks.
In summary, our contributions are threefold: 

    
  * We construct a continual offline RL benchmark that contains 90 tasks in the current stage, and we will actively incorporate more datasets for all researchers.
    
  * We investigate the possibility of integrating experience rehearsal and diffuser, then propose the Continual Diffuser () to balance plasticity and stability.
    
  * Extensive experiments on a series of tasks show that  can achieve a promising plasticity-stability trade-off and outperform existing baselines on most tasks.










§ RESULTS


In this section, we will introduce environmental settings and evaluation metrics Section <ref> and <ref>.
Then, in Section <ref> and  <ref>, we first introduce a novel continual offline RL benchmark, including the task description and the corresponding dataset statistics, and introduce various baselines.
Finally, in Section <ref> and <ref>, we report the comparison results, ablation study, and parameters sensitivity analysis.




 §.§ Environmental Settings

Following the same setting as prior works <cit.>, we conduct thorough experiments on Continual World and Gym-MuJoCo benchmarks.
In Continual World, we adopt the task setting of CW10 and CW20 where CW20 means two concatenated CW10.
All CW tasks are version v1.
Besides, we also select Ant-dir for evaluation, which includes 40 tasks, and we arbitrarily select
four tasks (tasks-10-15-19-25) for training and evaluation. See Appendix <ref> for more details.



 §.§ Evaluation Metrics

In order to compare the performance on a series of tasks, we follow previous studies <cit.> and adopt the totally average success rate P(ρ) (higher is better), forward transfer FT (higher is better), forgetting F (lower is better), and the total performance P+FT-F (higher is better) as evaluation metrics.

Suppose that we use p_i(ρ) to represent the average success rate on task i at gradient update step ρ and each task train Δ gradient steps, then the total average success rate P(ρ)=∑_i=1^Ip_i(ρ), where p_i(ρ) ∈ [0, 1].
The forward transfer FT denotes the normalized AUC area between the training curve and the reference curve. 
Note that FT_i<1 and it might also be negative.
Mathematically, FT=1/I∑_iFT_i=1/I∑_iAUC_i-AUC_ref,i/1-AUC_ref,i, where we set AUC_ref,i=0.5 and AUC_i=(p_i(i·Δ)+p_i((i+1)·Δ))/2 for simplicity.
The forgetting F_i is defined as the performance decrease between p_i((i+1)·Δ) and p_I-1(I·Δ), thus F=1/I∑_i^IF_i.






 §.§ Novel Benchmark for Continual Offline RL


To take advantage of the potential of diffusion models, we propose a benchmark for continual offline RL (CORL), comprising datasets from 90 tasks, including 88 Continual World tasks and 2 Gym-MuJoCo tasks <cit.>. 
For the Gym-MuJoCo domain, there are 42 environmental variants, which are constructed by altering the agent goals.
In order to collect the offline datasets, we trained Soft Actor-Critic (SAC) on each task for approximately 1M time steps <cit.>.

Continual World <cit.> is a popular testbed that is constructed based on Meta-World <cit.> and consists of realistic robotic manipulation such as Pushing, Reaching, and Door Opening.
CW is convenient for training and evaluating the abilities of forward transfer and forgetting because the state and action space are the same across all tasks.
Firstly, we will define the task-incremental CORL (TICORL), task-incremental CORL (TICORL), and task-incremental CORL (TICORL) <cit.>.
In RL, we call the CL setting CICORL, where the CL tasks are constructed in the same environment with different goals, such as different directions or velocities.
We call the CL setting TICORL, where the CL tasks are indeed different environments but with the same purposes. For instance, the CL settings with the purpose of pushing blocks (e.g., “push wall” and “push mug” tasks in Continual World) in different robotic control tasks formulate the TICORL.
Finally, we can use the tasks of different purposes, such as push, pull, turn, and press blocks, to construct the DICORL.
For example, CW10 and CW20 form the mixed TICORL and DICORL setups because the task sequence contains multiple purposes. 
Additionally, Gym-MuJoCo's 42 environmental variants facilitate constructing a CICORL setup.
Researchers can use these datasets in any sequence or length for CL tasks to test the plasticity-stability trade-off of their methods. 
We also provide multiple quality datasets, such as `medium' and `expert,' in our benchmark.
We list the information statistics of our benchmark in Table <ref> and <ref>, and Figure <ref> and <ref>, where the episodic time limit is set to 200, and the evaluation time step is set to 1M and 0.4M for different qualities datasets.


Ant-dir is an 8-joint ant environment. 
The different tasks are defined according to the target direction, where the agent should maximize its return with maximal speed in the pre-defined direction. 
As shown in Table <ref>, there are 40 tasks (distinguished with “task id”) with different uniformly sampled goal directions in Ant-dir. 
For each task, the dataset contains approximately 200k transitions, where the observation and action dimensions are 27 and 8, respectively.
We found that the Ant-dir datasets have been used by many researchers <cit.>, so we incorporate them into our benchmark.
Moreover, we report the mean return information of each sub-task in Table <ref> and Figure <ref>.
As for Cheetah-dir, it only contains two tasks that represent forward and backward goal directions.
Compared with Ant-dir, Cheetah-dir possesses lower observation and action space.



 §.§ Baselines

We compare our method () with various representative baselines, encompassing structure-based, regularization-based, and rehearsal-based methods.
In structure-based methods, we select LoRA <cit.>, PackNet <cit.>, and Multitask.
For regularization-based methods, we select L2, EWC <cit.>, MAS <cit.>, and VCL <cit.> for evaluation.
Rehearsal-based baselines include t-DGR <cit.>, DGR <cit.>, CRIL <cit.>, A-GEM <cit.>, and IL <cit.>.
Besides, we also include several diffusion-based methods <cit.> and Multitask methods, such as MTDIFF <cit.> for the evaluation.










 §.§ Main Results

Ant-dir Results.   
To show the effectiveness of our method in reducing catastrophic forgetting, we compare our method with other diffusion-based methods on the Ant-dir tasks ordered by 10-15-19-25.
As shown in Table <ref> (d) and Figure <ref>, the results illustrate: 
1) Directly applying previous diffusion-based methods into continual offline RL will lead to severe catastrophic forgetting because the scores of Diffuser-w/o rehearsal and DD-w/o rehearsal are far behind .
2) Extending the technique of LoRA into the diffusion model may not always work. The reason lies in that the parameter quantity size is small, which inspires us to construct diffuser foundation models in future work.
3) Rehearsal can bring significant improvements on diffuser as  approaches the score of Multitask . 




Online Continual World Results.   
Considering that offline datasets prohibit further exploration in the environments, which may hinder the capability of some baselines that are designed for online training.
We conduct CW10 and CW20 experiments of these methods under the online continual RL setting.
Similarly, we constrain the interaction as 500k time steps for each task and report the comparison results in Figure <ref> (a) and Table <ref> (a).
The results show that our method () surpasses other baselines by a large margin, which illustrates the superior performance over balancing plasticity and stability.
Besides, it is indeed that some methods, such as EWC, are more suitable for online training by comparing the performances in Figure <ref> (a) and (b). 
Additionally, we also report the comparison under mixed-quality datasets CL setting in Table <ref> (c).
Please refer to Appendix <ref> for the comparison of model plasticity and generation acceleration details.


Offline Continual World Results.   
This section presents the comparison between  and six representative continual RL methods on CW10 and CW20 benchmarks.
In order to show the capabilities of plasticity (quick adaptation to unseen tasks) and stability (lasting retention of previous knowledge), we keep the size of training samples, number of gradient updates, and computation constant.
Figure <ref> (b) and Table <ref> (b) summarize the results of CW10 and CW20 tasks.
We observe that our method can quickly master these manipulation tasks and remember the acquired knowledge when new tasks arrive, while the baselines (except for Multitask) struggle between plasticity and stability because the performance of these baselines fluctuates among tasks.
Moreover, after 5M gradient steps, our method still remembers how to solve the same task it learns, which shows small forgetting. 
The results of the table also show that though some baselines exhibit high forward transfer, the average success rate is lower than our method, and they forget knowledge fleetly. 


 





 §.§ Ablation Study
 
To show the effectiveness of experience rehearsal, we conduct an ablation study of  in CW and Ant-dir tasks.
We compare our method with and without experience rehearsal and find that experience rehearsal indeed brings significant performance gain. 
For example,  achieves 76.82% performance gain compared with -w/o rehearsal.
In CW 20 tasks,  reaches mean success rate from 20% to 98% when incorporating experience rehearsal.
Refer to Table <ref> for more results.


Sensitivity of Key Hyperparameters.   
In the experiments, we introduce the key hyper-parameters: the rehearsal frequency (υ) and rehearsal sample diversity (ξ).
The larger υ will aggravate the catastrophic forgetting because the model can access previous samples after a longer training process.
A large value of ξ will improve the performance and increase the storage burden, while a small value is more cost-efficient for longer CL tasks but is more challenging to hold the performance.
We conduct the sensitivity of the hyperparameters on the CW and Ant-dir environments, and the results are shown in Figure <ref> and Figure <ref>.
According to the results, our method can still reach good performance with the variation of υ and ξ.













§ DISCUSSION




 §.§ Catastrophic Forgetting of Diffuser


Previous diffusion-based methods <cit.>, such as DD and Diffuser, are usually proposed to solve a single task, which is not in line with the real-world situation where the task will dynamically change.
Thus, it is meaningful but challenging to train a diffuser that can adapt to new tasks (plasticity) while retaining historical knowledge.
When we directly extend the original diffusion-based method in continual offline RL, we can imagine that severe catastrophic forgetting will arise in the performance because there are no mechanisms to retain preceding knowledge. 
As shown in Figure <ref>, in order to show the catastrophic forgetting, we compare our method and the representative diffusion-based methods on Ant-dir, where we arbitrarily select four tasks, task-10, task-15, task-19, and task-25, to form the CL setting.
Diffuser-w/o rehearsal and DD-w/o rehearsal represent the original method Diffuser and DD, respectively.
Multitask  and MTDIFF are the multitask baselines, which can access all training datasets in any time step, and -RCR represents we use return condition for decision generation during the training stage.
-LoRA denotes that we train  with the technique of low-rank adaptation.
IL-rehearsal is the imitation learning with rehearsal.
The results show that previous diffusion-based methods exhibit severe catastrophic forgetting when the datasets arrive sequentially, and at the same time, the good performance of  illustrates experience rehearsal is effective in reducing catastrophic forgetting.




 §.§ Reducing Catastrophic Forgetting with Experience Rehearsal


In Section <ref>, we illustrate the effectiveness of experience rehearsal through the experiments on our proposed offline CL benchmark, which contains 90 tasks for evaluation.
From the perspective of the CL tasks quantity, we evaluate carious quantity settings, such as 4 tasks for Ant-dir, 4 tasks for CW4, 10 tasks for CW10, and 20 tasks for CW20.
From the perspective of classification of traditional CL settings, our experimental settings contain CICORL, TICORL, and DICORL.
In the Ant-dir environment, we select 10-15-19-25 task sequence as the CL setting and conduct the experiment compared with other diffusion-based methods.
From the results shown in Figure <ref>, we can see distinct catastrophic forgetting on the recent diffusion-based method, though they show strong performance in other offline RL tasks <cit.>.
To borrow the merits of diffusion models' strong expression on offline RL and equip them with the ability to reduce catastrophic forgetting, we propose to use experience rehearsal to master the CORL.
Detailed architecture is shown in Figure <ref>, and we postpone the method description in Section <ref>.



Apart from the Ant-dir environment, we also report the performance on more complex CL tasks, i.e., CW10 and CW20, in Table <ref>.
Considering that most baselines are trained in online mode in their original papers, we first select the online baselines and compare their mean success rate with our method.
The results (Table <ref> and Figure <ref>) show that our method () surpasses other baselines by a large margin, which illustrates the superior performance over balancing plasticity and stability.
Besides, we also compare our method with these baselines trained with offline datasets, where the results show that our method can quickly master these manipulation tasks and remember the acquired knowledge when new tasks arrive, while the baselines (except for Multitask) struggle between plasticity and stability because the performance of these baselines fluctuates among tasks.
When the previous tasks appear once again after 5M training steps, the baselines show different levels of catastrophic forgetting because the performance decreases after 5M steps. 
However, our method still remembers how to solve the same task it learned before, which shows small forgetting.
Moreover, we also conduct mixed-quality dataset experiments to show our method's capability of learning from sub-optimal offline datasets.
For more details, please refer to Appendix <ref>.

To investigate the influence of key hyperparameters, we report the performance of the rehearsal frequency (υ) and rehearsal sample diversity (ξ) in Figure <ref> and Figure <ref>, where larger υ corresponds to aggravated catastrophic forgetting and a larger value of ξ will improve the performance and increase the storage burden.
In practice, we find that usually υ=2 and ξ=10% indicate good performance and pose small challenges for the computation and memory burden (see Appendix <ref> for memory and efficiency analysis.).





§ METHODS




 §.§ Continual Offline RL


In this paper, we focus on the task-incremental setting of task-aware continual learning in the offline RL field where the different tasks come successively for training <cit.>.
Each task is defined as a corresponding Markov Decision Process (MDP) ℳ=⟨𝒮, 𝒜, 𝒫, ℛ, γ⟩, where 𝒮 and 𝒜 represent the state and action space, respectively, 𝒫: 𝒮×𝒜→Δ(𝒮) denotes the Markovian transition probability, ℛ: 𝒮×𝒜×𝒮→ℝ is the reward function, and γ∈ [0, 1) is the discount factor.
In order to distinguish different tasks, we use subscript i for task i, such as ℳ_i, 𝒮_i, 𝒜_i, 𝒫_i, ℛ_i, and γ_i.
At each time step t in task i, the agent receives a state s_i,t from the environment and produces an action a_i,t with a stochastic or deterministic policy π. 
Then a reward r_i,t=r(s_i,t, a_i,t) from the environment serves as the feedback to the executed action of the agent.
Continual offline RL aims to find an optimal policy that can maximize the discounted return ∑_i^I𝔼_π[∑_t=0^∞γ^tr(s_i,t, a_i,t)] <cit.> on all tasks with previously collected dataset {D_i}_i∈ I.




 §.§ Conditional Diffusion Probabilistic Models


In this paper, diffusion-based models are proposed to model the distribution of trajectory τ, where each trajectory can be regarded as a data point.
Then we can use diffusion models to learn the trajectory distribution q(τ)=∫ q(τ^0:K) dτ^1:K with a predefined forward diffusion process q(τ^k|τ^k-1)=𝒩(τ^k;√(α_k)τ^k-1,(1-α_k)I) and the trainable reverse process p_θ(τ^k-1|τ^k)=𝒩(τ^k-1;μ_θ(τ^k,k), Σ_k), where k∈ [1, K] is the diffusion step, √(α_k) and √(1-α_k) control the drift and diffusion coefficients, μ_θ(τ^k)=1/√(α_k)(τ_k-β_k/√(1-α̅_k)ϵ_θ(τ^k,k)), Σ_k=1-α̅_k-1/1-α̅_kβ_kI, and α_k + β_k = 1.
ϵ_θ(τ^k, k) represents the noising model <cit.>.
According to <cit.>, we can train ϵ_θ(τ^k, k) with the below simplified objective 

    ℒ(θ)=𝔼_k∼ U(1, 2, ..., K), ϵ∼𝒩(0,I), τ^0∼ D[||ϵ-ϵ_θ(τ^k, k)||_2^2],

where k is the diffusion time step, U is uniform distribution, ϵ is multivariant Gaussian noise, τ^0=τ is sampled from the replay buffer D, and θ is the parameters of model ϵ_θ. 

Conditions play a vital role in conditional generation because this method makes the outputs of diffusion models controllable.
We can also use two conditions methods, classifier-guided and classifier-free, to train diffusion models p_θ(τ^k-1|τ^k, 𝒞) <cit.>.
The classifier-guided method separates the training of the unconditional diffusion model and conditional guide and then combines them together, i.e., p_θ, ϕ(τ^k-1|τ^k, 𝒞)∝ p_θ(τ^k-1|τ^k)p_ϕ(𝒞|τ^k).
The corresponding sampling process is p(τ^k-1|τ^k, 𝒞)=𝒩(μ_θ+Σ_k ·∇ log p_ϕ(𝒞|τ), Σ_k).
Compared with classifier-guided, the classifier-free method implicitly builds the correlation between the trajectories and conditions in the training phase by learning unconditional and conditional noise ϵ_θ(τ^k, ∅, k) and ϵ_θ(τ^k, 𝒞, k), where ∅ is usually the zero vector <cit.>. 
Then the perturbed noise at each diffusion time step is calculated by ϵ_θ(τ^k, ∅, k)+ω(ϵ_θ(τ^k, 𝒞, k)-ϵ_θ(τ^k, ∅, k)).
In this paper, we adopt the classifier-free guidance due to its simplicity, controllability, and higher performance <cit.>.





 §.§ Continual Diffuser


In this section, we introduce the Continual Diffuser (), as shown in Figure <ref>, which contains classifier-free task-conditional training, experience rehearsal, and conditional generation for decision.

Data Organization.    
In RL, we leverage the characteristic of the diffusion model that can capture joint distributions in high-dimensional continual space by formulating the training data from single-step transition to multi-step sequences.
Specifically, we have I tasks, and each task ℳ_i consists of N trajectories {τ_i}_1^N, where the τ_i,n={s_i,t,n, a_i,t,n} will be split into equaling sequences with T_e time steps as the discrepancy of trajectories may occur across tasks.
In the following parts, we slightly abuse this notation τ_i to represent the sequence data with length T_e sampled from task i 's dataset D_i and τ̂_i to denote the generative sequence.

Task Condition.   
In order to distinguish different tasks, we propose to use environment-related information as the task condition.
For example, in the Ant-dir environment, the agent's goal is to maximize its speed in the pre-defined direction, which is given as the goal in the specific tasks.
So, we propose to use this information as condition 𝒞_task to train our model.
In each diffusion step k, the task condition 𝒞_task will pass through a task embedding function to obtain task embedding, which will be fed into the diffusion model jointly with diffusion time step embedding.  
Apart from the task conditions that are used implicitly in the training, we also need explicit observation conditions.
We use the first state s_i,t,n of the T_e length sampled sequence τ_i,n = {s_i,t,n, a_i,t,n, s_i,t+1,n, a_i,t+1,n, ..., s_i,t+T_e-1,n, a_i,t+T_e-1,n} as the conditions.
Then at each diffusion generation step, after we obtain the generated sequences {ŝ_i,t,n, â_i,t,n, ..., ŝ_i,t+T_e-1,n, â_i,t+T_e-1,n}^k, the first observation ŝ_i,t,n is directly replaced by s_i,t,n, i.e., τ̂_i,n^k = {s_i,t,n, â_i,t,n, ..., ŝ_i,t+T_e-1,n, â_i,t+T_e-1,n}^k.

Training Objective.    
Following the previous studies of the diffusion model <cit.>, the training and generation for each task i are defined as 

    ℒ_i(θ)=𝔼_k∼ U(1, K), ϵ∼𝒩(0,I), τ^0_i∼ D_i[||ϵ-ϵ_θ(τ_i^k,𝒞_task i, k)||_2^2],


    τ_i^k-1=√(α̅_k-1)β_k/1-α̅_k·τ̅_i+√(α)_k (1-α̅_k-1)/1-α̅_kτ_i^k+|Σ_k|z,

where z∼𝒩(0,I), τ̅_i=τ_i^k-√(1-α̅_k)ϵ̅/√(α̅_k), |Σ_k|=1-α̅_k-1/1-α̅_kβ_k, and ϵ̅=ϵ_θ(τ_i^k, ∅, k)+ω(ϵ_θ(τ_i^k, 𝒞_task, k)-ϵ_θ(τ_i^k, ∅, k)).

Experience Rehearsal.   
In this paper, we propose periodic rehearsal to strengthen the knowledge of previous tasks, which mimics the memory consolidation mechanism of hippocampus replay inside biological systems.
When a new dataset D_i of task i arrives, we preserve a small portion ξ of the entire dataset, donated as 𝒟_i.
For the small training dataset 𝒟_i, it is easy to overfit these data for most rehearsal-based methods.
Fortunately, inspired by the distributional robust optimization, increasing the hardness of the samples will hinder memory overfitting.
The discrete type of diffusion process τ^k=√(α_k)τ^k-1+√(1-α_k)ϵ can be reformulated as the corresponding continuous forward process dτ=-1/2β(t)τ dt+√(β(t))dW, where W is the standard Wiener process (a.k.a. Brownian motion).
This process gradually inserts directional noise (i.e., increasing the hardness) to induce transformation from trajectory distribution to Gaussian distribution.
So rehearsal-based diffusers naturally possess the capability of reducing memory overfitting, and the total objective function is 

    min_∀θ∈Θ[𝔼_τ_j∈ D_jℒ_j(θ,τ_j,𝒞_task j)+𝔼_τ_i∈𝒟_i, i<jℒ_i(θ,τ_i,𝒞_task i)]

In practice, we usually set the rehearsal frequency υ as 2 gradient steps and the portion ξ as 10%.


Architecture.   
In this paper, we adopt temporal Unet with one-dimensional convolution blocks as the diffusion model to predict noises.
Specifically, temporal Unet contains several down-sampling blocks, a middle block, several up-sampling blocks, a time embedding block, and a task embedding block.
We train the time embedding block and task embedding block to generate time and task embeddings that are added to the observation-action sequence

    τ_i,t:t+T_e-1,n=[       s_i,t,n     s_i,t+1,n           ... s_i,t+T_e-1,n;       a_i,t,n     a_i,t+1,n           ... a_i,t+T_e-1,n;               ].

In the return conditional diffusion models, we replace the task embedding block with the return embedding block.
Also, following the implementation of low-rank adaptation in Natural Language Processing <cit.>, we increase the LoRA module in down-sampling, middle, and up-sampling blocks to construct the LoRA variant -LoRA.



 §.§ Conclusion


First of all, to facilitate the development of the continual offline RL community, a continual offline benchmark that contains 90 tasks is constructed based on Continual World and Gym-MuJoCo. 
Based on our benchmark, we propose Continual Diffuser (), an effective continual offline RL method that possesses the capabilities of plasticity and stability with experience rehearsal.
Finally, extensive experiments illustrate the superior plasticity-stability trade-off when compared with representative continual RL baselines.




§ CODE AND DATA AVAILABILITY

The code and data are available in GitHub at <https://github.com/JF-Hu/Continual_Diffuser>.



§ ACKNOWLEDGEMENT

We would like to thank Lijun Bian for her contributions to the figures and tables of this manuscript.
We thank Runliang Niu for his contributions to providing help on the computing resource.





plainnat






§ SUPPLEMENTARY MATERIAL




 §.§ Pseudocode of Continual Diffuser

The pseudocode for  training is shown in Algorithm <ref>.
First of all, we process the datasets of I tasks before training, including splitting the trajectories into equal sequences and normalizing the sequences to facilitate learning.
As shown in lines 9-24, for each task i, we check the task index in the whole task sequence and sample different samples from the different buffers.
For example, for task i,i>0, we will perform experience rehearsal every υ train steps by sampling data from 𝒟_j, j∈0, ..., i-1, where j is sampled from U(0, i-1).
Then, the networks ϵ_θ, f_task(ϕ), and f_time(φ) are updated according to Equation (<ref>) and Equation (<ref>).
After training on task i, we preserve a small portion (ξ) of the dataset of buffer D_i as task i's rehearsal buffer.
During the evaluation of multiple tasks (shown in Algorithm <ref>), we successively generate decisions with  and calculate the evaluation metrics.




 §.§ Implement Details



Compute.    Experiments are carried out on NVIDIA GeForce RTX 3090 GPUs and NVIDIA A10 GPUs.
Besides, the CPU type is Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz. 
Each run of the experiments spanned about 24-72 hours, depending on the algorithm and the length of task sequences.

Hyperparameters.    
In the implementation, we select the maximum diffusion steps as 200, and the default structure is Unet.
Then, in order to speed up the generation efficiency during evaluation, we consider the speed-up technique of DDIM <cit.> and realize it in our method, thus accomplishing 19.043x acceleration compared to the original generation method.
The sequence length is set to 48 in all experiments, where a larger sequence length can capture a more sophisticated distribution of trajectories and may also increase the computation burden.
We set the LoRA dimension as 64 for each module of down-sampling, middle, and up-sampling blocks, and the percent of LoRA parameters is approximately 12% in our experiments.











 §.§ Related Work


Diffusion-Based Models for RL.   
Diffusion models have made big progress in many fields, such as image synthesis and text generation <cit.>.
Recently, a series of works have demonstrated the tremendous potential of diffusion-based models in offline RL tasks such as goal-based planning, composable constraint combination, scalable trajectory generation, and complex skill synthesis <cit.>.
For example, <cit.> propose to use the value function as the guide during trajectory generation, effectively reducing the effects of out-of-distribution actions and reaching remarkable performance in offline RL tasks.

Besides, diffusion models can also be used as policies to model the multimodal distribution from states to actions and as planners to perform long-horizon planning <cit.>.
For instance, <cit.> use diffusion models as policies to model the distribution from states to actions, while <cit.> endow diffusion models with the ability to perform planning and data augmentation with different task-specific prompts.

Continual Learning in RL.   
Continual learning (CL) aims to solve multi-tasks that come sequentially with explicit boundaries (task-aware CL) or implicit boundaries (task-free CL) and achieve no catastrophic forgetting and good task transferring (i.e., plasticity-stability dilemma) at the same time <cit.>.
Multitask learning methods <cit.> are usually regarded as the upper bound of continual learning.
Existing studies for continual RL can be roughly classified into three categories: Structure-based methods focus on novel model structures such as sub-networks, mixture-of-experts, hypernetworks, and low-rank adaptation <cit.>. 
Regularization-based methods propose using auxiliary regularization loss to constrain the policy optimization and avoid catastrophic forgetting during training <cit.>. 
Rehearsal-based methods preserve experiences of previous tasks or train generative models that can produce pseudo-samples to maintain knowledge of past tasks <cit.>.
Besides, recent plasticity-preserving studies <cit.> reveal that the plasticity of models can be enhanced by weight re-initialization and noisification when facing the early interactions overfitting within a single task.


Offline RL.   
Offline RL mainly focuses on how to train optimal policies with previously collected large datasets without expensive and risky data collection processes <cit.>. 
It, however, remains a huge challenge for training when facing the distribution shift between the learned policy and the data-collected policy and the overestimation of out-of-distribution (OOD) actions <cit.>.
To solve these issues, previous studies on offline-RL tasks generally rely on methods from constrained optimization, safe learning, imitation learning, and amendatory estimation <cit.>.  
Besides, planning and optimizing in the world model with limited interactions also serves as a promising way to train satisfactory policies <cit.>.
Recently, sequential modeling has been proposed to fit the joint state-action distribution over the trajectories with transformer-based models and diffusion-based models <cit.>. 




 §.§ Additional Experiments






Offline Continual World Results on CW10.    
We report the performance on CW10 in Figure <ref> when the baselines are trained with offline datasets.
The results show that the learning speed of our method () is much more efficient than other baselines when executing the same gradient updates.
Besides, we can observe that the performance of generative methods is more effective than non-generative methods, which shows the powerful expressiveness of generative models in modeling complex environments and generating pseudo-samples with high fidelity.

Online Continual World Results on CW10.    
Apart from the offline comparison, we also modified the original baselines and conducted experiments on CW10, where several new online baselines were introduced.
Similarly, the results in Figure <ref> also show that our method () surpasses the baselines by a large margin, illustrating the superiority of .
We do not incorporate several offline baselines trained with generative models into online comparison because the generative process consumes much more time for interaction, which exceeds the tolerable range of training.
These baselines trained with generative models are more suitable for training on offline datasets.

Mixed Dataset Training Analysis.    
We can classify the training under the sub-optimal demonstrations into two situations.
You can click here to return to Section <ref> quickly for continual reading of the main body.

The first is learning from noise datasets. In order to simulate the training under the sub-optimal demonstrations, we insert noise into the observations of the current dataset to obtain sub-optimal demonstrations, i.e., o̅ = o + clip(η * 𝒩(0, I), ρ). The larger noise denotes datasets with lower quality. We report the results in Table <ref>. The results illustrate that the performance decreases with the noise increasing, which inspires us to find additional techniques to reduce the influence of the noise on samples, such as adding an extra denoising module before diffuser training.





The second is learning from datasets sampled with mixed-quality policies. 
We construct the `medium' datasets on several Continual World tasks (CW4) to show the performance on the mixed-quality datasets, where the trajectories come from a series of behavior policies during the training stage. 
With the training stage going, we update the policy network many times, and each gradient update step will be regarded as generating a new behavior policy. Then, the performance of the policy will be improved. 
Next, we use the behavior policies whose performance ranges from medium to well-trained performance to collect `medium' datasets, i.e., the `medium' datasets contain unsuccessful trajectories and successful trajectories simultaneously (Refer to Table <ref> for more statistics.).
Based on the mixed-quality CW4 datasets, we adopt IL as the baseline and compare our method with IL. 
The corresponding experimental results are shown in Table <ref> (c). 
The results show that our method () can achieve better performance than the baseline in the `medium' dataset quality setting, which shows its effectiveness.


Plasticity Comparison.    
In order to compare the plasticity of our method and representative plasticity-preserving methods <cit.>, we conduct the experiments on the Ant-dir environment with task setting as `10-15-19-25', which is the same as the setting in the main body.
The results are reported in Table <ref>, where the final performance means evaluation on all tasks after the whole training on all tasks and the performance gain of plasticity (task-level) is calculated according to mean(P(train15test15) - P(train10test15) + P(train19test19) - P(train15test19) + P(train25test25) - P(train19test25)). 
The results illustrate that our model reaches better final performance than PLASTIC and SAM. Besides, in the task-level plasticity performance comparison, our method also obtains a higher score. 
Although PLASTIC and SAM do not perform well here, it's worth noting that PLASTIC and SAM are not designed to resolve continual learning under changing tasks but to address early interactions overfitting within a single task. 
The granularity of plasticity referred to in  is larger than that in PLASTIC and SAM.
Click here to return to Section <ref> quickly for continual reading of the main body.



Parameters Sensitivity on Ant-dir.    
In Section <ref>, we conduct the parameter sensitivity analysis on CW to show the effects of rehearsal frequency υ and rehearsal diversity ξ.
We also report the results of parameters sensitivity on Ant-dir in Figure <ref> and Table <ref>, where υ=inf means we do not perform rehearsal during training.
The results show that with the increase of υ, the performance declines because the model can not use previous datasets to strengthen its memory in time.






Efficiency Analysis of Generation Speed.    
The generation process of diffusion models is indeed computationally intensive because the mechanism of generation requires multiple rounds to generate a sequence. However, we can draw inspiration from previous studies <cit.> in related domains and accelerate the generation process. For example, we can reduce the reverse diffusion step from 200 to 10. To show the efficiency of accelerating during the generation process, we conduct a comparison of generation speed. We report the results in Table <ref>, where the 200 diffusion steps setting is the original version, and the 10 diffusion steps setting is our accelerated version. In the experiments of our manuscript, we adopt the 10 diffusion steps setting, which improves the sampling speed (19.043×) with a larger margin than the original sampling version. It's worth noting that our implemented accelerate technique can also use other diffusion steps settings, but we find that 10 diffusion steps setting performs well on performance and generation efficiency.






Ablation Study on Mixed Datasets.    
In Table <ref>, we report the effects of rehearsal sample diversity ξ on the `medium' datasets. From the results, we can see that increasing the rehearsal sample diversity is beneficial to the performance, which is in line with the experiments in the main body of our manuscript. Besides, the results also show that our method () can reach a better plasticity-stability trade-off than the baseline in the `medium' dataset quality setting.



Computation Costs Analysis.    
In order to show the consumption of computational costs, we report the comparison of computation costs during the training stage in Table <ref>, where we obtain the statistical data with `wandb.'
The results show that increasing the rehearsal samples does not significantly increase computation costs and training time.








 §.§ Statistics of Continual Offline RL Benchmarks





To take advantage of the potential of diffusion models, we first collect an offline benchmark that contains dozens of tasks from multiple domains, such as Continual World and Gym-MuJoCo <cit.>.
In order to collect the interaction data, we trained Soft Actor-Critic on each task for approximately 1M time steps <cit.>.
Totally, the benchmark contains 90 tasks, where 88 tasks come from Continual World, 2 tasks come from Gym-MuJoCo.


Specifically, CW <cit.> tasks are constructed based on Meta-World <cit.>.
CW consists of many realistic robotic manipulation 
tasks such as Pushing, Reaching, Door Opening, Pick, and Place.
CW is convenient for training and evaluating the abilities of forward transfer and forgetting because the state and action space are the same across all the tasks.
In our benchmark, we collect “expert” and “medium” datasets, where the episodic time limit is set to 200, and the evaluation time step is set to 1M and 0.4M for “expert" and “medium” datasets, respectively.
Thus, we obtain 5000 and 2000 episodes for these two quality tasks, as shown in Table <ref> and Table <ref>, in which we also report the mean success rate of these two qualities dataset.
Besides, we also provide the return information of all datasets in Figure <ref> and Figure <ref>.
Out of these tasks defined in Meta-World, we usually select ten tasks from them as the setting of continual learning, i.e., CW10, and CW20 denotes the setting of two CW10.

Aligning with the traditional definition of various CL settings <cit.>, this benchmark supports constructing task-incremental CORL (TICORL), domain-incremental CORL (DICORL), and class-incremental CORL (CICORL) settings. 
Researchers can use these datasets in any sequence or length for CL tasks to test the plasticity-stability trade-off of their proposed methods. 
The future expansion plan of this benchmark will gather datasets such as `random' and `full' training qualities datasets to bolster training robust CL agents. 


Ant-dir is an 8-joint ant environment. 
The different tasks are defined according to the target direction, where the agent should maximize its return with maximal speed in the pre-defined direction. 
As shown in Table <ref>, there are 40 tasks (distinguished with “task id”) with different uniformly sampled goal directions in Ant-dir. 
For each task, the dataset contains approximately 200k transitions, where the observation and action dimensions are 27 and 8, respectively.
We found that the Ant-dir datasets have been used by many researchers <cit.>, so we incorporate them into our benchmark.
Moreover, we report the mean return information of each sub-task in Table <ref> and Figure <ref>.
As for Cheetah-dir, it only contains two tasks that represent forward and backward goal directions.
Compared with Ant-dir, Cheetah-dir possesses lower observation and action space.












 





