#  Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement
Learning

Xu-Hui Liu  Tian-Shuo Liu  Shengyi Jiang  Ruifeng Chen  Zhilong Zhang  Xinwei
Chen  Yang Yu

###### Abstract

Combining offline and online reinforcement learning (RL) techniques is indeed
crucial for achieving efficient and safe learning where data acquisition is
expensive. Existing methods replay offline data directly in the online phase,
resulting in a significant challenge of data distribution shift and
subsequently causing inefficiency in online fine-tuning. To address this
issue, we introduce an innovative approach, Energy-guided DIffusion Sampling
(EDIS), which utilizes a diffusion model to extract prior knowledge from the
offline dataset and employs energy functions to distill this knowledge for
enhanced data generation in the online phase. The theoretical analysis
demonstrates that EDIS exhibits reduced suboptimality compared to solely
utilizing online data or directly reusing offline data. EDIS is a plug-in
approach and can be combined with existing methods in offline-to-online RL
setting. By implementing EDIS to off-the-shelf methods Cal-QL and IQL, we
observe a notable 20% average improvement in empirical performance on MuJoCo,
AntMaze, and Adroit environments. Code is available at
https://github.com/liuxhym/EDIS.

Reinforcement Learning, Diffusion Model

##  1 Introduction

Reinforcement learning (RL) (Sutton & Barto, 2018) has demonstrated remarkable
efficacy in diverse decision-making tasks, such as sequential recommendation
systems (Wang et al., 2018; Zhao et al., 2018), robotic locomotion skill
learning (Peng et al., 2020; Haarnoja et al., 2018) and operations research
(Hubbs et al., 2020; Wang et al., 2023c, 2024; Ling et al., 2024). Notably, RL
methods, including online (Mnih et al., 2015; Liu et al., 2021; Chen et al.,
2024) and offline (Fujimoto et al., 2019; Kumar et al., 2020; Jin et al.,
2022; Jia et al., 2024) ones, have been employed to achieve superior
performance. However, online RL methods often demand extensive data collection
through interacting with the environment, a process that can be time-consuming
or risky. In contrast, offline RL methods utilize pre-existing datasets to
train a new policy, avoiding the need for resource-intensive online
interactions but often suffering from suboptimality due to limited data.

To mitigate the challenges incurred due to data limitations in online and
offline RL, researchers have proposed the offline-to-online setting, aiming to
overcome the cost of online interaction and the suboptimality inherent in
offline learning. In this setting, the agent undergoes two learning phases:
first, it learns from an offline dataset, and then it fine-tunes through a
limited number of online interactions (Lee et al., 2021; Kostrikov et al.,
2022; Nakamoto et al., 2023). After the offline phase, value functions or
policies are derived and used to initialize the online phase. Therefore, the
algorithm can utilize prior knowledge in the offline dataset to help reduce
the cost of the online phase. However, a significant drawback in this paradigm
arises from the incomplete utilization of the offline dataset, as the
information extracted is limited to the pre-trained policy.

While previous works have effectively addressed the performance drop because
of the objective mismatch of the two-stage learning in offline-to-online
settings (Nakamoto et al., 2023; Lee et al., 2021; Nair et al., 2020), the
incomplete utilization of the prior knowledge in the offline dataset remains
an unexplored challenge. Some studies directly replayed samples in the online
phase for data augmentation (Lee et al., 2021), leading to performance
improvements. Nevertheless, this approach neglects the distribution shift
issue, as the data distribution in the offline dataset may differ from that
induced by the current policy. Distribution shift has shown adverse effects
even in off-policy reinforcement learning (Sinha et al., 2022; Liu et al.,
2021), which incorporates a replay buffer, let alone when utilizing an offline
dataset. Consequently, a fundamental question arises:

Is it feasible to generate samples without distribution shift based on an
offline dataset?

Model-based methods (Wang et al., 2022; Rafailov et al., 2023; Chen et al.,
2023b) generate such samples by first learning a model and using it for
rollout. However, this style of generation inevitably suffers from the issue
of compounding errors: the error of the learned model itself will accumulate
during the multi-step rollout. Therefore, we aim to generate samples with the
desired distribution directly. Motivated by the recent achievements of the
diffusion model in image generation (Ho et al., 2020; Karras et al., 2022), we
want to explore its potential application in RL sample generation. However,
utilizing a diffusion model trained on an offline dataset introduces a
challenge—it can only generate samples adhering to the dataset distribution,
thus still being susceptible to distribution shift issues.

The desired distribution for RL has three crucial characteristics: i) the
state distribution should align with that in the online training phase, ii)
actions should be consistent with the current policy, and iii) the next states
should conform to the transition function. To achieve this, we formulate three
distinct energy functions to guide the diffusion sampling process, ensuring
alignment with the aforementioned features. The developed algorithm is named
as Energy-Guided DIffusion Sampling (EDIS). By presenting the new algorithm,
we showcase the feasibility of generating additional, useful samples while
mitigating distribution shift, which will effectively help leverage the
offline dataset.

The theoretical analysis shows that EDIS exhibits reduced suboptimality
compared to solely utilizing online data or directly replaying offline data.
Additionally, it circumvents the compounding error issue commonly associated
with model-based methods. When incorporating EDIS into existing methods, Cal-
QL (Nakamoto et al., 2023) and IQL (Kostrikov et al., 2022), we observe 20%
performance improvement on average across MuJoCo, AntMaze, and Adroit
environments. Furthermore, our experiments demonstrate that purely model-based
methods fail to achieve performance improvements due to the compounding errors
in the model during rollouts, leading to an inaccurate state distribution. The
ablation study further validates the effectiveness of the energy functions to
guide the diffusion models.

##  2 Preliminaries

###  2.1 Markov Decision Process and RL

Let $M=({\mathcal{S}},{\mathcal{A}},P,r,\gamma,\rho_{0})$ be an MDP, where
${\mathcal{S}}$ is the state space, ${\mathcal{A}}$ is the action space,
$P:{\mathcal{S}}\times{\mathcal{A}}\to\Delta({\mathcal{S}})$ is the transition
function ($\Delta(\cdot)$ is the probability simplex),
$r:{\mathcal{S}}\times{\mathcal{A}}\to[0,R_{\max}]$ is the reward function,
$\gamma\in[0,1)$ is the discount faction and $\rho_{0}$ is the initial
distribution over states. A policy $\pi:{\mathcal{S}}\to\Delta({\mathcal{A}})$
describes a distribution over actions for each state. The goal of RL is to
learn the best policy $\pi^{*}$ that maximizes cumulative discounted reward,
i.e., $\sum_{t}\mathbb{E}_{a_{t}\sim\pi^{*}}\gamma^{t}r(s_{t},a_{t})$. The
value function and Q function of policy $\pi$ are
$V^{\pi}(s)=\sum_{t}\mathbb{E}_{a_{t}\sim\pi(s_{t})}[\gamma^{t}r(s_{t},a_{t})|s_{0}=s]$,
$Q^{\pi}(s,a)=\sum_{t}\mathbb{E}_{a_{t}\sim\pi(s_{t})}[\gamma^{t}r(s_{t},a_{t})|s_{0}=s,a_{0}=a]$.
$V^{*}$ and $Q^{*}$ be the shorthand for $V^{\pi^{*}}$ and $Q^{\pi^{*}}$
respectively. To facilitate later analysis, we introduce the discounted
stationary state distribution
$d^{\pi}(s)=\sum_{t}\gamma^{t}\textnormal{Pr}(s_{t}=s;\pi)$ and the discounted
stationary state-action distribution
$d^{\pi}(s,a)=\sum_{t}\gamma^{t}\textnormal{Pr}(s_{t}=s,a_{t}=a;\pi)$. There
are, in general, two learning paradigms of RL: online RL, where the agent can
learn from interacting with the environment; and offline RL, where the agent
can only learn from a given dataset ${\mathcal{D}}=\\{(s,a,r,s^{\prime})\\}$,
possibly collected by another policy.

###  2.2 Generative Modeling by Diffusion Model

Diffusion models (Ho et al., 2020; Karras et al., 2022) are a class of
generative models inspired by non-equilibrium thermodynamics. Given a dataset
$\\{\bm{x}_{0}^{(i)}\\}_{i=1}^{N}$ with $N$ samples of $D$-dimensional random
variable $\bm{x}$ from an unknown data distribution $p_{0}(\bm{x}_{0})$.
Diffusion models gradually add Gaussian noise at time 0 to $T$ according to
noise levels $\sigma_{\max}=\sigma_{T}>\sigma_{T-1}>\cdots>\sigma_{0}=0$ so
that at each noise level, $\bm{x}_{t}^{(i)}\sim p(\bm{x}_{t};\sigma_{t})$. The
distribution of the endpoint
$\bm{x}_{T}^{(i)}\sim{\mathcal{N}}(0,\sigma_{\max}^{2}\bm{I})$ is
indistinguishable from pure Gaussian noise, while the starting point
$\bm{x}^{(i)}_{0}$ aligns with the distribution of the original dataset. Then
the reverse process starts from a Gaussian distribution and iteratively
denoises samples using the trained model, ultimately recovering the target
distribution. The process can be interpreted as _SDE_ s:

$\displaystyle\mathrm{d}\bm{x}_{\pm}=-\dot{\sigma}(t)\sigma(t)\nabla_{\bm{x}}$
$\displaystyle\log p(\bm{x};\sigma(t))\mathrm{d}t$
$\displaystyle\pm\beta(t)\sigma^{2}(t)\nabla_{\bm{x}}$ $\displaystyle\log
p(\bm{x};\sigma(t))dt+\sqrt{2\beta(t)}\sigma(t)\mathrm{d}\omega_{t},$ (1)

where $\omega_{t}$ is a standard Wiener process, $d\bm{x}_{+}$ and
$d\bm{x}_{-}$ are separate SDEs for moving forward and backward in time. The
only unknown term is the _score function_ $\nabla_{\bm{x}}\log p_{t}(\cdot)$.
Karras et al. (2022) considers training a denoiser
$D_{\theta}(\bm{x}_{t};\sigma)$ on an L2 denoising minimization objective:

$L_{\textnormal{VLB}}(\theta):=\mathbb{E}_{\bm{x}\sim
p_{0}(\bm{x}_{0}),\epsilon\sim{\mathcal{N}}(0,\sigma^{2}I)}\left\|D_{\theta}(\bm{x}+\epsilon;\sigma)-\bm{x}\right\|^{2}_{2},$
(2)

where $\sigma$ is the standard deviation of Gaussian noise. After training a
diffusion model for estimating the score function with $\nabla_{\bm{x}}\log
p_{0}(\bm{x};\sigma)=(D_{\theta}(\bm{x};\sigma)-\bm{x})/\sigma^{2}$ , we can
fastly generate samples by solving the backward _SDE_.

##  3 EDIS: Energy-Guided Diffusion Sampling

To extract prior knowledge from the offline dataset and generate samples to
conform to the online data distribution, we introduce our innovative approach,
named Energy-guided DIffusion Sampling (EDIS). At the heart of our method is
to accurately generate a desired online data distribution, denoted as
$q_{\pi}(s,a,s^{\prime})$, from pre-gathered data. The distribution does not
include reward $r$ because we assume that the reward function $r(s,a)$ is
accessible, either directly or through learning from the dataset. To achieve
this, we have integrated a diffusion model into our framework, capitalizing on
its exceptional capability for modeling complex distributions.

###  3.1 Distribution Adjustment via Energy Guidance

One challenge in this process is the inherent limitation of directly training
a diffusion model on an offline dataset. Such a model typically yields an
offline data distribution $p_{\mathcal{D}}(s,a,s^{\prime})$, which does not
align perfectly with online data and causes distribution shift issues. To
address this, our method needs to guide the diffusion sampling process towards
the online distribution. This is achieved by decomposing the online data
distribution into the following form:

$q_{\pi}(s,a,s^{\prime})\propto
p_{\theta}(s,a,s^{\prime})e^{-\mathcal{E}(s,a,s^{\prime})},$ (3)

where $p_{\theta}(s,a,s^{\prime})$ is the distribution generated by the
denoiser network, parameterized by $\theta$. $\mathcal{E}(s,a,s^{\prime})$ is
the energy function, which serves as the guidance to bridge the gap between
generated distribution and online data distribution. The following theorem
shows such an energy function exists.

######  Theorem 3.1.

Let $p_{\theta}(s)$ be the marginal distribution of
$p_{\theta}(s,a,s^{\prime})$, $p_{\theta}(a|s)$ and
$p_{\theta}(s^{\prime}|s,a)$ be the conditional distribution of
$p_{\theta}(s,a,s^{\prime})$ given $s$ and $(s,a)$. Eq. (3) is valid if the
energy function $\mathcal{E}(s,a,s^{\prime})$ is structured as follows:

${\mathcal{E}}(s,a,s^{\prime})={\mathcal{E}}_{1}(s)+{\mathcal{E}}_{2}(a|s)+{\mathcal{E}}_{3}(s^{\prime}|s,a),$
(4)

such that $e^{{\mathcal{E}}_{1}(s)}\propto\frac{p_{\theta}(s)}{d^{\pi}(s)}$,
$e^{{\mathcal{E}}_{2}(a|s)}\propto\frac{p_{\theta}(a|s)}{\pi(a|s)}$,
$e^{{\mathcal{E}}_{3}(s^{\prime}|s,a)}\propto\frac{p_{\theta}(s^{\prime}|s,a)}{T(s^{\prime}|s,a)}$.

This theorem indicates that the energy function can be decomposed into three
distinct parts. Each part is responsible for aligning the generated
distribution with different aspects of the online data: the online state
distribution, the current policy action distribution, and the environmental
dynamics.

###  3.2 Learning Energy Guidance by Contrastive Energy Prediction

First, we concentrate on the first component, ${\mathcal{E}}_{1}(s)$. We
assume that the energy is estimated using a neural network denoted as
${\mathcal{E}}_{\phi_{1}}(s)$. Let $K$ and $K_{\textrm{neg}}$ be two positive
numbers. Given $s_{1},s_{2},\dots,s_{K}$, $K$ i.i.d. samples drawn from the
distribution $p_{\theta}(s)$, and
$s_{i}^{1},s_{i}^{2},\dots,s_{i}^{K_{\textrm{neg}}}$, $K_{\textrm{Neg}}$
negative samples for $s_{i}$. We employ the Information Noise Contrastive
Estimation (InfoNCE) loss (van den Oord et al., 2018):

${\mathcal{L}}(\phi_{1})=-\sum_{i=1}^{K}\log\frac{e^{-{\mathcal{E}}_{\phi_{1}}(s_{i})}}{e^{-{\mathcal{E}}_{\phi_{1}}(s_{i})}+\sum_{j=1}^{K_{\textrm{neg}}}e^{-{\mathcal{E}}_{\phi_{1}}(s_{i}^{j})}},$
(5)

Then, we devise positive and negative samples to achieve the target energy
function established by Thm. 3.1. Suppose the distribution of positive samples
is $\mu(s)$, the distribution of negative samples is $\nu(s)$, the final
optimized results is
$e^{{\mathcal{E}}_{\phi_{1}}(s)}\propto\frac{\nu(s)}{\mu(s)}$ (van den Oord et
al., 2018). Compared to the function indicated by Thm. 3.1, the result can be
achieved by selecting $\mu(s)=d^{\pi}(s)$, $\nu(s)=p_{\theta}(s)$. Following
the approach of Sinha et al. (2022); Liu et al. (2021), we construct a
positive buffer, containing only a small set of trajectories from very recent
policies. The data distribution in this buffer can be viewed as an
approximation of the on-policy distribution $d^{\pi}(s)$. While
$p_{\theta}(s)$ is the distribution of the data generated during the denoising
steps. Therefore, the positive samples is sampled from the positive buffer and
the negative samples is sampled from the denoiser.

For the remaining terms ${\mathcal{E}}_{2}(a|s)$ and
${\mathcal{E}}_{3}(s^{\prime}|s,a)$, we also parameterize them as
${\mathcal{E}}_{\phi_{2}}(a|s)$ and
${\mathcal{E}}_{\phi_{3}}(s^{\prime}|s,a)$, and employ infoNCE loss to
estimate them. Similarly, the loss computation requires the generation of
positive and negative samples. Differently from ${\mathcal{E}}_{\phi_{1}}(s)$,
here the samples from the offline buffer can also used for positive-sample
generation. For ${\mathcal{E}}_{\phi_{2}}(a|s)$, the positive samples are
obtained by first sampling $s$ from the offline and online distribution and
outputting the action of the current policy on $s$. For
${\mathcal{E}}_{\phi_{3}}(s^{\prime}|s,a)$, positive samples are obtained by
sampling $(s,a,s^{\prime})$ from the from the offline and online distribution
and outputting $s^{\prime}$ directly. Negative samples for both
${\mathcal{E}}_{\phi_{2}}(a|s)$ and ${\mathcal{E}}_{\phi_{3}}(s^{\prime}|s,a)$
are generated using the denoiser. For each positive samples of
${\mathcal{E}}_{\phi_{2}}(a|s)$, the negative samples are generated
conditioned on the given states. While for each positive samples of
${\mathcal{E}}_{\phi_{3}}(s^{\prime}|s,a)$, they are generated conditioned on
the sampled states-action pairs. This method of conditional sample generation
is inspired by techniques used in decision-making scenarios (Ajay et al.,
2023; Janner et al., 2022). The loss functions of parameterized
${\mathcal{E}}_{\phi_{2}}(a)$ and ${\mathcal{E}}_{\phi_{3}}(s^{\prime})$ are:

$\displaystyle{\mathcal{L}}(\phi_{2})=-\sum_{k=1}^{K_{2}}\sum_{i=1}^{K_{1}}\log\frac{e^{-{\mathcal{E}}_{\phi_{2}}(a_{ik}|s_{i})}}{e^{-{\mathcal{E}}_{\phi_{2}}(a_{ik}|s_{i})}+\sum_{j=1}^{K_{\textrm{neg}}}e^{-{\mathcal{E}}_{\phi_{2}}(a_{ik}^{j}|s_{i})}},$
(6)
$\displaystyle{\mathcal{L}}(\phi_{3})=-\sum_{i=1}^{K}\log\frac{e^{-{\mathcal{E}}_{\phi_{3}}(s^{\prime}_{i}|s_{i},a_{i})}}{e^{-{\mathcal{E}}_{\phi_{3}}(s^{\prime}_{i}|s_{i},a_{i})}+\sum_{j=1}^{K_{\textrm{neg}}}e^{-{\mathcal{E}}_{\phi_{3}}(s_{i}^{j^{\prime}}|s_{i},a_{i})}},$
(7)

where $K_{1}$ is the number of states sampled, $K_{2}$ is the number of
actions sampled for each state.

###  3.3 Sampling under Energy Guidance

To realize this distribution with diffusion models, we need to calculate its
score function in the sampling process, taking into account the energy
function designed in Sec. 3.2:

$\displaystyle\nabla_{(s,a,s^{\prime})}\log q_{\pi}(s,a,s^{\prime})$ (8)
$\displaystyle=\nabla_{(s,a,s^{\prime})}\log
p_{\theta}(s,a,s^{\prime})-\nabla_{(s,a,s^{\prime})}{\mathcal{E}}(s,a,s^{\prime})$

In the denoising process, we need to obtain the score function at each
timestep. Denote the forward distribution at time $t$ starting from
$p_{0}(s,a,s^{\prime})$ as $p_{t}(s,a,s^{\prime})$. Remember that the denoiser
model $D_{\theta}(s,a,s^{\prime};\sigma)$ is designed to match the score with
the expression:

$\nabla\log
p_{\theta}(s,a,s^{\prime})=(D_{\theta}(s,a,s^{\prime};\sigma)-(s,a,s^{\prime}))/\sigma^{2}.$
(9)

Thus, we can obtain the gradient through the denoiser model. Then, the key
problem is to obtain the intermediate energy guidance at time $t$, which is
addressed in the following theorem.

######  Theorem 3.2 (Thm. 3.1 in (Lu et al., 2023b)).

Denote $q_{t}(x_{t}):=\int q_{t}(x_{t}|x_{0})q_{0}(x_{0})dx_{0}$ and
$p_{t}(x_{t}):=\int p_{t}(x_{t}|x_{0})p_{0}(x_{0})dx_{0}$ as the marginal
distributions at time t, and define

${\mathcal{E}}_{t}(x_{t}):=\left\\{\begin{aligned}
&{\mathcal{E}}(x_{0}),&t=0,\\\
&-\log\mathbb{E}_{q_{t}(x_{0}|x_{t})}[e^{-{\mathcal{E}}(x_{0})}],&t>0.\end{aligned}\right.$

Then the score functions satisfy

$\nabla_{x_{t}}\log q_{t}(x_{t})=\nabla_{x_{t}}\log
p_{t}(x_{t})-\nabla_{x_{t}}{\mathcal{E}}_{t}(x_{t}).$

Let $x_{0}$ denote the set of original positive samples. According to this
theorem, we aim to formulate the energy function at timestep $t$. Drawing
inspiration from Sec. 3.2, we opt for a contrastive learning approach. Within
this method, these positive samples for the energy function at $t$ timestep
are derived from the original samples subjected to $t$ steps of noise
addition, i.e., $x_{t}\sim p(\bm{x}_{i};\sigma_{t})$, where $\sigma_{t}$ is
chosen as (Karras et al., 2022):

$\sigma_{t}=\left(\sigma_{\max}^{\frac{1}{\rho}}+\frac{t}{T-1}(\sigma_{\min}^{\frac{1}{\rho}}-\sigma_{\max}^{\frac{1}{\rho}})\right)^{\rho}\quad\textrm{and}\quad\sigma_{T}=0,$

where $\rho$, $\sigma_{\max}$ and $\sigma_{\min}$ are pre-defined constant
numbers. As for the negative samples, they are generated by the denoiser at
time $t$. Then we start the denoising process defined in Eq. (2.2). The
psuedo-code of EDIS is demonstrated in Appx. B.

##  4 Theoretical Analysis

In this section, we provide theoretical analysis on the suboptimality bound of
our method and previous methods. We follow the assumptions of Fitted-Q-
iteration (Chen & Jiang, 2019): Let ${\mathcal{F}}$ be the Q function class
that satisfies realizability: $Q^{*}\in{\mathcal{F}}$. ${\mathcal{F}}$ is
closed under Bellman update: $\forall
f\in{\mathcal{F}},{\mathcal{T}}^{\pi}f\in{\mathcal{F}}$, where
${\mathcal{T}}^{\pi}f(s,a):=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim
P(s,a)}[V^{\pi}_{f}(s^{\prime})]$,
$V^{\pi}_{f}(s^{\prime}):=\mathbb{E}_{a^{\prime}\sim\pi}f(s^{\prime},a^{\prime})$
and $\pi$ can be any policy. Additionally, we assume ${\mathcal{F}}$ is
$L$-Lipschitz.

In the offline phase, we initialize a value function $Q_{0}$. Based on
$Q_{0}$, we perform online learning by collecting $n$ samples each iteration.
The naive method, which relies solely on the information from $n$ collected
samples to update the policy, is characterized by the following suboptimality
bound.

######  Theorem 4.1.

Suppose the learned Q function after the offline phase satisfies
$\left(\left\|Q^{*}-Q_{0}\right\|_{\infty}+\left\|Q^{\pi}-Q_{0}\right\|_{\infty}\right)\leq\epsilon_{Q}$,
where $Q^{*}$ is the Q function of the optimal policy $\pi^{*}$. With
probability at leaset $1-\delta$, the output policy $\pi$ after one iteration
of online phase satisfies

$\displaystyle J(\pi^{*})-J(\pi)\leq\frac{2\gamma\epsilon_{Q}}{1-\gamma}+$
$\displaystyle\quad\quad\frac{\sqrt{C_{\pi,\pi^{*}}}+1}{1+\gamma}\frac{1}{(1-\gamma)^{2.5}}\sqrt{\frac{56R^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}},$

where $C_{\pi,\pi^{*}}=\max_{s,a}\frac{\pi_{1}(a|s)}{\pi_{2}(a|s)}$, $n$ is
the number of collected samples in one iteration.

Considering the performance bound after a single iteration is reasonable, as
subsequent iterations can be analyzed by treating the output value function as
the new initialization. The suboptimality after one iteration is characterized
by the initialization error, augmented by a term of
$O\left(\sqrt{\frac{1}{n}}\right)$. Notably, this error is nonnegligible
because of the relatively small size of the online interaction samples.

A natural approach for leveraging offline data is augmenting the online
dataset with the available offline data. To establish the suboptimality bound
for such a method, we need to introduce the concept of concentratability
coefficient.

######  Definition 4.2 (Concentratability Coefficient).

The concentratbility coefficient $C$ between state distribution $\nu$ and
$\mu$ is defined as $\forall s\in{\mathcal{S}},\frac{\nu(s)}{\mu(s)}\leq C.$

Assuming the concentratability coefficient between the state distribution of
current policy $d^{\pi}$ and offline dataset $D$ is $C_{d}$, we have the
following theorem:

######  Theorem 4.3.

Under the conditions of Thm. 4.1, by replaying offline data in the online
iteration, the output policy $\pi$ after one iteration satisfies

$\displaystyle J(\pi^{*})-J(\pi)\leq\frac{2\gamma\epsilon_{Q}}{1-\gamma}+$
$\displaystyle\quad\quad\frac{\sqrt{\widetilde{C_{d}}C_{\pi,\pi^{*}}}+1}{1+\gamma}\frac{1}{(1-\gamma)^{2.5}}\sqrt{\frac{56R^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3(n+N)}},$

where $\widetilde{C_{d}}=\frac{(n+N)C_{d}}{N+nC_{d}}$, $N$ is the number of
samples in offline dataset.

The error bound is characterized by an order of
$O\left(\sqrt{\frac{1}{n+N}}\right)$. Given the substantial difference in
size, where $N$ is notably larger than $n$, replaying offline data shows
advantages to the previous method. This result explains the improved
performance resulting from online dataset augmentation. However, the
introduction of the concentration coefficient $C_{d}$ in the new method
reflects the consequences of the distribution shift issue. This coefficient
can assume a significantly large value, particularly if the offline data fails
to adequately cover a substantial portion of the state space. It detriments
the potential performance improvement attainable through enhanced utilization
of prior knowledge.

Based on the diffusion model generator, our new method has access to synthetic
samples $(\tilde{s},\tilde{a},\tilde{r},\tilde{s}^{\prime})$ during the online
interaction phase. Let the distribution of $\tilde{s}$ be denoted as
$\tilde{d}^{\pi}$, and the conditional distribution of $\tilde{s}^{\prime}$ as
$\widetilde{T}(\cdot|\tilde{s},\tilde{a})$. The state distribution error and
model error of the diffusion model are defined
as:$\epsilon_{d}=\left\|d^{\pi}-\tilde{d}^{\pi}\right\|$ , and
$\epsilon^{d}_{m}=\left\|T(\cdot|s,a)-\widetilde{T}(\cdot|s,a)\right\|_{\tilde{d}^{\pi}}$.

######  Theorem 4.4.

Under the conditions of Thm. 4.1, we assume the data generator generates data
with state distribution error $\epsilon_{d}$ and model error $\epsilon_{m}$,
then the output policy $\pi$ after one iteration of online phase satisfies

$\displaystyle J(\pi^{*})-J(\pi)\leq\frac{2\gamma}{1-\gamma}\epsilon_{Q}+$
$\displaystyle\quad\
\left(\sqrt{C_{\pi,\pi^{*}}}+1\right)\left(\frac{1}{(1-\gamma)^{2}}R_{\max}\epsilon_{d}+\frac{\gamma
L}{(1-\gamma)^{2}}\epsilon^{d}_{m}\right),$

where $L$ is Lipschitz constant of ${\mathcal{F}}$.

Here, we simplify the analysis by disregarding the generation errors
associated with $\tilde{a}$ and $\tilde{r}$. This simplification is justified
as modeling these errors is considerably more straightforward compared to
modeling the error associated with $s$ and $\tilde{s}$. Additionally,
considering these two errors leads to a similar analysis and does not alter
the order of the error term. According to the theorem, the error bound is not
directly related to the number of samples, but it is highly dependent on the
error of the diffusion model. Based on the theory of supervised learning (von
Luxburg & Schölkopf, 2011), the model error $\epsilon_{m}$ is
$O\left(\sqrt{\frac{1}{n+N}}\right)$. The state distribution error
$\epsilon_{d}$ is the distribution matching error, which is also
$O\left(\sqrt{\frac{1}{n+N}}\right)$ (Xu et al., 2021). Therefore, our method
exhibits the same convergence rate as the previous method without introducing
the distribution shift issue identified by the concentration coefficient
$C_{d}$.

Model-based method can also generate synthetic online samples. Assume the
model error of the model is
$\epsilon^{t}_{m}=\left\|T(\cdot|s,a)-\widetilde{T}(\cdot|s,a)\right\|_{d^{\pi}}$.

######  Theorem 4.5.

Under the conditions of Thm. 4.1, we assume the model has an error
$\epsilon^{t}_{m}$, then the output policy $\pi$ after one iteration of online
phase satisfies

$\displaystyle J(\pi^{*})-J(\pi)\leq\frac{2\gamma}{1-\gamma}\epsilon_{Q}+$
$\displaystyle\quad\
\left(\sqrt{C_{\pi,\pi^{*}}}+1\right)\left(\frac{R_{\max}}{(1-\gamma)^{3}}\epsilon_{m}^{t}+\frac{\gamma
L}{(1-\gamma)^{2}}\epsilon^{t}_{m}\right).$

The error of model-based approach is irrelevant to the $C_{d}$ term,
illustrating that incorporating prior knowledge about transitions from the
offline dataset can address the issue of distribution shift. Nevertheless,
this method is subject to a compounding error, with an error bound expressed
as $O\left(\frac{1}{(1-\gamma)^{3}}\right)$. This limitation implies that
model-based methods may not surpass the performance of model-free methods
(Rafailov et al., 2023). In contrast, the error bound for EDIS is lower, at
$O\left(\frac{1}{(1-\gamma)^{2}}\right)$, which effectively reduces the impact
of compounding error. In summary, EDIS overcomes all the issues previous
methods have in theory.

Table 1: Enhanced performance achieved by EDIS after 0.2M online fine-tuning
on base algorithms Cal-QL and IQL. Each result is the average score over five
random seeds $\pm$ standard deviation.

Dataset | Cal-QL | IQL | Avg.
---|---|---|---
Base | Ours | Base | Ours | Base | Ours
hopper-random-v2 | 17.6$\pm$3.1 | 98.1$\pm$12.3 | 10.0$\pm$1.7 | 12.1$\pm$4.0 | 13.8$\pm$2.4 | 55.1$\pm$8.2
hopper-medium-replay-v2 | 102.2$\pm$4.6 | 109.9$\pm$0.8 | 99.0$\pm$4.7 | 101.1$\pm$1.6 | 100.6$\pm$4.7 | 105.5$\pm$1.2
hopper-medium-v2 | 97.6$\pm$1.4 | 105.0$\pm$4.1 | 59.2$\pm$7.9 | 73.1$\pm$4.4 | 78.4$\pm$4.7 | 89.1$\pm$4.3
hopper-medium-expert-v2 | 107.9$\pm$9.6 | 109.7$\pm$1.4 | 90.0$\pm$22.1 | 105.7$\pm$6.7 | 99.0$\pm$15.9 | 107.7$\pm$4.1
halfcheetah-random-v2 | 74.8$\pm$3.2 | 86.3$\pm$1.8 | 36.7$\pm$3.0 | 38.9$\pm$1.9 | 55.8$\pm$3.1 | 62.6$\pm$1.9
halfcheetah-medium-replay-v2 | 76.6$\pm$1.2 | 86.7$\pm$1.4 | 45.6$\pm$0.4 | 47.1$\pm$0.3 | 61.1$\pm$0.8 | 66.9$\pm$0.9
halfcheetah-medium-v2 | 72.3$\pm$2.1 | 83.9$\pm$1.0 | 48.6$\pm$0.2 | 49.8$\pm$0.2 | 60.5$\pm$1.2 | 66.9$\pm$0.6
halfcheetah-medium-expert-v2 | 91.0$\pm$0.6 | 98.6$\pm$0.5 | 87.9$\pm$4.3 | 85.4$\pm$2.7 | 89.5$\pm$2.5 | 92.0$\pm$1.6
walker2d-random-v2 | 15.1$\pm$3.5 | 61.6$\pm$12.6 | 6.5$\pm$0.7 | 16.2$\pm$2.9 | 10.8$\pm$2.1 | 38.9$\pm$7.8
walker2d-medium-replay-v2 | 87.3$\pm$8.5 | 112.9$\pm$6.4 | 83.6$\pm$2.0 | 95.3$\pm$1.4 | 85.5$\pm$5.3 | 104.1$\pm$3.9
walker2d-medium-v2 | 84.2$\pm$0.3 | 103.5$\pm$1.8 | 83.6$\pm$2.1 | 85.2$\pm$1.3 | 83.9$\pm$1.2 | 94.4$\pm$1.6
walker2d-medium-expert-v2 | 111.1$\pm$0.6 | 118.5$\pm$4.0 | 108.9$\pm$2.9 | 107.5$\pm$4.5 | 110.0$\pm$1.8 | 113.0$\pm$4.3
locomotion total | 937.7 | 1174.7 | 759.6 | 856.3 | 848.7 | 1015.5
antmaze-umaze-v2 | 96.3$\pm$1.4 | 98.9$\pm$1.3 | 79.2$\pm$4.1 | 81.1$\pm$3.4 | 87.8$\pm$2.8 | 90.0$\pm$2.4
antmaze-umaze-diverse-v2 | 93.4$\pm$4.6 | 95.9$\pm$2.8 | 51.3$\pm$4.5 | 66.7$\pm$5.0 | 72.4$\pm$4.6 | 81.3$\pm$3.9
antmaze-medium-diverse-v2 | 81.4$\pm$3.9 | 89.3$\pm$4.8 | 75.6$\pm$1.9 | 81.8$\pm$4.8 | 78.5$\pm$2.9 | 85.6$\pm$4.8
antmaze-medium-play-v2 | 86.8$\pm$1.6 | 93.9$\pm$2.7 | 81.0$\pm$2.2 | 86.2$\pm$1.3 | 83.9$\pm$1.9 | 90.1$\pm$2.0
antmaze-large-play-v2 | 42.5$\pm$5.2 | 66.1$\pm$8.2 | 39.2$\pm$7.2 | 40.0$\pm$5.3 | 40.9$\pm$6.2 | 53.1$\pm$6.8
antmaze-large-diverse-v2 | 42.3$\pm$2.2 | 57.1$\pm$2.8 | 45.0$\pm$8.7 | 52.1$\pm$2.6 | 43.7$\pm$5.5 | 54.6$\pm$2.7
antmaze total | 442.7 | 501.2 | 371.3 | 407.9 | 407.0 | 454.6
relocate-human-v1 | -0.4$\pm$0.2 | 0.2$\pm$0.2 | 1.4$\pm$0.3 | 1.7$\pm$0.4 | 0.5$\pm$0.3 | 1.0$\pm$0.3
pen-human-v1 | 68.4$\pm$8.7 | 95.6$\pm$6.2 | 91.2$\pm$5.2 | 103.6$\pm$4.2 | 79.8$\pm$7.0 | 99.6$\pm$5.2
door-human-v1 | 0.1$\pm$0.2 | 58.4$\pm$17.6 | 20.7$\pm$4.6 | 25.6$\pm$3.3 | 10.4$\pm$2.4 | 42.0$\pm$10.5
adroit total | 68.1 | 154.2 | 113.3 | 130.9 | 90.7 | 142.3
total | 1448.5 | 1830.1 | 1244.2 | 1395.1 | 1346.4 | 1612.4

##  5 Experiments

In this section, we empirically validate the effectiveness of EDIS. Sec. 5.1
showcases the considerable performance enhancement achieved by EDIS when
integrated with off-the-shelf offline-to-online algorithms. Sec. 5.2
investigates the reasons for this improvement, highlighting two key factors:
i) the robust distribution modeling capability of diffusion models, and ii)
the focus on modeling data distribution instead of the transition function,
which effectively mitigates compounding errors. Sec. 5.3 includes an ablation
study on three energy functions to underscore their critical role in our
algorithm. In this section, every experiment result is averaged over five
random seeds.

Figure 1: Comparison of EDIS and traditional model-based method. Diffusion
transition and MLP transition mean the transition function is modeled by
diffusion model and MLP respectively.

###  5.1 Enhanced Performance Achieved by EDIS

We evaluate the performance of EDIS on three benchmark tasks from D4RL (Fu et
al., 2020): MuJoCo Locomotion, AntMaze Navigation, and Adroit Manipulation. We
implement EDIS on top of base algorithms Cal-QL (Nakamoto et al., 2023), a
state-of-the-art offline-to-online method that effectively calibrates over-
conservatism of CQL (Kumar et al., 2020), and IQL (Kostrikov et al., 2022),
which adopts AWR-style policy learning in both phases. The 0.2M fine-tuning
results after 1M pre-training are presented in Tab. 1.

Notably, the integration of EDIS with Cal-QL yields a substantial 26.3%
improvement in overall performance, demonstrating significant enhancements in
the Adroit domain and MuJoCo random datasets. The latter are characterized by
a scarcity of successful demonstrations, creating a substantial gap between
offline and online distributions that poses a challenge to previous methods.
EDIS addresses this issue by directly generating online samples, overcoming
the distribution shift issue. Even combined with IQL, where the conservatism
cannot be entirely dropped during online fine-tuning due to its reliance on
in-sample data, EDIS still achieves an average performance improvement of
12.1%. For more information on implementation details, please refer to Appx.
C.

###  5.2 Comparisons between EDIS and Basic Model-based Methods

While EDIS demonstrates superior performance on the D4RL benchmarks, a natural
question arises: what contributes to this improvement? We hypothesize two main
contributors to EDIS’s success: firstly, the superior representational
capabilities of diffusion models, and secondly, the innovative modeling
approach that focuses on the distribution of $(s,a,s^{\prime})$ tuples instead
of modeling the transition function as is common in traditional model-based
methods. To substantiate our hypothesis, we have undertaken further
experiments involving two models tasked with representing the transition
function of the environments. The models have equivalent capacity as EDIS
(i.e., having the same number of parameters), allowing for a direct
comparison. These models are then used to rollout current policies within the
learned model, thereby augmenting online data. One model employs a multi-layer
perceptron (MLP) structure, while the other utilizes a diffusion model. The
base algorithm is Cal-QL and the results are shown in Fig. 1. It can be seen
that the MLP model does not exhibit enhanced performance compared to the
original algorithm. According to the theory outlined in Sec. 4, this suggests
that the performance drop induced by distribution shift and MLP model error is
similar.

In contrast, the diffusion transition model shows some improvement, indicating
that diffusion model suffers smaller model error than MLP model because of its
superior distribution modeling ability. Despite this, simply employing a
powerful diffusion model in conjunction with traditional data augmentation
methods could not match the performance achieved by EDIS. This is attributed
to the compounding error introduced by rolling out policies in a misspecified
transition model. The unique data generation approach of EDIS, which directly
generates online samples, proves crucial in achieving its remarkable
performance. To visualize the effect of compounding error reduction, we
compare the state distribution generated by these models. The result is
defered to Appx. C.2 because of space limitation. In summary, the performance
enhancement observed with EDIS stems not only from the capability of the
diffusion model but also from its distinctive data generation methodology.

###  5.3 Ablation for Energy Functions in EDIS

Table 2: Divergence comparisons for energy function ablation study (lower is
better). Each result is the average score over five random seeds $\pm$
standard deviation.
Dataset | State Divergence | Action Divergence | Transition Divergence
---|---|---|---
w/o energy | w/ energy | w/o energy | w/ energy | w/o energy | w/ energy
hopper-radnom-v2 |  0.85$\pm$0.02 | 0.73$\pm$0.03 |  0.51$\pm$0.04 | 0.39$\pm$0.02 |  0.69$\pm$0.03 | 0.66$\pm$0.04
antmaze-medium-diverse-v2 |  0.98$\pm$0.00 | 0.91$\pm$0.02 |  0.38$\pm$0.08 | 0.27$\pm$0.08 |  0.75$\pm$0.14 | 0.64$\pm$0.14
door-human-v1 |  0.40$\pm$0.04 | 0.24$\pm$0.02 |  0.54$\pm$0.07 | 0.41$\pm$0.08 |  2.57$\pm$0.06 | 2.52$\pm$0.05

Figure 2: Energy Module Ablation Study of EDIS

The energy functions, as described in Sec. 3.1, align each component in the
generated distribution with their online counterparts, including the state
distribution, the action distribution under the current policy, and the
transition fidelity. We conduct ablation studies on each energy function by
omitting its guidance during the reverse-time _SDE_ in hopper-random-v2,
antmaze-medium-diverse-v2, and door-human-v1. We compare the divergence
between the distribution generated by diffusion models and the real
environments. For the state divergence, we calculate the JS divergence between
the two distributions. The details of JS divergence and its calculation state
distribution are shown in Appx. C.4. For action divergence and state
divergence, we calculate the mean square loss of the action or next state
generated by the diffusion model and the current policy or the real dynamics.
As shown in Tab. 2, the guidance of the energy function effectively decreases
the divergence from the online sample distribution, real-world transitions,
and actions induced by the policy interacting with the environment.

Furthermore, the results in Fig. 2 demonstrate that a larger divergence of the
three parts detriments the final performance. Particularly, in environments
like hopper-random-v2, EDIS struggles to excel without the state energy
function, demonstrating the ineffectiveness of merging unrelated offline data
directly into the online fine-tuning phase.

##  6 Related Work

In this section, we provide a brief summary of related work, the complete
version is in Appx. D.

Offline-to-online Reinforcement Learning. Offline-to-online reinforcement
learning methods are developed to mitigate the dichotomy between the costly
exploration in online RL and the typically suboptimal performance of offline
RL. The learning process is typically divided into two phases: Warming-up the
policy and value functions in the offline phase and using them as
initialization in the online phase (Nair et al., 2020; Wang et al., 2023a;
Nakamoto et al., 2023; Lee et al., 2021). These approaches often employ
offline RL methods based on policy constraints or pessimism in the offline
phase (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2020).
However, the conservatism conflicts with the online phase and may induce
performance degradation. Various strategies have been implemented to tackle
this issue, such as policy expansion (Zhang et al., 2023), value function
calibration (Nakamoto et al., 2023), Q-ensemble techniques (Lee et al., 2021),
and constraint methods (Nair et al., 2020; Kostrikov et al., 2022; Li et al.,
2023). Despite the advancements, there has been less focus on the crucial
aspect of integrating useful data during the fine-tuning phase to enhance
training efficiency. Standard practices include enriching the replay buffer
with offline data (Nakamoto et al., 2023; Zhang et al., 2023), adopting
balanced sampling methods for managing both online and offline data sources
(Lee et al., 2021; Ball et al., 2023), or building models to performance
branch rollout (Rafailov et al., 2023). However, directly replaying the
offline data causes a distribution shift, and adopting balanced sampling
methods introduces large variance (Nachum et al., 2019) while rolling in the
built model suffers from compounding error (Janner et al., 2019). In contrast,
our work breaks new ground by proposing a diffusion-based generator
specifically designed to generate useful samples by thoroughly leveraging
prior knowledge in offline data.

Diffusion Model in Reinforcement Learning. Diffusion models have demonstrated
exceptional capabilities in modeling distribution (Saharia et al., 2022;
Nichol et al., 2022; Nichol & Dhariwal, 2021). Within the reinforcement
learning research, Diffuser (Janner et al., 2022) uses a diffusion model as a
trajectory generator and learns a separate return model to guide the reverse
diffusion process toward samples of high-return trajectories. Decision
Diffuser (Ajay et al., 2023) introduces conditional diffusion with reward or
constraint guidance for decision-making tasks, further boosting Diffuser’s
performance. Expanding the application of diffusion models, SYNTHER (Lu et
al., 2023a) focuses on leveraging the diffusion model for upsampling data in
both online and offline reinforcement learning scenarios. Recently, several
concurrent work also investigates generating samples with certain data
distribution by diffusion models. PolyGRAD (Rigter et al., 2023) and PGD
(Jackson et al., 2024) embed the policy for classifier-guided trajectory
generation, aiming at on-policy world modeling. However, they still use the
diffusion model to model the transition function rather than the data
distribution directly, which does not eliminate the issue of compounding
error. Finally, in the context of offline-to-online reinforcement learning,
our research pioneers the utilization of diffusion-based models to actively
generate valuable samples. This distinction from the passive reuse of offline
data marks EDIS as a novel approach, emphasizing the active role of diffusion
models in sample generation for offline-to-online RL.

##  7 Conclusion

This study underscores the crucial role of harnessing offline data effectively
to augment online fine-tuning. We introduce Energy-guided Diffusion Sampling
(EDIS), a powerful approach leveraging a state-of-the-art generative model
with energy guidance. In contrast to existing strategies that primarily rely
on initializing with offline-derived policies and Q functions or replaying
offline data, EDIS takes a proactive stance by generating synthetic data
within the online data distribution, drawing information from all available
data sources. Theoretical analysis verifies that EDIS outperforms prior
methods in addressing distribution shift challenges and effectively mitigates
compounding errors. As a versatile solution, EDIS seamlessly integrates with
prevalent offline-to-online frameworks. When combined with methods like Cal-QL
and IQL, EDIS showcases substantial performance enhancements. This superior
generative modeling capabilities of EDIS mark it as a scalable approach,
providing promising potential for designing data-efficient learning strategies
in more complex environments with high-dimensional state or action spaces.

## Acknowledgement

This work is supported by the National Science Foundation of China (61921006).
The authors thank anonymous reviewers for their helpful discussions and
suggestions for improving the article.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none of
which we feel must be specifically highlighted here.

## References

  * Ajay et al. (2021) Ajay, A., Kumar, A., Agrawal, P., Levine, S., and Nachum, O.  OPAL: offline primitive discovery for accelerating offline reinforcement learning.  In _Proceedings of the 9th International Conference on Learning Representations (ICLR’21)_ , Virtual Event, 2021.
  * Ajay et al. (2023) Ajay, A., Du, Y., Gupta, A., Tenenbaum, J. B., Jaakkola, T. S., and Agrawal, P.  Is conditional generative modeling all you need for decision making?  In _The Eleventh International Conference on Learning Representations (ICLR’23)_ , Kigali, Rwanda, 2023.
  * Ball et al. (2023) Ball, P. J., Smith, L. M., Kostrikov, I., and Levine, S.  Efficient online reinforcement learning with offline data.  In _International Conference on Machine Learning (ICML’23)_ , Honolulu, USA, 2023.
  * Chen et al. (2023a) Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J.  Offline reinforcement learning via high-fidelity generative behavior modeling.  In _The 11th International Conference on Learning Representations (ICLR’23)_ , Kigali, Rwanda, 2023a.
  * Chen & Jiang (2019) Chen, J. and Jiang, N.  Information-theoretic considerations in batch reinforcement learning.  In _Proceedings of the 36th International Conference on Machine Learning (ICML’19)_ , Long Beach, USA, 2019.
  * Chen et al. (2024) Chen, R., Liu, X., Liu, T., Jiang, S., Xu, F., and Yu, Y.  Foresight distribution adjustment for off-policy reinforcement learning.  In _Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS’24)_ , pp. 317–325, Auckland, New Zealand, 2024.
  * Chen et al. (2023b) Chen, X., Yu, Y., Zhu, Z., Yu, Z., Chen, Z., Wang, C., Wu, Y., Qin, R., Wu, H., Ding, R., and Huang, F.  Adversarial counterfactual environment model learning.  In _Advances in Neural Information Processing Systems 36 (NeurIPS’23)_ , New Orleans, LA, 2023b.
  * Fu et al. (2020) Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S.  D4RL: datasets for deep data-driven reinforcement learning.  _CoRR_ , abs/2004.07219, 2020.
  * Fujimoto & Gu (2021) Fujimoto, S. and Gu, S. S.  A minimalist approach to offline reinforcement learning.  In _Advances in Neural Information Processing Systems 34 (NeurIPS’21)_ , Virtual Event, 2021.
  * Fujimoto et al. (2019) Fujimoto, S., Meger, D., and Precup, D.  Off-policy deep reinforcement learning without exploration.  In _Proceedings of the 36th International Conference on Machine Learning (ICML’19)_ , Long Beach, USA, 2019.
  * Goodfellow et al. (2014) Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. C., and Bengio, Y.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems 27 (NeurIPS’24)_ , pp. 2672–2680, Montreal, Canada, 2014.
  * Gupta et al. (2019) Gupta, A., Kumar, V., Lynch, C., Levine, S., and Hausman, K.  Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning.  In _Proceedings of the 3rd Annual Conference on Robot Learning (CoRL’19)_ , Osaka, Japan, 2019.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _Proceedings of the 35th International Conference on Machine Learning (ICML’18)_ , Stockholmsmässan, Sweden, 2018.
  * Hester et al. (2018) Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Osband, I., et al.  Deep q-learning from demonstrations.  In _Proceedings of the 32nd AAAI conference on artificial intelligence (AAAI’18)_ , New Orleans, USA, 2018.
  * Ho et al. (2020) Ho, J., Jain, A., and Abbeel, P.  Denoising diffusion probabilistic models.  In _Proceedings of the 33rd Neural Information Processing Systems (NeurIPS’20)_ , Virtual Event, 2020.
  * Hubbs et al. (2020) Hubbs, C. D., Perez, H. D., Sarwar, O., Sahinidis, N. V., Grossmann, I. E., and Wassick, J. M.  Or-gym: A reinforcement learning library for operations research problems.  _CoRR_ , abs/2008.06319, 2020.
  * Jackson et al. (2024) Jackson, M. T., Matthews, M. T., Lu, C., Ellis, B., Whiteson, S., and Foerster, J. N.  Policy-guided diffusion.  _CoRR_ , abs/2404.06356, 2024.
  * Janner et al. (2019) Janner, M., Fu, J., Zhang, M., and Levine, S.  When to trust your model: Model-based policy optimization.  In _Proceedings of the 32nd Neural Information Processing Systems (NeurIPS’19)_ , Vancouver, Canada, 2019.
  * Janner et al. (2022) Janner, M., Du, Y., Tenenbaum, J., and Levine, S.  Planning with diffusion for flexible behavior synthesis.  In _Proceedings of the 39th International Conference on Machine Learning (ICML’22)_ , Baltimore, USA, 2022.
  * Jia et al. (2024) Jia, C., Zhang, F., Li, Y., Gao, C., Liu, X., Yuan, L., Zhang, Z., and Yu, Y.  Disentangling policy from offline task representation learning via adversarial data augmentation.  In _Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS’24)_ , pp. 944–953, Auckland, New Zealand, 2024.
  * Jin et al. (2022) Jin, X., Liu, X., Jiang, S., and Yu, Y.  Hybrid value estimation for off-policy evaluation and offline reinforcement learning.  _CoRR_ , abs/2206.02000, 2022.
  * Karras et al. (2022) Karras, T., Aittala, M., Aila, T., and Laine, S.  Elucidating the design space of diffusion-based generative models.  In _Proceedings of the 36th Neural Information Processing Systems (NeurIPS’22)_ , LA, USA, 2022.
  * Kostrikov et al. (2022) Kostrikov, I., Nair, A., and Levine, S.  Offline reinforcement learning with implicit q-learning.  In _Proceedings of the 10-th International Conference on Learning Representations (ICLR’22)_ , Virtual Event, 2022.
  * Kumar et al. (2020) Kumar, A., Zhou, A., Tucker, G., and Levine, S.  Conservative q-learning for offline reinforcement learning.  In _Advances in Neural Information Processing Systems 33 (NeurIPS’20)_ , Virtual Event, 2020.
  * Lee et al. (2021) Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J.  Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.  In _Conference on Robot Learning (CoRL’21)_ , London, UK, 2021.
  * Li et al. (2023) Li, J., Hu, X., Xu, H., Liu, J., Zhan, X., and Zhang, Y.  PROTO: iterative policy regularized offline-to-online reinforcement learning.  _CoRR_ , abs/2305.15669, 2023.
  * Ling et al. (2024) Ling, H., Wang, Z., and Wang, J.  Learning to stop cut generation for efficient mixed-integer linear programming.  _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’24)_ , 38(18):20759–20767, Mar. 2024.
  * Liu et al. (2021) Liu, X.-H., Xue, Z., Pang, J., Jiang, S., Xu, F., and Yu, Y.  Regret minimization experience replay in off-policy reinforcement learning.  In _Proceedings of 34th conference on Neural Information Processing Systems (NeurIPS’21)_ , Virtual Event, 2021.
  * Lu et al. (2023a) Lu, C., Ball, P. J., and Parker-Holder, J.  Synthetic experience replay.  _CoRR_ , abs/2303.06614, 2023a.
  * Lu et al. (2023b) Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J.  Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning.  In _Proceedings of the 40th International Conference on Machine Learning(ICML’23)_ , volume 202, pp. 22825–22855, Honolulu, HI, 2023b.
  * Mazoure et al. (2023) Mazoure, B., Talbott, W., Bautista, M. Á., Hjelm, R. D., Toshev, A., and Susskind, J. M.  Value function estimation using conditional diffusion models for control.  _CoRR_ , abs/2306.07290, 2023.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M. A., Fidjeland, A., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 2015.
  * Nachum et al. (2019) Nachum, O., Chow, Y., Dai, B., and Li, L.  Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections.  In _Proceedings of the 32nd Neural Information Processing Systems (NeurIPS’19)_ , pp. 2315–2325, Vancouver, Canada, 2019.
  * Nair et al. (2020) Nair, A., Dalal, M., Gupta, A., and Levine, S.  Accelerating online reinforcement learning with offline datasets.  _CoRR_ , abs/2006.09359, 2020.
  * Nakamoto et al. (2023) Nakamoto, M., Zhai, Y., Singh, A., Mark, M. S., Ma, Y., Finn, C., Kumar, A., and Levine, S.  Cal-ql: Calibrated offline RL pre-training for efficient online fine-tuning.  _CoRR_ , abs/2303.05479, 2023.
  * Nichol & Dhariwal (2021) Nichol, A. Q. and Dhariwal, P.  Improved denoising diffusion probabilistic models.  In Meila, M. and Zhang, T. (eds.), _Proceedings of the 38th International Conference on Machine Learning (ICML’21)_ , Virtual Event, 2021.
  * Nichol et al. (2022) Nichol, A. Q., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P., McGrew, B., Sutskever, I., and Chen, M.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In _Proceedings of the 39th International Conference on Machine Learning (ICML’22)_ , Baltimore, USA, 2022.
  * Peng et al. (2020) Peng, X. B., Coumans, E., Zhang, T., Lee, T. E., Tan, J., and Levine, S.  Learning agile robotic locomotion skills by imitating animals.  In _Proceedings of the 14th Robotics: Science and Systems (RSS’20)_ , Virtual Event, 2020.
  * Rafailov et al. (2023) Rafailov, R., Hatch, K. B., Kolev, V., Martin, J. D., Phielipp, M., and Finn, C.  Moto: Offline pre-training to online fine-tuning for model-based robot learning.  In _Conference on Robot Learning (CoRL’23)_ , pp. 3654–3671, 2023.
  * Rajeswaran et al. (2018) Rajeswaran, A., Kumar, V., Gupta, A., Vezzani, G., Schulman, J., Todorov, E., and Levine, S.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  In _Proceedings of the 12th Robotics: Science and Systems (RSS’18)_ , PA, USA, 2018.
  * Rigter et al. (2023) Rigter, M., Yamada, J., and Posner, I.  World models via policy-guided trajectory diffusion.  _CoRR_ , abs/2312.08533, 2023.
  * Saharia et al. (2022) Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, S. K. S., Lopes, R. G., Ayan, B. K., Salimans, T., Ho, J., Fleet, D. J., and Norouzi, M.  Photorealistic text-to-image diffusion models with deep language understanding.  In _Proceedings of the 35th Neural Information Processing Systems (NeurIPS’22)_ , New Orleans, USA, 2022.
  * Schaal (1996) Schaal, S.  Learning from demonstration.  _Advances in neural information processing systems_ , 9, 1996.
  * Sinha et al. (2022) Sinha, S., Song, J., Garg, A., and Ermon, S.  Experience replay with likelihood-free importance weights.  In _Learning for Dynamics and Control Conference (L4RC’22)_ , Stanford, USA, 2022.
  * Song et al. (2022) Song, Y., Zhou, Y., Sekhari, A., Bagnell, J. A., Krishnamurthy, A., and Sun, W.  Hybrid rl: Using both offline and online data can make rl efficient.  In _The 11th International Conference on Learning Representations (ICLR’22)_ , Virtual Event, 2022.
  * Sutton & Barto (2018) Sutton, R. S. and Barto, A. G.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Tarasov et al. (2022) Tarasov, D., Nikulin, A., Akimov, D., Kurenkov, V., and Kolesnikov, S.  CORL: Research-oriented deep offline reinforcement learning library.  In _3rd Offline RL Workshop: Offline RL as a ”Launchpad”_ , 2022.
  * van den Oord et al. (2018) van den Oord, A., Li, Y., and Vinyals, O.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748, 2018.
  * Vecerik et al. (2017) Vecerik, M., Hester, T., Scholz, J., Wang, F., Pietquin, O., Piot, B., Heess, N., Rothörl, T., Lampe, T., and Riedmiller, M.  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _CoRR_ , abs/1707.08817, 2017.
  * von Luxburg & Schölkopf (2011) von Luxburg, U. and Schölkopf, B.  Statistical learning theory: Models, concepts, and results.  In _Inductive Logic_ , volume 10, pp. 651–706. Elsevier, 2011.
  * Wang et al. (2024) Wang, J., Wang, Z., Li, X., Kuang, Y., Shi, Z., Zhu, F., Yuan, M., Zeng, J., Zhang, Y., and Wu, F.  Learning to cut via hierarchical sequence/set model for efficient mixed-integer programming.  _CoRR_ , abs/2008.06319, 2024.
  * Wang et al. (2023a) Wang, S., Yang, Q., Gao, J., Lin, M. G., Chen, H., Wu, L., Jia, N., Song, S., and Huang, G.  Train once, get a family: State-adaptive balances for offline-to-online reinforcement learning.  _CoRR_ , abs/2310.17966, 2023a.
  * Wang et al. (2018) Wang, X., Chen, Y., Yang, J., Wu, L., Wu, Z., and Xie, X.  A reinforcement learning framework for explainable recommendation.  In _Proceedings of the 18th International Conference on Data Mining (ICDM’18)_ , Singapore, 2018.
  * Wang et al. (2022) Wang, Z., Wang, J., Zhou, Q., Li, B., and Li, H.  Sample-efficient reinforcement learning via conservative model-based actor-critic.  _Proceedings of the AAAI Conference on Artificial Intelligence (AAAI’22)_ , 36(8):8612–8620, Jun. 2022.
  * Wang et al. (2023b) Wang, Z., Hunt, J. J., and Zhou, M.  Diffusion policies as an expressive policy class for offline reinforcement learning.  In _The Eleventh International Conference on Learning Representations (ICLR’23)_ , Kigali, Rwanda, 2023b.
  * Wang et al. (2023c) Wang, Z., Li, X., Wang, J., Kuang, Y., Yuan, M., Zeng, J., Zhang, Y., and Wu, F.  Learning cut selection for mixed-integer linear programming via hierarchical sequence model.  In _The Eleventh International Conference on Learning Representations (ICLR’23)_ , 2023c.
  * Xu et al. (2021) Xu, T., Li, Z., and Yu, Y.  Nearly minimax optimal adversarial imitation learning with known and unknown transitions.  _CoRR abs/2106.10424_ , 2021.
  * Zhang et al. (2023) Zhang, H., Xu, W., and Yu, H.  Policy expansion for bridging offline-to-online reinforcement learning.  In _The Eleventh International Conference on Learning Representations(ICLR’23)_ , Kigali, Rwanda, 2023.
  * Zhao et al. (2018) Zhao, X., Zhang, L., Ding, Z., Xia, L., Tang, J., and Yin, D.  Recommendations with negative feedback via pairwise deep reinforcement learning.  In _Proceedings of the 24th International Conference on Knowledge Discovery & Data Mining (KDD’18)_, London, UK, 2018.
  * Zhao et al. (2022) Zhao, Y., Boney, R., Ilin, A., Kannala, J., and Pajarinen, J.  Adaptive behavior cloning regularization for stable offline-to-online reinforcement learning.  _CoRR_ , abs/2210.13846, 2022.
  * Zhu et al. (2019) Zhu, H., Gupta, A., Rajeswaran, A., Levine, S., and Kumar, V.  Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost.  In _Proceedings of the 36th International Conference on Robotics and Automation (ICRA’19)_ , Montreal, Canada, 2019.
  * Zhu et al. (2018) Zhu, Y., Wang, Z., Merel, J., Rusu, A. A., Erez, T., Cabi, S., Tunyasuvunakool, S., Kramár, J., Hadsell, R., de Freitas, N., and Heess, N.  Reinforcement and imitation learning for diverse visuomotor skills.  In _Proceedings of the 12th Robotics: Science and Systems (RSS’18)_ , PA, USA, 2018.

##  Appendix A Proofs and additional Theory

###  A.1 Proof of Thm. 3.1

Because $e^{{\mathcal{E}}_{1}(s)}\propto\frac{p_{\theta}(s)}{d^{\pi}(s)}$,
$e^{{\mathcal{E}}_{2}(a|s)}\propto\frac{p_{\theta}(a|s)}{\pi(a|s))}$,
$e^{{\mathcal{E}}_{3}(s^{\prime}|s,a)}\propto\frac{p_{\theta}(s^{\prime}|s,a)}{T^{\pi}(s^{\prime}|s,a))}$,
we have $e^{{\mathcal{E}}_{1}(s)}=k_{1}\frac{p_{\theta}(s)}{d^{\pi}(s)}$,
$e^{{\mathcal{E}}_{2}(a|s)}=k_{2}\frac{p_{\theta}(a|s)}{\pi(a|s)}$,
$e^{{\mathcal{E}}_{3}(s^{\prime}|s,a)}=k_{3}\frac{p_{\theta}(s^{\prime}|s,a)}{T(s^{\prime}|s,a)}$,
where $k_{1}$, $k_{2}$ and $k_{3}$ are arbitrary positive constants.

$\displaystyle\quad\
e^{-({\mathcal{E}}_{1}(s)+{\mathcal{E}}_{2}(a|s)+{\mathcal{E}}_{3}(s^{\prime}|s,a))}$
$\displaystyle=k_{1}k_{2}k_{3}\frac{d^{\pi}(s)\pi(a|s)T(s^{\prime}|s,a)}{p_{\theta}(s)p_{\theta}(a|s)p_{\theta}(s^{\prime}|s,a)}$
$\displaystyle=k_{1}k_{2}k_{3}\frac{q_{\pi}(s,a,s^{\prime})}{p_{\theta}(s,a,s^{\prime})},$

where the last equation comes from
$q_{\pi}(s,a,s^{\prime})=d^{\pi}(s)\pi(a|s)T(s^{\prime}|s,a)$ and
$p_{\theta}(s,a,s^{\prime})=p_{\theta}(s)p_{\theta}(a|s)p_{\theta}(s^{\prime}|s,a)$.
Because $k_{1}k_{2}k_{3}>0$, we conclude
$e^{-({\mathcal{E}}_{1}(s)+{\mathcal{E}}_{2}(a|s)+{\mathcal{E}}_{3}(s^{\prime}|s,a))}\propto\frac{q_{\pi}(s,a,s^{\prime})}{p_{\theta}(s,a,s^{\prime})}$,
then $p_{\theta}(s,a,s^{\prime})e^{-{\mathcal{E}}(s,a,s^{\prime})}\propto
q_{\pi}(s,a,s^{\prime})$, and thus Eq. (3) holds.

###  A.2 Proof of Thm. 4.1

Before proof this theorem, we introduce some new notations.

${\mathcal{T}}^{\pi}f(s,a):=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim
P(s,a)}[V^{\pi}_{f}(s^{\prime})],$

where
$V^{\pi}_{f}(s^{\prime}):=\mathbb{E}_{a^{\prime}\sim\pi}f(s^{\prime},a^{\prime})$.
Recursively define
$({\mathcal{T}}^{\pi})^{j}=({\mathcal{T}}^{\pi})({\mathcal{T}}^{\pi})^{j-1}$.

Similarly, we define

${\mathcal{T}}f(s,a):=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim
P(s,a)}[V_{f}(s^{\prime})],$

where $V_{f}(s^{\prime}):=\max_{a^{\prime}}f(s^{\prime},a^{\prime})$.

Given a function $f:X\mapsto{\mathbb{R}}$, define its norm $\|\cdot\|_{\mu}$
with respect to distribution $\mu$ as:

$\|f\|_{\mu}=\sqrt{\sum_{x}f^{2}(x)\mu(x)}.$

Suppose $\nu(s)$ is a state state distribution $\pi(a|s)$ is an action
distribution based on given state $s$, then
$\nu\times\pi(s,a)=\nu(s)\pi(a|s)$.

######  Lemma A.1 (Theorem 1 of (Sinha et al., 2022)).

The Bellman operator ${\mathcal{T}}$ is a $\gamma$-contraction with respect to
the $\|\cdot\|_{d^{\pi}}$ norm, i.e.,

$\left\|{\mathcal{T}}^{\pi}f-{\mathcal{T}}^{\pi}f^{\prime}\right\|_{d^{\pi}}\leq\gamma\left\|f-f^{\prime}\right\|_{d^{\pi}},\quad\forall
f,f^{\prime}\in{\mathcal{F}}.$

######  Lemma A.2.

Let $\nu$ be any admissible state distribution, $\pi_{1}$ and $\pi_{2}$ be two
policies, and
$C_{\pi_{1},\pi_{2}}=\max_{s,a}\frac{\pi_{1}(a|s)}{\pi_{2}(a|s)}$,
$\left\|\cdot\right\|_{\nu\times\pi_{1}}\leq\sqrt{C_{\pi_{1},\pi_{2}}}\left\|\cdot\right\|_{\nu\times\pi_{2}}$.

###### Proof.

For any function $f:{\mathcal{S}}\times{\mathcal{A}}\to{\mathbb{R}}$, we have

$\displaystyle\left\|f\right\|_{\nu\times\pi_{1}}$
$\displaystyle=\left(\sum_{s,a}|f(s,a)|^{2}\nu(s)\pi_{1}(a|s)\right)^{1/2}$
$\displaystyle\leq\left(\sum_{s,a}|f(s,a)|^{2}C_{\pi_{1},\pi_{2}}\nu(s)\pi_{2}(a|s)\right)^{1/2}$
$\displaystyle=\sqrt{C_{\pi_{1},\pi_{2}}}\left(\sum_{s,a}|f(s,a)|^{2}\nu(s)\pi_{2}(a|s)\right)^{1/2}$
$\displaystyle=\sqrt{C_{\pi_{1},\pi_{2}}}\left\|f\right\|_{\nu\times\pi_{2}}.$

∎

To simplify the notation, we use $\widehat{{\mathcal{T}}}^{\pi}$ to denote the
Bellman operator under limited data, i.e.,

$\widehat{{\mathcal{T}}}^{\pi}f=\operatorname*{arg\,min}_{g}\sum_{i=1}^{N}(g-r-\gamma
V^{\pi}_{f})^{2}.$

######  Lemma A.3.

If $f\in{\mathcal{F}}$, with probability at least $1-\delta$, we have

$\left\|(\widehat{{\mathcal{T}}}^{\pi})^{j}f-({\mathcal{T}}^{\pi})^{j}f\right\|_{d^{\pi}}\leq\frac{1-\gamma^{j}}{1-\gamma}\frac{56V^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}.$

###### Proof.

The proof is based on Lem. 16 of (Chen & Jiang, 2019).

First, we consider
$\left\|\widehat{{\mathcal{T}}}^{\pi}f-{\mathcal{T}}^{\pi}f\right\|_{d^{\pi}}$.
Define

$\displaystyle L_{d^{\pi}}(g,h)=\mathbb{E}_{s,a\sim d^{\pi}}(g-r-\gamma
V^{\pi}_{h})^{2}$ $\displaystyle L_{D}(g,h)=\mathbb{E}_{s,a\sim D}(g-r-\gamma
V_{h}^{\pi})^{2}$

Then we have ${\mathcal{T}}f=\operatorname*{arg\,min}_{g}L_{d^{\pi}}(g,f)$ and
$\widehat{{\mathcal{T}}}f=\operatorname*{arg\,min}_{g}L_{D}(g,f)$.

To simplify the notations, we define

$X(g,f,g^{*}):=(g(s,a)-r-\gamma
V^{\pi}_{f}(s^{\prime}))^{2}-(g^{*}(s,a)-r-V_{f}^{\pi}(s^{\prime}))^{2}$

Next, we bound the variance of $X(g,f,g^{*})$,

$\displaystyle{\mathbb{V}}\left[X(g,f,g^{*})\right]$
$\displaystyle\leq\mathbb{E}\left[\left\|X^{2}(g,f,g^{*})\right\|\right]$
$\displaystyle=\mathbb{E}\left[\left((g(s,a)-r-\gamma
V^{\pi}_{f}(s^{\prime}))^{2}-(g^{*}(s,a)-r-V_{f}^{\pi}(s^{\prime}))^{2}\right)\right]$
$\displaystyle=\mathbb{E}\left[(g(s,a)-g^{*}(s,a))^{2}(g(s,a)+g^{*}(s,a)-2r-2\gamma
V_{f}^{\pi}(s^{\prime}))^{2}\right]$ $\displaystyle\leq
4V_{\max}^{2}\mathbb{E}\left[(g(s,a)-g^{*}(s,a))^{2}\right]$
$\displaystyle=4V_{\max}^{2}\left\|g-g^{*}\right\|^{2}_{d^{\pi}}$

Note that

$\displaystyle\left\|g-g^{*}\right\|^{2}_{d^{\pi}}$
$\displaystyle\overset{(a)}{\leq}2(\|g-r-\gamma
V_{f}^{\pi}\|^{2}_{d^{\pi}}+\|r+\gamma V_{f}^{\pi}-g^{*}\|_{d^{\pi}}^{2})$
$\displaystyle=2\mathbb{E}[X(g,f,g^{*})],$

where (a) holds because $(a+b)^{2}\leq 2(a^{2}+b^{2})$.

Finally, we apply Bernstein’s inequality and union bound over all
$f\in{\mathcal{F}}$. With probability at least $1-\delta$, we have

$\displaystyle\quad\
\mathbb{E}[X(g,f,g^{*})-\frac{1}{n}\sum_{i=1}^{n}X_{i}(g,f,g^{*})$
$\displaystyle\leq\sqrt{\frac{2{\mathbb{V}}\left[X(g,f,g^{*})\right]\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{n}}+\frac{4V^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}$
$\displaystyle\leq\sqrt{\frac{16V_{\max}^{2}\mathbb{E}[X(g,f,g^{*})\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{n}}+\frac{4V^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}$

Since $\widehat{{\mathcal{T}}}^{\pi}f$ minimizes $L_{D}(\cdot,f)$, it also
minimizes $\frac{1}{n}X_{i}(\cdot,f,g^{*})$. This is because the two
objectives only differ by a constant ${\mathcal{L}}_{D}(\cdot,f)$. Therefore,
we have

$\frac{1}{n}\sum_{i=1}^{n}X_{i}(\widehat{{\mathcal{T}}}^{\pi}f,f,{\mathcal{T}}^{\pi}f)\leq\frac{1}{n}\sum_{i=1}^{n}X_{i}({\mathcal{T}}^{\pi}f,f,{\mathcal{T}}^{\pi}f)=0.$

Then

$\mathbb{E}[X(\widehat{{\mathcal{T}}}^{\pi},f,{\mathcal{T}}^{\pi}f)\leq\sqrt{\frac{16V_{\max}^{2}\mathbb{E}[X(g,f,g^{*})\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{n}}+\frac{4V^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}.$

Solving for the quadratic formula, we have

$\mathbb{E}[X(\widehat{{\mathcal{T}}}^{\pi},f,{\mathcal{T}}^{\pi}f)\leq\frac{56V^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}.$

Noticing that

$\displaystyle\left\|\widehat{{\mathcal{T}}}^{\pi}f-{\mathcal{T}}^{\pi}f\right\|^{2}_{d^{\pi}}$
$\displaystyle={\mathcal{L}}_{d^{\pi}}(\widehat{{\mathcal{T}}}^{\pi}f,f)+{\mathcal{L}}_{d^{\pi}}({\mathcal{T}}^{\pi}f,f)$
(10)
$\displaystyle=\mathbb{E}[X(\widehat{{\mathcal{T}}}^{\pi}f,f,{\mathcal{T}}^{\pi}f)]\leq\epsilon,$

where
$\epsilon=\frac{56V^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}$.

For
$\left\|(\widehat{{\mathcal{T}}}^{\pi})^{2}f-({\mathcal{T}}^{\pi})^{2}f\right\|^{2}_{d^{\pi}}$,
note that

$\displaystyle\left\|(\widehat{{\mathcal{T}}}^{\pi})^{2}f-({\mathcal{T}}^{\pi})^{2}f\right\|^{2}_{d^{\pi}}$
$\displaystyle=\left\|\widehat{{\mathcal{T}}}^{\pi}(\widehat{{\mathcal{T}}}^{\pi}f)-{\mathcal{T}}^{\pi}({\mathcal{T}}^{\pi}f)\right\|^{2}_{d^{\pi}}$
$\displaystyle\leq\left\|\widehat{{\mathcal{T}}}^{\pi}(\widehat{{\mathcal{T}}}^{\pi}f)-{\mathcal{T}}^{\pi}(\widehat{{\mathcal{T}}}^{\pi}f)\right\|^{2}_{d^{\pi}}+\left\|{\mathcal{T}}^{\pi}(\widehat{{\mathcal{T}}}^{\pi}f)-{\mathcal{T}}^{\pi}({\mathcal{T}}^{\pi}f)\right\|^{2}_{d^{\pi}}$
$\displaystyle\overset{(a)}{\leq}\epsilon+\left\|{\mathcal{T}}^{\pi}(\widehat{{\mathcal{T}}}^{\pi}f)-{\mathcal{T}}^{\pi}({\mathcal{T}}^{\pi}f)\right\|^{2}_{d^{\pi}}$
$\displaystyle\overset{(b)}{\leq}\epsilon+\gamma^{2}\left\|\widehat{{\mathcal{T}}}^{\pi}f-{\mathcal{T}}^{\pi}f\right\|^{2}_{d^{\pi}}$
$\displaystyle\overset{(c)}{\leq}\epsilon+\gamma^{2}\epsilon,$

where (a) and (c) use Eq. (10), (b) use Lem. A.1.

Recursively, we have

$\left\|(\widehat{{\mathcal{T}}}^{\pi})^{2}f-({\mathcal{T}}^{\pi})^{2}f\right\|^{2}_{d^{\pi}}\leq\sum_{i=0}^{j}\gamma^{2i}\epsilon=\frac{1-\gamma^{2j}}{1-\gamma^{2}}\frac{56V^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}.$

∎

######  Lemma A.4 (Lemma 14 in (Chen & Jiang, 2019)).

Assume $f,f^{\prime}\in{\mathcal{F}}$ and define
$\pi_{f,f^{\prime}}:=\operatorname*{arg\,max}_{a\in{\mathcal{A}}}\max\\{f(s,a),f^{\prime}(s,a)\\}$.
Then we have $\forall\nu\in\Delta({\mathcal{S}}\times{\mathcal{A}})$,

$\left\|{\mathcal{T}}f-{\mathcal{T}}f^{\prime}\right\|_{\nu}\leq\gamma\left\|f-f^{\prime}\right\|_{\nu\times\pi_{f,f^{\prime}}}.$

Suppose $\pi$ is induced by $Q_{0}$,

$\displaystyle J(\pi^{*})-J(\pi)$
$\displaystyle=\sum_{t=0}^{\infty}\mathbb{E}_{s\sim
d^{\pi}}\gamma^{t}[V^{*}(s)-Q^{*}(s,\pi)]$ (11)
$\displaystyle\leq\sum_{t=0}^{\infty}\gamma^{t}\mathbb{E}_{s\sim
d^{\pi}}[Q^{*}(s,\pi^{*})-\widetilde{Q}(s,\pi^{*})+\widetilde{Q}(s,\pi)-Q^{*}(s,\pi)]$
$\displaystyle\leq\sum_{t=0}^{\infty}\left(\sum_{s,a}d^{\pi}(s)\pi^{*}(a|s)\left|Q^{*}(s,a)-\widetilde{Q}(s,a)\right|+\sum_{s,a}d^{\pi}(s)\pi(a|s)\left|Q^{*}(s,a)-\widetilde{Q}(s,a)\right|\right)$
$\displaystyle\leq\sum_{t=0}^{\infty}\gamma^{t}\left(\left\|Q^{*}-\widetilde{Q}\right\|_{d^{\pi}\times\pi^{*}}+\left\|Q^{*}-\widetilde{Q}\right\|_{d^{\pi}}\right)$

$\displaystyle\quad\ \left\|Q^{*}-\widetilde{Q}\right\|_{\nu}$ (12)
$\displaystyle=\left\|Q^{*}-{\mathcal{T}}Q_{0}+{\mathcal{T}}Q_{0}-Q^{\pi}+Q^{\pi}-\widetilde{Q}\right\|_{\nu}$
$\displaystyle\leq\left\|Q^{*}-{\mathcal{T}}Q_{0}\right\|_{\nu}+\left\|{\mathcal{T}}Q_{0}-Q^{\pi}\right\|_{\nu}+\left\|Q^{\pi}-\widetilde{Q}\right\|_{\nu}$
$\displaystyle=\left\|Q^{*}-{\mathcal{T}}Q_{0}\right\|_{\nu}+\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{\nu}+\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-\widetilde{Q}\right\|_{\nu}+\left\|{\mathcal{T}}Q_{0}-Q^{\pi}\right\|_{\nu}$
$\displaystyle=\left\|{\mathcal{T}}Q^{*}-{\mathcal{T}}Q_{0}\right\|_{\nu}+\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{\nu}+\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-\widetilde{Q}\right\|_{\nu}+\left\|{\mathcal{T}}Q_{0}-Q^{\pi}\right\|_{\nu}$
$\displaystyle\overset{(a)}{\leq}\gamma\left\|Q^{*}-Q_{0}\right\|_{\nu\times\pi_{Q^{*},Q_{0}}}+\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{\nu}+\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-\widetilde{Q}\right\|_{\nu}+\left\|{\mathcal{T}}Q_{0}-Q^{\pi}\right\|_{\nu}$
$\displaystyle=\gamma\left\|Q^{*}-Q_{0}\right\|_{\nu\times\pi_{Q^{*},Q_{0}}}+\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{\nu}+\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-(\widehat{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{\nu}+\left\|{\mathcal{T}}Q_{0}-Q^{\pi}\right\|_{\nu},$

where (a) uses Lem. A.4. The last equality comes from the fact that
$\widetilde{Q}=(\widehat{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}$.

When $\nu=d^{\pi}\times\pi^{*}$, we can use Lem. A.2 to obtain

$\left\|\cdot\right\|_{\nu}\leq\sqrt{C_{\pi,\pi^{*}}}\left\|\cdot\right\|_{d^{\pi}}.$

When $\nu=d^{\pi}\times\pi_{Q^{*},Q_{0}}$, note that $\pi_{Q^{*},Q_{0}}$ is
more similar to $\pi$ than $\pi^{*}$, so

$\left\|\cdot\right\|_{\nu\times\pi_{Q^{*},Q_{0}}}\leq\sqrt{C_{\pi,\pi^{*}}}\left\|\cdot\right\|_{d^{\pi}}.$

Therefore, we only need to consider how to bound these terms when
$\nu=d^{\pi}$.

According to Lem. A.1,

$\|{\mathcal{T}}Q_{0}-Q^{\pi}\|_{d^{\pi}}\overset{(b)}{=}\|{\mathcal{T}}Q_{0}-{\mathcal{T}}^{\pi}Q^{\pi}\|_{d^{\pi}}=\|{\mathcal{T}}^{\pi}Q_{0}-{\mathcal{T}}^{\pi}Q^{\pi}\|_{d^{\pi}}\overset{(c)}{\leq}\gamma\|Q^{\pi}-Q_{0}\|_{d^{\pi}},$

where (b) comes from $\pi(a|s)=\operatorname*{arg\,max}Q_{0}(s,a)$, (c) uses
Lem. A.1.

Let $j\to\infty$, we can apply Lem. A.3 to term
$\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-(\widehat{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}$,

$\left\|({\mathcal{T}}^{\pi_{k}})^{\infty}Q_{0}-(\widehat{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{{d^{\pi}}}\leq\epsilon^{\prime},$

where
$\epsilon^{\prime}=\sqrt{\frac{1}{1-\gamma^{2}}\frac{56V^{2}_{\max}\log\frac{|{\mathcal{F}}|}{\delta}}{3n}}$
and the inequality holds with probability at least $1-\delta$.

For the term
$\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}$, note
that

$\displaystyle\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}$
$\displaystyle=\left\|{\mathcal{T}}^{\pi}Q^{\pi}-{\mathcal{T}}^{\pi}({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}$
$\displaystyle\leq\gamma\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}$

The inequality uses Lem. A.1. Recursively, we get
$\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}=0$.

To sum up, we have

$\displaystyle J(\pi^{*})-J(\pi)$
$\displaystyle\leq\sum_{t=0}^{\infty}\gamma^{t}\left(\left\|Q^{*}-\widetilde{Q}\right\|_{d^{\pi}\times\pi^{*}}+\left\|Q^{*}-\widetilde{Q}\right\|_{d^{\pi}}\right)$
$\displaystyle\leq\frac{2\gamma}{1-\gamma}\left(\left\|Q^{*}-Q_{0}\right\|_{d^{\pi}\times\pi_{Q^{*},Q_{0}}}+\left\|Q^{\pi}-Q_{0}\right\|_{d^{\pi}}\right)+\frac{\sqrt{C_{\pi,\pi^{*}}}+1}{1+\gamma}\frac{1}{(1-\gamma)^{2.5}}\sqrt{\frac{56R^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}}$
$\displaystyle\leq\frac{2\gamma}{1-\gamma}\left(\left\|Q^{*}-Q_{0}\right\|_{\infty}+\left\|Q^{\pi}-Q_{0}\right\|_{\infty}\right)+\frac{\sqrt{C_{\pi,\pi^{*}}}+1}{1+\gamma}\frac{1}{(1-\gamma)^{2.5}}\sqrt{\frac{56R^{2}_{\max}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3n}}.$

Then we conclude the proof.

###  A.3 Proof of Thm. 4.3

Let the state distribution of the combined buffer as $d(s)$, the
concentratability coefficient between $d^{\pi}$ and $d$ as $C_{d}$, i.e.,
$C_{d}=\max_{s}\frac{d^{\pi}(s)}{d(s)}$. We introduce the following lemmas.

######  Lemma A.5.

$D_{1}$, $D_{2}$ are two dataset containing $N$ samples and $n$ samples,
respectively. If the distribution of $D_{1}$ is $\mu$ and $D_{2}$ is $\nu$,
the concentratabilty coefficient between $\nu$ and $\mu$ is $C$, then the
concentratability coefficient between $\nu$ and combined buffer $\rho$ is
$\frac{(N+n)C}{N+nC}$.

###### Proof.

The distribution of combined buffer is $\rho=\frac{N\mu(s)+n\nu(s)}{N+n}$.

$\displaystyle\max_{s}\frac{\nu(s)}{\rho(s)}$
$\displaystyle=\max_{s}\frac{(N+n)\nu(s)}{N\mu(s)+n\nu(s)}$
$\displaystyle=\max_{s}\frac{N+n}{N\frac{\mu(s)}{\nu(s)}+n}$
$\displaystyle=\frac{(N+n)C}{N+nC}.$

∎

######  Lemma A.6 (Lemma 12 in (Chen & Jiang, 2019)).

Let $\mu(s)$ be any admissible distribution, $C$ be the concentratablility
coefficient of $\mu(s)$ and $\nu(s)$, then
$\|\cdot\|_{\nu}\leq\sqrt{C}\|\cdot\|_{\mu}$.

Note that Eq. (11) and (12) hold whatever the distribution of training data
is. By replacing $\nu$ in Eq. (12) with $d\times\pi^{*}$ and
$d\times\pi_{Q^{*},Q_{0}}$, and apply Lem. A.6, Lem. A.1 and Lem. A.5:

$\displaystyle\left\|\cdot\right\|_{d\times\pi^{*}}\leq\sqrt{\widetilde{C_{d}}C_{\pi,\pi^{*}}}\left\|\cdot\right\|_{d^{\pi}},$
$\displaystyle\left\|\cdot\right\|_{d\times\pi_{Q^{*},Q_{0}}}\leq\sqrt{\widetilde{C_{d}}C_{\pi,\pi^{*}}}\left\|\cdot\right\|_{d^{\pi}},$

where $\widetilde{C_{d}}=\frac{(N+n)C_{d}}{N+nC_{d}}$.

Similar to the derivation of last section, we have

$\displaystyle
J(\pi^{*})-J(\pi)\leq\frac{2\gamma}{1-\gamma}\left(\left\|Q^{*}-Q_{0}\right\|_{\infty}+\left\|Q^{\pi}-Q_{0}\right\|_{\infty}\right)+\frac{\sqrt{\widetilde{C_{d}}C_{\pi,\pi^{*}}}+1}{(1-\gamma)^{4}}\frac{56R_{\max}^{2}\log\frac{|{\mathcal{F}}|^{2}}{\delta}}{3(n+N)}$

###  A.4 Proof of Thm. 4.4

Define th transition function learned by the model is $\widetilde{T}$. To
simplify the notation, we use $\widetilde{{\mathcal{T}}}^{\pi}$ to denote the
Bellman operator under learned model, i.e.,

$\widehat{{\mathcal{T}}}^{\pi}f=\operatorname*{arg\,min}_{g}\sum_{i=1}^{N}(g(s,a)-r(s,a)-\gamma
V^{\pi}_{f}(\tilde{s}^{\prime}))^{2},$

where $\tilde{s}^{\prime}\sim\widetilde{T}(s^{\prime}|s,a)$.

######  Lemma A.7.

$\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-(\widetilde{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{\hat{d}^{\pi}}\leq\frac{\gamma}{1-\gamma}L\epsilon^{d}_{m}$

###### Proof.

Note that

$\left\|{\mathcal{T}}^{\pi}Q_{0}-\widetilde{{\mathcal{T}}}^{\pi}Q_{0}\right\|_{\hat{d}^{\pi}}=\gamma\left\|Q_{0}(s^{\prime},\pi)-Q_{0}(\tilde{s}^{\prime},\pi)\right\|_{\hat{d}^{\pi}},$

where $s^{\prime}\sim T$, $\tilde{s}^{\prime}\sim\widetilde{T}$.

Since $Q_{0}$ is $L$-Lipschitz,

$\displaystyle\left\|Q_{0}(s^{\prime},\pi)-Q_{0}(\tilde{s}^{\prime},\pi)\right\|_{\hat{d}^{\pi}}\leq
L\left\|s^{\prime}-\tilde{s}^{\prime}\right\|_{\hat{d}^{\pi}}\leq
L\epsilon^{d}_{m}$

Then,

$\displaystyle\quad\
\left\|({\mathcal{T}}^{\pi})^{2}Q_{0}-(\widetilde{{\mathcal{T}}}^{\pi})^{2}Q_{0}\right\|_{\hat{d}^{\pi}}$
$\displaystyle=\gamma\left\|({\mathcal{T}}^{\pi}Q_{0})(s^{\prime},\pi)-(\widetilde{{\mathcal{T}}}^{\pi}Q_{0})(\tilde{s}^{\prime},\pi)\right\|_{\hat{d}^{\pi}}$
$\displaystyle\leq\gamma\left\|({\mathcal{T}}^{\pi}Q_{0})(s^{\prime},\pi)-(\widetilde{{\mathcal{T}}}^{\pi}Q_{0})(s^{\prime},\pi)\right\|_{\hat{d}^{\pi}}+\left\|(\widetilde{{\mathcal{T}}}^{\pi}Q_{0})(s^{\prime},\pi)-(\widetilde{{\mathcal{T}}}^{\pi}Q_{0})(\tilde{s}^{\prime},\pi)\right\|_{\hat{d}^{\pi}}$
$\displaystyle\overset{(a)}{=}\gamma\left\|({\mathcal{T}}^{\pi}Q_{0})-(\widetilde{{\mathcal{T}}}^{\pi}Q_{0})\right\|_{\hat{d}^{\pi}}+\left\|(\widetilde{{\mathcal{T}}}^{\pi}Q_{0})(s^{\prime},\pi)-(\widetilde{{\mathcal{T}}}^{\pi}Q_{0})(\tilde{s}^{\prime},\pi)\right\|_{\hat{d}^{\pi}}$
$\displaystyle\leq\gamma^{2}L\epsilon^{d}_{m}+\gamma L\epsilon^{d}_{m},$

where (a) is because $d^{\pi}$ is stationary distribution.

Similarly, we have

$\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-\widetilde{{\mathcal{T}}}^{\infty}Q_{0}\right\|_{\hat{d}^{\pi}}\leq\sum_{t=1}^{\infty}\gamma^{t}L\epsilon^{d}_{m}=\frac{\gamma}{1-\gamma}L\epsilon^{d}_{m}.$

∎

######  Lemma A.8.

Let $\nu$ and $\mu$ be any state distribution,
$f:{\mathcal{S}}\to{\mathbb{R}}$ be any function with $|f|\leq C$,
$\left|\left\|f\right\|_{\nu}-\left\|f\right\|_{\mu}\right|\leq
C\left\|\nu-\mu\right\|$.

###### Proof.

$\displaystyle\left|\left\|f\right\|_{\nu}-\left\|f\right\|_{\mu}\right|$
$\displaystyle=\left|\sqrt{\sum_{s}|f(s)|^{2}\nu(s)}-\sqrt{\sum_{s}|f(s)|^{2}\mu(s)}\right|$
$\displaystyle\overset{(a)}{=}\frac{\left|\|f\|_{\nu}^{2}-\|f\|_{\mu}^{2}\right|}{\|f\|_{\nu}+\|f\|_{\mu}}$
$\displaystyle\overset{(b)}{\leq}\sqrt{\left|\|f\|_{\nu}^{2}-\|f\|_{\mu}^{2}\right|}$
$\displaystyle\leq\sqrt{C^{2}|\nu(s)-\mu(s)|}$
$\displaystyle=C\sqrt{\left\|\nu-\mu\right\|}$

(a) is because $a^{1/2}-b^{1/2}=\frac{a-b}{\sqrt{a}+\sqrt{b}}$. (b) is because

$\sqrt{\left|\|f\|_{\nu}^{2}-\|f\|_{\mu}^{2}\right|}\leq\|f\|_{\nu}+\|f\|_{\mu}.$

∎

Similar to the analysis in the previous section,
$\left\|Q^{*}-\widetilde{Q}\right\|_{\nu}$ can be decomposed into three terms.

$\left\|Q^{*}-\widetilde{Q}\right\|_{\nu}\leq\gamma\left\|Q^{*}-Q_{0}\right\|_{\nu\times\pi_{Q^{*},Q_{0}}}+\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{\nu}+\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-(\widetilde{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{\nu}$
(13)

According to last section, we only need to consider $\nu=d^{\pi}$, and obtain
that $\left\|Q^{\pi}-({\mathcal{T}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}=0$.

Therefore, we focus on the last term
$\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-(\widetilde{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}$.

$\displaystyle\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-(\widetilde{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{d^{\pi}}$
$\displaystyle\overset{(a)}{\leq}V_{\max}\epsilon_{d}+\left\|({\mathcal{T}}^{\pi})^{\infty}Q_{0}-(\widetilde{{\mathcal{T}}}^{\pi})^{\infty}Q_{0}\right\|_{\hat{d}^{\pi}}$
$\displaystyle\overset{(b)}{\leq}V_{\max}\epsilon_{d}+\frac{\gamma}{1-\gamma}L\epsilon^{d}_{m},$

where (a) uses Lem. A.8, (b) uses Lem. A.7.

Then we have

$\displaystyle J(\pi^{*})-J(\pi)$
$\displaystyle\leq\frac{2\gamma}{1-\gamma}\left(\left\|Q^{*}-Q_{0}\right\|_{\infty}+\left\|Q^{\pi}-Q_{0}\right\|_{\infty}\right)+\left(\sqrt{C_{\pi,\pi^{*}}}+1\right)\left(\frac{1}{1-\gamma}V_{\max}\epsilon_{d}+\frac{\gamma}{(1-\gamma)^{2}}L\epsilon^{d}_{m}\right)$
$\displaystyle=\frac{2\gamma}{1-\gamma}\left(\left\|Q^{*}-Q_{0}\right\|_{\infty}+\left\|Q^{\pi}-Q_{0}\right\|_{\infty}\right)+\left(\sqrt{C_{\pi,\pi^{*}}}+1\right)\left(\frac{1}{(1-\gamma)^{2}}R_{\max}\epsilon_{d}+\frac{\gamma}{(1-\gamma)^{2}}L\epsilon^{d}_{m}\right)$

###  A.5 Proof of Thm. 4.5

Suppose the dataset generated by the traditional model is
$\\{(s,a,r,s^{\prime})\\}$. Let the distribution of $s$ in the dataset be
$\tilde{d}^{\pi}$. The next lemma aims to bound the error between
$\tilde{d}^{\pi}$ and the real distribution $d^{\pi}$:

######  Lemma A.9 (Lemma B.2 in (Janner et al., 2019)).

The distance in the state marginal distribution is bounded as:

$\left\|\tilde{d}^{\pi}-d^{\pi}\right\|\leq\frac{1}{1-\gamma}\epsilon^{t}_{m}.$

By replacing $\epsilon_{d}$ with $\frac{1}{1-\gamma}\epsilon^{t}_{m}$, we get
the final result:

$\displaystyle J(\pi^{*})-J(\pi)$
$\displaystyle\leq\frac{2\gamma}{1-\gamma}\left(\left\|Q^{*}-Q_{0}\right\|_{\infty}+\left\|Q^{\pi}-Q_{0}\right\|_{\infty}\right)+\left(\sqrt{C_{\pi,\pi^{*}}}+1\right)\left(\frac{1}{(1-\gamma)^{3}}R_{\max}\epsilon_{m}^{t}+\frac{\gamma}{(1-\gamma)^{2}}L\epsilon^{t}_{m}\right)$

##  Appendix B Pseudo-Code

Algorithm 1 Energy-Guided Diffusion Sampling in Offline-to-Online
Reinforcement Learning

0: Offline phase loss function
$\\{{\mathcal{L}}^{Q_{\psi}}_{\text{offline}},{\mathcal{L}}^{\pi_{\omega}}_{\text{offline}}\\}$,
online phase loss function
$\\{{\mathcal{L}}^{Q_{\psi}}_{\text{online}},{\mathcal{L}}^{\pi_{\omega}}_{\text{online}}\\}$,
energy-based models
$\\{{\mathcal{E}}_{\phi_{1}},{\mathcal{E}}_{\phi_{2}},{\mathcal{E}}_{\phi_{3}}\\}$,
noise prediction model $D_{\theta}$.

Initialize $\psi,\theta,\phi_{1},\phi_{2},\phi_{3},\omega$, offline replay
buffer ${\mathcal{D}}_{\text{offline}}$, online replay buffer
${\mathcal{D}}_{\text{online}}$, diffusion replay buffer
${\mathcal{D}}_{\text{diffusion}}$.

while in _offline training phase_ do

% offline policy training using batches from the offline replay buffer
${\mathcal{D}}_{\text{offline}}$

$\psi\leftarrow\psi-\lambda_{Q}\nabla_{\psi}{\mathcal{L}}_{\text{offline}}^{Q}(\psi),\omega\leftarrow\omega-\lambda_{\pi}\nabla_{\omega}{\mathcal{L}}_{\text{offline}}^{\pi}(\omega)$.

end while

while in _online training phase_ do

for each environment step do

${\mathcal{D}}_{\text{online}}\leftarrow{\mathcal{D}}_{\text{online}}\cup\\{(s,a,s^{\prime},r)\\}$

end for

if step meets $D_{\theta}$ update frequency then

Sample data from
${\mathcal{D}}_{\text{offline}}\cup{\mathcal{D}}_{\text{online}}.$

Update $D_{\theta}$ by minimizing Eq. (2).

Sample positive samples from ${\mathcal{D}}_{\text{online}}$ and generate
negative samples with $D_{\theta}$ based on the positive samples.

Update ${\mathcal{E}}_{\phi_{1}}$ by minimizing Eq. (5).

Sample positive samples from
${\mathcal{D}}_{\text{online}}\cup{\mathcal{D}}_{\text{offline}}$ and generate
negative samples.

Update ${\mathcal{E}}_{\phi_{2}}$ by minimizing Eq. (6).

Sample positive samples from
${\mathcal{D}}_{\text{online}}\cup{\mathcal{D}}_{\text{offline}}$ and generate
negative samples.

Update ${\mathcal{E}}_{\phi_{3}}$ by minimizing Eq. (7).

Score-based sampling with energy guidance and store them as
${\mathcal{D}}^{\epsilon},{\mathcal{D}}_{\text{diffusion}}\leftarrow{\mathcal{D}}\cup{\mathcal{D}}^{\epsilon}$.

end if

for each gradient step do

Sample data from
${\mathcal{D}}_{\text{online}}\cup{\mathcal{D}}_{\text{diffusion}}$.

$\psi\leftarrow\psi-\lambda_{Q}\nabla_{\psi}{\mathcal{L}}_{\text{online}}^{Q}(\psi)$

$\omega\leftarrow\omega-\lambda_{\pi}\nabla_{\omega}{\mathcal{L}}_{\text{online}}^{\pi}(\omega)$

end for

end while

##  Appendix C Experimental Details

###  C.1 Task Description

Adroit Manipulation. Our empirical evaluation on Adroit manipulation contains
3 domains: pen, door, relocate, where the RL agent is required to solve
dexterous manipulation tasks including rotating a pen in specific directions,
opening a door, and moving a ball, respectively. The offline datasets are
human-v1 datasets in D4RL (Fu et al., 2020) benchmark, which only contain a
few successful non-markovian human demonstrations and thus is pretty difficult
for most offline RL approaches to acquire reasonable pre-training
performances.

AntMaze Navigation. Our tests on Antmaze navigation benchmark consists of four
datasets, namely umaze-v2, medium-diverse-v2, medium-play-v2 and large-play-v2
from the D4RL (Fu et al., 2020). The objective is for an ant to learn how to
walk and navigate from the starting point to the destination in a maze
environment, with only sparse rewards provided. This task poses a challenge
for online RL algorithms to explore high-quality data effectively without the
support of offline datasets or additional domain knowledge.

MuJoCo Locomotion. MuJoCo locomotion encompasses several standard locomotion
tasks commonly utilized in RL research, such as Hopper, Halfcheetah, Walker2d.
In each task, the RL agent is tasked with controlling a robot to achieve
forward movement. The D4RL (Fu et al., 2020) benchmark provides four types of
datasets with varying quality for each task: random-v2, medium-v2, medium-
replay-v2, medium-expert-v2.

###  C.2 Details on Comparison with Model-based Methods

Figure 3: Visualization of the Maze MDP.

We conduct the experiment on comparison of distribution differences between
the real online distribution and the distribution generated by models a Maze
environment, which is visualized in Figure 3. With going up, down, left, and
right as selectable actions, the agent starts at the upper left corner and the
exit is at the lower right corner. The black blocks are occupied and
inaccessible. The agent’s goal is to reach the exit as quickly as possible,
with every step the agent incurs a penalty or, when finally reaching the exit,
a reward.

We visualize the distribution difference in Fig. 4. To quantify this
difference, we compute the distribution difference as the deviation in the
number of times the agent visited each state, normalized by a factor of 1000.
Table 4 explicitly illustrates the divergence values, underscoring that our
EDIS effectively generates the intended distribution. In contrast, model-based
approaches, particularly those utilizing MLP, struggle to accurately emulate
the actual online distribution. While the distribution generated by the
transition modeled by diffusion model shows better result, verifying diffusion
model has better capability of modeling distributions. It is noteworthy that
the offline dataset distribution exhibits a substantial divergence from the
online distribution, rendering direct replay of the offline dataset
impractical. Despite the online buffer having a distribution comparable to the
real one, its limited dataset size poses a challenge to achieving optimal
sample efficiency.

###  C.3 Details and Hyperparameters for EDIS

We use the PyTorch implementation of Cal-QL and IQL from
https://github.com/tinkoff-ai/CORL, and primarily followed the author’s
recommended parameters (Tarasov et al., 2022). The hyperparameters used in our
EDIS module are detailed in the Tab. 3:

Table 3: Hyperparameters and their values in EDIS Hyperparameter |  Value
---|---
Network Type (Denoising) |  Residual MLP
Denoising Network Depth |  $6$ layers
Denoising Steps |  $128$ steps
Denoising Network Learning Rate |  $3\times 10^{-4}$
Denoising Network Hidden Dimension |  $1024$ units
Denoising Network Batch Size |  $256$ samples
Denoising Network Activation Function |  ReLU
Denoising Network Optimizer |  Adam
Learning Rate Schedule (Denoising Network) |  Cosine Annealing
Training Epochs (Denoising Network) |  $50,000$ epochs
Training Interval Environment Step (Denoising Network) |  Every $10,000$ steps
Energy Network Hidden Dimension |  256 units
Negative Samples (Energy Network Training) |  $10$
Energy Network Learning Rate |  $1\times 10^{-3}$
Energy Network Activation Function |  ReLU
Energy Network Optimizer |  Adam

(a) Online (Real) Distribution

(b) MLP Transition Distribution

(c) Diffusion Distribution

(d) Diffusion Transition Distribution

(e) Offline Distribution

(f) Online Buffer Distribution

Figure 4: Illustration of the distribution differences between the actual online distribution and the three generated distributions in a Maze environment. Table 4: Comparison of distribution divergence between the actual online distribution and generated data distributions in different methods | Offline | Interaction | MLP Transition | Diffusion Transition | Diffusion
---|---|---|---|---|---
Divergence | 0.29 | 0.18 | 0.44 | 0.24 | 0.30

###  C.4 Additional Ablation Study

We conducted additional ablation studies on each energy function by omitting
their guidance during the reverse-time _SDE_ in a total of six environments,
including hopper-random-v2, halfcheetah-medium-replay-v2, antmaze-medium-
play-v2, antmaze-medium-diverse-v2, and door-human-v1. It can be observed that
each energy function plays a functional role and reduces the corresponding
divergence value.

The state divergence here is the JS divergence between the two state
distributions, which is defined as:

$D_{\textnormal{JS}}(p,q)=\frac{1}{2}\left(D_{\mathrm{KL}}(p,(p+q)/2)+D_{\mathrm{KL}}(q,(p+q)/2)\right),$

where $p$ and $q$ are two distributions and $D_{\mathrm{KL}}$ is

$D_{\mathrm{KL}}(p,q)=\int\log(p(x)/q(x))dx.$

If the two distributions are similar, JS divergence will approach zero, or it
will approach one. To calculate it, we apply the techniques of GAN (Goodfellow
et al., 2014). We learn a discriminator by minimizing the following loss
function

$V(D)=\mathbb{E}_{x\sim p}[\log D(x)]+\mathbb{E}_{z\sim q}[\log(1-D(z)],$

and JS divergence can be derived according to the following formula:

$\max_{D}V(D)=-\log 4+2D_{\textnormal{JS}}(p,q).$

Table 5: Divergence comparisons for energy function ablation study
Dataset | State Divergence | Action Divergence | Transition Divergence
---|---|---|---
w/o energy | w/ energy | w/o energy | w/ energy | w/o energy | w/ energy
hopper-radnom-v2 |  0.85$\pm$0.02 | 0.73$\pm$0.03 |  0.51$\pm$0.04 | 0.39$\pm$0.02 |  0.69$\pm$0.03 | 0.66$\pm$0.04
halfcheetah-medium-replay-v2 |  0.98$\pm$0.01 | 0.50$\pm$0.05 |  0.43$\pm$0.15 | 0.31$\pm$0.03 |  1.01$\pm$0.08 | 0.88$\pm$0.07
walker2d-medium-expert-v2 |  0.91$\pm$0.02 | 0.65$\pm$0.12 |  1.62$\pm$0.06 | 1.32$\pm$0.03 |  0.51$\pm$0.05 | 0.33$\pm$0.08
antmaze-medium-play-v2 |  0.99$\pm$0.01 | 0.95$\pm$0.02 |  0.68$\pm$0.09 | 0.54$\pm$0.05 |  2.05$\pm$0.13 | 1.85$\pm$0.14
antmaze-medium-diverse-v2 |  0.98$\pm$0.00 | 0.91$\pm$0.02 |  0.38$\pm$0.08 | 0.27$\pm$0.08 |  0.75$\pm$0.14 | 0.64$\pm$0.14
door-human-v1 |  0.40$\pm$0.04 | 0.24$\pm$0.02 |  0.54$\pm$0.07 | 0.41$\pm$0.08 |  2.57$\pm$0.06 | 2.52$\pm$0.05

Figure 5: Energy Module Abaltion Study of EDIS

The aggregated learning curves, summarizing outcomes across all environments
from Tab.1, are displayed in Fig 6 and Fig. 7. They compare the performance of
Cal-QL and IQL, both augmented with EDIS, against their respective base
algorithms. For the evaluation of the normalized score in sparse reward
domains, we computed a metric that represents the goal achievement rate for
each method. For example, in the adroit environment door-human, we assessed
the success rate of opening the door.

Figure 6: Aggregated learning curves of Cal-QL and Cal-QL-EDIS on MuJoCo
Locomotion, AntMaze Navigation and Adroit Manipulation tasks

Figure 7: Aggregated learning curves of IQL and IQL-EDIS on MuJoCo Locomotion,
AntMaze Navigation and Adroit Manipulation tasks

###  C.5 Experiments on Visual Environment

We conduct extra experiments on DMC, which is pixel-based state observations.
Our EDIS operates on encoded representations, which are relatively low-
dimensional. By integrating EDIS with CQL+SAC, a commonly used offline to
online baseline, we observed significant improvements on walker-walk and
cheetah-run tasks. In future work, we will explore even more complex
environments. This expansion will help further refine our approach and
validate its generalization.

Table 6: Experiments on the remaining antmaze environments. Environment | Cal-QL | Cal-QL-EDIS | IQL | IQL-EDIS
---|---|---|---|---
antmaze-umaze-diverse-v2 |  93.4$\pm$4.6 | 95.9$\pm$2.8 |  51.3$\pm$4.5 | 66.7$\pm$5.0
antmaze-large-diverse-v2 |  42.3$\pm$2.2 | 57.1$\pm$ 2.8 |  45.0$\pm$8.7 | 52.1$\pm$2.6

###  C.6 Experiments on Sensitivity Analysis

In our sensitivity analysis, we evaluated the effect of varying the diffusion
model’s denoising steps and the number of negative samples. We tested
denoising steps within the range of 64 to 256 and found comparable performance
across this spectrum, indicating that EDIS is robust to changes in this
hyperparameter. On the other hand, reducing the number of negative samples
from 10 to 5 resulted in a noticeable decline in performance. However,
maintaining a minimum of 10 negative samples ensures that performance remains
consistent and unaffected.

Figure 8: Sensitivity analysis on the training steps for the diffusion model.
Figure 9: Sensitivity analysis on the number of negative samples.

###  C.7 Computational Resources

We train EDIS integrated with base algorithms on an NVIDIA RTX 4090, with
approximately 4 hours required for 0.2M fine-tuning on MuJoCo Locomotion and
Adroit Manipulation, while 6 hours for AntMaze Navigation. Specifically, the
time cost for each time training our EDIS every time for $50000$ epochs is
about $10$ minutes. The detailed computational consumption is shown in Tab. 6.
As pointed out in (Karras et al., 2022), the sampling time is faster than
prior diffusion designs, which is much shorter compared with training. The
introduction of the diffusion model does indeed entail an inevitable increase
in computational and time costs. However, this tradeoff between improved
performance and higher computational cost is a common consideration in
diffusion model research. As the field progresses, we anticipate that better
solutions will emerge. In our future work, we aim to further refine and
optimize the extra costs.

Table 7: Computational consumption of different algorithms. Algorithm | Online phase training time | Maximal GPU memory
---|---|---
Cal-QL (MuJoCo) | 3h | 2GB
Cal-QL-EDIS (MuJoCo) | 5h | 2GB
Cal-QL (AntMaze) | 3h | 2GB
Cal-QL-EDIS (AntMaze) | 5h | 3GB
Cal-QL (Adroit) | 3h | 2GB
Cal-QL-EDIS (Adroit) | 6h | 3GB
IQL (MuJoCo) | 2h | 2GB
IQL-EDIS (MuJoCo) | 3h | 3GB
IQL (AntMaze) | 3h | 2GB
IQL-EDIS (AntMaze) | 7h | 3GB
IQL (Adroit) | 2h | 2GB
IQL-EDIS (Adroit) | 5h | 3GB

##  Appendix D Additional Related Work

###  D.1 Offline-to-online Reinforcement Learning.

Offline-to-online reinforcement learning methods are developed to mitigate the
dichotomy between the costly exploration in online RL and the typically
suboptimal performance of offline RL. Previous existing methods do this in a
variety of ways: incorporating the offline data into the replay buffer of
online RL (Schaal, 1996; Vecerik et al., 2017; Song et al., 2022; Hester et
al., 2018), imitating offline policy as auxiliary losses (Zhu et al., 2018;
Rajeswaran et al., 2018; Zhu et al., 2019), or extracting a high-level skill
space for downstream online RL (Gupta et al., 2019; Ajay et al., 2021).
Although these methods improve the sample efficiency of online RL from
scratch, they cannot eliminate the need to actively rollout poor policies for
data collection (Nakamoto et al., 2023).

Another line of work, typically divide the learning process into two phases.
Warming up the policy and value functions in the offline phase and use them as
initialization in the online phase (Zhao et al., 2022; Nair et al., 2020; Wang
et al., 2023a; Nakamoto et al., 2023; Lee et al., 2021; Kostrikov et al.,
2022). These approaches often employ offline RL methods based on policy
constraints or pessimism (Fujimoto et al., 2019; Fujimoto & Gu, 2021; Kumar et
al., 2020) in the offline phase. However, the conservatism conflicts with the
online phase and may induce performance degradation. Various strategies have
been implemented to tackle this issue, such as policy expansion (Zhang et al.,
2023), value function calibration (Nakamoto et al., 2023), Q-ensemble
techniques (Lee et al., 2021), and constraint methods (Nair et al., 2020;
Kostrikov et al., 2022; Li et al., 2023).

Despite the advancements, there has been less focus on the crucial aspect of
integrating useful data during the fine-tuning phase to enhance training
efficiency. Standard practices includes enriching the replay buffer with
offline data (Nakamoto et al., 2023; Zhang et al., 2023), adopting balanced
sampling methods for managing both online and offline data sources (Lee et
al., 2021; Ball et al., 2023), or building models to performance branch
rollout (Rafailov et al., 2023). However, directly replaying the offline data
causes a distribution shift, and adopting balanced sampling methods introduces
large variance (Nachum et al., 2019) while rolling in the built model suffers
from compounding error (Janner et al., 2019). In contrast, our work breaks new
ground by proposing a diffusion-based generator specifically designed to
generate useful samples by thoroughly leveraging prior knowledge in offline
data.

###  D.2 Diffusion Model in Reinforcement Learning

Diffusion models have demonstrated exceptional capabilities in modeling
distribution (Saharia et al., 2022; Nichol et al., 2022; Nichol & Dhariwal,
2021). Within the reinforcement learning research, Diffuser (Janner et al.,
2022) uses a diffusion model as a trajectory generator and learns a separate
return model to guide the reverse diffusion process toward samples of high-
return trajectories. The consequent work, Decision Diffuser (Ajay et al.,
2023) introduces conditional diffusion with reward or constraint guidance for
decision-making tasks, further boosting Diffuser’s performance. Diffusion-QL
(Wang et al., 2023b) tracks the gradients of the actions sampled from the
behavior diffusion policy to guide generated actions to high Q-value area.
SfBC (Chen et al., 2023a) and Diffusion-QL both employ the technique of
resampling actions from multiple behavior action candidates using predicted
Q-values as sampling weights. Expanding the application of diffusion models,
SYNTHER (Lu et al., 2023a) focuses on leveraging the diffusion model for
upsampling data in both online and offline reinforcement learning scenarios.
DVF (Mazoure et al., 2023) introduces a diffusion model to learn the
transition dynamics, which can then be used to estimate the value function.
Recently, several concurrent work also investigates generating samples with
certain data distribution by diffusion models. PolyGRAD (Rigter et al., 2023)
and PGD (Jackson et al., 2024) embed the policy for classifier-guided
trajectory generation, aiming at on-policy world modeling. However, these
studies do not focus on the offline-to-online setting, and they model the
transition function rather than the distribution directly, which does not
eliminate the issue of compounding error. In the context of offline-to-online
reinforcement learning, our research pioneers the utilization of diffusion-
based models to actively generate valuable samples. This departure from
passive reuse of offline data marks a novel approach, emphasizing the active
role of diffusion models in sample generation for enhanced learning outcomes.