ifaamas [AAMAS ’24]Proc. of the 23rd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2024)May 6 – 10, 2024 Auckland, New
ZealandN. Alechina, V. Dignum, M. Dastani, J.S. Sichman (eds.) 2024 2024 1021
Applied Artificial Intelligence Institute, Deakin University Geelong Australia
Applied Artificial Intelligence Institute, Deakin University Geelong Australia
Applied Artificial Intelligence Institute, Deakin University Geelong Australia

# Policy Learning for Off-Dynamics RL with Deficient Support

Linh Le Pham Van<EMAIL_ADDRESS>,  Hung The Tran
<EMAIL_ADDRESS>and  Sunil Gupta<EMAIL_ADDRESS>
###### Abstract.

Reinforcement Learning (RL) can effectively learn complex policies. However,
learning these policies often demands extensive trial-and-error interactions
with the environment. In many real-world scenarios, this approach is not
practical due to the high costs of data collection and safety concerns. As a
result, a common strategy is to transfer a policy trained in a low-cost, rapid
source simulator to a real-world target environment. However, this process
poses challenges. Simulators, no matter how advanced, cannot perfectly
replicate the intricacies of the real world, leading to dynamics discrepancies
between the source and target environments. Past research posited that the
source domain must encompass all possible target transitions, a condition we
term full support. However, expecting full support is often unrealistic,
especially in scenarios where significant dynamics discrepancies arise. In
this paper, our emphasis shifts to addressing large dynamics mismatch
adaptation. We move away from the stringent full support condition of earlier
research, focusing instead on crafting an effective policy for the target
domain. Our proposed approach is simple but effective. It is anchored in the
central concepts of the skewing and extension of source support towards target
support to mitigate support deficiencies. Through comprehensive testing on a
varied set of benchmarks, our method’s efficacy stands out, showcasing notable
improvements over previous techniques.

###### Key words and phrases:

Off-Dynamics; Deficient Support; Transfer Learning; Reinforcement Learning

##  1\. Introduction

Reinforcement Learning (RL) has shown its capacity to acquire intricate
behaviors in numerous real-world challenges Mnih et al. (2015); Levine et al.
(2016); Schrittwieser et al. (2020). However, it requires numerous trial-and-
error interactions with the environment, which may not be feasible due to the
high costs of data collection or safety concerns in many real-world scenarios
(e.g. robotics, autonomous driving, medical treatment, etc). Training the
policy in an alternative source environment, e.g. a simulator, which is both
safer and faster, and using a limited set of data from the real-world target
environment has, therefore, become a common approach. This approach is usually
known as Off-Dynamics Reinforcement Learning Eysenbach et al. (2020).

Previous work on this problem includes Ljung (1998); Chebotar et al. (2019);
Peng et al. (2018); Andrychowicz et al. (2020). Ljung (1998) and Chebotar et
al. (2019) propose an approach to align the source dynamics to the target
dynamics by real-world data using the system identification method. These
methods require a detailed understanding of the target domain (e.g. knowing
the physics behind the systems). Peng et al. (2018) train the policy on a set
of randomized simulators to yield a robust policy. Again, the set of
randomizations is chosen carefully based on the detailed understanding of the
target domain. The requirement of having a detailed understanding of target
and source domains limits the applicability of such methods. _Thus new methods
that do not rely on such detailed knowledge are required_.

The other recent works such as Hanna and Stone (2017); Desai et al. (2020b);
Karnan et al. (2020); Desai et al. (2020a) learn the target policy via
learning an action transformation function which maps the actions suggested by
the source policy to make them suitable for the target domain. In a related
approach, Eysenbach et al. (2020); Liu et al. (2022) use the dynamics
discrepancy term as an additional reward to prevent the policy from exploiting
the dynamics mismatch area. However, all these works make a strong assumption
that the source domain (e.g. a simulator) encompasses all possible target
(real-world) transitions, a condition which we call _Full support_. However,
full support condition rarely holds in practice as a simulator no matter how
advanced cannot perfectly replicate the intricacies of the real world, and
thus can not cover all the transitions in the target domain. For example, an
autonomous driving vehicle may face changed conditions such as new kind of
places (i.e. highway, city, countryside), weather (i.e. sunny, rainy, hazy),
or time (i.e. day, night), resulting in only a fraction of target transitions
in the support of the source domain. In Section 5, we show that the existing
methods fail drastically when a source domain does not fully support the
target domain. _Therefore, the problem of off-dynamics reinforcement learning
under deficient support remains an open problem_.

In this paper, we address the aforementioned challenges relaxing both the
detailed domain understanding and full support requirements in Off-dynamics RL
to deal with source support deficiency. Under this setting, we propose an
effective method to reduce source deficiency by creating a _modified source_
domain using two operations: (1) by _skewing_ the source transitions to
support the target domain with higher probability, (2) and _extending_ the
source transitions towards the target transitions to improve the source
support for the target. The skewing is guided by an importance weighting which
is learned by solving an optimization problem and the source support extension
is done by following the MixUp scheme (Zhang et al., 2017). Finally, we
utilize both the skewed and mixup transitions and adjust the rewards to
compensate for the dynamics discrepancy between the modified source domain and
the target domain, and use this modified data for target policy learning.

Our main contributions are:

  * •

We are the first to address the policy learning for off-dynamics RL with
deficient support, which is a novel problem and is encountered in numerous
real-world scenarios.

  * •

We conduct a theoretical analysis of off-policy RL under the setting of
deficent support, offering valuable practical insights for the development of
effective policy learning algorithm.

  * •

We propose DADS, a practical and effective algorithm for off-dynamics RL with
deficient support via source skewing and extension operations.

  * •

Finally, we demonstrate the superior performance of our proposed method over
the existing methods through a diverse set of experiments.

##  2\. Related Work

Domain adaptation in RL: Domain adaptation in RL is needed when there is a
difference in the observation space, transition dynamics, or reward function.
In this paper, we study domain adaptation with dynamics mismatch. System
identification method Ljung (1998); Kolev and Todorov (2015); Yu et al.
(2017); Chebotar et al. (2019) is a direct approach to align the source
dynamics with the observed target data. However, these methods typically
require a model of the source environment and a large set of target data to
adjust the parameters of the source environment to align it with the target
domain. Another approach, domain randomization Peng et al. (2018);
Andrychowicz et al. (2020); Sadeghi and Levine (2016); Tobin et al. (2017),
involves training RL policies over a collection of randomized simulated source
domains. However, this approach often exhibits sensitivity to the selection of
randomized parameters or parameter distributions Eysenbach et al. (2020).

In contrast, ground action transformation techniques Hanna and Stone (2017);
Karnan et al. (2020); Desai et al. (2020b, a); Zhang et al. (2021) eliminate
the need for a parameterized simulator or manually selected randomized
dynamics parameters. These techniques aim to rectify dynamics mismatch by
learning action transformations of the source policy using the target data.
Such action transformation techniques require the existence of an accurate
action transformation policy, which may be infeasible when the dynamics
mismatch between the source and target is large e.g. when the source domain
lacks transitions seen in the target domain. In the absence of an accurate
action transformation policy, such methods exhibit poor performance.

Xu et al. (2023) introduce value-guided data filtering that removes the
transitions that have high value discrepancies during policy training.
Recently, Eysenbach et al. (2020); Liu et al. (2022) proposed using dynamics
discrepancy to correct the reward during training the policy. However, these
works rely on a full support condition, that the source domain must contain
all possible transitions in the target domain, which rarely holds in real-
world scenarios, thus preventing handling the large dynamics gap problems. Our
study takes advantage of the reward correction but relaxes the full support
condition.

Mixup in RL: MixUp was first introduced in Zhang et al. (2017) in the
supervised learning setting as a novel data augmentation method that improves
the generalizability of the deep learning models by training them on convex
linear combinations of dataset samples. In the RL problems, Wang et al.
(2020); Sander et al. (2022); Lin et al. (2021); Zhang and Guo (2022) have
demonstrated that MixUp helps to improve the generalizability and sample
efficiency of the learned policy. From a different perspective, we employ
MixUp to expand the support of the source domain to cover the support of the
target domain, reducing the support deficiency problem.

Deficient support in Off-policy Bandits: The deficient support problem has
been explored in off-policy bandit settings, where it refers to the lack of
data for certain actions under the logging policy compared to the target
policy Sachdeva et al. (2020); Tran-The et al. (2021); Felicioni et al.
(2022); Saito et al. (2023). However, this existing work focuses primarily on
the policy space. In contrast, our work tackles the novel challenge of
deficient support in the context of off-dynamics policy learning for
reinforcement learning. We address the gap between the source and target
transition dynamics distributions, which is a critical issue for effective
policy adaptation in RL.

##  3\. Problem Setting and Preliminaries

Figure 1. Off-dynamics online policy learning with deficient support. Given
the source domain and target domain with limited online interaction, we
propose skewing the source dynamics, which enables us to sample the source
transitions that are closely aligned with the target dynamics and employ MixUp
procedure to expand the source support set towards the target support set.
Then we adopt the Reward correction to compensate the policy with an
additional reward for encouraging dynamic-consistent behaviors.

Background: In this section, we introduce our notation and a formal definition
of off-dynamics online policy learning. We consider two infinite-horizon
Markov Decision Processes (MDPs)
$\mathcal{M}_{src}:=(\mathcal{S},\mathcal{A},P_{src},r,\gamma,\rho_{0})$ and
$\mathcal{M}_{tar}:=(\mathcal{S},\mathcal{A},P_{tar},r,\gamma,\rho_{0})$
representing source domain and target domain, respectively. In our setting, we
assume that the two domains share the same state space $\mathcal{S}$, action
space $\mathcal{A}$, reward function
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, discount factor
$\gamma\in[0,1)$, and the initial state distribution
$\rho_{0}:\mathcal{S}\rightarrow[0,1]$; the only difference between two
domains is in their transition dynamics, $P_{src}(s^{\prime}|s,a)$ and
$P_{tar}(s^{\prime}|s,a)$.

A policy $\pi:\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A})$ is defined as a
map from states to a probability distribution over actions. Then, we denote
the probability that the policy $\pi$ encounters state $s$ at the time step
$t$ in an MDP $\mathcal{M}$ as $\mathrm{P}^{\pi}_{\mathcal{M},t}(s)$, and the
normalized state-action occupancy of state-action pair $(s,a)$ in
$\mathcal{M}$ is
$\rho^{\pi}_{\mathcal{M}}(s,a):=(1-\gamma)\sum^{\infty}_{t=0}\mathrm{P}^{\pi}_{\mathcal{M},t}(s)\pi(a|s)$.
The performance of a policy $\pi$ in the MDP $\mathcal{M}$ is defined as
$\eta_{\mathcal{M}}(\pi)=\mathbb{E}_{s,a\sim\rho_{\mathcal{M}}^{\pi}}[r(s,a)]$.
The value function on the MDP $\mathcal{M}$ and policy $\pi$ is defined as
$V_{\mathcal{M}}^{\pi}(s):=\underset{\pi,P}{\mathbb{E}}\left[\sum_{t=0}^{\infty}\gamma^{t}r\left(s_{t},a_{t}\right)\mid
s_{0}=s\right]$.

We focus on the Off-Dynamics Online (ODO) policy learning, which is formally
defined as follows:

######  Definition 1 (Off-Dynamics Online Policy Learning).

Given a source domain represented by $\mathcal{M}_{src}$ and a target domain
represented by $\mathcal{M}_{tar}$ with distinct dynamics functions, our goal
is to leverage source interactions and a small amount of target interactions
to derive a good policy that achieves high reward in $\mathcal{M}_{tar}$.

We highlight that the previous methods often requires the assumption of a full
support condition, that implies every possible transition in the target domain
$\mathcal{M}_{tar}$ is covered by the source domain $\mathcal{M}_{src}$. We
formally define the full support condition as follows:

######  Definition 2 (Full support).

We say that a source domain $\mathcal{M}_{src}$ has full support for a target
domain $\mathcal{M}_{tar}$ if every transition with non-zero probability in
the target domain $\mathcal{M}_{tar}$ also has a non-zero probability in the
source domain $\mathcal{M}_{src}$: $P_{tar}(s^{\prime}|s,a)>0\Rightarrow
P_{src}(s^{\prime}|s,a)>0,\forall s,s^{\prime}\in\mathcal{S},a\in\mathcal{A}.$

In the ODO policy learning with the full support condition holds, the target
performance can be guaranteed as follows:

######  Proposition 3 (Performance bound).

Let $\mathcal{M}_{src}$ and $\mathcal{M}_{tar}$ are the source domain and
target domain with different dynamics $P_{src}$ and $P_{tar}$ respectively.
The performance difference of any policy $\pi$ in $\mathcal{M}_{src}$ and
$\mathcal{M}_{tar}$ can be bounded as follows:

$\displaystyle\left|\eta_{M_{tar}}(\pi)-\eta_{M_{src}}(\pi)\right|$ (1)
$\displaystyle\leq\frac{\gamma
r_{max}}{(1-\gamma)^{2}}\cdot\underbrace{\sqrt{2\mathbb{E}_{\rho_{tar}^{\pi}}\left[D_{KL}(P_{tar}(.|s,a),P_{src}(.|s,a))\right]}}_{(a)}.$

The performance bound, as outlined in Proposition 3, depends on the dynamics
discrepancy term $(a)$. A recent approach called DARC (Eysenbach et al., 2020)
uses the dynamics discrepancy between source and target domains as an
incremental reward, to prevent the policy from exploiting areas in the source
that have a high dynamics mismatch with the target domain.

However, the _full support condition_ is stringent and might not hold in many
real-world scenarios. When this condition is not met, it results in the
challenge of deficient support, which we formally define as:

######  Definition 4 (Deficient support).

We say source MDP $\mathcal{M}_{src}$ has support deficiency for target MDP
$\mathcal{M}_{tar}$ if there exists a set
$\\{(s^{\prime},s,a)\\}\neq\emptyset$ such that for each transition
$(s^{\prime},s,a)$ belongs to it, we have $P_{tar}(s^{\prime}|s,a)>0$ but
$P_{src}(s^{\prime}|s,a)=0$.

The deficient support assumption does not require the source domain to
encompass all potential target transitions. Thus, when deficient support
happens, it poses a challenge due to the uncovered target areas. In this
paper, we relax the full support assumption and propose a method for the off-
dynamics online (ODO) policy learning with _deficient support_.

Under the deficient support problem, we derive the performance bound as
follows:

######  Proposition 5 (Performance bound under Defficient Support).

Let $\mathcal{M}_{src}$ and $\mathcal{M}_{tar}$ are source domain and target
domain with different dynamics $P_{src}$ and $P_{tar}$ respectively. For each
state-action pair $s,a$, denote
$S_{s,a}^{0}=\\{[s^{\prime}_{0_{i}},s^{\prime}_{0_{j}}]\\}$ contains intervals
where $P_{src}(.|s,a)=0$, and
$S_{s,a}^{1}=\\{[s^{\prime}_{1_{i}},s^{\prime}_{1_{j}}]\\}$ includes intervals
where $P_{src}(.|s,a)>0$, and $S_{s,a}^{0}\cup
S_{s,a}^{1}=supp(P_{tar}(.|s,a))$. The performance difference of any policy
$\pi$ in $\mathcal{M}_{src}$ and $\mathcal{M}_{tar}$ can be bounded as
follows:

$\displaystyle\left|\eta_{\mathcal{M}_{tar}}(\pi)-\eta_{\mathcal{M}_{src}}(\pi)\right|$
(2) $\displaystyle\leq\frac{\gamma
r_{max}}{(1-\gamma)^{2}}\cdot\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\left[\sum_{S_{s,a}^{1}}\left|\int_{s^{\prime}_{1_{i}}}^{s^{\prime}_{1_{j}}}P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)ds^{\prime}\right|\right]$
$\displaystyle+\frac{\gamma}{1-\gamma}\cdot\underbrace{\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\left[\sum_{S_{s,a}^{0}}\int_{s^{\prime}_{0_{i}}}^{s^{\prime}_{0_{j}}}P_{tar}(s^{\prime}|s,a)\cdot\left|V_{src}^{\pi}(s^{\prime})\right|ds^{\prime}\right]}_{\text{support
deficiency}}.$

Proposition 5 highlights the gap between $\eta_{\mathcal{M}_{tar}}(\pi)$ and
$\eta_{\mathcal{M}_{src}}(\pi)$ due to support deficiency. Notably, the In Eq
(1) emerges as a special instance of In Eq (2) when full support is assumed.
In this special case, the support deficiency term in (2), which quantifies the
target value $V_{tar}$ on the unsupported set, vanishes. Based on Proposition
5, guaranteeing performance on the target domain hinges on minimizing both the
dynamics discrepancy within the supported region and the support deficiency.
With this dual objective in mind, the next section introduces our method aimed
at simultaneously reducing both terms to ensure robust performance guarantees.

##  4\. Proposed Method

In this section, we introduce a novel approach to address the challenge of
off-dynamics policy learning in the presence of support deficiency. Our method
aims to create a modified source domain that has minimum source deficiency
w.r.t to the target. Our method has three primary steps: (1) skewing the
source transitions to maximize its support overlap with the target domain; (2)
extrapolating the source transitions to extend the source support all the way
up to the target domain. This is done using the MixUp procedure by creating
new synthetic transitions between the source transitions and the target
transitions via their convex combinations; and (3) combining the source and
MixUp transitions to form a modified source transition set, adjusting their
rewards similarly to Eysenbach et al. (2020), and use these modified
transitions to train the target policy. Our method is depicted in Figure 1.
Our approach aims to minimize the second and third terms in the performance
bound in In Eq (2). The second term is minimised by iteratively _skewing_ the
source support towards the target. The third term in the performance bound is
minimized by _extending_ the source support toward the target via mix up as it
can generate the samples in the unsupported region.

###  4.1. Skewing Source Dynamics

We present the skewing source dynamics strategy, which enables us to sample
the source transitions that are closely aligned with the target dynamics.
Specifically, we learn a dynamics distribution $P(s^{\prime}|s,a)$ that is
close to the target dynamics distribution $P_{tar}(s^{\prime}|s,a)$ but not
significantly far from the source dynamics $P_{src}(s^{\prime}|s,a)$, measured
in terms of the KL divergence. This is formulated as the following constrained
function optimization problem:

$\displaystyle\min_{P\in\mathcal{P}}D_{\mathrm{KL}}(P(.|s,a)||P_{tar}(.|s,a))\text{
}\forall s\in\mathcal{S},a\in\mathcal{A}.$ (3) $\displaystyle\text{s.t
}D_{\mathrm{KL}}(P(.|s,a)||P_{src}(.|s,a))\leq\epsilon\text{ }\forall
s\in\mathcal{S},a\in\mathcal{A}.$
$\displaystyle\int_{s^{\prime}}P(s^{\prime}|s,a)ds^{\prime}=1\text{ }\forall
s\in\mathcal{S},a\in\mathcal{A}.$

where $\mathcal{P}$ is a family of all transition dynamics distributions and
we have $P_{tar}(.|s,a)\in\mathcal{P}$ and $P_{src}(.|s,a)\in\mathcal{P}$. The
first constraint with KL divergence and the parameter $\epsilon$ regularizes
the dynamics function $P(s^{\prime}|s,a)$ to stay close to the source dynamics
$P_{src}(s^{\prime}|s,a)$, while the second constraint is to ensure
$P(s^{\prime}|s,a)$ is a valid probability density function.

To solve the constrained optimization problem, we create a Lagrangian-based
objective function and then solve for the optimal transition dynamics
$P^{*}(s^{\prime}|s,a)$ by taking a derivative and equating it to zero. With a
few steps of analysis, we obtain the optimal skewed dynamics function
$P^{*}(s^{\prime}|s,a)$ as follows:

$\displaystyle P^{*}(s^{\prime}|s,a)$ $\displaystyle\propto
P_{src}(s^{\prime}|s,a)\exp[\frac{1}{1+\mu}\log\frac{P_{tar}(s^{\prime}|s,a)}{P_{src}(s^{\prime}|s,a)}].$
(4)

The parameter $\mu$ serves as the Lagrange multiplier linked to the KL
constraint in (LABEL:eq:dynamic_optimization_first), and it essentially
dictates the degree of constraint strength or how far the optimal solution can
deviate from the source. The full derivation is provided in the _Appendix_.

To effectively sample the skewed distribution $P^{*}$, we leverage the
existing source domain data $P_{src}$. We achieve this by re-weighting samples
from $P_{src}$ based on the density ratio between the skewed transition
dynamics and the original (source) transition dynamics. In particular, we
propose a sampling approach where source transitions $(s,a,s^{\prime})$ are
chosen with probabilities proportional to the density ratio
$w(s,a,s^{\prime})$ as follows:

$\begin{split}w(s,a,s^{\prime})&=\exp\left[\frac{1}{1+\mu}(\log\frac{P_{tar}(s^{\prime}|s,a)}{P_{src}(s^{\prime}|s,a)})\right]\\\
&\propto P^{*}(s^{\prime}|s,a)/P_{src}(s^{\prime}|s,a).\end{split}$ (5)

Concretely, we define the sampling probability of the $i$-th source transition
$(s,a,s^{\prime})$ as follows:

$p^{i}(s,a,s^{\prime})=\frac{w^{i}(s,a,s^{\prime})}{\sum_{k}w^{k}(s,a,s^{\prime})}.$
(6)

where $w^{i}(s,a,s^{\prime})$ is the priority weight of source transition $i$.

Estimating the ratio of source and target transition dynamics: Calculating the
sampling probability of each source transition requires estimating the density
ratio between the target dynamics and the source dynamics for each source
transition. Similar to Eysenbach et al. (2020), we adopt the probabilistic
classification technique Sugiyama et al. (2012) to estimate this density
ratio. Specifically, we use a pair of binary classifiers,
$q_{\theta_{SAS}}(.|s,a,s^{\prime})$ and $q_{\theta_{SA}}(.|s,a)$, which
distinguish whether a transition $(s,a,s^{\prime})$ (or a state-action pair
$(s,a)$) comes from the source or target domain. The density ratio is computed
as follows:

$\displaystyle\log\frac{P_{tar}(s^{\prime}|s,a)}{P_{src}(s^{\prime}|s,a)}=$
$\displaystyle\log\frac{q_{\theta_{SAS}}(\text{target}|s,a,s^{\prime})}{q_{\theta_{SAS}}(\text{source}|s,a,s^{\prime})}$
(7)
$\displaystyle+\log\frac{q_{\theta_{SA}}(\text{source}|s,a)}{q_{\theta_{SA}}(\text{target}|s,a)}.$

The two classifiers $q_{\theta_{SAS}}(.|s,a,s^{\prime})$ and
$q_{\theta_{SA}}(.|s,a)$ are learned with the standard cross-entropy loss
using the source and target data.

###  4.2. Extending Source Support

The deficient support presents the existence of uncovered target areas. Our
idea is to extend the source support toward the target support by employing
the MixUp procedure, thus filling the uncovered target support. Specifically,
we utilize the skewing source transitions from the previous step and mix them
up with target transitions to create MixUp transitions. While we should
ideally be mixing up $s^{\prime}$ from source and target domains conditioned
on the same state-action pair $(s,a)$, since we deal with continuous state and
action spaces, it is challenging to find an identical pair. Even a nearest
neighbor approach can result in fairly distant $(s,a)$ pairs from the source
and target domains. To avoid this problem, we mix up the quadruples
$(s,a,r,s^{\prime})$ of the source with those of the target.

Given a source transition $x_{src}=(s,a,r,s^{\prime})_{src}$ (obtained from
the skewing step) and a target transition $x_{tar}=(s,a,r,s^{\prime})_{tar}$,
we use MixUp to generate a synthetic transition by taking convex combination
between $x_{src}$ and $x_{tar}$ as follows:

$x_{mix}=\lambda x_{src}+(1-\lambda)x_{tar}$ (8)

where $\lambda$ is sampled from a Beta distribution as $\lambda\sim
B(\alpha,\alpha),\text{ with }\alpha>0$. As suggested by Zhang et al. (2017),
we set $\alpha=0.2$. Note that if either source or target transition
encounters a terminal state, we do not employ interpolation, and simply use
the target transition instead. This is mainly done to avoid non-binary
terminal signal (Sander et al., 2022).

Let $\\{x^{i}_{src}=(s,a,r,s^{\prime})^{i}_{src}\\}_{i=1}^{N}$ be a batch of
$N$ source transitions sampled from the source data $D_{src}$ with
probabilities $p^{i}(s,a,s^{\prime})$ as in Eq (6). We sample uniformly $N$
target transitions $\\{x^{i}_{tar}=(s,a,r,s^{\prime})^{i}_{tar}\\}_{i=1}^{N}$
from the target data $D_{tar}$. We then sample a batch
$\\{\lambda_{i}\\}_{i=1}^{N}$ from a Beta distribution, and perform MixUp
using each pair of source and target transitions.

###  4.3. Reward modification

While the modified source now supports the target transitions better, the
transition probability densities of the modified source and target domain
still may be different. Thus, following the scheme in Eysenbach et al. (2020),
we adjust the reward for each transition by adding an incremental term $\Delta
r$, as follows:

$\Delta r(s,a,s^{\prime})=\log P_{tar}(s^{\prime}|s,a)-\log
P_{msrc}(s^{\prime}|s,a)$ (9)

where $P_{msrc}(s^{\prime}|s,a)$ denotes the transition dynamics of the
modified source. In practice, to obtain $\Delta r(s,a,s^{\prime})$ for each
transition $(s,a,s^{\prime})$, we employ a similar density ratio estimate as
in section 4.1, using two binary classifiers
$q_{\phi_{SAS}}(.|s,a,s^{\prime})$ and $q_{\phi_{SA}}(.|s,a)$ as follows:

$\displaystyle\Delta r(s,a,s^{\prime})=$
$\displaystyle\log\frac{q_{\phi_{SAS}}(\text{target}|s,a,s^{\prime})}{q_{\phi_{SAS}}(\text{modified
source}|s,a,s^{\prime})}$ (10)
$\displaystyle+\log\frac{q_{\phi_{SA}}(\text{modified
source}|s,a)}{q_{\phi_{SA}}(\text{target}|s,a)}.$

Then each transition in the batch of modified source transitions is modified
as $(s,a,r+\Delta r,s^{\prime})$ and used to train the target policy.

###  4.4. Algorithm

Algorithm 1 Online Dynamics Adaptation with Deficient Support in RL (DADS)

1: Input: Source MDP $\mathcal{M}_{src}$ and target $\mathcal{M}_{tar}$; ratio
$r$ of experience from source vs. target; the batch size $N$.

2: Initialize: The source data $\mathcal{D}_{src}$ and the target data
$\mathcal{D}_{tar}$; policy $\pi$; parameters $\theta$ for classifiers that
distinguish source and target domain $q_{\theta_{SAS}},q_{\theta_{SA}}$; and
parameters $\phi$ for classifiers that distinguish modified source and target
domain $q_{\phi_{SAS}},q_{\phi_{SA}}$.

3: for $t=1,\dots,\text{num iterations}$ do

4:
$\mathcal{D}_{src}\leftarrow\mathcal{D}_{src}\cup\text{ROLLOUT}(\pi,\mathcal{M}_{src})$.

5: if $t\mod r==0$ then

6:
$\mathcal{D}_{tar}\leftarrow\mathcal{D}_{tar}\cup\text{ROLLOUT}(\pi,\mathcal{M}_{tar})$.

7: end if

8: Sample $N$ source transitions
$\\{(s,a,r,s^{\prime})^{i}_{src}\\}^{N}_{i=1}$ with each transition’s
probability $p^{i}(s,a,s^{\prime})$ computed via Eq (6) from
$\mathcal{D}_{src}$ .

9: Update transition priority in $D_{source}$ via Eq (7).

10: Sample $N$ target transitions
$\\{(s,a,r,s^{\prime})^{i}_{tar}\\}^{N}_{i=1}$ uniformly from
$\mathcal{D}_{tar}$.

11: Create MixUp transitions
$\\{(s,a,r,s^{\prime})^{i}_{mix}\\}^{N}_{i=1}\leftarrow\text{MixUp}(\\{(s,a,r,s^{\prime})^{i}_{src}\\}^{N}_{i=1},\\{(s,a,r,s^{\prime})^{i}_{tar}\\}^{N}_{i=1})$
via Eq (8).

12: Modify the reward for each transition in source and mixup batch with
$\Delta r$ via Eq (10).

13: Train the source-target classifiers
$\theta\leftarrow\theta-\eta\nabla_{\theta}\text{Cross-
entropyLoss}(\mathcal{D}_{src},\mathcal{D}_{tar},\theta)$.

14: Train the modified_source-target classifiers
$\phi\leftarrow\phi-\eta\nabla_{\phi}\text{Cross-
entropyLoss}(\mathcal{D}_{src},\mathcal{D}_{tar},\phi)$.

15: Train the policy $\pi$ with modified source and mixup transitions using
any standard policy learning algorithm (e.g. SAC).

16: end for

17: return  $\pi$.

We summarize the above steps as our proposed method in Algorithm 1. We perform
the skew operation in Lines 8 and 9, and the MixUp procedure in Line 11. We
perform reward correction in Line 12 and learn two pair of domain classifiers
in Lines 13 and 14. Finally, in Line 15, we use a standard RL algorithm for
policy learning. In our experiment, we use Soft Actor-Critic (SAC) Haarnoja et
al. (2018) algorithm. To simplify the algorithm, we find that simply using a
fixed constant for the Lagrange multiplier $\mu$ is also effective, rather
than adaptively updating $\mu$. In the implementation, to sample efficiently
from Eq (6), we employ the _sum-tree_ data structure similar to Schaul et al.
(2015), which allows $O(\log N)$ complexity for updates and sampling.

Discussion: We discuss the key signficance of our two operations: _Skew_ and
_Extension_. Skewing source transitions shifts their probability mass towards
the target transitions without changing the source domain’s support or
generating any new samples. Unlike skewing, extension of source support using
MixUp can generate new synthetic samples from outside the source support
closer to the target domain. Both schemes have their individual strengths and
limitations. In particular, the skewing operation boosts the sampling of
source transitions that are close to the target domain. However, these
transitions can not fill the uncovered target areas in the source domain. On
the other hand, the MixUp operation can generate novel synthetic samples that
can expand the source support to cover unseen target transitions and thus can
bring significant improvement in target policy learning. However, a downside
can be that randomly mixing up a target transition with any source transition
may lead to synthetic samples that do not lie in the target transition
manifold, which could at times degrade the performance. In our case, when we
use skewing and MixUp together, skewing helps to improve the performance of
MixUp by rejecting the source transitions that are unlikely to occur in the
target domain before mixing them. Thus, both skewing and Mixup have
independent and effective roles in our algorithm. We empirically demonstrate
their effectiveness in Section 5.

##  5\. Experiments

In this section, we provide an empirical analysis of our proposed approach
across different levels of deficient support: small support overlap (small),
medium support overlap (medium), and large support overlap (large).
Furthermore, through ablation studies, we delve deeper into the significance
of each component in our method.

Figure 2. Visualization of the source and target noise distributions in Walker
benchmark in three distinct deficient support levels. Figure 3. The target
return of different methods in four Mujoco benchmarks with different deficient
support levels: large overlapping support (Top row), medium overlapping
support (Middle row), and small overlapping support (Bottom row). The solid
curves are the average target returns over 5 runs with different random seeds,
and the shaded areas represent standard deviation.

Environments: We use four simulated robot benchmarks from Mujoco Gym Todorov
et al. (2012); Brockman et al. (2016): Ant, HalfCheetah, Hopper and Walker.
For each benchmark, we establish three levels of deficient support between the
source and target domains: small overlapping, medium overlapping, and large
overlapping. More specifically, for each benchmark, we first sample the noise
$\xi$ from a pre-defined distribution $p_{\xi}$ and then add it to
$s^{\prime}$ as follows:

$s^{\prime}\leftarrow s^{\prime}+\xi,\text{ where }\xi\sim p_{\xi}.$ (11)

We then update the value of the next state $s^{\prime}$ in the environment. By
adding noises from distinct distributions with different support sets, we
simulate different levels of support deficiency between the source and the
target domain. Specifically, the noise applied to the source domain only
partially overlaps with the noise introduced to the target domain. Thus, after
adding different noises to source and target, the source dynamics has the
support deficiency w.r.t the target dynamics. Moreover, we adjust this
overlapping region to create three different levels of support deficiency.
Figure 7 illustrates the noises added to the source and the target domain at
three deficient support levels in the Walker environment. Comprehensive
details regarding the environments are provided in the _Appendix_.

Baselines: We compare our algorithm with six baselines. DARC Eysenbach et al.
(2020), which uses the additional reward term to encourage the policy to not
use source transitions with low likelihood. GARAT Desai et al. (2020a) that
learns the grounded source environment obtained via action transformation and
then trains the policy on the learned environment. The Finetune baseline first
trains a policy on the source domain and then finetunes it with the limited
transitions from the target domain. The IW (Importance Weighting) baseline
trains the policy on importance-weighted samples from the source domain.
Finally, the RL on Target trains the policy only using the target samples and
can serve as Oracle. RL on Source trains the policy only using source samples.
We run all algorithms with the same five random seeds. More details about the
baseline settings are in the _Appendix_.

###  5.1. Off-dynamics Policy Evaluation

In Figure 3, we illustrate the off-dynamics policy performance of all methods
across four Mujoco environments for the three support deficiency levels.
Across all tasks, the performances of RL on Source are considerably lower than
RL on Target performances, suggesting that directly transferring trained
policies from the source to the target domain yields unsatisfactory
performance when support deficiency is present.

In cases with substantial support overlap (as seen in large overlap setting),
the performance differences between the methods are not too high. However, as
support overlap decreases (in medium and small overlap settings), our
approach, DADS, consistently excels in most tasks. Specifically, GARAT
performances are significantly low for all tasks, supporting our intuition
that its grounded action environment can be inaccurate and infeasible for
policy learning under the dynamics mismatch and deficient support problems.
The performances of DARC and IW drop significantly when the support overlap
decreases (from a large level to a small level). Our method outperforms DARC
and IW baselines for most of the tasks. We surpass the Finetune baseline in
nine out of twelve cases, excluding HalfCheetah. While Finetune performs on
par with our method on large and medium overlapping levels in HalfCheetah, it
outperforms our method in the small overlap cases. We believe that this is
because the agent never dies in the HalfCheetah environment, which helps it to
transfer any learnings from the source domain to the target domain without any
problem. Nonetheless, our method stands out as the only approach that
asymptotically matches the performance of RL on Target (i.e. Oracle) and even
surpasses RL on Target in six out of twelve tasks.

###  5.2. Ablation studies

In this section, we analyze the impact of each component and hyperparameter in
our method. We provide the results for the Walker environment. The results for
the other environments in all settings are provided in the _Appendix_.

Figure 4. Comparison between our DADS method, and its variant with out Skewing
opertation.

####  5.2.1. The impact of Skewing operation:

To validate the effect of the skewing operation, we compared our method to a
variant that does not use this component. As shown in Figure 4, not including
the skewing operation leads to a notable drop in performance and makes target
return unstable, indicating the critical role of this component in our method.

####  5.2.2. The impact of MixUp operation:

We evaluate the impact of the MixUp operation by removing it from our approach
and retraining the policy. As shown in Figure 5, excluding MixUp results in a
significant performance drop in target return, especially in the settings
where support deficiencies are large (medium and small overlap settings). This
verifies the effectiveness of MixUp component in our method.

Notably, omitting either the Skewing or MixUp operations results in a
significant reduction in target returns. This observation confirms the
effectiveness of both operations.

Figure 5. Comparison between our DADS method, and its variant without MixUp.

####  5.2.3. The impact of $\mu$:

Figure 6. The adaptation performance of DADS with different values of $\mu$.

Our method only has one hyperparameter $\mu$ that controls the strength of the
source dynamics regularization in Equation
LABEL:eq:dynamic_optimization_first. We conduct experiments with different
source dynamics regularization $\mu$ values ($0,1/3,2/3,1,2,4$), where $\mu=0$
mean that we ignore the source dynamics regularization and increasing values
of $\mu$ indicate higher weight for the source dynamics regularization. The
results are shown in Figure 6. We can see that the best range for $\mu$ is
from $2/3$ to $1$. If we decrease the value of $\mu$ to $0$, the policy
performance in the target domain drops significantly. The reason is that
reducing or ignoring the source dynamics regularization results in sampling
transitions that are too close to the target domain, which reduces the
diversity of samples for the subsequent MixUp operation. On the other hand, a
high value of $\mu$ might also result in reduced, unstable performance as
there are not enough source transitions that are close to the target domain.
This could also adversely affect MixUp operations. Thus source dynamics
constraint is important for the effective performance of our algorithm. In our
experiments, we used $\mu=1$ due to the highest target return values.

##  6\. Conclusion

In this paper, we have addressed the problem of off-dynamics RL under
deficient support, which is widely encountered in many real-world
applications. To the best of our knowledge, ours is the first work on this
problem. We proposed DADS, a simple, yet effective method, that reduces the
support deficiency of the source domain by modifying it through two
operations: skewing and extension. The skewing is learned by solving an
optimization problem and the extension is performed by using a Mixup operation
between source and target transitions. Extensive experiments have demonstrated
the effectiveness of our method compared to the existing state-of-the-art
approaches for off-dynamics RL.

## References

  * (1)
  * Andrychowicz et al. (2020) OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. 2020\.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ 39, 1 (2020), 3–20.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016.  Openai gym.  _arXiv preprint arXiv:1606.01540_ (2016).
  * Chebotar et al. (2019) Yevgen Chebotar, Ankur Handa, Viktor Makoviychuk, Miles Macklin, Jan Issac, Nathan Ratliff, and Dieter Fox. 2019.  Closing the sim-to-real loop: Adapting simulation randomization with real world experience. In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 8973–8979.
  * Desai et al. (2020a) Siddharth Desai, Ishan Durugkar, Haresh Karnan, Garrett Warnell, Josiah Hanna, and Peter Stone. 2020a.  An imitation from observation approach to transfer learning with dynamics mismatch.  _Advances in Neural Information Processing Systems_ 33 (2020), 3917–3929.
  * Desai et al. (2020b) Siddharth Desai, Haresh Karnan, Josiah P Hanna, Garrett Warnell, and Peter Stone. 2020b.  Stochastic grounded action transformation for robot learning in simulation. In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 6106–6111.
  * Eysenbach et al. (2020) Benjamin Eysenbach, Swapnil Asawa, Shreyas Chaudhari, Sergey Levine, and Ruslan Salakhutdinov. 2020.  Off-dynamics reinforcement learning: Training for transfer with domain classifiers.  _arXiv preprint arXiv:2006.13916_ (2020).
  * Felicioni et al. (2022) Nicolò Felicioni, Maurizio Ferrari Dacrema, Marcello Restelli, and Paolo Cremonesi. 2022.  Off-policy evaluation with deficient support using side information.  _Advances in Neural Information Processing Systems_ 35 (2022), 30250–30264.
  * Fujimoto and Gu (2021) Scott Fujimoto and Shixiang Shane Gu. 2021.  A minimalist approach to offline reinforcement learning.  _Advances in neural information processing systems_ 34 (2021), 20132–20145.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In _International conference on machine learning_. PMLR, 1861–1870.
  * Hanna and Stone (2017) Josiah Hanna and Peter Stone. 2017.  Grounded action transformation for robot learning in simulation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 31.
  * Huang et al. (2022) Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. 2022.  CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms.  _Journal of Machine Learning Research_ 23, 274 (2022), 1–18.  http://jmlr.org/papers/v23/21-1342.html
  * Karnan et al. (2020) Haresh Karnan, Siddharth Desai, Josiah P Hanna, Garrett Warnell, and Peter Stone. 2020.  Reinforced grounded action transformation for sim-to-real transfer. In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 4397–4402.
  * Kolev and Todorov (2015) Svetoslav Kolev and Emanuel Todorov. 2015.  Physically consistent state estimation and system identification for contacts. In _2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids)_. IEEE, 1036–1043.
  * Levine et al. (2016) Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. 2016.  End-to-end training of deep visuomotor policies.  _The Journal of Machine Learning Research_ 17, 1 (2016), 1334–1373.
  * Lin et al. (2021) Junfan Lin, Zhongzhan Huang, Keze Wang, Xiaodan Liang, Weiwei Chen, and Liang Lin. 2021.  Continuous Transition: Improving Sample Efficiency for Continuous Control Problems via MixUp. In _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 9490–9497.
  * Liu et al. (2022) Jinxin Liu, Hongyin Zhang, and Donglin Wang. 2022.  Dara: Dynamics-aware reward augmentation in offline reinforcement learning.  _arXiv preprint arXiv:2203.06662_ (2022).
  * Ljung (1998) Lennart Ljung. 1998.  System identification.  In _Signal analysis and prediction_. Springer, 163–173.
  * Luo et al. (2018) Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. 2018.  Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees.  _arXiv preprint arXiv:1807.03858_ (2018).
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _nature_ 518, 7540 (2015), 529–533.
  * Peng et al. (2018) Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. 2018.  Sim-to-real transfer of robotic control with dynamics randomization. In _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 3803–3810.
  * Peng et al. (2019) Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. 2019.  Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.  _arXiv preprint arXiv:1910.00177_ (2019).
  * Sachdeva et al. (2020) Noveen Sachdeva, Yi Su, and Thorsten Joachims. 2020.  Off-policy bandits with deficient support. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 965–975.
  * Sadeghi and Levine (2016) Fereshteh Sadeghi and Sergey Levine. 2016.  Cad2rl: Real single-image flight without a single real image.  _arXiv preprint arXiv:1611.04201_ (2016).
  * Saito et al. (2023) Yuta Saito, Qingyang Ren, and Thorsten Joachims. 2023.  Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling.  _arXiv preprint arXiv:2305.08062_ (2023).
  * Sander et al. (2022) Ryan Sander, Wilko Schwarting, Tim Seyde, Igor Gilitschenski, Sertac Karaman, and Daniela Rus. 2022.  Neighborhood Mixup Experience Replay: Local Convex Interpolation for Improved Sample Efficiency in Continuous Control Tasks. In _Learning for Dynamics and Control Conference_. PMLR, 954–967.
  * Schaul et al. (2015) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. 2015.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ (2015).
  * Schrittwieser et al. (2020) Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. 2020\.  Mastering atari, go, chess and shogi by planning with a learned model.  _Nature_ 588, 7839 (2020), 604–609.
  * Sugiyama et al. (2012) Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. 2012.  _Density ratio estimation in machine learning_.  Cambridge University Press.
  * Tobin et al. (2017) Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. 2017.  Domain randomization for transferring deep neural networks from simulation to the real world. In _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 23–30.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa. 2012.  Mujoco: A physics engine for model-based control. In _2012 IEEE/RSJ international conference on intelligent robots and systems_. IEEE, 5026–5033.
  * Tran-The et al. (2021) Hung Tran-The, Sunil Gupta, Thanh Nguyen-Tang, Santu Rana, and Svetha Venkatesh. 2021.  Combining Online Learning and Offline Learning for Contextual Bandits with Deficient Support.  _arXiv preprint arXiv:2107.11533_ (2021).
  * Varuna Jayasiri (2020) Nipun Wijerathne Varuna Jayasiri. 2020.  labml.ai Annotated Paper Implementations.
  * Wang et al. (2020) Kaixin Wang, Bingyi Kang, Jie Shao, and Jiashi Feng. 2020.  Improving generalization in reinforcement learning with mixture regularization.  _Advances in Neural Information Processing Systems_ 33 (2020), 7968–7978.
  * Xu et al. (2023) Kang Xu, Chenjia Bai, Xiaoteng Ma, Dong Wang, Bin Zhao, Zhen Wang, Xuelong Li, and Wei Li. 2023.  Cross-Domain Policy Adaptation via Value-Guided Data Filtering.  _arXiv preprint arXiv:2305.17625_ (2023).
  * Yu et al. (2017) Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. 2017.  Preparing for the unknown: Learning a universal policy with online system identification.  _arXiv preprint arXiv:1702.02453_ (2017).
  * Zhang et al. (2021) Grace Zhang, Linghan Zhong, Youngwoon Lee, and Joseph J Lim. 2021.  Policy transfer across visual and dynamics domain gaps via iterative grounding.  _arXiv preprint arXiv:2107.00339_ (2021).
  * Zhang et al. (2017) Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017.  mixup: Beyond empirical risk minimization.  _arXiv preprint arXiv:1710.09412_ (2017).
  * Zhang and Guo (2022) Hanping Zhang and Yuhong Guo. 2022.  Generalization of Reinforcement Learning with Policy-Aware Adversarial Data Augmentation. In _Decision Awareness in Reinforcement Learning Workshop at ICML 2022_.

##  Appendix A Proof of Performance Bounds

In this section, we provide the proof of the performance bounds in our main
paper. We also present the telescoping lemma used for our theoretical results.
See 3 Proof. From the Lemma 6, we have:

$\displaystyle\eta_{\mathcal{M}_{tar}}(\pi)-\eta_{\mathcal{M}_{src}}(\pi)$
$\displaystyle=\frac{\gamma}{1-\gamma}\cdot\mathbb{E}_{\rho_{tar}^{\pi}}\left[\int_{s^{\prime}}P_{tar}(s^{\prime}|s,a)V_{src}^{\pi}(s^{\prime})ds^{\prime}-P_{src}(s^{\prime}|s,a)V_{src}^{\pi}(s^{\prime})ds^{\prime}\right]$
(12)

So, we have

$\displaystyle\left|\eta_{\mathcal{M}_{tar}}(\pi)-\eta_{\mathcal{M}_{src}}(\pi)\right|$
$\displaystyle=\frac{\gamma}{1-\gamma}\cdot\left|\mathbb{E}_{\rho_{tar}^{\pi}}\left[\int_{s^{\prime}}P_{tar}(s^{\prime}|s,a)V_{src}^{\pi}(s^{\prime})ds^{\prime}-P_{src}(s^{\prime}|s,a)V_{src}^{\pi}(s^{\prime})ds^{\prime}\right]\right|$
(13)
$\displaystyle\stackrel{{\scriptstyle(\mathrm{i})}}{{\leq}}\frac{\gamma}{1-\gamma}\cdot\mathbb{E}_{\rho_{tar}^{\pi}}\left[\int_{s^{\prime}}\left|\left(P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)\right)V_{src}^{\pi}\right|ds^{\prime}\right]$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{ii})}}{{\leq}}\frac{\gamma
r_{max}}{(1-\gamma)^{2}}\cdot\mathbb{E}_{\rho_{tar}^{\pi}}\left[\int_{s^{\prime}}\left|P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)\right|ds^{\prime}\right]$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{iii})}}{{=}}\frac{2\gamma
r_{max}}{(1-\gamma)^{2}}\cdot\mathbb{E}_{\rho_{tar}^{\pi}}\left[D_{TV}(P_{tar}(.|s,a),P_{src}(.|s,a))\right]$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{iv})}}{{\leq}}\frac{\gamma
r_{max}}{(1-\gamma)^{2}}\cdot\sqrt{2\mathbb{E}_{\rho_{tar}^{\pi}}\left[D_{KL}(P_{tar}(.|s,a),P_{src}(.|s,a))\right]},$

where (i) follows the absolute value of expectation inequality
$\left|\mathbb{E}\left[X\right]\right|\leq\mathbb{E}\left[\left|X\right|\right]$,
(ii) uses the bound of value function $V^{\pi}_{src}(s)\leq
r_{max}/(1-\gamma)$, (iii) is from the total variation distance of probability
measures, that
$\int_{s^{\prime}}\left|P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)\right|ds^{\prime}=2\cdot
D_{TV}\left(P_{tar}(.|s,a)|P_{src}(.|s,a)\right)$, and (iv) directly applies
the Pinsker’s inequality.

See 5

Proof. Given $s\in\mathcal{S}$, $a\in\mathcal{A}$, and transition dynamics
$P_{src}(.|s,a)$, $P_{tar}(.|s,a)$, we split the support set of the target
dynamics $P_{tar}(.|s,a)$ into two distinct sets. The first set,
$S_{s,a}^{0}=\\{[s^{\prime}_{0_{i}},s^{\prime}_{0_{j}}]\\}$, includes
intervals where $P_{src}(.|s,a)=0$. The second set,
$S_{s,a}^{1}=\\{[s^{\prime}_{1_{i}},s^{\prime}_{1_{j}}]\\}$, includes
intervals where $P_{src}(.|s,a)>0$. From the Lemma 6, we have:

$\displaystyle\left|\eta_{tar}(\pi)-\eta_{src}(\pi)\right|$
$\displaystyle=\frac{\gamma}{1-\gamma}\cdot\left|\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\left[\int_{s^{\prime}}P_{tar}(s^{\prime}|s,a)V_{src}^{\pi}(s^{\prime})ds^{\prime}-\int_{s^{\prime}}P_{src}(s^{\prime}|s,a)V_{src}^{\pi}(s,a)ds^{\prime}\right]\right|$
(14)
$\displaystyle=\frac{\gamma}{1-\gamma}\cdot\bigg{|}\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\bigg{[}\sum_{S_{s,a}^{1}}\int_{s^{\prime}_{1_{i}}}^{s^{\prime}_{1_{j}}}V_{src}^{\pi}(s^{\prime})\cdot\big{(}P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)\big{)}ds^{\prime}+\sum_{S_{s,a}^{0}}\int_{s^{\prime}_{0_{i}}}^{s^{\prime}_{0_{j}}}P_{tar}(s^{\prime}|s,a)\cdot
V_{src}^{\pi}(s^{\prime})ds^{\prime}\bigg{]}\bigg{|}$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{i})}}{{\leq}}\frac{\gamma}{1-\gamma}\cdot\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\bigg{[}\bigg{|}\sum_{S_{s,a}^{1}}\int_{s^{\prime}_{1_{i}}}^{s^{\prime}_{1_{j}}}V_{src}^{\pi}(s^{\prime})\cdot\big{(}P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)\big{)}ds^{\prime}+\sum_{S_{s,a}^{0}}\int_{s^{\prime}_{0_{i}}}^{s^{\prime}_{0_{j}}}P_{tar}(s^{\prime}|s,a)\cdot
V_{src}^{\pi}(s^{\prime})ds^{\prime}\bigg{|}\bigg{]}$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{ii})}}{{\leq}}\frac{\gamma}{1-\gamma}\cdot\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\bigg{[}\left|\sum_{S_{s,a}^{1}}\int_{s^{\prime}_{1_{i}}}^{s^{\prime}_{1_{j}}}V_{src}^{\pi}(s^{\prime})\cdot\left(P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)\right)ds^{\prime}\right|$
$\displaystyle+\left|\sum_{S_{s,a}^{0}}\int_{s^{\prime}_{0_{i}}}^{s^{\prime}_{0_{j}}}P_{tar}(s^{\prime}|s,a)\cdot
V_{src}^{\pi}(s^{\prime})ds^{\prime}\right|\bigg{]}$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{iii})}}{{\leq}}\frac{\gamma}{1-\gamma}\cdot\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\left[\sum_{S_{s,a}^{1}}\left|\int_{s^{\prime}_{1_{i}}}^{s^{\prime}_{1_{j}}}V_{src}^{\pi}(s^{\prime})\cdot\left(P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)\right)ds^{\prime}\right|\right]$
$\displaystyle+\frac{\gamma}{1-\gamma}\cdot\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\left[\sum_{S_{s,a}^{0}}\left|\int_{s^{\prime}_{0_{i}}}^{s^{\prime}_{0_{j}}}P_{tar}(s^{\prime}|s,a)\cdot
V_{src}^{\pi}(s^{\prime})ds^{\prime}\right|\right]$
$\displaystyle\stackrel{{\scriptstyle(\mathrm{iv})}}{{\leq}}\frac{\gamma
r_{max}}{(1-\gamma)^{2}}\cdot\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\left[\sum_{S_{s,a}^{1}}\left|\int_{s^{\prime}_{1_{i}}}^{s^{\prime}_{1_{j}}}P_{tar}(s^{\prime}|s,a)-P_{src}(s^{\prime}|s,a)ds^{\prime}\right|\right]$
$\displaystyle+\frac{\gamma}{1-\gamma}\cdot\mathbb{E}_{\rho_{tar}^{\pi}(s,a)}\left[\sum_{S_{s,a}^{0}}\int_{s^{\prime}_{0_{i}}}^{s^{\prime}_{0_{j}}}P_{tar}(s^{\prime}|s,a)\cdot\left|V_{src}^{\pi}(s^{\prime})\right|ds^{\prime}\right],$

where (i) uses the fact that
$\left|\mathbb{E}\left[X\right]\right|\leq\mathbb{E}\left[\left|X\right|\right]$,
(i) and (iii) are from the triangle inequalities
$\left|x+y\right|\leq\left|x\right|+\left|y\right|$ and
$\left|\sum_{i=1}^{N}(x_{i})\right|\leq\sum_{i=1}^{N}\left(\left|x_{i}\right|\right)$,
and (iv) is from the bound of value function $V^{\pi}_{src}(s)\leq
r_{max}/(1-\gamma)$.

######  Lemma 6 (Telescoping Lemma, Lemma 4.3 in Luo et al. (2018)).

Let $\mathcal{M}_{1}:=(\mathcal{S},\mathcal{A},P_{1},r,\gamma,\rho_{0})$ and
$\mathcal{M}_{2}:=(\mathcal{S},\mathcal{A},P_{2},r,\gamma,\rho_{0})$ be two
MDPs with different dynamcis $P_{1}$ and $P_{2}$. Given a policy $\pi$, we
have

$\displaystyle\eta_{\mathcal{M}_{1}}(\pi)-\eta_{\mathcal{M}_{2}}(\pi)$ (15)
$\displaystyle=\frac{\gamma}{1-\gamma}\mathbb{E}_{s,a\sim\rho_{\mathcal{M}_{1}}^{\pi}}\left[\mathbb{E}_{s^{\prime}\sim
P_{1}}\left[V_{\mathcal{M}_{2}}^{\pi}\left(s^{\prime}\right)\right]-\mathbb{E}_{s^{\prime}\sim
P_{2}}\left[V_{\mathcal{M}_{2}}^{\pi}\left(s^{\prime}\right)\right]\right]$

##  Appendix B Derivation of Skewing Source Dynamics

In the Skewing source dynamics, we find the dynamics function
$P(s^{\prime}|s,a)$ that is closely aligned with the target dynamics
$P_{tar}(s^{\prime}|s,a)$ but not significantly far from the source dynamics
$P_{src}(s^{\prime}|s,a)$, measured in terms of the KL divergence. This is
formulated as the subsequent constrained function optimization problem:

$\displaystyle\min_{P\in\mathcal{P}}D_{\mathrm{KL}}(P(.|s,a)||P_{tar}(.|s,a))\text{
}\forall s\in\mathcal{S},a\in\mathcal{A}.$ (16) $\displaystyle\text{s.t
}D_{\mathrm{KL}}(P(.|s,a)||P_{src}(.|s,a))\leq\epsilon\text{ }\forall
s\in\mathcal{S},a\in\mathcal{A}.$
$\displaystyle\int_{s^{\prime}}P(s^{\prime}|s,a)ds^{\prime}=1\text{ }\forall
s\in\mathcal{S},a\in\mathcal{A}.$

where $\mathcal{P}$ is a family of all transition dynamics distributions, and
we have $P_{tar}(.|s,a)\in\mathcal{P}$ and $P_{src}(.|s,a)\in\mathcal{P}$. The
first constraint with KL divergence and the parameter $\epsilon$ regularizes
the dynamics function $P(s^{\prime}|s,a)$ to stay close to the source dynamics
$P_{src}(s^{\prime}|s,a)$, while the second constraint is to ensure
$P(s^{\prime}|s,a)$ is a valid probability density function. Since imposing
the pointwise KL in the above objective across all states and actions is
computationally challenging, we relax by enforcing them only in expectation
with $\rho^{\pi}_{src}(s,a)$ is the state-action distribution in the source
domain $\mathcal{M}_{src}$. Thus, we obtain:

$\displaystyle\min_{P\in\mathcal{P}}\int_{s,a}\rho^{\pi}_{src}(s,a)D_{\mathrm{KL}}(P(.|s,a)||P_{tar}(.|s,a))dsda.$
(17) $\displaystyle\text{s.t
}\int_{s,a}\rho^{\pi}_{src}(s,a)D_{\mathrm{KL}}(P(.|s,a)||P_{src}(.|s,a))dsda\leq\epsilon.$
$\displaystyle\int_{s^{\prime}}P(s^{\prime}|s,a)ds^{\prime}=1\text{ }\forall
s\in\mathcal{S},a\in\mathcal{A}.$

Similar with Peng et al. (2019), from (LABEL:eq:dynamics_optimization), we
form the Lagrangian as follows:

$\displaystyle\mathcal{L}(P,\mu,\upsilon)$
$\displaystyle=\int_{s,a}\rho^{\pi}_{src}(s,a)D_{\mathrm{KL}}(P(.|s,a)||P_{tar}(.|s,a))dsda+\mu[\int_{s,a}\rho^{\pi}_{src}(s,a)D_{\mathrm{KL}}(P(.|s,a)||P_{src}(.|s,a))dsda-\epsilon]$
(18)
$\displaystyle+\int_{s,a}\upsilon_{sa}[1-\int_{s^{\prime}}P(s^{\prime}|s,a)ds^{\prime}]dsda.$

where $\mu$ and $\upsilon=\\{\upsilon_{sa}|\forall
s\in\mathcal{S},a\in\mathcal{A}\\}$ corresponds to the Lagrange multipliers.
Differentiating $\mathcal{L}(P,\mu,\upsilon)$ with respect to
$P(s^{\prime}|s,a)$ we have:

$\displaystyle\frac{\partial\mathcal{L}}{\partial P}$
$\displaystyle=\rho^{\pi}_{src}(s,a)[\log P(s^{\prime}|s,a)-\log
P_{tar}(s^{\prime}|s,a)+1]+\mu\rho^{\pi}_{src}(s,a)[\log
P(s^{\prime}|s,a)-\log P_{src}(s^{\prime}|s,a)+1]-\upsilon_{sa}.$ (19)
$\displaystyle=\log
P(s^{\prime}|s,a)\rho^{\pi}_{src}(s,a)(1+\mu)+\rho^{\pi}_{src}(s,a)(1+\mu)-\rho^{\pi}_{src}(s,a)[\log
P_{tar}(s^{\prime}|s,a)+\mu\log P_{src}(s^{\prime}|s,a)]-\upsilon_{sa}.$

Setting to zero and solving for $P(s^{\prime}|s,a)$ gives:

$\begin{split}\log P(s^{\prime}|s,a)=\frac{1}{1+\mu}[\log
P_{tar}(s^{\prime}|s,a)+\mu\log
P_{src}(s^{\prime}|s,a)]+\frac{\upsilon_{sa}}{\rho^{\pi}_{src}(s,a)(1+\mu)}-1.\end{split}$
(20)

or

$\begin{split}P(s^{\prime}|s,a)=P_{src}(s^{\prime}|s,a)\exp[\frac{1}{1+\mu}(\log
P_{tar}(s^{\prime}|s,a)-\log
P_{src}(s^{\prime}|s,a))]\exp[\frac{\upsilon_{s,a}}{\rho^{\pi}_{src}(s,a)(1+\mu)}-1].\end{split}$
(21)

Since $\int_{s^{\prime}}P(s^{\prime}|s,a)ds^{\prime}=1$, the second
exponential term in the Eq (21) is the partition function $Z(s,a)$ that
normalizes the condition state action distribution as follows:

$\begin{split}Z(s,a)&=\exp[1-\frac{\upsilon_{s,a}}{\rho^{\pi}_{src}(s,a)(1+\mu)}]\\\
&=\text{
}\int_{s^{\prime}}P_{src}(s^{\prime}|s,a)\exp[\frac{1}{1+\mu}\log\frac{P_{tar}(s^{\prime}|s,a)}{P_{src}(s^{\prime}|s,a)}]ds^{\prime}.\end{split}$
(22)

Finally, the optimal $P^{*}(s^{\prime}|s,a)$ is given by:

$\displaystyle P^{*}(s^{\prime}|s,a)$ $\displaystyle\propto
P_{src}(s^{\prime}|s,a)\exp[\frac{1}{1+\mu}\log\frac{P_{tar}(s^{\prime}|s,a)}{P_{src}(s^{\prime}|s,a)}].$
(23)

##  Appendix C Experimental Details and Hyperparameters

We use Python 3.9, Pytorch 1.13.1, and Gym 0.23.1. All experiments are
conducted on an Ubuntu 20.4 server with 36 cores CPU, 767GB RAM, and a V100
32GB GPU with CUDA version 12.0. Our source code is provided in the
supplementary material and will be made publicly accessible upon publication.

###  C.1. Environments

|  | Large overlapping level | Medium overlapping level | Small overlapping level
---|---|---|---|---
Ant | Source |  |  $p_{\xi_{pos}}$: $\mathcal{U}(-0.01,0.01)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(0.0,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.01,0.01)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(0.0,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.01,0.01)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(0.0,0.005)$
| Target |  |  $p_{\xi_{pos}}$: $\mathcal{U}(-0.005,0.015)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.0125,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(0.0,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.015,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(0.005,0.025)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.0175,0.005)$
Hopper | Source |  |  $p_{\xi_{pos}}$: $\mathcal{U}(-0.01,0.01)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(0.0,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.01,0.01)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(0.0,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.01,0.01)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(0.0,0.005)$
| Target |  |  $p_{\xi_{pos}}$: $\mathcal{U}(-0.005,0.015)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.0125,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(0.0,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.015,0.005)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(0.005,0.025)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.0175,0.005)$
Walker | Source |  |  $p_{\xi_{pos}}$: $\mathcal{U}(-0.02,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{U}(-0.02,0.02)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.02,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{U}(-0.02,0.02)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.02,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{U}(-0.02,0.02)$
| Targe |  |  $p_{\xi_{pos}}$: $\mathcal{N}(0.015,0.004)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.015,0.004)$
|  $p_{\xi_{pos}}$: $\mathcal{N}(0.02,0.004)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.02,0.004)$
|  $p_{\xi_{pos}}$: $\mathcal{N}(0.025,0.004)$
---
$p_{\xi_{pos}}$: $\mathcal{N}(-0.025,0.004)$
HalfCheetah | Source |  |  $p_{\xi_{pos}}$: $\mathcal{U}(-0.02,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{U}(-0.02,0.02)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.02,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{U}(-0.02,0.02)$
|  $p_{\xi_{pos}}$: $\mathcal{U}(-0.02,0.02)$
---
$p_{\xi_{vel}}$: $\mathcal{U}(-0.02,0.02)$
| Target |  |  $p_{\xi_{pos}}$: $\mathcal{N}(-0.015,0.004)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.015,0.004)$
|  $p_{\xi_{pos}}$: $\mathcal{N}(-0.02,0.004)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.02,0.004)$
|  $p_{\xi_{pos}}$: $\mathcal{N}(-0.025,0.004)$
---
$p_{\xi_{vel}}$: $\mathcal{N}(-0.025,0.004)$
Table 1. Details of noise distributions for all environments. We note
$\mathcal{U}(a,b)$ is the Uniform distribution with two boundaries parameters
$a\text{ and }b$, and $\mathcal{N}(m,s)$ is the Gaussian distribution with
mean is $m$ and standard deviation $s$. Figure 7. Visualization of the source
and target noise distributions in the Walker benchmark in three distinct
deficient support levels.

We use four simulated robot benchmarks from Mujoco Gym Todorov et al. (2012);
Brockman et al. (2016): Ant-v2, Hopper-v2, Walker-v2, and HalfCheetah-v2. For
each benchmark, we establish three levels of deficient support between the
source and target domains: small, medium, and large. More specifically, for
each benchmark, we first sample the noise $\xi$ from a pre-defined
distribution $p_{\xi}$ and then add it to $s^{\prime}$ as follows:

$s^{\prime}\leftarrow s^{\prime}+\xi,\text{ where }\xi\sim p_{\xi}$

We then update the value of the next state $s^{\prime}$ in the environment. By
adding noises from distinct distributions with different support sets, we
simulate different levels of support deficiency between the source and the
target domain. Specifically, the noise applied to the source domain only
partially overlaps with the noise introduced to the target domain. Thus, after
adding different noises to source and target, the source dynamics has the
support deficiency w.r.t the target dynamics. Moreover, we adjust this
overlapping region to create three different levels of support deficiency.
Figure 7 visualizes the noises added to the source and the target domain at
three deficient support levels in the Walker environment. Note that the next
state $s^{\prime}$ is a vector of position feature $s^{\prime}_{pos}$ and
velocity feature $s^{\prime}_{vel}$. We use the noise distribution
$p_{\xi_{pos}}$ and $p_{\xi_{vel}}$ for each postion feature
$s^{\prime}_{pos}$ and velocity feature $s^{\prime}_{vel}$ respectively. A
sample implementation of the environment is shown in Example 1.

⬇

1import numpy as np

2from gym import utils

3from gym.envs.mujoco import mujoco_env

4

5class Walker2dNoiseEnv(mujoco_env.MujocoEnv, utils.EzPickle):

6 def __init__(self, pos_noise_dis, vel_noise_dis):

7 self.pos_noise_dis = pos_noise_dis

8 self.vel_noise_dis = vel_noise_dis

9 mujoco_env.MujocoEnv.__init__(self, "agent.xml", 4)

10 utils.EzPickle.__init__(self)

11

12 def step(self, a):

13 posbefore = self.sim.data.qpos[0]

14 self.do_simulation(a, self.frame_skip)

15 # adding noise to the qpos and qvel

16 qpos_noise = self.pos_noise_dis(size=self.model.nq)

17 qvel_noise = self.vel_noise_dis(size=self.model.nv)

18 qpos = self.sim.data.qpos + qpos_noise

19 qvel = self.sim.data.qvel + qvel_noise

20 # set the new qpos and qvel after adding noise

21 self.set_state(qpos, qvel)

22 # calculate the reward based on the state, next state and action

23 posafter, height, ang = self.sim.data.qpos[0:3]

24 alive_bonus = 1.0

25 reward = (posafter - posbefore) / self.dt

26 reward += alive_bonus

27 reward -= 1e-3 * np.square(a).sum()

28 done = not (height > 0.8 and height < 2.0 and ang > -1.0 and ang < 1.0)

29 ob = self._get_obs()

30 return ob, reward, done, {}

Example 1: A sample Python pseudo implementation of the Walker environment.

We provide the details of noise distributions for all environments in Table 1.

###  C.2. Algorithm’s Implementations

This section provides the implementation details and hyperparameters of all
methods.

##### DADS

We use Soft Actor-Critic (SAC) Haarnoja et al. (2018) as the RL algorithm for
policy learning. Our setup is based on SAC’s implementation by Huang et al.
(2022). We use three-layer MLPs with 256 hidden units for policy and value
functions. We set the learning rate at $3e^{-4}$ and batch size at $128$. The
discount factor $\gamma$ is set to $0.99$ in all environments. The target
smoothing coefficient $\tau$ is 0.005. The initial temperature coefficient is
$1.0$ and is learnable. We set the gradient step for the policy as 1, and the
target update interval is $1$. The buffer size is $10^{6}$. For each domain
classifier, we employ two-layer MLPs with 256 hidden units. We use the Maximum
cross-entropy loss to train all of the domain classifiers. As suggested in
Eysenbach et al. (2020), we add the standard Gaussian noise to the input of
the classifiers during the classifier’s training. We warm-started _DADS_ by
applying RL on the source task without modifying the reward for the first
$10^{5}$ iterations. The implementation of the sampling method in the Skewing
operation is based on Schaul et al. (2015) implemented in Varuna Jayasiri
(2020). We train our algorithm with $10^{6}$ steps and collect data from the
target domain after $10$ steps, i.e., the ratio $r$ of experience from source
vs. target is 10.

##### DARC

We follow the implementation of _DARC_ algorithm from the official
code111https://github.com/google-research/google-research/tree/master/darc. We
run _DARC_ with the default configuration reported in the paper Eysenbach et
al. (2020) and the same SAC backbone as _DADS_. We train _DARC_ with $10^{6}$
steps and collect target data after $10$ steps.

##### GARAT

We utilize the author’s official _GARAT_ implementation from their GitHub
repository222https://github.com/HareshKarnan/GARAT-NeurIPS20. We integrate our
environments into the code and execute it using the default hyperparameters
mentioned in the paper Desai et al. (2020a). For a fair comparison with other
methods, we increase the target transitions in _GARAT_ to $10^{5}$ for
learning the grounded source environment and then train the policy with
$10^{5}$ interactions with the grounded source environments.

##### RL on Target and RL on Source

Two baselines are implemented with the same SAC backbone with our method
_DADS_. The only difference is that _RL on Target_ is trained with the target
domain, while _RL on Source_ uses the source domain. Both baselines are
trained with $10^{6}$ steps.

##### Importance weight

The _Importance Weight (IW)_ baseline calculates importance weights by
approximating $P_{tar}(s^{\prime}|s,a)/P_{src}(s^{\prime}|s,a)$ as
$\exp(\Delta r)$. These weights are then used to weight transitions in the SAC
actor and critic losses.

##### Finetune

We first train the SAC on the source domain with $10^{6}$ environment
interactions. Then, we transfer SAC to the target domain and continue the
training with $10^{5}$ target interactions.

Following Fujimoto and Gu (2021), we evaluate all methods every 5000 time
steps during training, with each evaluation comprising 10 episodes.

##  Appendix D Additional Experiments

In this section, we provide the additional results for ablation studies in all
tasks.

###  D.1. The impact of Skewing operation

We assess the effect of the Skewing operation across all tasks. As shown in
Figure 9, the absence of skewing operation results in a notable performance
drop in target return in almost environments, indicating the importance of
this component in our method.

###  D.2. The impact of MixUp operation

We evaluate the impact of the MixUp operation by removing it from our approach
and retraining the policy. As shown in Figure 9, excluding MixUp results in a
significant performance drop in target return. This observation verifies the
effectiveness of the MixUp component in our method.

###  D.3. The impact of $\mu$

Figure 10 illustrates the results of an ablation study about the impact of
hyperparameter $\mu$, which controls the weight for the source dynamics
regularization. We used $\mu$ = 1 in our experiments due to the highest target
return values.

Figure 8. Comparison between our DADS method and its variant without Skewing
operation in all environments. Figure 9. Comparison between our DADS method
and its variant without using MixUp in all environments. Figure 10. The
adaptation performance of DADS with different values of $\mu$ in all tasks.