# Continual Domain Randomization

Josip Josifovski1∗, Sayantan Auddy2∗, Mohammadhossein Malmir1,
Justus Piater2,4, Alois Knoll1 and Nicolás Navarro-Guerrero3 1 School of
Computation Information and Technology, Technical University of Munich,
Germany.2 Department of Computer Science, University of Innsbruck, Austria.3
L3S Research Center, Leibniz Universität Hannover, Germany.4 Digital Science
Center (DiSC), University of Innsbruck, Austria.$*$ Equal contribution

###### Abstract

Domain Randomization (DR) is commonly used for sim2real transfer of
reinforcement learning (RL) policies in robotics. Most DR approaches require a
simulator with a fixed set of tunable parameters from the start of the
training, from which the parameters are randomized simultaneously to train a
robust model for use in the real world. However, the combined randomization of
many parameters increases the task difficulty and might result in sub-optimal
policies. To address this problem and to provide a more flexible training
process, we propose _Continual Domain Randomization_ (CDR) for RL that
combines domain randomization with continual learning to enable sequential
training in simulation on a subset of randomization parameters at a time.
Starting from a model trained in a non-randomized simulation where the task is
easier to solve, the model is trained on a sequence of randomizations, and
continual learning is employed to remember the effects of previous
randomizations. Our robotic reaching and grasping tasks experiments show that
the model trained in this fashion learns effectively in simulation and
performs robustly on the real robot while matching or outperforming baselines
that employ combined randomization or sequential randomization without
continual learning. Our code and videos are available at https://continual-
dr.github.io/.

###### Index Terms:

Domain randomization, sim2real transfer, continual reinforcement learning,
robotic manipulation

##  I Introduction

Deep reinforcement learning (DRL) in robotics research relies heavily on
simulated environments, mainly because of the sample inefficiency of DRL
algorithms and the absence of safe exploration guarantees, which make it
impractical to train such algorithms on physical robots directly. While
simulations provide virtually unlimited training data and eliminate safety
concerns during training, their approximative nature can significantly degrade
the model’s performance on the real system in sim2real transfer [1] – a
phenomenon known as the _reality gap_.

To bridge the reality gap, one can try to attain _zero-shot_ sim2real transfer
by crafting an accurate simulation and perfectly mimicking the behavior of the
real system [1]. Unfortunately, even for elementary systems, simulating
reality with sufficient precision is rarely possible. Alternatively, a
simulation-trained DRL model can be _finetuned_ on the target to improve
performance. This _domain adaptation_ approach updates the policy learned in
simulation and usually leads to improved performance. However, it makes the
policy system-specific and requires at least some real-world training, which
can be unsafe and time-consuming [2].

Due to the impracticality of developing perfect simulations and the
limitations of domain adaptation, recent works on _domain randomization_ [3]
rely on the randomization of simulation parameters. Here, instead of
accurately simulating every parameter of the real system, the relevant
simulation parameters are randomized to cover the difference between the real
world and simulation. However, it can be challenging to precisely identify all
the essential randomization parameters at the start. Moreover, the excessive
randomization of many parameters jointly increases the task complexity and the
agent’s uncertainty, making it difficult to find a good policy in simulation
[4].

To address these limitations, we propose _Continual Domain Randomization_
(CDR), which combines _domain randomization_ [3] with _continual learning_ [5]
(CL) for sim2real transfer. Viewing domain randomization as a CL problem
provides greater flexibility in the initial choice of the randomization
parameters. Additionally, randomizing a subset of all parameters at a time
reduces the risks of excessive randomization. In CDR, we start with an agent
trained in an idealized simulation (with all randomizations disabled) and
successively train the agent in differently randomized simulations. We treat
each randomization as a separate CL _task_ and enable the DRL agent to
continually train under each randomization without _catastrophically
forgetting_ [5] the knowledge gained from previous training runs (i.e., past
_tasks_). We implement an offline and an online version of CDR by combining
the Proximal Policy Optimization (PPO) [6] algorithm with the regularization-
based CL algorithm _Elastic Weight Consolidation_ (EWC) [7] and its online
variant [8]. Nevertheless, CDR is a general strategy that can be implemented
with other DRL or CL algorithms.

We perform experiments for sim2real transfer of two different robotics tasks,
reaching and grasping, and compare the real-world performance of policies
learned using CDR against finetuning, full-randomization, and the ideal model
trained without randomization. Our results show that CDR can match or
outperform baselines that jointly randomize all parameters throughout the
training process. At the same time, it is more stable and less sensitive to
the order of randomizations than sequential baselines like finetuning, which
only optimizes the policy for the newest randomization and ignores previous
ones. These results highlight the potential of CDR as a general and flexible
framework for helping bridge the reality gap in DRL-based robotics.

##  II Related Work

Even though there are some cases of sim2real transfer where measuring and
precisely simulating all relevant parameters of the real system may be
possible [9, 10], due to wear and tear in the robot’s hardware or changes in
external conditions like temperature or humidity, they can vary significantly
[1]. Intuitively, providing some degree of variability in the simulator
parameters and training a model that can deal with a range of parameter values
would lead to a more robust sim2real transfer [11]. For this reason, _domain
randomization_ (DR) [3, 12, 13, 14, 15, 16, 17, 18] has become a widely used
sim2real technique. However, inappropriate and excessive randomization can
increase the uncertainty of the system and the difficulty of solving the task
in simulation, leading to sub-optimal policies and longer training times [4].

One approach to circumvent those problems is _automatic domain randomization_
[14]. Here, the randomization increases progressively as the policy under the
current randomization improves the performance. In contrast, _active domain
randomization_[15] starts from wide randomization ranges and learns the most
informative ranges for training a robust policy. While the previous approaches
assume that only simulated data is available, data from the real system can be
used in offline fashion [19, 17] to find suitable parameter distributions, or
to identify which parameters are relevant for sim2real transfer through causal
discovery [18].

Continual learning (CL) algorithms enable knowledge accumulation from a
sequence of tasks without _catastrophic forgetting_ [5]. CL has traditionally
targeted supervised learning scenarios using static datasets or reinforcement
learning problems with discrete actions [7, 8]. More recently, CL has been
applied to more challenging scenarios such as robot learning, both in
simulation [20, 21] as well as in the real world [22, 23]. Continual
reinforcement learning (CRL) in continuous action and observation spaces [24]
has also emerged as a popular means of training robots to learn multiple
tasks. CRL scenarios can be of different types [24]:  (i) the state transition
and action space for different tasks remain the same, but the reward function
changes (the agent needs to perform fundamentally different actions such as
pushing, reaching, etc., for each task) [25];  (ii) the state transition
function changes, but the reward function and action space remain the same.
Here, the environment dynamics change, but the ultimate goal of the agent
remains the same [26]. The sim2real scenario belongs to the latter.
_Progressive networks_ [27] have also been used for continual sim2real
transfer [28]. For instance, as shown by Rusu et al., [28], an initial neural
network can be trained in simulation to solve a task, and a new network, with
lateral connections to the previous network for feature reusing, is
instantiated and trained on the real robot. Another approach used for sim2real
transfer is Policy distillation [29]. For instance, Traoré et al., [30]
trained a separate policy and generated a dataset of observation-action pairs
for each task in simulation. The datasets are then used in a supervised
learning fashion to train a single model that can solve the tasks in the real
environment. Unlike other domain randomization approaches, in CDR no fixed set
of randomization parameters during training is assumed – the model can train
on a subset of randomization parameters at a time and adapt to new
randomization parameters later. This is achieved by combining sequential
randomization with continual learning, which retains the effects of previous
randomizations and avoids overfitting to the last set of parameters. Unlike
other continual sim2real approaches, CDR uses a single neural network trained
across all tasks. Additionally, the online CDR variant avoids the linear
increase in memory for keeping a separate dataset or network parameters per
task. Finally, CDR can easily be combined with methods like _automated_ [14]
or _active_ [15] DR to find suitable ranges for the subset of randomization
parameters within a single training run in the sequence.

##  III Methodology

Figure 1: Overview of our proposed CDR approach. CDR-$\lambda$ (blue arrow)
uses a set of network snapshots and Fisher matrices (one for each past task)
for continual learning, while CDR-$\mathcal{O}\lambda$ (green arrow) uses only
a single parameter snapshot and Fisher matrix. Other baselines are shown with
gray arrows.

###  III-A Problem Description

Let $\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,S_{0},\gamma,T)$ define a Markov
Decision Process (MDP) where $\mathcal{S}$ denotes the state space,
$\mathcal{A}$ is the action space,
$P(s_{t+1}\in\mathcal{S}|s_{t}\in\mathcal{S},a_{t}\in\mathcal{A})$ is the
state transition probability, $R$ is the reward function, $S_{0}$ defines an
initial state distribution, $\gamma$ is the reward discount factor and $T$
defines the finite horizon. The objective of reinforcement learning is to find
the parameters $\theta$ for a parameterized policy $\pi_{\theta}(a_{t}|s_{t})$
to maximize the expected return
$\mathbb{E}_{\pi_{\theta}}[\sum^{T}_{t=0}\gamma^{t}R(s_{t},a_{t})]$ where
$s_{0}\sim S_{0}$, $s_{t+1}\sim P$ and $a_{t}\sim\pi_{\theta}(a_{t}|s_{t})$.
Let $\Phi$ denote a real system with unknown dynamics, and
$\mathcal{E}=\\{\epsilon_{1},\cdots,\epsilon_{n}\\}$ denote the set of all
randomization parameters of a simulator $\Sigma$ of $\Phi$. The goal is to
optimize $\theta$ using $\Sigma$ such that it maximizes the expected return on
$\Phi$ (zero-shot sim2real transfer).

###  III-B Continual Domain Randomization

In our suggested approach, _Continual Domain Randomization_ (CDR), the policy
parameters $\theta=\tilde{\theta}$ are initialized and optimized in an
idealized simulation $\Psi_{\varnothing}$ (with all randomizations disabled).
Afterward, we sequentially optimize the resulting $\theta$ with CL on a
sequence of simulations where a subset of randomization parameters is enabled
each time, as shown in Fig. 1:
$\tilde{\theta}\xrightarrow[]{\Psi_{\varnothing}}\theta_{\varnothing}\xrightarrow[]{\Psi_{\epsilon_{1}}}\theta_{1}\cdots\xrightarrow[]{\Psi_{\epsilon_{n}}}\theta_{n}$.

CDR-$\lambda$: combines the PPO [6] algorithm with the popular regularization-
based CL algorithm Elastic Weight Consolidation (EWC) [7], $\lambda$ denotes
the regularization process with EWC. EWC is an approximate Bayesian method
that quantifies the importance of neural network parameters for remembering
previous knowledge using the empirical _Fisher information matrix_ (FIM). The
FIM is used to estimate the posterior distribution over the network
parameters, approximating it as a multivariate Gaussian centered around the
maximum a posteriori (MAP) estimate. The diagonal of the covariance matrix is
then used to determine parameter importance. The loss function for learning
task $C$ after tasks $A$ and $B$ is:

$\mathcal{L}(\theta)=\mathcal{L}_{C}(\theta)+\lambda_{B}||\theta-\theta^{*}_{B}||^{2}_{F_{B}}+\lambda_{A}||\theta-\theta^{*}_{A}||^{2}_{F_{A}}$
(1)

where $\mathcal{L}_{C}$ is the task-specific loss for task $C$, $\lambda_{A}$
and $\lambda_{B}$ are the regularization constants, $\theta^{*}_{A}$ and
$\theta^{*}_{B}$ are the saved MAP parameters after learning tasks $A$ and
task $B$ respectively, and $F_{A}$ and $F_{B}$ are the diagonal Fisher
information matrices for the first two tasks. In this paper, we adapt PPO for
CL by regularizing the parameters of _only_ the actor network with EWC. After
learning each task, we roll out trajectories of the PPO agent trained on the
most recent environment and store these in a replay buffer. After that, we
compute the (diagonal) empirical Fisher information for the latest task using
the squared gradients of the log-likelihoods of the action [31] predicted by
the PPO policy:

$F(\theta)=\cfrac{1}{n}\sum\limits_{i}^{n}\nabla_{\theta}\log
p_{\theta}(y_{i}|x_{i})\nabla_{\theta}\log
p_{\theta}(y_{i}|x_{i})^{\mathrm{T}}$ (2)

here, $n$ denotes the number of samples in the replay buffer, $x_{i}$ and
$y_{i}$ denote the input and output of the policy network, and $\theta$
denotes the parameters of the policy network. Since the critic does not
contribute to the log-likelihood computation, the parameters of the critic are
not regularized. In contrast to previous implementations of PPO and EWC [32],
but similar to online-EWC [8], we normalize the elements of $F$ to be in the
range [0.0,1.0] after each task. This normalization decouples the
regularization constant $\lambda$ from the arbitrary scaling of the Fisher
information of individual tasks and allows us to use of a single value of
$\lambda$ for all tasks.

A disadvantage of regular EWC is that for each randomization (task), a
separate snapshot of the actor network parameters and the corresponding
empirical diagonal Fisher information needs to be saved. This results in the
undesired linear scaling of storage requirements. To counter this, we suggest
combining PPO with online-EWC [8] creating CDR-$\mathcal{O}\lambda$. After
learning a task, a replay buffer is filled, and the empirical Fisher
information is computed with Eq. 2. Thus, instead of maintaining a separate
copy of the network and Fisher information for each task, only one copy of the
network after the most recent task is kept, and the network is updated using

$\mathcal{L}_{i}(\theta)=\tilde{\mathcal{L}}_{i}(\theta)+\cfrac{1}{2}||\theta-\theta_{i-1}^{*}||^{2}_{\gamma
F^{*}_{i-1}}$ (3)

where $\mathcal{L}_{i}(\theta)$ is the overall loss for the current task,
$\tilde{\mathcal{L}}_{i}(\theta)$ is the task-specific PPO loss,
$\theta_{i-1}^{*}$ is the snapshot of the network after the previous task, and
$F^{*}_{i-1}$ is the _online_ Fisher information after the previous task.
$F^{*}_{i-1}$ and the Fisher information of the current task $F_{i}$ are used
to update the online Fisher information $F_{i}^{*}$ using a hyperparameter
$\gamma$:

$F_{i}^{*}=\gamma F^{*}_{i-1}+F_{i}$ (4)

We again normalize $F_{i}$ to be in the range [0.0,1.0] to avoid any arbitrary
scaling. Similar to CDR-$\lambda$, we only regularize the PPO actor network
for CDR-$\mathcal{O}\lambda$.

###  III-C Baselines

We implement multiple baselines [4] with PPO [6] (Fig. 1):

  1. (i)

Ideal: Training is conducted in the _ideal_ simulation environment without any
randomization.

  2. (ii)

Finetuning: The model trained in the ideal simulation is used as a starting
point and then successively _finetuned_ on each of the randomization
parameters, one at a time, without any continual learning.

  3. (iii)

Randomized: The model is trained by activating all the randomization
parameters together at the same time.

##  IV Experiment Setup

For the experiments, we consider the _reach-and-balance_ [4] task with a
2-joint-controlled robotic arm, and a robotic _grasping_ task with a
kinematically redundant, 7-joint-controlled arm and a three-fingered gripper
[33]. Both tasks are defined in continuous state and action spaces, and the RL
agent exercises fine-grained direct control over the robot’s joint velocities.
We intentionally use joint space control, because it requires minimal
information about the real system’s dynamics [34]. Moreover, in this case the
agents are supposed to learn to compensate for dynamic effects, e.g., inertial
forces [34] through randomization, which is particularly important and part of
our sim2real transfer analysis.

###  IV-A Reacher Task

The _reacher_ task is defined as:

$\mathbf{s}_{t}=[\Delta{x}_{t},\Delta{y}_{t},\Delta{z}_{t},{q}_{1},{q}_{2}]\in\mathbb{R}^{5}$
(5)
$\mathbf{a}_{t}=[\dot{q}_{1}^{\prime},\dot{q}_{2}^{\prime}]\in\mathbb{R}^{2}$
(6) $r_{t}=\begin{cases}-d_{t}&\text{ if $C$ is false}\\\
-d_{t}\times(T-t)&\text{ otherwise}\end{cases}\in\mathbb{R}$ (7)

where $\mathbf{s}_{t}$ is the agent’s state at timestep $t$, $\Delta{x}$,
$\Delta{y}$ and $\Delta{z}$ are the distances between the end-effector and the
target along the corresponding coordinate axes, and ${q_{i}}$ are the joint
positions. $\mathbf{a}_{t}$ is the set of joint velocities the agent commands
at timestep $t$. The reward $r_{t}$ at each timestep is defined as the
negative Euclidean distance ${d}_{t}$ between the end-effector and the target,
unless the robot collides with the floor or reaches joint limits, in which
case the terminal condition $C$ becomes true, the episode terminates and the
reward is the negative distance at that timestep multiplied with the remaining
timesteps to the maximal episode length $T$.

###  IV-B Grasper Task

The _grasping_ task is defined as:

$\displaystyle\mathbf{s}_{t}=[$
$\displaystyle\Delta{x}_{t},\Delta{y}_{t},\Delta{z}_{t},{q}_{1},{q}_{2},{q}_{3},{q}_{4},{q}_{5},{q}_{6},{q}_{7},$
(8)
$\displaystyle\Delta{fx}_{t},\Delta{fy}_{t},\Delta{fz}_{t},\Delta{ux}_{t},\Delta{uy}_{t},\Delta{uz}_{t}]\in\mathbb{R}^{16}$
$\mathbf{a}_{t}=[\dot{q}_{1}^{\prime},\dot{q}_{2}^{\prime},\dot{q}_{3}^{\prime},\dot{q}_{4}^{\prime},\dot{q}_{5}^{\prime},\dot{q}_{6}^{\prime},\dot{q}_{7}^{\prime},]\in\mathbb{R}^{7}$
(9) $r_{t}=\begin{cases}-{dp}_{t}-0.5\times{du}_{t}-c&\text{ if $t<T$}\\\
-{dp}_{t}-0.5\times{du}_{t}-c+g&\text{if $t=T$}\\\ \end{cases}\in\mathbb{R}$
(10)

where $\mathbf{s}_{t}$ is the state of the agent at timestep $t$,
$\Delta{x_{p}}$, $\Delta{y_{p}}$ and $\Delta{z_{p}}$ are the distances between
the end-effector and the center of the object along the corresponding
coordinate axes, and ${q_{1-7}}$ are the joint positions. $\Delta{fx}$,
$\Delta{fy}$, $\Delta{fz}$ are the distances between the forward unit
direction vector of the object and the forward unit direction vector of the
end-effector along each axis, and $\Delta{ux}$, $\Delta{uy}$, $\Delta{uz}$ are
the distances between the up unit direction vector of the plane on which the
object is placed and the up unit direction vector of the end-effector along
each axis. $\mathbf{a}_{t}$ is the set of joint velocities the agent commands
at time $t$. Each episode has a fixed duration $T$. If a collision occurs or
the robot reaches joint limits during the episode, zero velocities are
commanded until the episode ends. At the end of the episode, if there were no
collisions and the joint limits were not reached, the robot stops moving and
the gripper closes in an attempt to grasp the object. The reward at timestep
$t$ is defined in terms of the negative Euclidean distance ${dp}_{t}$ between
the end-effector and the center of the object, the negative Euclidean distance
${du}_{t}$ between the up unit vectors of the gripper and the floor, and a
penalty coefficient $c$ that is 0 by default and 1 if there is a collision or
joint limits are reached during the episode. At the end of the episode, if the
grasping attempt was successful, the term $g$ is set to 10, otherwise it is 0.

###  IV-C Robot Platform

The simulated and real-world environments are shown in Fig. 2. We use the
Unity-based platform introduced in [26] for the simulation. For both _reacher_
and _grasper_ , we use the _KUKA LBR iiwa 14_ robotic manipulator [35], and
for the grasping task we use the 3-Finger Adaptive Robotic Gripper (3F model)
by Robotiq in pinch mode, relying on the gripper parameters and meshes from
the URDF data provided by the ROS-Industrial [36] package. The real robot is
controlled with the IIWA stack [37] via Robot Operating System (ROS) [38]. The
box used for the grasping experiments is a cube with a side of 50mm.

Figure 2: The simulated and real environments for reaching and grasping.
Figure 3: Effects of different randomization parameters on sim2real transfer
for the reaching task.

###  IV-D Training Procedure

We use three randomisation parameters commonly used in the literature [4],
[13]:  (i) Latency $L$: random delay in observing the current state;  (ii)
Torque $T$: the joint stiffness and damping coefficients are randomized,
resulting in a randomized torque for reaching the commanded velocity; and
(iii) Noise $N$: noise drawn from a uniform distribution is randomly
incorporated to simulate sensor inaccuracies.  The implementation is identical
to [4] and the ranges of the randomization parameters are provided in Tab. I.

The efficacy of sim2real transfer under sequential training strategies like
finetuning or CDR may be influenced by the ordering of randomization
parameters, as each parameter can have varying effects on the transfer process
[4]. Instead of unnecessarily testing all possible permutations of $L$, $N$,
and $T$, we empirically evaluate the importance of each parameter for sim2real
transfer on the reaching task to determine the best and worst case ordering
scenario. In the best scenario, the parameter that leads to the best sim2real
transfer is at the end of the training sequence, while in the worst case
scenario, it is at the beginning. For the empirical evaluation, we perform
training on a single-parameter randomized simulation from scratch and evaluate
the models in ideal simulation and on the real system. The results from 5
independent seeds for each parameter, as well as the baselines where no
parameter or all parameters are randomized, are presented in Fig. 3. Based on
the results, we define $TLN$ and $NLT$ orderings for the sequential training
strategies in the experiments.

TABLE I: Training hyperparameters.

Parameter (PPO) | Value | Parameter | Value
---|---|---|---
Bias-var tradeoff ($\lambda$) | 0.95 | Torque ($T$) |  | stiff: [$10^{1}$, $10^{3}$]
---
damp: [$10^{1}$, $10^{3}$]
Disc. factor ($\gamma$) | 0.99 | Latency ($L$) | [0, 1] sec
Learning rate ($\alpha$) | $25\times 10^{-5}$ | Noise ($N$) | [0, 10] %
Val. fun. coeff. ($c_{1}$) | 0.5 | EWC reg. const. $\lambda$ | $5\times 10^{3}$
Ent. coeff. ($c_{2}$) | 0.0 | EWC buffer limit | 100
Clip range ($\epsilon$) | 0.1 | EWC replay batch | 32
Epochs | 10 | EWC replay samples | 5000
Mini-batches | 32 | online-EWC $\gamma$ | 0.95

We extend the PPO [6] implementation in _StableBaselines3_ [39] to implement
the methods described in Sec. III. We use the default network architecture
(2-layer MLP, 64 units/layer, $tanh$ activations). Due to the different state
and action space dimensions, the network for _grasper_ is much larger than
that for _reacher_. We use the same PPO-specific hyperparameters as in [4].
For the EWC hyperparameters, we start from literature-recommended values [7]
and adjust them empirically. All hyperparameters are listed in Tab. I.

For _reacher_ , we use an episode length of 500 timesteps, with a timestep
duration of 20ms (50Hz control frequency), for a total simulated time of 10s
per episode. The maximal allowed joint speed is 20 deg/sec. For _grasper_ ,
due to safety concerns on the real system, we reduce the maximum joint speed
to 10 deg/sec, adapt the timestep duration to 50ms (20Hz control frequency),
and the episode length to 200 timesteps, for the same total simulated time per
episode of 10s. Under the _ideal_ training strategy, the simulator has all
randomization parameters disabled throughout the entire training duration,
i.e., there is no latency or noise, and the commanded joint velocities are
reached instantaneously. Under the fully randomized strategy, all
randomization parameters are simultaneously enabled. We start with a model
pre-trained in ideal simulation for the CDR and finetuning strategies. Then,
we train them sequentially on each of the $N$, $L$, and $T$-only randomized
simulations for equal duration. For the reaching task, the total training
duration is $4\times 10^{6}$ timesteps, while for the grasping task it is
$10\times 10^{6}$ timesteps due to the higher task complexity and a larger
number of trainable parameters.

###  IV-E Evaluation Procedure and Metrics

While the training takes place only in simulation, evaluation takes place both
in simulation (ideal setting) and on the real robot, see Fig. 2. In both
cases, the same episode duration, control frequency, robot’s starting
configuration, maximal joint speeds, joint limits, and object/target positions
per task are used, leading to identical conditions in the simulation and the
real system, such that any difference in performance of the evaluated model
can be attributed to the sim2real gap. We report multiple metrics detailed in
Tab. II, both in simulation and the real system.

TABLE II: Evaluation metrics.  Cumulative episodic reward ($r_{\mathrm{ep}}$):
The sum of the rewards collected by the agent during an evaluation rollout of
$T$ steps.  $r_{\mathrm{ep}}=\sum\limits_{t=0}^{T}r_{t}$ (11)
---
Continuity cost ($\mathcal{C}$) [40]: A measure of the smoothness of motion.
$\mathcal{C}=\mathbb{E}_{t}\left[\left(\cfrac{a_{t+1}-a_{t}}{\Delta^{a}_{\mathrm{max}}}\right)^{2}\right]$
(12) where $a_{t+1}$ and $a_{t}$ are consecutive actions, and
$\Delta^{a}_{\mathrm{max}}$ denotes the maximum difference between consecutive
actions during the evaluation rollout. $\mathcal{C}$ ranges between 0
(constant actions) to 1 (each consecutive action reaches opposing limits in
the action space). Lower values denote a smoother robot motion and are
desirable in the real system.
Distance to the target ($d_{\mathrm{tgt}}$): The average Euclidean distance of
the center of the robot’s end-effector $p_{t}$ to the target
$p_{\mathrm{tgt}}$ in the second half of the episode, when it is expected that
the target is already reached. This is related to its stabilization strength
[41], i.e., the ability to balance and avoid jittering motions around the
target once it is reached. Lower values denote better performance.
$d_{\mathrm{tgt}}=\cfrac{2}{T}\sum\limits_{t=T/2}^{t=T}||p_{t}-p_{\mathrm{tgt}}||^{2}$
(13)
Grasping success ($\mathcal{G}$): Measures the percentage of successful grasps
during multiple evaluation rollouts. Values range from 0 to 1, where one means
that every grasp attempt was successful.

##  V Experiment Results

###  V-A Reacher

The training progress of agents trained under different strategies and
randomization orders to solve the _reaching_ task is presented in Fig. 4. Each
model is trained ten times with independent seeds and their performance
throughout training is evaluated in non-randomized simulation. The _reaching_
task is relatively easy to solve, especially in the absence of randomizations,
which is why it takes less time for the _Ideal_ baseline to converge w.r.t.
the _Ramdomized_ one. For the sequentially trained models the ordering of the
randomization parameters greatly affects the extent to which the policy can be
improved. The policy improves faster when torque randomization is first
($TLN$) due to the lower uncertainty in the agent’s state compared to the
sequence starting with noise randomization ($NLT$). _Finetuning_ adapts much
faster than the CL models (_CDR- $\lambda$, CDR-$\mathcal{O}\lambda$_) and
reaches performance in simulation similar to the _Ideal_ model. Among the CL
models, _CDR- $\lambda$_ changes more slowly and reaches a lower performance
in simulation compared to _CDR- $\mathcal{O}\lambda$_ since there is a
separate regularization for each randomization parameter under EWC training,
which reduces the ability of the policy to change. _CDR- $\mathcal{O}\lambda$_
adapts faster and converges to a similar performance as the fully-randomized
models.

The evaluation results in a non-randomized simulation and on the real system
for each randomization strategy for the _reaching_ task at the end of training
are presented in Tab. III. The _Ideal_ model performs best in simulation.
However, it overfits to the simulation, and this leads to poor transfer to the
real system, resulting in lower rewards ($r_{\mathrm{ep}}$), high continuity
cost ($\mathcal{C}$), and unstable jittering motions around the target which
results in high average distance to the target ($d_{\mathrm{tgt}}$).
_Randomized_ has lower rewards in simulation but achieves very good sim2real
transfer, leading to higher rewards, low continuity cost, and lowest distance
to target on the real system. The CDR models also achieve very good sim2real
transfer and perform comparably to or better than the fully _Randomized_
baseline. Due to the stronger EWC regularization during training,
CDR-$\lambda$ reaches lower rewards and higher distance to target w.r.t. the
CDR-$\mathcal{O}\lambda$ strategy, while in has comparable continuity cost to
CDR-$\mathcal{O}\lambda$. Overall, CDR-$\mathcal{O}\lambda$ achieves the best
rewards on the real system and has distance and continuity cost values similar
to the fully _Randomized_ baseline. _Finetuning_ performs comparably to
_Ideal_ in simulation but better on the real system. However, due to the lack
of any regularization, it finds over-confident policies in simulation, leading
to much worse sim2real transfer than the CDR strategies. The qualitative
differences in performance between the different strategies are also
noticeable in the supplementary video.

Figure 4: Training progress for the reaching task. Finetuing, CDR-$\lambda$
and CDR-$\mathcal{O}\lambda$ start from the Ideal model at $10^{6}$ timesteps,
and are sequentially trained on each randomization for $10^{6}$ steps, shown
by vertical dotted lines. The max reward is the best reward achieved by any
agent, and the min reward is the one achieved by an agent that executes random
actions. TABLE III: Reacher evaluation in simulation and reality. Metrics are
Episodic reward $r_{\mathrm{ep}}$, dist. to target $d_{\mathrm{tgt}}$, and
continuity cost $\mathcal{C}$.

| $r_{\mathrm{ep}}\uparrow$ | $d_{\mathrm{tgt}}\downarrow$ | $\mathcal{C}\downarrow$
---|---|---|---
Strategy | Sim | Real | Sim | Real | Sim | Real
CDR-$\lambda$ (NLT) | -140.59$\pm$15.21 | -157.24$\pm$17.23 | 0.06 | 0.07 | 0.00 | 0.00
CDR-$\lambda$ (TLN) | -140.37$\pm$15.15 | -157.72$\pm$18.59 | 0.06 | 0.07 | 0.00 | 0.00
CDR-$\mathcal{O}\lambda$ (NLT) | -128.82$\pm$4.88 | -143.88$\pm$4.82 | 0.02 | 0.02 | 0.00 | 0.00
CDR-$\mathcal{O}\lambda$ (TLN) | -130.97$\pm$4.26 | -146.95$\pm$4.18 | 0.03 | 0.03 | 0.00 | 0.00
Finetuning (NLT) | -119.57$\pm$0.29 | -197.78$\pm$7.23 | 0.00 | 0.19 | 0.01 | 0.16
Finetuning (TLN) | -121.86$\pm$1.09 | -196.75$\pm$8.24 | 0.01 | 0.18 | 0.00 | 0.11
Ideal | -119.27$\pm$0.38 | -208.46$\pm$2.17 | 0.00 | 0.20 | 0.01 | 0.30
Randomized | -129.16$\pm$1.94 | -146.52$\pm$3.11 | 0.02 | 0.02 | 0.00 | 0.00

Figure 5: Training progress for the grasping task. Finetuning, CDR-$\lambda$
and CDR-$\mathcal{O}\lambda$ start from the Ideal model at $4\times 10^{6}$
timesteps, and are sequentially trained on each randomization for $2\times
10^{6}$ steps (vertical dotted lines). The max reward is the best reward
achieved by any agent, and the min reward is the one achieved by an agent that
executes random actions. TABLE IV: Grasper evaluation in simulation and
reality. Metrics are Episodic reward $r_{\mathrm{ep}}$, continuity cost
$\mathcal{C}$, and grasping success $\mathcal{G}$.

| $r_{\mathrm{ep}}\uparrow$ | $\mathcal{C}\downarrow$ | $\mathcal{G}\uparrow$
---|---|---|---
Strategy | Sim | Real | Sim | Real | Sim | Real
CDR-$\lambda$ (NLT) | -25.84$\pm$15.10 | -24.27$\pm$7.96 | 0.06 | 0.15 | 0.40 | 0.20
CDR-$\lambda$ (TLN) | -27.63$\pm$20.25 | -23.25$\pm$6.89 | 0.08 | 0.11 | 0.40 | 0.20
CDR-$\mathcal{O}\lambda$ (NLT) | -12.74$\pm$6.34 | -22.78$\pm$21.97 | 0.05 | 0.12 | 0.78 | 0.60
CDR-$\mathcal{O}\lambda$ (TLN) | -44.81$\pm$32.47 | -23.61$\pm$6.11 | 0.08 | 0.09 | 0.23 | 0.20
Finetuning (NLT) | -12.96$\pm$5.99 | -14.99$\pm$5.56 | 0.07 | 0.09 | 0.59 | 0.40
Finetuning (TLN) | -42.88$\pm$35.37 | -27.24$\pm$15.30 | 0.08 | 0.02 | 0.33 | 0.40
Ideal | -12.43$\pm$7.69 | -32.90$\pm$41.40 | 0.06 | 0.45 | 0.77 | 0.60
Randomized | -28.42$\pm$6.00 | -25.89$\pm$7.28 | 0.04 | 0.01 | 0.20 | 0.40

###  V-B Grasper

The results of the training progress of the different strategies for the
_grasping_ task evaluated in the non-randomized simulation are presented in
Fig. 5. Each experiment is repeated five times with independent seeds.
Compared to _reaching_ , the _grasping_ task is more challenging to solve due
to the larger model sizes, the interaction with the object, and the high
precision needed for successful grasping.

With the _Ideal_ model, grasping the object and achieving a high reward is
easier, while with the fully _Randomized_ model, it is more difficult due to
the high uncertainty of the agent’s state. Among the sequential training
strategies, it can be seen that the task is significantly more challenging to
solve when the noise randomization $N$ is encountered in the $TLN$ or $NLT$
sequences. The policy found at the previous randomization step significantly
changes to adapt to the noise-randomized simulation. Overall, _Finetuning_ has
higher deviations during training, while the deviations are lower and more
stable throughout training for the CDR strategies, with _CDR- $\lambda$_ being
more consistent in comparison to _CDR- $\mathcal{O}\lambda$_. This is likely
due to the separate regularization for each randomization parameter.

The evaluation results for _grasping_ in ideal simulation and the real world
after training in the simulation are presented in Tab. IV. Similar to the
_reaching_ task, _Ideal_ achieves high reward in simulation but achieves low
average reward and high continuity cost (unsafe jittering behavior, see
supplementary video) on the real system, even though the object is grasped in
most runs. _Randomized_ has moderate success in finding a good policy in
simulation and has the lowest grasp success in non-randomized simulation.
However, it has good sim2real transfer and achieves the lowest continuity
cost. From the sequential strategies, we notice that they are sensitive to the
ordering and achieve higher success rates but also higher continuity cost when
torque is randomized last in the sequence ($NLT$), and lower continuity cost
but also lower grasp success when noise is at the end of the sequence ($TLN$).
If we look at the rewards on the real system, _Finetuning_ deviates
significantly and, depending on the sequence, can achieve the best or worst
reward from any model that involves randomization, with a difference of about
12%. On the other hand, the CDR models are less susceptible to the ordering of
randomization, as they achieve similar rewards regardless of the sequence.
Moreover, the rewards of the CDR models lie within the range of the best and
worst _Finetuning_ strategy and are better than the reward of the _Randomized_
strategy.

###  V-C Effect of Continual Learning Regularization Strength

In the context of regularization-based CL methods used to implement CDR, the
regularization strength $\lambda$ plays a key role. A very low value makes the
CDR models behave like _Finetuning_ , while a very high value makes it
difficult to adapt to new randomizations after being initially trained in
ideal simulation. We perform additional experiments to investigate the effect
of the regularization strength during continual learning of the different
randomizations by our CDR models. For this study, we employ the same training
procedure as in the _reaching_ experiment in Sec. V-A. We use the same
hyperparameters (Tab. I) and only change the value of the EWC $\lambda$
parameter, testing a range of different values of $\lambda=x^{y}$ with
$x\in\\{1,5\\}$ and $y\in\\{0,\cdots,4\\}$. We train models of _CDR-
$\lambda$_ and _CDR- $\mathcal{O}\lambda$_ in the ideal simulation followed by
two different randomization sequences $TLN$ and $NLT$. We repeated them three
times with independent seeds. The evaluation results of the models in the
ideal simulation and on the real system are shown in Fig. 6.

Figure 6: Effect of the EWC regularization constant $\lambda$ on episodic
reward ($r_{\mathrm{ep}}$), distance to target ($d_{\mathrm{tgt}}$) and
continuity cost ($\mathcal{C}$) for sim2real transfer.

CDR models perform well in simulation for lower values of $\lambda$. However,
they become sensitive to the order of randomizations and transfer poorly to
the real system. On the other hand, higher values of $\lambda$ prevent changes
to the model pre-trained on non-randomized simulation used as a starting point
and perform poorly both in simulation and on the real system. In this case,
the continuity cost on the real system is low because the agent performs slow
moves toward the target but does not learn to balance and overshoots the
target. The middle ranges of $\lambda$ perform reasonably well in simulation
and give the best results on the real system while being relatively robust to
the order of randomizations.

##  VI Discussion and Conclusion

Our experimental results show that full randomization reduces the possibility
of finding a good policy in simulation compared to a non-randomized
simulation, and it also takes longer to converge. Nevertheless, randomization
is important for transferring the model to the real system. Sequential
randomization starting from a model pre-trained on non-randomized simulation
offers a middle ground between no randomization and full randomization.
However, the ordering of the randomizations can affect sim2real transfer.
Unfortunately, in zero-shot transfer (without testing on the target system),
the effect of each randomization parameter cannot be known with certainty.
This can be detrimental since common sequential randomization strategies
overfit to the latest randomization they were trained on.

CDR is less susceptible to the order of randomizations and can transfer the
effects of earlier randomizations in the sequence to the final model. This can
be done either explicitly with CDR-$\lambda$ where the transfer comes at the
price of linear increase of the memory, or implicitly with
CDR-$\mathcal{O}\lambda$ where memory does not increase but the regularization
effect for continual learning is weaker. This paper uses CDR for _zero-shot_
sim2real transfer. However, CDR can also be adapted to provide more flexible
_domain adaptation_ behavior by viewing the real system as the final
randomization in the training sequence. EWC-$\lambda$ can be set to 0 so that
the policy is finetuned to the real system dynamics or to values greater than
0 in cases where the system parameters can vary over time due to wear and tear
or external conditions and the policy needs to stay more general.

A limitation of our current training approach on one randomization at a time
is that we factorize the space of randomization parameters. The complex
interactions between separate randomizations in the sequence may not be
captured, which may lead to sub-optimal solutions. A possible remedy for this
is to define the randomization tasks as sets of different and possibly
intersecting groups of randomization parameters. Furthermore, this paper does
not address how to define the ranges for the randomization parameters.
Excessive randomization is possible even when a single parameter is randomized
at a time, e.g., 10% noise randomization translates to $\pm$ 200mm uncertainty
of the box’s position and makes the grasping task challenging to solve,
leading to poor grasp success on the real system. To address this issue, CDR
could be combined with _automated domain randomization_ [14] or _active domain
randomization_ [15] to find suitable ranges for each randomization parameter.

In summary, CDR provides a flexible framework for zero-shot sim2real transfer
that does not require all randomization parameters to be defined or
implemented ahead of time. It enables randomization parameter decoupling and
continual model adaptation on new randomizations if required. In future work,
we want to address the limitations mentioned above, as well as to implement
CDR with other RL and regularization-based CL approaches and apply it in the
context of vision-based domain randomization.

## References

  * [1] W. Zhao, J. P. Queralta, and T. Westerlund, “Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: A Survey,” in _IEEE Symposium Series on Computational Intelligence (SSCI)_ , Canberra, ACT, Australia, Dec. 2020, pp. 737–744.
  * [2] G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester, “Challenges of Real-World Reinforcement Learning: Definitions, Benchmarks and Analysis,” _Machine Learning_ , vol. 110, no. 9, pp. 2419–2468, 2021.
  * [3] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2017, pp. 23–30.
  * [4] J. Josifovski, M. Malmir, N. Klarmann, B. L. Zagar, N. Navarro-Guerrero, and A. Knoll, “Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Tasks,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. Kyoto, Japan: IEEE, Oct. 2022, pp. 10 193–10 200.
  * [5] G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, and S. Wermter, “Continual Lifelong Learning with Neural Networks: A Review,” _Neural Networks_ , vol. 113, pp. 54–71, May 2019.
  * [6] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms, Tech. Rep. arXiv: 1707.06347, July 2017.
  * [7] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming Catastrophic Forgetting in Neural Networks,” _Proceedings of the National Academy of Sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [8] J. Schwarz, W. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell, “Progress & Compress: A Scalable Framework for Continual Learning,” in _International Conference on Machine Learning (ICML)_ , vol. 80. PMLR, 2018, pp. 4528–4537.
  * [9] S. James and E. Johns, “3D Simulation for Robot Arm Control with Deep Q-Learning,” in _NIPS Workshop: Deep Learning for Action and Interaction_ , Barcelona, Spain, Dec. 2016.
  * [10] M. Kaspar, J. D. Muñoz Osorio, and J. Bock, “Sim2Real Transfer for Reinforcement Learning Without Dynamics Randomization,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. Las Vegas, NV, USA: IEEE, Oct. 2020, pp. 4383–4388.
  * [11] F. Muratore, F. Ramos, G. Turk, W. Yu, M. Gienger, and J. Peters, “Robot Learning from Randomized Simulations: A Review,” _Frontiers in Robotics and AI_ , vol. 9, no. 799893, Apr. 2022.
  * [12] J. Josifovski, M. Kerzel, C. Pregizer, L. Posniak, and S. Wermter, “Object Detection and Pose Estimation Based on Convolutional Neural Networks Trained with Synthetic Data,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Madrid, Spain, Oct. 2018, pp. 6269–6276.
  * [13] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , Brisbane, QLD, Australia, 2018, pp. 3803–3810, iSSN: 2577-087X.
  * [14] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang, “Solving Rubik’s Cube with a Robot Hand, Tech. Rep. arXiv:1910.07113, Oct. 2019.
  * [15] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull, “Active Domain Randomization,” in _Conference on Robot Learning (CoRL)_ , vol. 100. Osaka, Japan: PMLR, May 2020, pp. 1162–1176.
  * [16] F. Ramos, R. Possas, and D. Fox, “BayesSim: Adaptive Domain Randomization Via Probabilistic Inference for Robotics Simulators,” in _Robotics: Science and Systems (R:SS)_ , vol. 15, June 2019.
  * [17] F. Muratore, T. Gruner, F. Wiese, B. Belousov, M. Gienger, and J. Peters, “Neural Posterior Domain Randomization,” in _Conference on Robot Learning (CoRL)_ , vol. 164. PMLR, Nov. 2021, pp. 1532–1542.
  * [18] P. Huang, X. Zhang, Z. Cao, S. Liu, M. Xu, W. Ding, J. Francis, B. Chen, and D. Zhao, “What Went Wrong? Closing the Sim-to-Real Gap Via Differentiable Causal Discovery,” in _Conference on Robot Learning (CoRL)_ , Atlanta, GA, USA, Aug. 2023.
  * [19] G. Tiboni, K. Arndt, and V. Kyrki, “DROPO: Sim-to-Real Transfer with Offline Domain Randomization,” _Robotics and Autonomous Systems_ , vol. 166, p. 104432, Aug. 2023.
  * [20] Y. Huang, K. Xie, H. Bharadhwaj, and F. Shkurti, “Continual Model-Based Reinforcement Learning with Hypernetworks,” in _IEEE International Conference on Robotics and Automation (ICRA)_ , Xi’an, China, May 2021, pp. 799–805.
  * [21] P. Schöpf, S. Auddy, J. Hollenstein, and A. Rodríguez-Sánchez, “Hypernetwork-PPO for Continual Reinforcement Learning,” in _Deep Reinforcement Learning Workshop NeurIPS 2022_ , Dec. 2022.
  * [22] S. Auddy, J. Hollenstein, M. Saveriano, A. Rodríguez-Sánchez, and J. Piater, “Continual Learning from Demonstration of Robotics Skills,” _Robotics and Autonomous Systems_ , vol. 165, p. 104427, 2023\.
  * [23] ——, “Scalable and Efficient Continual Learning from Demonstration Via a Hypernetwork-Generated Stable Dynamics Model, Tech. Rep. arXiv:2311.03600, Jan. 2024, eprint: 2311.03600.
  * [24] K. Khetarpal, M. Riemer, I. Rish, and D. Precup, “Towards Continual Reinforcement Learning: A Review and Perspectives,” _Journal of Artificial Intelligence Research_ , vol. 75, pp. 1401–1476, Dec. 2022.
  * [25] Y. Luo, Y. Wang, K. Dong, Q. Zhang, E. Cheng, Z. Sun, and B. Song, “Relay Hindsight Experience Replay: Self-Guided Continual Reinforcement Learning for Sequential Object Manipulation Tasks with Sparse Rewards,” _Neurocomputing_ , vol. 557, p. 126620, Nov. 2023.
  * [26] J. Josifovski, M. Malmir, N. Klarmann, and A. Knoll, “Continual Learning on Incremental Simulations for Real-World Robotic Manipulation Tasks,” in _2nd R:SS Workshop on Closing the Reality Gap in Sim2Real Transfer for Robotics_ , Corvallis, OR, USA, July 2020, p. 3.
  * [27] A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell, “Progressive Neural Networks, Tech. Rep. arXiv:1606.04671, Oct. 2016.
  * [28] A. A. Rusu, M. Večerík, T. Rothörl, N. Heess, R. Pascanu, and R. Hadsell, “Sim-to-Real Robot Learning from Pixels with Progressive Nets,” in _Annual Conference on Robot Learning (CoRL)_ , vol. 78. Mountain View, CA, USA: PMLR, 2017, pp. 262–270.
  * [29] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, J. Kirkpatrick, R. Pascanu, V. Mnih, K. Kavukcuoglu, and R. Hadsell, “Policy Distillation,” in _International Conference on Learning Representations (ICLR)_. San Juan, Puerto Rico: arXiv, May 2016, _eprint: 1511.06295.
  * [30] R. Traoré, H. Caselles-Dupré, T. Lesort, T. Sun, N. Díaz-Rodríguez, and D. Filliat, “Continual Reinforcement Learning Deployed in Real-Life Using Policy Distillation and Sim2real Transfer,” in _ICML Workshop on Multi-Task and Lifelong Learning_. arXiv, June 2019.
  * [31] S. Kessler, J. Parker-Holder, P. Ball, S. Zohren, and S. J. Roberts, “UNCLEAR: A Straightforward Method for Continual Reinforcement Learning,” in _ICML Workshop on Continual Learning_ , vol. 108\. Vienna, Austria: PMLR, 2020.
  * [32] A. Gaurav, “Safety-Oriented Stability Biases for Continual Learning,” Master’s thesis, University of Waterloo, 2020.
  * [33] “Robotiq Gripper.” [Online]. Available: http://robotiq.com/products/industrial-robot-hand/
  * [34] P. Varin, L. Grossman, and S. Kuindersma, “A Comparison of Action Spaces for Learning Manipulation Tasks,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , Macau,China, 2019, pp. 6015–6021.
  * [35] Y. R. Stürz, L. M. Affolter, and R. S. Smith, “Parameter Identification of the KUKA LBR iiwa Robot Including Constraints on Physical Feasibility,” _IFAC-PapersOnLine_ , vol. 50, no. 1, pp. 6863–6868, 2017.
  * [36] S. Edwards and C. Lewis, “ROS-Industrial — Applying the Robot Operating System (ROS) to Industrial Applications,” in _ECHORD Workshop at the IEEE International Conference on Robotics and Automation (ICRA)_ , St. Paul, MN, USA, 2012.
  * [37] C. Hennersperger, B. Fuerst, S. Virga, O. Zettinig, B. Frisch, T. Neff, and N. Navab, “Towards MRI-Based Autonomous Robotic Us Acquisitions: A First Feasibility Study,” _IEEE Transactions on Medical Imaging_ , vol. 36, no. 2, pp. 538–548, Feb. 2017.
  * [38] M. Quigley, B. Gerkey, K. Conley, J. Faust, T. Foote, J. Leibs, E. Berger, R. Wheeler, and A. Ng, “ROS: An Open-Source Robot Operating System,” in _ICRA Workshop on Open Source Software_ , vol. 3, 2009, p. 6.
  * [39] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable Baselines,” 2018.
  * [40] A. Raffin, J. Kober, and F. Stulp, “Smooth Exploration for Robotic Reinforcement Learning,” in _Conference on Robot Learning (CoRL)_ , vol. 164. London, UK: PMLR, Jan. 2022, pp. 1634–1644.
  * [41] M. Malmir, J. Josifovski, N. Klarmann, and A. Knoll, “DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control, Tech. Rep. arXiv:2306.09010, 2023\.