11institutetext: Faculdade de Engenharia da Universidade do Porto, Porto,
Portugal 22institutetext: INESC TEC, Porto, Portugal 22email:
{ruben.m.queiros, luis.m.martins, helder.m.fontes<EMAIL_ADDRESS>
# RateRL: A Framework for Developing RL-Based Rate Adaptation Algorithms in
ns-3

Ruben Queiros 1122 0000-0001-7804-0246 Luís Ferreira 1122  0009-0002-9107-0639
Helder Fontes 22 0000-0002-7672-8335 Rui Campos 1122 0000-0001-9419-6670

###### Abstract

The increasing complexity of recent Wi-Fi amendments is making the use of
traditional algorithms and heuristics unfeasible to address the Rate
Adaptation (RA) problem. This is due to the large combination of configuration
parameters along with the high variability of the wireless channel. Recently,
several works have proposed the usage of Reinforcement Learning (RL)
techniques to address the problem. However, the proposed solutions lack
sufficient technical explanation. Also, the lack of standard frameworks
enabling the reproducibility of results and the limited availability of source
code, makes the fair comparison with state of the art approaches a challenge.
This paper proposes a framework, named RateRL, that integrates state of the
art libraries with the well-known Network Simulator 3 (ns-3) to enable the
implementation and evaluation of RL-based RA algorithms. To the best of our
knowledge, RateRL is the first tool available to assist researchers during the
implementation, validation and evaluation phases of RL-based RA algorithms and
enable the fair comparison between competing algorithms.

###### Keywords:

Wireless Networks ns-3 Deep Reinforcement Learning Machine Learning Tool.

##  1 Introduction

The new configuration parameters available in the most recent Wi-Fi amendments
allied to the high variability and asymmetry of the radio channel, make Rate
Adaptation (RA) and the optimal configuration of these parameters challenging.
Recent RA algorithms in Wi-Fi are often developed considering simplified
simulations that are not always well described. This poses a challenge for the
implementation and comparison of different RA algorithms. For example, the
Network Simulator 3 (ns-3) implements a significant amount of existing state-
of-the-art RA algorithms. However, only a few of these algorithms can be used
for recent Wi-Fi versions subsequent to IEEE 802.11n. Additionally, many
recent RA algorithms are based on Machine Learning (ML), which can make them
difficult to implement and understand. In some cases, the authors of these
algorithms do not even provide the source code or the training dataset, which
poses an obstacle to accurately reproduce the obtained results. When
developing novel ML-based solutions the authors should consider the following
practices: 1) clearly describe the problem that the proposal is trying to
solve and justify any underlying assumptions; 2) systematically devise
procedures that ensure the reproducibility of results; 3) clearly state and
justify the metrics used for the evaluation; 4) expose the dataset used in the
training process of the algorithm; and 5) have one or more baseline models to
compare with.

The use of Reinforcement Learning (RL) and other ML techniques within the
wireless networks research area has emerged as a way to further improve the
network Quality of Service (QoS). Leveraging these techniques, the research
community has been optimising the network performance, reducing latency, and
ensuring efficient resource allocation [15, 10]. A high percentage of the
current telecommunications infrastructures have begun to invest and test ML
algorithms for supporting the network operation and business decisions [11].
However, the techniques employed are not standardized, resulting in a lack of
foundational principles that would enable the transfer of knowledge from
previous initiatives to more recent ones. When compared with other fields that
pioneered the use of ML, such as computer vision or natural language
processing, wireless networks research started using ML at a later stage.
Thus, it is common within the existing literature to identify ML-based
solutions that lack the use of good practices stated above. A predominant
challenge relates to the scarcity of technical details provided in published
works, which hampers the reproducibility of results. Furthermore, the limited
availability to the source code and the dataset used, precludes the validation
and extension of prior findings. These issues collectively emphasize the need
for a more cohesive and systematic approach to integrate RL into wireless
networks.

The main contribution of this paper is RateRL, a framework to support the
development of RL-based RA algorithms. We illustrate the use of RateRL through
a practical use case, employing the Data Driven Algorithm for Rate Adaptation
(DARA) proposed in [13]. Our framework integrates well-known RL libraries,
such as TensorFlow Agents and OpenAI Gym [2] with ns-3 [14], using ns3-gym [7]
to interface all the components. RateRL enables the task automation to
identify the hyperparameter configuration that maximizes the expected
cumulative reward of the RL algorithm, while offering real-time feedback of
the training process. The code and ns-3 scripts are publicly available,
facilitating the efforts of future research works to build upon RateRL.

The rest of the paper is organised as follows. Section II presents the
background. Section III addresses the related work. Section IV explains the
RateRL framework. Section V illustrates the use of RateRL considering DARA as
the use case. Finally, Section V provides some concluding remarks and points
out the future work.

##  2 Background

This section refers to some background concepts that are necessary to
understand RateRL. It starts with an overview of RL, namely the Q-learning
algorithm and existing frameworks to implement these techniques. Then, a brief
description of the network simulator ns-3 is provided.

###  2.1 Reinforcement Learning

The RL model, represented in Figure 1, is composed of the agent and the
environment elements. The communication between them is based on state, action
and reward signals. The agent learns the decisions (actions) based on the
observations (states) received from the environment and its decision is
evaluated taking into account the reward. The objective of RL is to learn a
policy of actions to take for a given environment state that maximises the
overall cumulative reward for every (state, action) pair. There are multiple
learning algorithms available in the literature. The recent works that address
the RA problem [13, 5, 12, 4] use the classic Q-learning algorithm and show
better results when compared to traditional algorithms such as Minstrel-HT
[6], the default RA algorithm for Linux systems.

Figure 1: Reinforcement Learning loop diagram.

Q-learning [16] is a model-free algorithm, which means that it does not use
any prior knowledge of the environment and learns through trial and error.
Q-learning only works with a discrete action space. The objective of
Q-learning is to learn the optimal policy that maximises the expected
cumulative reward. Q-learning learns and updates its Q-function values via
trial and error using Eq. 1, where $Q(s,a)$ is the expected cumulative reward
when the agent selects action $a$ in state $s$ considering that future actions
are selected according to the learnt policy. $r(s,a)$ is the reward for taking
action $a$ in state $s$, and $\max_{a\in A}Q(s_{new},a),\forall a\in A$ is the
maximum possible reward of the new state $s_{new}$, which is the result of the
current action; $s_{new}$ is the new state and $a_{all}$ represents every
$a\in A$. The learning rate $\alpha$ determines the rate at which new values
update the total Q-value. Finally, the discount factor $\gamma\in[0,1]$
determines the importance of future rewards in the calculation of the expected
cumulative reward.

$Q(s,a)\leftarrow(1-\alpha)Q(s,a)+\alpha[r(s,a)+\gamma\max_{\forall a\in
A}Q(s_{new},a)]$ (1)

###  2.2 Reinforcement Learning Frameworks

There are multiple and extensively validated frameworks available to implement
Q-learning or other RL-based learning algorithm. TensorFlow and PyTorch are
two prominent deep learning frameworks widely adopted for building and
training ML models, including RL agents. TensorFlow, developed by Google,
offers a comprehensive ecosystem of tools and libraries that facilitate the
creation of neural networks and the optimisation of their parameters.
TensorFlow Agents (TF-Agents) is an extension of TensorFlow specifically
designed to enable the construction of RL agents. TF-Agents [8] provides
reusable components for defining agent behaviour, defining the environment
interaction loop, and implementing various state-of-the-art RL algorithms.
Similarly, PyTorch, developed by Facebook’s AI Research lab, provides a
dynamic computational graph that simplifies model development and
experimentation. Gym is an open-source toolkit developed by OpenAI, designed
to facilitate the development and testing of RL algorithms. It provides a
collection of environments (simulated scenarios) that allow researchers and
developers to experiment with and benchmark various RL techniques as well as
to develop their own custom environment making it a valuable resource that can
interface with the popular network simulator ns-3.

###  2.3 The Network Simulator 3

ns-3 [14] is a well-known open-source network simulation tool and one of the
most used wireless network simulators. ns-3 was driven by a desire to model
networks in a way that best suits network research and learning. It aims to
provide highly accurate and scalable network simulation capabilities for
studying various aspects of networking protocols, communications methods, and
network behaviour. Hence, it was not developed with ML or artificial
intelligence in mind. There is currently no official framework for integrating
with prevalent ML tools. Initial community efforts [7, 17] were made to bridge
the gap between the network simulator and popular ML frameworks such as
TensorFlow, PyTorch and Gym. The resulting tools, named ns3-ai and ns3-gym,
are detailed in Section 3.

##  3 Related Work

The related work is presented in this section. First, we overview existing
reinforcement learning frameworks for networking and then we review the state
of the art in RL-based solutions for the RA problem.

###  3.1 Reinforcement Learning Frameworks for Networking

To the best of our knowledge, few frameworks that integrate Reinforcement
Learning frameworks with network simulators have been proposed in the state of
the art. In [7] the authors develop a sockets-based interface between ns-3 and
OpenAI Gym to encourage the usage of RL in networking research. To overcome
the lack of flexibility imposed by ns3-gym, the authors in [17] proposed
ns3-ai, a Python module that allows the interconnection between any artificial
intelligence framework with ns-3, using a higher efficiency mechanism based on
shared-memory. However, since this is a high flexibility and efficiency data
interaction framework researchers using ns3-ai have to adapt to implement
their own development environment to address their specific problem, resulting
in slower algorithm development or simplistic simulation environments, which
reduces the solution realism. On the other hand, the authors in [1] developed
the PRISMA framework, which is tailored to the distributed packet routing
problem, on top of ns3-gym, extending the problem to a Multi-Agent Deep
Reinforcement Learning approach. Therefore, researchers would need to modify
PRISMA’s design if the objective was to address the RA problem.

###  3.2 Reinforcement Learning for Rate Adaptation

Different solutions have been proposed to solve the RA problem, in particular
for Wi-Fi networks. Some use classical heuristic approaches [6, 3] while
recent works have been using RL-based techniques [9, 4, 5, 12, 13]. In [4],
the authors developed a Q-learning based link adaptation solution that
addresses RA together with other configuration parameters such as the channel
bandwidth and number of spatial streams, outperforming state of the art
solutions in terms of throughput. Despite implementing it in a network
interface card and evaluating their solution in an experimental setting, the
authors do not mention any standard framework that was used to implement their
solution. Also, the source code is not publicly available. In [5, 12] the
authors use ns-3 to implement their RL-based RA solutions with the help of
ns3-gym and ns3-ai frameworks. However, both works do not provide any
information with regards to the training process or hyperparameter
configuration. Moreover, despite simplistic simulation scenarios for
evaluation, they are not sufficiently well described, posing an obstacle to
accurately reproduce the obtained results.

###  3.3 Summary

Despite the good results of recent works that boost the overall network
performance, the difficulty to replicate the results of the proposed solutions
is common among existing works. Typically, they lack implementation details –
i.e. the code is not open source – and training process description. Moreover,
within the identified works, the results of the proposed solutions are not
compared with other RL-based RA algorithms.

##  4 RateRL Framework

In this section, we present the RateRL framework, including its architecture
and components.

The RateRL framework was designed considering design principles similar to the
PRISMA framework [1], such as the achievement of realistic wireless networks
simulation environments due to the usage of ns-3 and the development with a
modular approach, which makes fast prototyping of RL-based RA algorithms
possible.

Figure 2: RateRL architecture diagram.

In Figure 2 we present the RateRL architecture and how its components interact
with each other. The programming languages used to code RateRL were Python and
C++. The Python files hold every essential setup procedures and parsing
mechanisms required for the proper execution of the system, the core ML
components and most of the data collection processes. The files written in
C++, are mainly related to the configuration and utilisation of the ns-3
simulator together with some files that interact with the ns3-gym interface.

We now detail the RateRL components and instruct the reader on how to use
RateRL. The starting point of the RateRL framework is the start_script file. A
detailed description of the role of each component is presented below:

  * •

start_script checks if the configuration file is valid. It also sets up the
results folder where the logs of the simulation or training are stored.
Finally, it starts the agent component.

  * •

config is a JSON file that holds the configuration parameters relevant for
each of the other files within the framework, such as the agent, gym and sim.

  * •

agent interacts with the TF-Agents framework using the environment data that
is retrieved from the ns-3 simulation. It starts the ns-3 environment using
the ns3-gym interface.

  * •

sim defines the ns-3 simulation script. It uses the lib folder, where multiple
utility functions are defined to ease aspects such as the simulation
configuration and data collection.

  * •

gym configures the ns-3 environment defining the observation and action data
shapes. It executes the action that comes from the agent, and establishes the
data collection methods that are used to collect the observation from the ns-3
simulation as well as the metrics to calculate the reward.

The installation process is documented in the public repository, inside the
install folder, and it assumes the user does not have any of the required
dependencies installed. The user can run the agent in two different modes:

  * •

Training Mode starts by filling the replay buffer with the trajectories
collected from the simulation. A trajectory consists of a time step (i.e., the
initial observation of the environment), the action step (i.e., the action
that was taken considering the previous time step), and the next time step
(i.e., the new observation and the reward that was obtained using the previous
action step). This replay buffer is filled while the simulation is running
until the end of the episode, which in this case is the end of the simulation.
The agent is trained during this process, grabbing randomly from the replay
buffer an amount of trajectories defined with the hyperparameter batch size,
updating the weights accordingly and increasing the train step counter. The
user can then adapt the amount of episodes the simulation runs, how frequent
the training happens and the way epsilon greedy adjusts over the training
process. When the training is over, or if the user wants to pause it, it is
possible to save the progress with a checkpoint so that the current state of
the policy can be recovered later.

  * •

Evaluation Mode loads the trained Policy and assumes a fixed epsilon greedy
factor of 0 to avoid exploratory attempts. However, this mode is not prepared
for simulation scenarios that dynamically change requiring an online learning
approach. This will be the subject of future work.

Regardless of the mode used, RateRL saves simulation logs with the throughput
of every existing communication link as well as the nodes positions, with a
configurable periodicity.

##  5 Using the RateRL Framework

In this section we use the implementation, training and evaluation of the DARA
algorithm [13] as a use case to illustrate the utilisation of the RateRL
framework.

###  5.1 DARA Overview

DARA [13] is a RL-based RA algorithm developed for the IEEE 802.11n amendment.
It considers scenarios with Single Input Single Output and fixed channel
bandwidth of 20 MHz, using long Guard Interval. The valid actions are the
first 8 Modulation and Coding Schemes (MCS). The state is the average SNR
value considering the Acknowledgement frames that originate from the receiver
node. Finally, the reward is a function of the Frame Success Ratio (FSR) and
the chosen MCS, to value the highest possible MCS without compromising the
FSR.

###  5.2 Simulation Settings

We configured the preliminary validation scenario defined in [13], . In this
scenario, we have a stationary node and a moving node. In the beginning of the
simulation, the nodes start close to each other, and their distance increases
throughout the simulation period. In this way we stimulate the algorithm with
a wide range of SNR values. The algorithm is then compared in terms of
throughput with other RA algorithms implemented in ns-3 such as Minstrel-HT
(MIN) and the Ideal (ID) algorithm. All the other main simulation
configuration parameters are presented in Table 1.

Table 1: Simulation Configuration Parameters. Configuration Parameter | Value
---|---
Wi-Fi Standard | IEEE 802.11n
Propagation Delay Model | Constant Speed
Propagation Loss Model | Friis
Frequency | 5180 MHz
Channel Bandwidth | 20 MHz
Transmission Power | 20 dBm
Wi-Fi MAC | Ad-hoc
Traffic | UDP, generated above link capacity
Packet Size | 1400 Bytes of UDP Payload

###  5.3 Training and Hyperparameter Tuning

DARA was trained and evaluated on a ASUS ROG G14 Laptop with a Ryzen 9 5900HS
(8 cores up to 4.6 GHz), 32 GB RAM and a NVIDIA RTX 3060 GPU. In this
illustrative example the hyperparameter configuration chosen is defined in
Table 2. The simulations were 60 seconds long, for a total of 15 episodes.

Table 2: DQN Learning Algorithm main Parameters. Parameter |  Value
---|---
Observation Space |  One-dimensional scaled float (0.0-1.0)
Action Space |  One-dimensional integer (0-7)
Optimiser |  Adam
Loss Function |  Mean Square Error
Epsilon Greedy |  Fixed at 0.1
Discount Factor |  Fixed at 0.5
Replay buffer |  size of $10^{6}$
Batch Size |  64

The hyperparameter tuning is an essential part of any machine learning model
training. To this end, we assess how different values could benefit the final
DARA performance. In this work, comparisons between different values of
learning rate and the hidden layer architecture of the neural network were
carried out.

These comparisons were not extensive, thus the performance of DARA could be
further improved with a more in depth tuning, despite the simple scenario,
which shows that good results are achievable with few training episodes.
However, the objective of this work is to show that RateRL can be used to
compare different hyperparamenter configurations and assess their impact in
the performance of the algorithm.

(a) Learning Rate comparison training.

(b) Hidden Layer architecture training.

Figure 3: Hyperparameter tuning trainings.

Figure 3(a) shows the cumulative reward over 15 training episodes with 4
different learning rate configurations, using two hidden layers with 32 units
each. The results show that a learning rate of 0.01 is consistently better
than the other options. After defining the used learning rate value,
additional trainings were performed to fine tune the hidden layer
architecture.

Figure 3(b) shows the results of this training, with 5 different options being
evaluated. Despite the similarities in performance it was decided to choose as
the final configuration the one which finished with highest cumulative reward
by the end of the 15 episodes training. Therefore, a learning rate of 0.01 and
a neural network with 3 hidden layers of 16 units each was defined.

###  5.4 Simulation Results

Using the resulting policy from the training that was detailed in the previous
sections, we used RateRL to evaluate how the performance of DARA is compared
to other popular RA algorithms such as Minstrel and Ideal. Fig 4(a) shows the
throughput throughout the simulation period and Fig 4(b) its complementary
cumulative distribution function.

The results show that RateRL can be used to evaluate the performance and
comparison of RL-based RA algorithms with other state of the art RA solutions.
The average throughput of Ideal was of 13.45 Mbit/s and Minstrel was of 13.07
Mbit/s. DARA achieved an average throughput of 13.52 Mbit/s, an increase of
3.4% over Minstrel and similar throughput when compared with Ideal. To
conclude, we managed to successfully implement train and evaluate DARA using
RateRL.

(a) Throughput

(b) Throughput CCDF

Figure 4: Simulation Results using the resulting hyperparamenter
configuration.

##  6 Conclusion

This paper presented RateRL, the first framework designed for assiting the
development, validation and evaluation of RL-based RA algorithms. We
demonstrated the use of RateRL in the whole development cycle of an RL-based
RA algorithm, using the state of the art DARA algorithm as a use case. Our
objective with RateRL is to provide a framework for developing future RL-based
RA solutions and enable their direct comparison with state of the art or
related solutions. The RateRL framework is open source and it is publicly
available on Gitlab 111https://gitlab.inesctec.pt/pub/ctm-win/raterl.

As future work, we plan to migrate to ns3-ai to support other popular ML
frameworks. Also, we aim to extend RateRL to use other popular RL algorithms
such as Deep Deterministic Policy Gradient and Proximal Policy optimisation.

## Acknowledgements

This work is financed by National Funds through the Portuguese funding agency,
FCT - Fundação para a Ciência e Tecnologia, under the PhD grant 2022.10093.BD.

## References

  * [1] Alliche, R.A., Barros, T.D.S., Aparicio-Pardo, R., Sassatelli, L.: Prisma: a packet routing simulator for multi-agent reinforcement learning. In: 2022 IFIP Networking Conference (IFIP Networking). pp. 1–6. IEEE (2022)
  * [2] Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W.: Openai gym (2016)
  * [3] Byeon, S., Yoon, K., Yang, C., Choi, S.: Strale: Mobility-aware phy rate and frame aggregation length adaptation in wlans. In: IEEE INFOCOM 2017 - IEEE Conference on Computer Communications. pp. 1–9 (2017). https://doi.org/10.1109/INFOCOM.2017.8056965
  * [4] Chen, S.C., Li, C.Y., Chiu, C.H.: An experience driven design for ieee 802.11ac rate adaptation based on reinforcement learning. In: IEEE INFOCOM 2021 - IEEE Conference on Computer Communications. pp. 1–10 (2021). https://doi.org/10.1109/INFOCOM42981.2021.9488876
  * [5] Cho, S.: Reinforcement learning for rate adaptation in csma/ca wireless networks. In: Advances in Computer Science and Ubiquitous Computing: CSA-CUTE 2019\. pp. 175–181. Springer (2021)
  * [6] FietKau, F.: Minstrel_HT: New rate control module for 802.11n [LWN.net] (3 2010), https://lwn.net/Articles/376765, [Accessed 20-May-2022]
  * [7] Gawłowicz, P., Zubow, A.: Ns-3 meets openai gym: The playground for machine learning in networking research. In: Proceedings of the 22nd International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems. pp. 113–120 (2019)
  * [8] Guadarrama, S., Korattikara, A., Ramirez, O., Castro, P., Holly, E., Fishman, S., Wang, K., Gonina, E., Wu, N., Kokiopoulou, E., Sbaiz, L., Smith, J., Bartók, G., Berent, J., Harris, C., Vanhoucke, V., Brevdo, E.: TF-Agents: A library for reinforcement learning in tensorflow. https://github.com/tensorflow/agents (2018), https://github.com/tensorflow/agents, [Online; accessed 25-June-2019]
  * [9] Karmakar, R., Chattopadhyay, S., Chakraborty, S.: Smartla: Reinforcement learning-based link adaptation for high throughput wireless access networks. Computer Communications 110, 1–25 (2017)
  * [10] Kulin, M., Kazaz, T., De Poorter, E., Moerman, I.: A survey on machine learning-based performance improvement of wireless networks: Phy, mac and network layer. Electronics 10(3), 318 (2021)
  * [11] Mahmoud, H.H.H., Ismail, T.: A review of machine learning use-cases in telecommunication industry in the 5g era. In: 2020 16th International Computer Engineering Conference (ICENCO). pp. 159–163 (2020). https://doi.org/10.1109/ICENCO49778.2020.9357376
  * [12] Pratama, M.H.B., Nakashima, T., Nagao, Y., Kurosaki, M., Ochi, H.: Experimental evaluation of rate adaptation using deep-q-network in ieee 802.11 wlan. In: 2023 IEEE 20th Consumer Communications & Networking Conference (CCNC). pp. 668–669. IEEE (2023)
  * [13] Queirós, R., Almeida, E.N., Fontes, H., Ruela, J., Campos, R.: Wi-fi rate adaptation using a simple deep reinforcement learning approach. In: 2022 IEEE Symposium on Computers and Communications (ISCC). pp. 1–3 (2022). https://doi.org/10.1109/ISCC55528.2022.9912784
  * [14] Riley, G.F., Henderson, T.R.: The ns-3 network simulator. In: Modeling and tools for network simulation, pp. 15–34. Springer (2010)
  * [15] Szott, S., Kosek-Szott, K., Gawłowicz, P., Gómez, J.T., Bellalta, B., Zubow, A., Dressler, F.: Wi-fi meets ml: A survey on improving ieee 802.11 performance with machine learning. IEEE Communications Surveys & Tutorials 24(3), 1843–1893 (2022)
  * [16] Watkins, C.J., Dayan, P.: Q-learning. Machine learning 8, 279–292 (1992)
  * [17] Yin, H., Liu, P., Liu, K., Cao, L., Zhang, L., Gao, Y., Hei, X.: Ns3-ai: Fostering artificial intelligence algorithms for networking research. In: Proceedings of the 2020 Workshop on Ns-3. p. 57–64. WNS3 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3389400.3389404, https://doi.org/10.1145/3389400.3389404


# Impedance Adaptation by Reinforcement Learning
with Contact Dynamic Movement Primitives

Chunyang Chang1,†, Kevin Haninger2,†, Yunlei Shi4,1, Chengjie Yuan3, Zhaopeng
Chen1, Jianwei Zhang4 1 with Agile Robots AG, Munich, Germany. Email:
<EMAIL_ADDRESS>with the Department of Automation at
Fraunhofer IPK, Berlin, Germany3 with Intel Asia-Pacific Research &
Development Ltd., Shanghai, China4 with TAMS (Technical Aspects of Multimodal
Systems), Department of Informatics, Universität Hamburg, Hamburg, Germany†
The first two authors contributed equally to this work. This research has
received funding from the German Research Foundation (DFG) and the National
Science Foundation of China (NSFC) in project Crossmodal Learning, DFG
TRR-169/NSFC 61621136008, partially supported by the European Union’s Horizon
2020 research and innovation programme under the Marie Sklodowska-Curie grant
agreement No 778602 ULTRACEPT and No 820689 SHERLOCK.

###### Abstract

Dynamic movement primitives (DMPs) allow complex position trajectories to be
efficiently demonstrated to a robot. In contact-rich tasks, where position
trajectories alone may not be safe or robust over variation in contact
geometry, DMPs have been extended to include force trajectories. However,
different task phases or degrees of freedom may require the tracking of either
position or force – e.g., once contact is made, it may be more important to
track the force demonstration trajectory in the contact direction. The robot
impedance balances between following a position or force reference trajectory,
where a high stiffness tracks position and a low stiffness tracks force. This
paper proposes using DMPs to learn position and force trajectories from
demonstrations, then adapting the impedance parameters online with a higher-
level control policy trained by reinforcement learning. This allows one-shot
demonstration of the task with DMPs, and improved robustness and performance
from the impedance adaptation. The approach is validated on peg-in-hole and
adhesive strip application tasks.

##  1 INTRODUCTION

A major challenge in robotics is reducing the commissioning effort for a new
task. For free-space tasks, the task is mostly solved by a position trajectory
of the robot, which can be taught by a variety of methods [1]. However,
contact-rich tasks, which require the application of force on a possibly
uncertain environment, are typically not solved with only position control.

Some tasks, e.g. polishing tasks, are more robustly represented as force
tasks, requiring a force trajectory to be applied in certain degrees of
freedom (DOF). Other tasks require a mix of position and force control, which
can be considered in a unified way with impedance control [2]. In impedance
control, a high impedance (i.e. high stiffness) better tracks a position
trajectory, and a low impedance better tracks a force trajectory [3].

Dynamic movement primitives (DMPs) allow the efficient teaching of a
trajectory - only one demonstration is needed. In the original formulation
[4], DMPs track a position trajectory, allowing a trajectory shape to be
generalized over changes in initial or goal positions. DMPs have also been
extended into contact DMPs [5], where force trajectories are also recorded.
This allows the DMP to be employed with an impedance- or force-controlled
robot [6, 7], improving safety and robustness over small changes in geometry.

However, different impedance behaviors are required in different applications
– due to, e.g. different part geometries, contact materials, or tolerances.
Impedance may also vary during a task as contact conditions change [8].
Adaptation of impedance is also important for humans, especially in unstable
tasks like screwdriving [9].

Figure 1: Experimental setup, including robot, additional F/T sensor, peg-in-
hole and adhesive tape assembly task. Insets on top right show the hole and
the adhesive strip end-effector.

Optimizing impedance parameters can improve the cycle time and robustness of a
task [10], reduce force overshoot [11] or reduce trajectory jerk [12]. There
are a wide variety of application-specific and ad-hoc methods for tuning
impedance, recently reviewed in [8]. Determining robot impedance from
demonstrations can be done by transferring from human demonstrations [13, 14],
an additional human interface [15], determined by a model predictive
controller [16], or determined by variance of the demonstrations [17].

Reinforcement learning (RL) also aims to reduce manual commissioning work,
allowing the robot to explore and learn a control policy. To improve sample
efficiency – how much trial-and-error is required – RL has been combined with
DMPs. In [18, 19], a DMP trajectory is taught and a position control gain
schedule is learned. RL has also been applied to learn a force trajectory [20]
or a residual force trajectory [21] for contact-rich tasks.

Compared with other contact DMPs [6], we learn online adaptation of the
impedance to balance the tracking of force and position trajectories. Compared
with other combinations of DMPs and RL, this paper uses RL to adapt impedance,
not learn a force trajectory [20, 21]. This paper first introduces the
teaching workflow and system architecture, then presents the individual
modules (admittance control, dynamic movement primitives, and reinforcement
learning), then presents the experimental results.

##  2 Problem Statement and System Architecture

A robot with a hand guiding mode and a force/torque (F/T) sensor, as seen in
Figure 1 is used. The commissioning process here contains three stages:

  1. 1.

Demonstration Collection: Using the hand guidance mode of the robot, the robot
is grasped on the robot side of the F/T sensor and task is demonstrated. The
position data from the robot is acquired along with the measurements from the
F/T sensor.

  2. 2.

Autonomous Data Collection: The demonstration is processed into a DMP, which
is executed autonomously. A control policy which maps position and force to a
change in robot impedance is learned by the soft actor-critic RL algorithm.

  3. 3.

Execution: The system is executed, with the DMP producing the position and
force trajectory, and the RL policy adjusting impedance according to measured
position and force.

An overview of data flow during the commissioning process can be seen in
Figure 2. The individual modules (admittance controller, DMPs, and RL policy),
will be introduced in the following sections.

Figure 2: Data flow diagram for the complete system

##  3 Admittance Control

To allow data collection where human and environment forces are separated, a
F/T sensor is used, as labelled in Figure 1. The human forces for the hand
guidance in demonstration are measured by the built-in joint torque sensors of
the robot. The environmental contact forces are measured with the flange F/T
sensor during the demonstration. This separation of human and environment
forces is established for contact-rich demonstrations [13, 20].

###  3.1 Admittance Controller

The admittance control law of

$\mathbf{M}\ddot{\mathbf{x}}_{e}+\mathbf{D}\dot{\mathbf{x}}_{e}+\mathbf{K}\mathbf{x}_{e}=\mathbf{F}_{ext}-\mathbf{F}_{d}$
(1)

is realized, where $\mathbf{x}_{e}$ is the pose deviation from planned
trajectory $\mathbf{x}_{d}$, $\mathbf{F}_{ext}$ is the measured external force
and $\mathbf{F}_{d}$ is the desired force-torque, all in the TCP coordinate
frame. The matrices $\mathbf{M}$, $\mathbf{D}$ and $\mathbf{K}$ are control
parameters, diagonal $6\times 6$ matrices. The admittance controller can be
seen in Figure 3.

####  3.1.1 Trade-off between position and force control

To motivate the use of impedance stiffness $\mathbf{K}$ to modulate between
force and position control, we examine (1) for a single DOF and various $K$.
Taking the Laplace domain form of (1), we find
$Ms^{2}+Ds+K=F_{ext}(s)-F_{d}(s)$. To find the steady-state behavior, we
consider $s\rightarrow 0$, and constant $F_{ext}$, $F_{d}$.

When $K\rightarrow\infty$, $x_{e}=K^{-1}(F_{ext}-F_{d})$, and it can be seen
that small $x_{e}$ results, so the commanded robot position is almost equal
the position trajectory, i.e. $x_{c}\approx x_{d}$. When $K\rightarrow 0$,
$F_{ext}-F_{d}\approx 0$, so the real force $F_{ext}$ tracks the desired force
$F_{d}$.

Figure 3: Control block diagram of the admittance controller

####  3.1.2 Orientation implementation

The linear and orientation components of a pose
$\mathbf{x}_{[\cdot]}=[\mathbf{x}^{p}_{[\cdot]},\mathbf{q}_{[\cdot]}]$, where
$\mathbf{q}_{[\cdot]}$ is a unit quaternion. A unit quaternion is
$\mathbf{q}=v+\mathbf{u}\in\mathcal{S}^{3}$, where $\mathcal{S}^{3}$ is a unit
sphere in $\mathbb{R}^{4}$, consisting of a scalar part $v\in\mathbb{R}$ and a
vector part $\mathbf{u}\in\mathbb{R}^{3}$. The multiplication, conjugation and
norm of quaternions are defined as

$\displaystyle\mathbf{q}_{1}*\mathbf{q}_{2}$
$\displaystyle=(v_{1}v_{2}\texttt{-}\mathbf{u}^{\top}_{1}\mathbf{u}_{2})\texttt{+}(v_{1}\mathbf{u}_{2}\texttt{+}v_{2}\mathbf{u}_{1}\texttt{+}\mathbf{u}_{1}\times\mathbf{u}_{2})$
(2) $\displaystyle\overline{\mathbf{q}}$
$\displaystyle=\overline{v+\mathbf{u}}=v-\mathbf{u}$ (3)
$\displaystyle||{\mathbf{q}}||$
$\displaystyle=\sqrt{\mathbf{q}*\overline{\mathbf{q}}}=\sqrt{v^{2}+||\mathbf{u}||^{2}},$
(4)

where $||\mathbf{u}||$ is the $\ell_{2}$ norm of $\mathbf{u}$. To map
$\mathcal{S}^{3}$ to $\mathbb{R}^{3}$, the quaternion logarithm function
$\log:\mathcal{S}^{3}\mapsto\mathbb{R}^{3}$ is given by

$\log(\mathbf{q})=\log(v+\mathbf{u})=\begin{cases}\arccos(v)\frac{\mathbf{u}}{||\mathbf{u}||},&||\mathbf{u}||\neq
0\\\ [0,0,0]^{\top},&otherwise\end{cases}$ (5)

Furthermore, there is a relationship between the quaternion logarithm and the
angular velocity as

$\boldsymbol{\omega}=2\Delta_{t}^{-1}\log(\mathbf{q}_{t+1}*\overline{\mathbf{q}_{t}}),$
(6)

where $\boldsymbol{\omega}\in\mathbb{R}^{3}$ denotes the angular velocity that
rotates quaternion $\mathbf{q}_{t}$ into $\mathbf{q}_{t+1}$ within time
$\Delta_{t}$.

Inversely, the exponential map $\exp:\mathbb{R}^{3}\mapsto\mathcal{S}^{3}$ is

$\exp(\mathbf{r})=\begin{cases}\cos(||\mathbf{r}||)+\sin(||\mathbf{r}||)\frac{\mathbf{r}}{||\mathbf{r}||},&||\mathbf{r}||\neq
0\\\ 1+[0,0,0]^{\top},&otherwise\end{cases}$ (7)

With the inverse of (6), the new orientation $\mathbf{q}_{t+1}$ can also be
derived with the known orientation $\mathbf{q}_{t}$ and angular velocity
$\boldsymbol{\omega}$ by

$\mathbf{q}_{t+1}=\exp(\frac{\boldsymbol{\omega}\Delta_{t}}{2})\mathbf{q}_{t}$
(8)

The orientation of admittance controller in (1) is calculated by solving for
angular acceleration $\dot{\mathbf{\omega}}_{e,t}$, integrating to angular
velocity $\mathbf{\omega}_{e,t}$, then using (8) to update the orientation
$\mathbf{q}_{e,t}$ to $\mathbf{q}_{e,t+1}$.

Finally, the commanded pose $\mathbf{x}_{c}$ could be specified by the
multiplication of desired pose and error pose, which is:

$\mathbf{X}_{c}=\mathbf{X}_{d}\mathbf{X}_{e}$ (9)

where $\mathbf{X}_{[\cdot]}$ is the $4\times 4$ transformation matrix of
corresponding $\mathbf{x}_{[\cdot]}$. The order is switched because
$\mathbf{X}_{e}$ is in TCP coordinates, and $\mathbf{X}_{d}$ is in base
coordinates.

###  3.2 Admittance Gains

The impedance characteristics in (1) are determined by the parameters
$\mathbf{M}$, $\mathbf{D}$ and $\mathbf{K}$ with the equation. This can be
rewritten as

$\mathbf{D}=2\zeta\sqrt{\mathbf{MK}}$ (10)

where $\zeta$ is a damping parameter, $\zeta>1$ is overdamping and $\zeta<1$
underdamping with more oscillation.

Here, the $M^{p}$ (mass) are $5$ kg for the linear components and the
orientation components $M^{o}$ (inertia) are $0.02$ kgm2. The value of
$\mathbf{K}$ is assigned by the RL policy, and is thus allowed to be updated
online. The critical damping is chosen by (10) with $\zeta=1$.

To maintain contact stability, limits on linear stiffness $K^{p}\in[20,2000]$
and orientation stiffness $K^{o}\in[1,40]$ were used. A maximum change in the
admittance controller of $|K^{p}_{t+1}-K^{p}_{t}|<40$ and
$|K^{o}_{t+1}-K^{o}_{t}|<1$ were enforced.

##  4 Dynamic Movement Primitives with Force

This section explains the construction and evaluation of the DMPs, including
both rotation and force information.

###  4.1 Position Dynamic Movement Primitives

A DMP is a representation for complex motor actions, motivated by differential
equations of well-defined attractor dynamics [22]. It is defined by point
attractor dynamics

$\tau^{2}\ddot{x}=\alpha_{x}(\beta_{x}(g-x)-\tau\dot{x})+f,$ (11)

where $x$ is the position of the system, $g$ is the goal position, $\dot{x}$
and $\ddot{x}$ are velocity and acceleration, $\alpha_{x}$ and $\beta_{x}$ are
gain terms that determine the damping and spring behavior of the system, $f$
is a forcing function, and $\tau$ adjusts the temporal behavior of the system.

The forcing function $f$ that models the nonlinear behavior is learned as a
function of phase variable $z$. It can be formulated using a linear
combination of $N$ nonlinear Radial Basis Functions (RBFs) as

$f(z)=(g-x_{0})\sum_{i=1}^{N}\psi_{i}(z)w_{i}z,$ (12)

where $x_{0}$ is the initial pose of the robot, and the term $(g-x_{0})$ is to
maintain the shape of the trajectory when the goal $g$ is changed, and

$\psi_{i}(z)=\frac{\exp{(-h_{i}(z-c_{i})^{2})}}{\sum_{j=1}^{N}\exp{(-h_{j}(z-c_{j})^{2})}},$
(13)

where $h_{i}$ and $c_{i}$ are constants that determine the width and centers
of the basis functions, respectively, and are manual parameters. Setting
$c_{i}=z_{0}\exp{(-\frac{\alpha_{z}}{\tau}\frac{i\cdot T}{N})}$ makes an even
distribution of RBFs over time, and the basis function bandwidth used here is
$h_{i}=\frac{N^{1.5}}{c_{i}\cdot\alpha_{z}}$, found by trial and error.

The phase variable $z$ evolves as

$z=z_{0}\exp{(-\frac{\alpha_{z}}{\tau}t)},$ (14)

where $\alpha_{z}$ is a constant controlling the speed, $t$ is the time step
in $\\{0,1,2,\dots,T\\}$, and $z_{0}$ is the initial value of $z$.

The target value of the forcing function is calculated from the demonstration
as

$f_{tar}(t)=\tau^{2}\ddot{x}_{demo}(t)\texttt{-}\alpha_{x}(\beta_{x}(g\texttt{-}x_{demo}(t))\texttt{-}\tau\dot{x}_{demo}(t)).$
(15)

Thus, in (12), the weights $\mathbf{w}=[w_{1},w_{2},w_{3},\dots,w_{N}]^{T}$
over basis functions could be learned to make the forcing term match
$f_{tar}(t)$ by a regression algorithm, such as locally weighted regression
(LWR) [23], giving a fit weight vector $\mathbf{w}$ of

$\mathbf{w}=(\mathbf{\Psi}^{\top}\mathbf{\Psi})^{-1}\mathbf{\Psi}^{\top}\mathbf{F}.$
(16)

For the attractor dynamics, $\mathbf{\Psi}=[\psi_{1},\dots,\psi_{T}]$ and
$\mathbf{F}$ is given by

$\mathbf{F}=[\frac{f_{tar}(0)}{(g-x_{0})z(0)},\dots,\frac{f_{tar}(T)}{(g-x_{0})z(T)}]^{\top}.$
(17)

###  4.2 Orientation with Quaternion Representation

The algorithms mentioned above are designed for the single DOF motion. To
handle rotations, the quaternion form of DMPs is used [24]. Thus, the
expression of (11) in quaternion form is given by

$\tau^{2}\dot{\boldsymbol{\omega}}=\alpha_{x}(\beta_{x}2\log(\mathbf{g}^{o}*\overline{\mathbf{q}})-\boldsymbol{\omega})+f^{o}(z),$
(18)

where $\boldsymbol{\omega}$ is the angular velocity and $\textbf{g}^{o}$ is
the quaternion representation of goal orientation. The corresponding forcing
function also has a similar form as (12)

$f^{o}(z)=\mathrm{diag}(\log(\mathbf{g}^{o}*\overline{\mathbf{q}}))\sum_{i=1}^{N}\psi_{i}(z)w_{i}^{o}z,$
(19)

with orientation weights $w_{i}^{o}$, which are fit in the same way as (15)
and (16).

###  4.3 Representation of Force-Torque Information

To integrate the force-torque demonstration information, linear combinations
of RBFs are utilized to regress the forces over Cartesian space as [6]

$F_{d}(z)=\sum_{i=1}^{N}\psi_{i}(z)w_{i}^{F}$ (20)

where the $z$ and $\psi_{i}(z)$ are calculated by the (14) and (13), and
$w_{i}^{F}$ are weights for the force/torque regression, calculated as in
(16).

###  4.4 DMP Commissioning and Execution

To commission, i.e. to find weights
$\mathbf{w}^{p},\mathbf{w}^{o},\mathbf{w}^{F}$, first, the phase variable
$z_{t}$ is calculated with (14), the RBFs $\psi$ of the forcing function $f$
are calculated using (13), velocity and acceleration calculated by the
numerical derivative and (6), the forcing functions are calculated with (15),
then the weight vector $\mathbf{w}$ is found by (16). The parameters used here
are $\alpha_{z}=1$, $\tau=25$, $\alpha_{y}=6.25$.

To execute, the phase variable $z$ is calculated by (14), the forcing
functions are calculated (12) and (19), then the position and orientation
trajectory by numerical integration on (11) and (18). The desired force is
calculated by (20).

##  5 Reinforcement Learning

The soft actor-critic (SAC) algorithm optimizes a stochastic policy in an off-
policy way [25]. The significant characteristic of SAC is determining
exploration noise, how much the actions deviate from the currently optimal
action, by entropy regularization. At a high level, SAC empirically finds an
approximately optimal policy $\pi^{*}:S\times A\rightarrow[0,1]$, a
distribution over state $\mathbf{s}\in S$ and action $\mathbf{a}\in A$, as

$\pi^{*}=\arg\max_{\pi}\sum_{t=0}^{T}\mathbb{E}_{\mathbf{s}_{t},\mathbf{a}_{t}}r(\mathbf{s}_{t},\mathbf{a}_{t})+\alpha\mathcal{H}(\pi(\cdot|\mathbf{s}_{t})),$
(21)

where
$\mathcal{H}(\pi(\cdot|\mathbf{s}_{t}))=\int\pi(\mathbf{a}_{t}|\mathbf{s}_{t})\log(\pi(\mathbf{a}_{t}|\mathbf{s}_{t})d\mathbf{a}_{t}$
is a measure for how random the policy $\pi$ is, and coefficient $\alpha>0$
increase the exploration noise.

SAC is here used to adapt the impedance according to the current force and
position. The state and actions for the agent are

$\displaystyle\mathbf{s}$
$\displaystyle=[\mathbf{x}^{p},\mathbf{q},\mathbf{F}_{ext}]$ (22)
$\displaystyle\mathbf{a}$ $\displaystyle=[\mathbf{K}^{p},\mathbf{K}^{o}],$
(23)

where $\mathbf{x}^{p}\in\mathbb{R}^{3}$ is linear position,
$\mathbf{q}\in\mathcal{S}^{3}$ is the orientation quaternion,
$\mathbf{F}_{ext}\in\mathbb{R}^{6}$ is the force/torque measurement, and
$\mathbf{K}^{p},\mathbf{K}^{o}\in\mathbb{R}^{3}$ are the linear and
orientation stiffness parameters, where
$\mathbf{K}=\mathrm{diag}([\mathbf{K}^{p},\mathbf{K}^{o}])$.

The SAC policy and critic networks are three linear fully-connected layers,
with 256 nodes per layer and ReLU activation functions. An entropy term of
$\alpha=0.05$ is used with learning rate scheduled from $3.3e-3$ to $3e-4$.
Other parameters are standard from [26].

Inspired by [27], the reward function is designed as

$\begin{split}r(\mathbf{s},\mathbf{a})=&\,\omega_{goal}\kappa(\mathbf{s})+\omega_{p}L(||\mathbf{x}^{p}-\mathbf{x}^{p}_{d}||/e_{term,p})\\\
&+\omega_{q}L(e^{o}/e_{term,q})+\omega_{F}L(||\mathbf{F}^{p}_{ext}-\mathbf{F}_{d}^{p}||/e_{term,F})\\\
&+\omega_{M}L(||\mathbf{F}^{o}_{ext}-\mathbf{F}^{o}_{d}||/e_{term,M})\end{split}$
(24)

where $F^{p}_{[\cdot]}$ are linear forces, $F^{o}_{[\cdot]}$ are the torques,
$e_{term,\cdot}$ is the defined allowed maximal difference,
$L(x)=1-x/\sqrt{3}$ to map to the range $0$ to $1$, $\omega_{[\cdot]}$ are
weighting hyperparameters, and $e^{o}$ is the orientation error in logarithm
form as

$e^{o}=\begin{cases}2\pi,&\mathbf{q}*\overline{\mathbf{q}}_{d}=-1+[0,0,0]^{\top}\\\
2||\log(\mathbf{q}*\overline{\mathbf{q}}_{d})||_{2},&otherwise\end{cases}$
(25)

because the logarithmic map has no discontinuity boundary except for a
singularity at $\mathbf{q}*\overline{\mathbf{q}}_{d}=-1+[0,0,0]^{\top}$.

Weighting hyperparameters of $\omega_{goal}=1$ and
$\omega_{p}=\omega_{q}=\omega_{F}=\omega_{M}=0.5$ are used here. The
terminating distance $e_{term,p}=0.01$, $e_{term,q}=0.1$, $e_{term,F}=3.0$ and
$e_{term,M}=1.0$. The first term $\kappa(\mathbf{s})$ is the completion term,
which is defined as

$\kappa(\mathbf{s})=\begin{cases}100,&task\ finished\\\ -50,&terminated\\\
-100,&error\\\ 0,&otherwise\end{cases}$ (26)

The purpose of this term is to encourage the agent to complete the task. If
the difference of any element in full state
$[\mathbf{x}^{p},\mathbf{q},\mathbf{F}_{ext}]$ exceeds the corresponding
$e_{term,\cdot}$, the movement would be terminated and the status would be
labeled as $terminated$; if an error such as joint discontinuity occurs and
robot is stopped during moving, the status is labeled as $error$; if the
movement is eventually completed, the status is then $finished$; otherwise,
there is no external reward added.

##  6 Experimental Validation

This section presents the implementation and evaluation on two contact-rich
tasks.

###  6.1 Implementation

####  6.1.1 Hardware

For the robot, Franka Emika Panda robot has been used with the Cartesian pose
interface at 1kHz. An external force-torque sensor SCHUNK Axia80 has been
mounted on the flange of the robot arm and connected to the higher-level
computer with an Ethernet connection. Then two different 3d printed end
effectors are mounted on the other side of the sensor according to the
experiments.

####  6.1.2 Software

To transfer data between sensor, robot and computer, the middleware software
Links and Nodes (LN, developed by DLR) is used. LN is similar to ROS, and
provides two communication paradigms, publishing and service calls. For
controlling the Franka robot, the Franka Control Interface (FCI) with C++ APIs
has been used.

To store the raw and modified data, Sqlite is used as the SQL database engine.
The reason is that it is flexible and provides Python and C++ APIs so that it
is possible to operate the data in both programming languages.

We used Gym [28] as the framework of RL with a custom environment and Stable
Baseline 3 [26] as the implementation of RL algorithm SAC.

###  6.2 Peg in hole task

We used insertion task seen in Figure 1 to test and evaluate the performance
of the full state representation of position and force trajectories. The
diameters of the peg and the hole are 20mm and 20.5mm, respectively, and
$K^{p}=600$, $K^{o}=13$ for the tests.

####  6.2.1 DMP in Cartesian space

Firstly, the DMP algorithm in Cartesian free space has been tested. After a
demonstration without force-torque involved, the raw demonstrated trajectory
has been recorded. Then the DMP modified trajectories have been generated
according to different number of basis functions. To better visualize the
results, we represent the translation of each trajectory in the 3d view as
follows in Figure 4.

Figure 4: Translational trajectories comparison in 3d view, axes units are
meters

As shown in Figure 4, the more basis functions there are, the higher the
similarity will be. In this case the trajectory with 10,000 BFs shows barely
any difference compared to raw data. If the number of BFs is relatively
smaller such as 1,000, the corresponding trajectory becomes smoother and
maintains the characteristics of the original trajectory to a certain extent.
With a much smaller number of BFs (100), the trajectory is the smoothest but
is significantly different from the raw trajectory.

####  6.2.2 Force-Torque Representation

We investigate if the LWR force regression is suited to hard contact. To test
this, we then demonstrate a trajectory with contact, and compare LWR generated
force-torque profiles with a range of number of basis functions, seen in
Figure 5.

The force curve changes quickly due to the environment being stiff, where
typical overshoots from contact transitions can be seen in the raw data
(blue). The 10,000 BFs most closely matches the raw data, but can cause errors
when used on the robot (joint acceleration discontinuity error). The 100 BF
shows the highest smoothness but lacks accuracy.

Figure 5: Force-Torque Profiles in X-axis, Raw Data and DMP Data with
Different Number of Basis Functions. The x-axis is milliseconds, and the
y-axis is $N$.

####  6.2.3 Force and Position Regression

To test the force-torque part explicitly, we pushed the end-effector down a
bit more and then move back after inserting the hole to get a proper value of
force, and performed insertion tests using 100, 1,000 and 10,000 BFs. 100
trials have been proceeded for each setting to obtain the success rate.
Success is defined by a final z error of $<5$ mm (hole depth is $10$ mm). The
results can be seen in Table 1.

Table 1: Success rate in different experimental conditions Peg-in-Hole | Tape, original | Tape, with 1 mm shift
---|---|---
# BFs | Succ. | Stiff. ($K^{p}$, $K^{o}$) | Succ. | Stiff. ($K^{p}$, $K^{o}$) | Succ.
100 | 92% | low(50, 0.5) | 0% | low(50, 0.5) | 0%
1,000 | 79% | middle(605, 13) | 72% | middle(605, 13) | 61%
10,000 | 50% | high(2000, 40) | 96% | high(2000, 40) | 73%
- | - | SAC | 97% | SAC | 93%
Figure 6: Learning Curve of SAC, with average $\sum r$ on the y-axis and
training steps on x-axis.

From Table 1, we see that the approach can achieve a good success rate, but
modifying the number of BFs affects the success rate. More BFs can cause
stability issues during contact due to more frequent changes in force (as seen
in Figure 5). The pose tracking is not as sensitive to the number of BFs as
the pose varies more slowly over time.

###  6.3 Adhesive strip application task

A task of pressing a tape into a press fit along a long, curved channel, as
shown in Figure 1, using the tape end effector in the inset of Figure 1 which
has a smooth steel ball at the end. To allow the experiment to reset
automatically, both ends of the tape are fixed to the fixture.

####  6.3.1 Evaluation of SAC Algorithm

First, the raw demonstration data were converted to the DMP with 500 BFs. The
SAC is applied with parameters seen in Section 5, and the learning curve can
be seen in Figure 6, where $40k$ training steps takes about $2$ hours,
including reset time.

Then we evaluated the trained RL model compared with the trajectories with
different fixed stiffness and get the success rate by 100 trials, with results
shown in the middle column of Table 1. The fixed stiffness values are given
for $K^{p}$ and $K^{o}$.

The success rate depends strongly on stiffness, where generally the success
rate increases with stiffness as this prevents the end-effector from slipping
off the tape. For the SAC agent, stiffness varies over time as can be seen in
Figure 7(a), which shows how the stiffness in the $y$ direction varies during
the contact-rich task. The $y$ direction is along the tape.

We can analyze the difference between actual force and desired force in y
axis, shown in Figure 7(b). From this figure, the most risky time period for
Fy in this task is around 10s to 11s, because $F_{d}$ increases rapidly in
this time, sometimes causing a discontinuity error. During this time period,
the stiffness increases, and the force trajectory is followed less, so the
risk of discontinuity error is reduced.

The stiffness in the three Cartesian directions are plotted in Figure 7(c),
where the $x$ is across the tape, $y$ along the tape, and $z$ into the tape.
During the main contact motion of the task, $K^{p}$ is higher in $x$, keeping
from slipping off the tape. $K^{p}$ is lower in $z$, which is the contact
direction, doing more force control. Near 10 seconds, the gains are affected
by the discontinuity error and breaking contact with the tape.

####  6.3.2 Evaluation of Robustness

The ability to tolerate small position errors in assembly is a key feature. To
test this, we manually added an offset of 1 mm to the base coordinate system
$y+$ axis in the controller, making the end-effector closer to the fixture.
The obtained results are shown in Table 1.

(a) $K^{p}$ in y

(b) $F_{ext}-F_{d}$ in y

(c) $K^{p}$ in x, y, and z

Figure 7: Trained SAC agent on tape task, where (a) shows $K^{p}$ on original
and offset, (b) shows the force error $F_{ext}-F_{d}$ on original and offset,
and (c) shows the Cartesian $K^{p}$ on original.

After adding the offset, on the one hand, the success rate of all experimental
groups has decreased because of the bigger contact force and higher
probability of impacting the fixture; on the other hand, the success rate with
SAC model has not decreased very much compared with other groups.

To explain the generalization of the model more clearly, we plotted
$\mathbf{F}_{ext}-\mathbf{F}_{d}$ and stiffness in the $y$ direction in the
offset scenario and compared it with the original scenario in Figures 7(a) and
7(b). These plots are manually time aligned, and show that a similar profile
is reached even though the timing was originally offset. Slightly larger
forces near 10-12 seconds in the offset task cause higher stiffness to ensure
successful completion of the task.

##  7 CONCLUSION

In this work, we proved that the proposed framework with the combination of
admittance controller, DMPs and RL is feasible as an efficient way to teach
contact-rich tasks. On the one hand, the DMPs could smooth the demonstrated
position and force trajectories, meanwhile provide an up-sampling function to
generate the trajectories from lower to higher frequency. On the other hand,
the RL policy could balance the ratio of position and force control by
adapting the impedance characteristics. Last but not least, the modified
admittance controller implements the motion of the robot using the data
provided by the other parts. The framework exhibits a reliable robustness to
tiny offset of position, making a wider range of applications possible.

However, the current framework has also potential to be improved further. The
proper number of BFs depends highly on the length of demonstrated trajectory
i.e. the duration of the demonstration. Thus, the autonomous adaption of
number of BFs could be considered in the future. Besides, the reward function
of RL could be improved to reach a better performance according to different
scenarios. Also the hyperparameters of the RL and DMPs could be further
researched to make the training faster and better.

## References

  * [1] H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 3, pp. 297–330, 2020.
  * [2] N. Hogan, “Impedance control: An approach to manipulation: Part i—theory,” 1985\.
  * [3] A. Bicchi and G. Tonietti, “Fast and ”Soft-Arm” Tactics,” _IEEE Robotics & Automation Magazine_, vol. 11, no. 2, pp. 22–33, Jun. 2004.
  * [4] A. J. Ijspeert, J. Nakanishi, and S. Schaal, “Movement imitation with nonlinear dynamical systems in humanoid robots,” in _Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292)_ , vol. 2. IEEE, 2002, pp. 1398–1403.
  * [5] B. Nemec, F. J. Abu-Dakka, B. Ridge, A. Ude, J. A. Jørgensen, T. R. Savarimuthu, J. Jouffroy, H. G. Petersen, and N. Krüger, “Transfer of assembly operations to new workpiece poses by adaptation to the desired force profile,” in _2013 16th International Conference on Advanced Robotics (ICAR)_ , Nov. 2013, pp. 1–7.
  * [6] F. J. Abu-Dakka, B. Nemec, J. A. Jørgensen, T. R. Savarimuthu, N. Krüger, and A. Ude, “Adaptation of manipulation skills in physical contact with the environment to reference force profiles,” _Autonomous Robots_ , vol. 39, no. 2, pp. 199–217, Aug. 2015.
  * [7] J. Kober, M. Gienger, and J. J. Steil, “Learning movement primitives for force interaction tasks,” in _Robotics and Automation (ICRA), 2015 IEEE International Conference On_. IEEE, 2015, pp. 3192–3199.
  * [8] F. J. Abu-Dakka and M. Saveriano, “Variable impedance control and learning – A review,” _arXiv:2010.06246 [cs]_ , Oct. 2020.
  * [9] Y. Li, G. Ganesh, N. Jarrassé, S. Haddadin, A. Albu-Schaeffer, and E. Burdet, “Force, Impedance, and Trajectory Learning for Contact Tooling and Haptic Identification,” _IEEE Transactions on Robotics_ , vol. 34, no. 5, pp. 1170–1182, 2018.
  * [10] C. C. Beltran-Hernandez, D. Petit, I. G. Ramirez-Alpizar, and K. Harada, “Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach,” _Applied Sciences_ , vol. 10, no. 19, p. 6923, Oct. 2020.
  * [11] L. Roveda, N. Pedrocchi, M. Beschi, and L. M. Tosatti, “High-accuracy robotized industrial assembly task control schema with force overshoots avoidance,” _Control Engineering Practice_ , vol. 71, pp. 142–153, 2018\.
  * [12] F. Dimeas and N. Aspragathos, “Reinforcement learning of variable admittance control for human-robot co-manipulation,” in _Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference On_. IEEE, 2015, pp. 1011–1016.
  * [13] T. Tang, H.-C. Lin, and M. Tomizuka, “A learning-based framework for robot peg-hole-insertion,” in _Dynamic Systems and Control Conference_ , vol. 57250. American Society of Mechanical Engineers, 2015, p. V002T27A002.
  * [14] F. J. Abu-Dakka, L. Rozo, and D. G. Caldwell, “Force-based variable impedance learning for robotic manipulation,” _Robotics and Autonomous Systems_ , vol. 109, pp. 156–167, Nov. 2018.
  * [15] L. Peternel, T. Petrič, and J. Babič, “Robotic assembly solution by human-in-the-loop teaching method based on real-time stiffness modulation,” _Autonomous Robots_ , vol. 42, no. 1, pp. 1–17, 2018.
  * [16] K. Haninger, C. Hegeler, and L. Peternel, “Model Predictive Control with Gaussian Processes for Flexible Multi-Modal Physical Human Robot Interaction,” _arXiv preprint arXiv:2110.12433_ , 2021.
  * [17] S. Calinon, I. Sardellitti, and D. G. Caldwell, “Learning-based control strategy for safe human-robot interaction exploiting task and robot redundancies,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 249–254.
  * [18] J. Buchli, F. Stulp, E. Theodorou, and S. Schaal, “Learning variable impedance control,” _The International Journal of Robotics Research_ , vol. 30, no. 7, pp. 820–833, Jun. 2011.
  * [19] F. Stulp, J. Buchli, A. Ellmer, M. Mistry, E. A. Theodorou, and S. Schaal, “Model-free reinforcement learning of impedance control in stochastic environments,” _IEEE Transactions on Autonomous Mental Development_ , vol. 4, no. 4, pp. 330–341, 2012.
  * [20] M. Hazara and V. Kyrki, “Reinforcement learning for improving imitated in-contact skills,” in _Humanoid Robots (Humanoids), 2016 IEEE-RAS 16th International Conference On_. IEEE, 2016, pp. 194–201.
  * [21] T. Davchev, K. S. Luck, M. Burke, F. Meier, S. Schaal, and S. Ramamoorthy, “Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4488–4495, Apr. 2022.
  * [22] S. Schaal, “Dynamic movement primitives-a framework for motor control in humans and humanoid robotics,” in _Adaptive motion of animals and machines_. Springer, 2006, pp. 261–280.
  * [23] W. S. Cleveland and S. J. Devlin, “Locally weighted regression: an approach to regression analysis by local fitting,” _Journal of the American statistical association_ , vol. 83, no. 403, pp. 596–610, 1988.
  * [24] A. Ude, B. Nemec, T. Petrić, and J. Morimoto, “Orientation in cartesian space dynamic movement primitives,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2014, pp. 2997–3004.
  * [25] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” _arXiv preprint arXiv:1801.01290_ , 2018.
  * [26] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning implementations,” _Journal of Machine Learning Research_ , 2021.
  * [27] C. C. Beltran-Hernandez, D. Petit, I. G. Ramirez-Alpizar, and K. Harada, “Variable compliance control for robotic peg-in-hole assembly: A deep-reinforcement-learning approach,” _Applied Sciences_ , vol. 10, no. 19, p. 6923, 2020.
  * [28] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” _arXiv preprint arXiv:1606.01540_ , 2016.