# A Reinforcement Learning Approach for Robotic Unloading from Visual
Observations

Vittorio Giammarino*1, Alberto Giammarino2 and Matthew Pearce3 *Part of this
work was done while Giammarino was an intern in the AI team at Pickle Robot
Co., 1280 Cambridge St., Cambridge, MA 02139,USA1Vittorio Giammarino is with
the Division of Systems Engineering, Boston University, Boston, MA 02446, USA,
<EMAIL_ADDRESS>Giammarino is an independent researcher
<EMAIL_ADDRESS>Pearce is in the AI team at Pickle
Robot Co., 1280 Cambridge St., Cambridge, MA 02139, USA,
<EMAIL_ADDRESS>
###### Abstract

In this work, we focus on a robotic unloading problem from visual
observations, where robots are required to autonomously unload stacks of
parcels using RGB-D images as their primary input source. While supervised and
imitation learning have accomplished good results in these types of tasks,
they heavily rely on labeled data, which are challenging to obtain in
realistic scenarios. Our study aims to develop a _sample efficient_ controller
framework that can learn unloading tasks _without the need for labeled data_
during the learning process. To tackle this challenge, we propose a
hierarchical controller structure that combines a high-level decision-making
module with classical motion control. The high-level module is trained using
Deep Reinforcement Learning (DRL), wherein we incorporate a safety bias
mechanism and design a reward function tailored to this task. Our experiments
demonstrate that both these elements play a crucial role in achieving improved
learning performance. Furthermore, to ensure reproducibility and establish a
benchmark for future research, we provide free access to our code and
simulation.

##  I Introduction

Robotic unloading is generally defined as the set of tasks in which robots are
deployed to unload items from containers, trucks, or other transportation
vehicles. The successful progress of this technology represents a compelling
opportunity for the future, as it can address various challenges encountered
in logistics, manufacturing, and warehousing. Within these industries,
unloading tasks involve physically demanding and repetitive actions that can
pose risks for human workers. In this regard, robotic unloading offers a way
to enhance workers’ safety and mitigate hazards associated with heavy lifting
and challenging work environments.

(a) Visual Observation.

(b) Environment.

Figure 1: Overview of our robotic unloading setup. Fig. 1(a) illustrates a
snapshot of the visual observations available to the robot for the decision-
making process. These observations are captured by a camera positioned on the
top-left side of the robot and consist in $720\times 1280$ high resolution
RGB-D images. Fig. 1(b) presents an overview of the unloading task. Stacks of
parcels are positioned in front of an industrial KUKA KR70 robotic manipulator
equipped with a suction gripper. Using the visual information depicted in Fig.
1(a), the robot has to select a parcel to pick, grasp it, and then place it on
the ground on its right side. The parcels in the scene are randomized in terms
of color within the brown spectrum. A video demonstrating the task is
available in the Supplementary Materials.

In this paper, we investigate robotic unloading tasks from visual observations
(see Fig. 1 for an overview of the environment). Specifically, our objective
is to enable a robotic manipulator to autonomously unload stacks of parcels by
using RGB-D images as primary input source. We formulate this problem as a
three-dimensional pick-and-place task, where the parcels, arranged in piles,
are picked from the stack and placed on a floor conveyor. Previous studies
have addressed pick-and-place by integrating objects’ pose estimation [1, 2]
with scripted planning and motion control [3]. While these systems demonstrate
robustness in structured environments, they are unsuitable for uncertain and
unstructured settings, which require improved generalization capabilities. In
order to address these challenges, recent years have witnessed a surge of
interest in machine learning techniques. In particular, end-to-end
Reinforcement Learning (RL) has been successfully used for pick-and-place in
[4, 5]. However, end-to-end RL faces difficulties in real-world scenarios due
to the large amount of data required to achieve acceptable performance. To
improve data efficiency, recent work has integrated RL with object-centric
assumptions such as keypoints [6, 7], embeddings [8] or dense descriptors [9,
10]. These representations are typically learned through supervised learning
[11], which often involves tedious and expensive data labeling or annotation
processes. Another line of research has explored Imitation Learning (IL), also
known as learning from demonstrations [12, 13, 14]. Despite achieving
promising results, IL remains a supervised learning approach that relies on
collecting expert demonstrations, which is akin to collecting labeled data and
can be costly, time-consuming, or even infeasible. As a result, the main goal
of this paper is to develop a _sample efficient_ controller framework that can
learn the robotic unloading task in Fig. 1, _without requiring any form of
labeled data_.

Towards addressing this problem, we propose a hierarchical controller
structure that separates the robot decision-making process from the low-level
module. The decision-making component, named high-level controller, is trained
using DRL [15, 16] and more specifically Deep Q-Learning (DQL) [17] from RGB
images. Meanwhile, the low-level module relies on classical trajectory
planning and motion control techniques. Within this framework, our work makes
two main contributions.

From an algorithmic perspective, our main novelties lie in the high-level
controller, aiming to improve the sample efficiency of our DQL pipeline.
First, we equip our Deep Q-Networks with a _safety bias mechanism_ with the
goal of biasing the decision policy towards safe end-effector configurations.
Note that an end-effector configuration is considered safe if it is reachable,
i.e., it is contained within the robot workspace. Additionally, we propose a
task-specific reward function _which takes into account the verticality_ of
our unloading task. In order to test the impact of these mechanisms on
learning performance, we conduct an ablation study and show how both these
elements are crucial to accomplish improved results in our task.

Our second contribution involves the development of a simulated environment
using the PyBullet physics simulator [18]. Simulators play a crucial role in
prototyping RL algorithms as they offer a cost-effective and risk-free testing
environment. This aspect is particularly valuable for industry-oriented tasks
like robotic unloading, which are challenging to replicate in a research
setting. Having a simulator available becomes essential in facilitating new
research in this domain. Therefore, we provide open access to our environment,
with the goal of establishing an interesting benchmark for future studies in
this field.

The remainder of the paper is organized as follows: Section II provides a
summary of the work related to robotic unloading. Section III introduces
notation and background on RL. Section IV provides a detailed description of
the simulation environment. Section V presents the hierarchical controller and
outlines the algorithm used to train the high-level controller. Finally,
Section VI presents our experimental results and Section VII concludes the
paper providing a general discussion on our findings.

##  II Related Work

In the following, we briefly review the most relevant work on robotic
unloading. One of the earliest studies in this field is presented in [19],
which offers an overview of the main technical challenges associated with
automatic unloading. This study proposes solutions based on a 3D laser scanner
perception system and scripted trajectories.

In recent years, there has been a significant focus on leveraging deep
learning (DL) techniques to enhance perception systems tailored to robotic
unloading tasks. Papers such as [20, 21] formulate algorithms for accurate
object detection and parcel segmentation within the context of robotic
unloading. However, these studies primarily concentrate on perception and do
not address the decision-making and control aspects of the robot.

Other studies have explored the integration of parcel segmentation with
robotic control problems. In [22] for instance, perception, planning, and
control are individually addressed and subsequently combined within a unified
framework. In particular, a RGB-D camera is attached to the robot gripper for
perception, and segmentation models are utilized to identify the safest object
in the scene. A customized motion planning framework is then employed for
control. Similarly, in [23], a 3D vision processing algorithm is introduced
for parcel segmentation, enabling on-the-fly determination of multiple
gripping strategies.

Regarding the decision-making problem, papers such as [24, 25] introduce a
reasoning framework based on decision trees for optimal unloading, which is
then combined with motion planning. However, these papers do not explicitly
address the perception problem. Similarly, in [26], a Q-learning-based
algorithm is proposed for optimal decision-making, assuming accurate
detection, perfect stacking, and uniform-sized boxes.

Compared to these studies, our research takes a comprehensive approach to the
unloading problem. We explore controllers that integrate perception, decision-
making, and motion control as an interconnected system and treat this
interconnection as a unified problem.

##  III Preliminaries

Unless indicated otherwise, we use uppercase letters (e.g., $S_{t}$) for
random variables, lowercase letters (e.g., $s_{t}$) for values of random
variables, script letters (e.g., $\mathcal{S}$) for sets and reference frames,
bold lowercase letters (e.g., $\bm{\theta}$) for vectors and bold uppercase
letters (e.g. $\bm{H}$) for matrices. We denote expectation as
$\mathbb{E}[\cdot]$.

Our decision-making process is modeled as a finite-horizon discounted Markov
Decision Process (MDP) described by the tuple
$(\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R},\rho_{0},\gamma)$, where
$\mathcal{S}$ is the set of states and $\mathcal{A}$ is the set of actions.
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow P(\mathcal{S})$ is the
transition probability function where $P(\mathcal{S})$ denotes the space of
probability distributions over $\mathcal{S}$,
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward
function which maps state-action pairs to scalar rewards, $\rho_{0}\in
P(\mathcal{S})$ is the initial state distribution, and $\gamma\in[0,1)$ the
discount factor. We define the agent as a stationary policy
$\pi:\mathcal{S}\rightarrow P(\mathcal{A})$, where $\pi(a|s)$ is the
probability of taking action $a$ in state $s$. When a function is
parameterized with parameters $\bm{\theta}\in\varTheta\subset\mathbb{R}^{k}$
we write $\pi_{\bm{\theta}}$.

### Reinforcement learning

Given an MDP and a stationary policy $\pi:\mathcal{S}\to P(\mathcal{A})$, the
RL objective is to learn an optimal policy, $\pi^{*}$, which maximizes the
expected total discounted reward

$\displaystyle\begin{split}J(\pi)=\mathbb{E}_{\tau}\Big{[}\sum_{t=0}^{T}\gamma^{t}\mathcal{R}(s_{t},a_{t})\Big{]},\end{split}$
(1)

where $\tau=(s_{0},a_{0},s_{1},a_{1},\dots,a_{T-1},s_{T})$ are trajectories
sampled according to $s_{0}\sim\rho_{0}$, $a_{t}\sim\pi(\cdot|s_{t})$ and
$s_{t+1}\sim\mathcal{T}(\cdot|s_{t},a_{t})$, and $T$ is the number of decision
steps in a single episode. Note that, neither the transition probability
function $\mathcal{T}$ nor the reward function $\mathcal{R}$ are available to
the learning agent. Therefore, the agent interacts with the environment,
collects transitions $(s_{t},a_{t},r_{t},s_{t+1})$, where
$r_{t}=\mathcal{R}(s_{t},a_{t})$, and exploits these transitions to estimate
$J(\pi)$ in (1) in order to learn $\pi^{*}$ as
$\pi^{*}\in\arg\max_{\pi}J(\pi)$.

Well-known algorithms solve the RL problem by estimating, and optimizing, the
value functions induced by the policy $\pi$. We define the state value
function as
$V^{\pi}(s)=\mathbb{E}_{\tau}[\sum_{t=0}^{T}\gamma^{t}\mathcal{R}(s_{t},a_{t})|S_{0}=s]$
and the state-action value function as
$Q^{\pi}(s,a)=\mathbb{E}_{\tau}[\sum_{t=0}^{T}\gamma^{t}\mathcal{R}(s_{t},a_{t})|S_{0}=s,A_{0}=a]$.
Furthermore, $V^{\pi}(s)=\mathbb{E}_{a\sim\pi(\cdot|s)}[Q^{\pi}(s,a)]$. Note
that, the optimal policy $\pi^{*}\in\arg\max_{\pi}J(\pi)$ induces the optimal
state value function

$V^{*}(s)=\max_{\pi}V^{\pi}(s),\ \ \ \forall s\in\mathcal{S}.$

Therefore, given $V^{*}(s)$ and assuming $\mathcal{T}$ is known, we can
retrieve $\pi^{*}(s)$ for all $s\in\mathcal{S}$ as the action
$a\in\mathcal{A}$ that leads to the highest expected return
$\mathbb{E}_{s_{t+1}\sim\mathcal{T}(\cdot|s_{t},a)}[V^{*}(s_{t+1})]$ for all
$s_{t}\in\mathcal{S}$.

In the RL setting, where $\mathcal{T}$ is unknown, Temporal Difference (TD)
methods [15] compute $V^{*}(s)$ and $\pi^{*}(s)$ by leveraging the following
iterative updating rule for $Q^{\pi}(s_{t},a_{t})$:

$\displaystyle Q^{\pi}(s_{t},a_{t})\leftarrow
Q^{\pi}(s_{t},a_{t})+\alpha(Y-Q^{\pi}(s_{t},a_{t})),$ (2)

where $\alpha$ is a learning rate and $Y$ is a target value as in a standard
regression problem.

Given a set of transitions $(s_{t},a_{t},r_{t},s_{t+1},a_{t+1})$, generated by
the interactions of the policy $\pi$ with the environment; setting
$Y=r_{t}+\gamma Q^{\pi}(s_{t+1},a_{t+1})$ yields the on-policy update typical
of SARSA [15]. On the other hand, setting
$Y=r_{t}+\gamma\max_{a}Q^{\pi}(s_{t+1},a)$ yields the off-policy update used
in $Q$-learning [15]. Provided the optimal state-action value function
$Q^{*}(s,a)$, then $V^{*}(s)=\max_{a}Q^{*}(s,a)$ and
$\pi^{*}(s)\in\arg\max_{a}Q^{*}(s,a)$ for all $s\in\mathcal{S}$.

##  IV Simulated Environment

Figure 2: The evolution of the scene is illustrated in two different episodes.
In the upper row episode, the agent follows an optimal decision policy,
successfully unloading all the parcels. In the lower row episode, the policy
is suboptimal and a single wrong decision impacts the subsequent organization
of the scene, affecting the agent’s final outcome and undermining the entire
unloading process.

In the following we provide a detailed description of our robotic unloading
task. The environment is summarized in Fig. 1 and is based on the PyBullet
physics simulator [18]. The simulated environment can be accessed at our
GitHub repository111https://github.com/VittorioGiammarino/RL-for-unloading-
from-pixels.

### Agent

The decision agent controls an industrial $6$-axis KUKA KR70 robotic
manipulator. The robot inertial frame corresponds to the world frame and is
denoted with $\mathcal{I}$. We refer to the end-effector pose with the
homogeneous transformation $\bm{H}_{\mathcal{I}\mathcal{E}}$, where
$\mathcal{E}$ is the robot end-effector frame. The robot end-effector is a
suction gripper with a contact sensor which enables the contact detection
between the gripper and the other objects of the scene. The unloading task
relies on the visual observations illustrated in Figure 1(a) and 2. These
images are captured by a RGB-D camera positioned at the top-left side of the
robot, providing $720\times 1280$ high-resolution images. The intrinsic and
extrinsic parameters of the camera are known, and along with the depth image,
are used to transform from the 2D pixel space to the robot inertial frame
$\mathcal{I}$ [27]. Additionally, the agent is provided with the robot
workspace boundaries in $\mathcal{I}$, denoted with $W_{\mathcal{I}}$.

### Task

For each episode, our task involves unloading a stack of $42$ parcels,
arranged as shown in Fig. 1. Each parcel is a cube with $0.25m$ edge length
and randomized color surface within the brown spectrum. At each decision step,
the agent is required to select a parcel, grasp it, and then place it on the
ground, where a floor conveyor will complete the unloading. The complexity of
this problem, compared to more classical pick-and-place or rearrangement
tasks, arises from the sequential nature of the decision-making process and
the limited margin of error allowed by the task. In this regard, Fig. 2
illustrates two episodes generated by different decision policies. In the
upper row, a successful unloading sequence is depicted, where the agent
successfully unloads all the $42$ parcels. Conversely, the lower row shows an
unsuccessful unloading sequence, where a single decision has a negative impact
on the subsequent scene, ultimately compromising the agent’s final outcome.
Note that the simulation environment is modelled as a finite-horizon MDP with
$T=42$, i.e., the number of decision steps is equivalent to the number of
initialized parcels at $t=0$. When a mistake is made, as shown in Fig. 2
(lower), we remove the $n$ parcels that land outside the agent’s workspace
$W_{\mathcal{I}}$, and the agent incurs a time penalty of $n$, i.e.
$t\leftarrow t+n$ rather than $t\leftarrow t+1$. Similarly, when the agent
attempts picks out of $W_{\mathcal{I}}$, a parcel is removed from the top of
the stack and $t\leftarrow t+1$. In other words, the agent is forced to skip a
step. This ensures $T=42$ for all the episodes, preventing infinite loops
during the training process. Note that infinite loops, like repeatedly
choosing parcels outside of $W_{\mathcal{I}}$, tend to occur frequently in the
initial stages of the learning process and can prevent progress if not
addressed.

##  V Methods

Figure 3: Summary of our hierarchical controller. The high-level controller
selects a pixel $a=(u,v)$ from an observation and computes a picking pose
denoted with $\bm{H}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)$. This pose is
passed to a planner, which generates a trajectory of Cartesian waypoints. The
low-level controller transforms this Cartesian trajectory into a joint space
trajectory solving an inverse kinematics problem. The resulting trajectory
serves as a reference for the PD controller of the actuators.

In this section, we introduce our hierarchical controller framework,
consisting of a high-level decision-making module and a low-level trajectory
planning and motion control module (cf. Fig. 3). Moreover, we provide a
comprehensive description of our DQL pipeline, used to train the agent’s
decision-making policy.

### High-level controller

Given a visual observation $s_{t}\in\mathcal{S}$, the goal of the high-level
controller is to provide a corresponding picking end-effector pose, denoted by
the homogeneous transformation
$\bm{H}^{\text{pick}}_{\mathcal{I}\mathcal{E}}$, which maximizes the expected
number of unloaded parcels over the episode. Hence, at each decision step $t$,
we want

$f(s_{t})\to\bm{H}^{\text{pick}}_{\mathcal{I}\mathcal{E}},\ \ \ t\in[0,T).$
(3)

The function $f$ in (3) is defined as a composition of two main steps. In the
first step, a policy $\pi_{\bm{\theta}}:\mathcal{S}\to P(\mathcal{A})$ takes a
$64\times 64$ RGB image of the scene and selects a single pixel
$a_{t}=(u,v)_{t}$. The $64\times 64$ image is a cropped and resized version of
the original $720\times 1280$ image in Fig. 1(a). Note that the original image
is cropped assuming prior knowledge of the area containing the stack of
parcels during the unloading process. Using an aligned depth image, the
selected $a_{t}=(u,v)_{t}$ is transformed into Cartesian coordinates,
representing the picking end-effector position in $\mathcal{I}$ denoted with
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$. The picking position
is then associated with a rotation matrix
$\bm{C}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$, corresponding to the
picking end-effector orientation. Picking position
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ and orientation
$\bm{C}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ yield the picking end-
effector pose $\bm{H}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$, where,
compared to the notation in (3), we explicitly state the dependency on
$a_{t}=(u,v)_{t}$. In this work, we assume a fixed end-effector orientation
$\bm{C}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$, which is computed to
be orthogonal to the plane of the stack front surface (the visible surface in
Fig. 1(a), 2 and 3). It is important to note that
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ and
$\bm{C}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ are interdependent
choices. In our setting, we condition
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ on
$\bm{C}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$, i.e.,
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ is learned given
$\bm{C}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$. The alternative
setting, where $\bm{C}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ is
conditioned on $\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$, is left
for future work.

### Learning algorithm

The picking policy $\pi_{\bm{\theta}}$ is learned via DQL [17]. We define $K$
critic networks as
$Q_{\bm{\theta}_{k}}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, parameterized
by a $43$-layer encoder-decoder residual network (ResNet) [28] where the same
encoder is shared across all the critic networks.

The critic networks output picking confidence values over the pixel space and
are trained to minimize:

$\displaystyle\mathcal{L}_{\bm{\theta}_{k}}(\mathcal{B})$
$\displaystyle=\mathbb{E}_{(s_{t},a_{t},r_{t},s_{t+1})\sim\mathcal{B}}[(y_{t}-Q_{\bm{\theta}_{k}}(s_{t},a_{t}))^{2}],$
(4) $\displaystyle y_{t}$
$\displaystyle=r_{t}+\gamma\max_{a}\min_{k}Q_{\bm{\bar{\theta}}_{k}}(s_{t+1},a),$
(5)

where $\mathcal{B}$ is a replay buffer (cf. [17]) and $\bm{\bar{\theta}}_{k}$
are the slow moving weights for the target critic networks [29]. We use
multiple critics and target networks to address the optimistic bias introduced
by function approximation [30, 31]. Moreover, we define two different reward
functions:

$\displaystyle\begin{split}\mathcal{R}_{b}(s,a)&=\begin{cases}1,\ \text{if
picking succeeds},\\\ 0,\ \text{otherwise},\end{cases}\\\
\mathcal{R}_{v}(s,a)&=\begin{cases}1+\lambda\cdot
z^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a),\ \text{if picking succeeds},\\\
0,\ \text{otherwise},\end{cases}\end{split}$ (6)

where $\lambda$ is a scalar hyperparameter, and
$z^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)$ is the $z$-coordinate in
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)$ where the $z$-axis
represents the up direction in $\mathcal{I}$. Both the functions in (6)
provide rewards when picking succeeds. Moreover, $\mathcal{R}_{v}(s,a)$ adds a
bonus proportional to the $z$-coordinate, i.e., the height, of the picking
position. This bonus incentivizes the unloading of parcels when in the upper
part of the stack. In other words, it disincentivizes behaviors which can lead
to the collapse of the stack as in the lower row episode in Fig. 2. We
experimentally show that $\mathcal{R}_{v}(s,a)$ remarkably improves
performance when compared to $\mathcal{R}_{b}(s,a)$ in our task.

The loss in (4)-(5) is computed by sampling interactions
$(s_{t},a_{t},r_{t},s_{t+1})$ from a replay buffer $\mathcal{B}$. During
training, the agent interacts with the environment according to the
exploration policy summarized in Algorithm 1. The safety bias layer, named
Mask in Algorithm 1, is defined as

$\displaystyle\text{Mask}(Q_{\bm{\bar{\theta}}_{k}}(s,a))=\begin{cases}Q_{\bm{\bar{\theta}}_{k}}(s,a)+b,\
\ \ \text{if $\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\in
W_{\mathcal{I}}$},\\\ Q_{\bm{\bar{\theta}}_{k}}(s,a),\ \ \
\text{otherwise},\end{cases}$ (7)

where $W_{\mathcal{I}}$ is the robot workspace and $b$ is a positive definite
safety bias treated as an hyperparameter. The main goal of Mask in (7) is to
bias the policy towards the actions with a corresponding
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\in W_{\mathcal{I}}$. We
experimentally show that this layer becomes crucial to improve performance and
efficiency in our task. During evaluation, given $s$, the actions are
generated following
$a\in\arg\max_{a}\text{Mask}(\min_{k}Q_{\bm{\bar{\theta}}_{k}}(s,a))$ where we
use Mask in (7) as additional safety measure. We provide a summary of the full
algorithm in Appendix.

Input:

$s$, $Q_{\bm{\bar{\theta}}_{k}}$, $\epsilon$, Mask, $W_{\mathcal{I}}$,
$\mathcal{U}$, $\sigma$: state ($64\times 64$ RGB image), target critic
networks, exploration probability, safety bias layer, robot workspace, uniform
distribution, and softmax function.

begin ExplorationAction[$s$]

$u\sim\mathcal{U}(0,1)$

if  _$u\leq\epsilon$_ then

$a\sim\mathcal{U}\\{a_{W_{\mathcal{I}}}\\}$ where $a_{W_{\mathcal{I}}}$
denotes the action $a$ such that
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\in W_{\mathcal{I}}$

else

$a\sim\sigma(\text{Mask}(\min_{k}Q_{\bm{\bar{\theta}}_{k}}(s,a)))$

return _$a$_

Algorithm 1 Exploration Policy

### Low-level module

The low-level module comprises a trajectory planner and a low-level
controller. The planner receives
$\bm{H}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$ from the high-level
controller and provides a set of $5$ end-effector waypoints to the low-level
controller. This set of waypoints consists of: $\bm{H}^{\text{pre-
pick}}_{\mathcal{I}\mathcal{E}}$,
$\bm{H}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$, $\bm{H}^{\text{post-
pick}}_{\mathcal{I}\mathcal{E}}$, $\bm{H}^{\text{pre-
place}}_{\mathcal{I}\mathcal{E}}$,
$\bm{H}^{\text{place}}_{\mathcal{I}\mathcal{E}}$, $\bm{H}^{\text{out-of-
camera}}_{\mathcal{I}\mathcal{E}}$. Specifically,
$\bm{H}^{\text{place}}_{\mathcal{I}\mathcal{E}}$ represents the placing pose
which is assumed to be known throughout the task. $\bm{H}^{\text{out-of-
camera}}_{\mathcal{I}\mathcal{E}}$ is the pose required for an unoccluded
image of the scene, as shown in Fig. 1(a). $\bm{H}^{\text{pre-
pick}}_{\mathcal{I}\mathcal{E}}$ is the pre-picking pose, $\bm{H}^{\text{post-
pick}}_{\mathcal{I}\mathcal{E}}$ the post-picking pose and $\bm{H}^{\text{pre-
place}}_{\mathcal{I}\mathcal{E}}$ the pre-placing pose. Between
$\bm{H}^{\text{pre-pick}}_{\mathcal{I}\mathcal{E}}$ and
$\bm{H}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a_{t})$, the gripper is
activated as soon as a contact with a parcel is detected, after which the end-
effector is moved towards $\bm{H}^{\text{pre-
place}}_{\mathcal{I}\mathcal{E}}$.

Provided a trajectory of Cartesian waypoints, the low-level controller
transforms this trajectory from Cartesian to joint space by solving an inverse
kinematics optimization problem. The joint space trajectory is then used as a
reference for the PD controller of the actuators.

##  VI Experiments

All our experiments focus on the unloading problem described in Section IV. We
evaluate four different versions of our DQL algorithm as summarized in Table
I. It is important to note that both $\mathcal{R}_{v}$ in (6) and Mask in (7)
contain two important hyperparameters, respectively $\lambda$ and $b$, which
are kept fixed throughout our experiments. In particular, we set $\lambda=2$
and $b=100$. This choice for $\lambda$ takes into account the maximum height
of the stack and ensures an appropriate trade-off between the two factors in
$\mathcal{R}_{v}$. As for $b$, its optimal value depends on the initialization
of the critic networks weights and on the considered reward function. The
experiments depicted in Fig. 4 show that $b=100$ yields good empirical results
for all the algorithms in which Mask is used.

TABLE I: A summary of the different algorithms tested in our experiments. Mask denotes the use of the safety bias layer in (7), while $\mathcal{R}_{v}$ denotes the use of $\mathcal{R}_{v}$ rather than $\mathcal{R}_{b}$ in (6). | Mask-off | Mask-off, v-reward | Mask-on | Mask-on, v-reward
---|---|---|---|---
Mask | ✗ | ✗ | ✓ | ✓
$\mathcal{R}_{v}$ | ✗ | ✓ | ✗ | ✓

As commonly done in the RL literature, in our experiments we randomize
training and evaluation episodes from the same distribution. The obtained
final results are summarized in Fig. 4 where the average normalized
performance over $6$ random seeds is illustrated. Specifically, the left
figure shows the average number of successful picks, and the right figure
shows the number of attempted picks with
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\notin W_{\mathcal{I}}$. All
the curves are normalized with respect to $42$, i.e., the maximum number of
parcels per episode. These results demonstrate that both Mask and
$\mathcal{R}_{v}$ remarkably improve performance, in particular when they are
jointly used during training. Specifically, by using $\mathcal{R}_{v}$ rather
than $\mathcal{R}_{b}$ in (6), the agent receives a more accurate feedback
about the characteristics of the task and requires fewer interactions to
achieve better results. Furthermore, by using Mask, we are able to effectively
reduce the number of attempted picks in which
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\notin W_{\mathcal{I}}$. This
leads to an improved exploration strategy, as the agent mainly focuses on
viable actions that lead to
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\in W_{\mathcal{I}}$.
Conversely, when Mask is not used, the number of actions with
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\notin W_{\mathcal{I}}$
increases, leading to less effective exploration strategies and slower
learning rate.

In Fig. 5, we show, for each version of our algorithm, the seed leading to the
best maximum performance among the seeds averaged in Fig. 4. These experiments
more clearly emphasize the effect of Mask and $\mathcal{R}_{v}$ on improving
our final results. In Fig 5, our best policy, trained with Mask-on, v-reward
(cf. Table I), achieves $99\%$ picking success over $3$ full evaluation
episodes.

The training curves in Fig. 4 and Fig. 5 are both summarized by the box plots
in Fig. 6. Additional results and all the used hyperparameters are provided in
Appendix. We refer to our GitHub repository for more implementation details.

Figure 4: Ablation experiments. (Left) the average normalized number of
successful picks, which represents the number of parcels successfully
unloaded. (Right) the average normalized number of picks attempted with
$\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\notin W_{\mathcal{I}}$. Both
results are averaged over $6$ seeds and the shaded area represents the
standard deviation over seeds. For each seed, we randomly initialize the
critic networks and train for $10^{5}$ steps. We evaluate the learned policy
every $10$ training episodes, i.e., $420$ steps, using average performance
over $3$ episodes. The characteristics of the tested algorithms are summarized
in Table I. Figure 5: Training curves for the seeds with the best maximum
performance. For each version of our algorithm, the best seed is chosen among
the seeds averaged in Fig. 4. The maximum performance is highlighted with
"$\filledstar$". The experiments are conducted as in Fig. 4.

(a) Box plot for the curves in Fig. 4.

(b) Box plot for the curves in Fig. 5.

Figure 6: The box plots are generated considering the training curves in Fig.
4 and Fig. 5 where each evaluation step is treated as a data sample.

##  VII Conclusion

In this study, we tackle the problem of robotic unloading from visual
observations and propose a hierarchical controller that does not require any
labeled data. Our hierarchical controller, depicted in Fig. 3, consists of a
high-level controller responsible for decision-making and a low-level module
for trajectory planning and low-level control. Optimal decision-making is
learned from RGB images using DQL and is converted in optimal end-effector
pose leveraging an aligned depth image. In our DQL algorithm, we introduce a
safety bias mechanism and a task-specific reward function which prove to be
essential in order to improve our experimental results. Finally, we develop
and make publicly available a simulated environment, with the goal of
providing a benchmark for future studies in this field.

### Limitations and future work

Despite the advancement in addressing robotic unloading, it is important to
understand the limitations of our approach. In the high level-controller, the
main issue involves the training stability of our DQL algorithm. This problem
is evident in Fig. 5, where the training progress does not show a monotonic
trend. Furthermore, this instability is the main reason for the slow rate of
improvement illustrated in Fig. 4. We consider this to be a crucial challenge
for optimal unloading policy learning, representing a significant area for
future research.

Regarding the simulated environment, we consider it as a valuable benchmark
for testing RL-based methods in unloading tasks. This perspective is
substantiated by the results in Fig. 4 and Fig. 5, where vanilla RL algorithms
often struggle to succeed, as evidenced in the Mask-off case. Moreover, prior
research has addressed unloading tasks similar to what we present in our study
[24, 26]. However, we acknowledge that our current simulation does not
encompass all the potential variables encountered in real-world unloading
scenarios. As a result, our future efforts will focus on improving our
simulated environment in order to introduce more randomized settings, where
the parcels can have different textures, size and shape. We emphasize this as
an important research avenue towards improving performance in real-world
scenarios, where parcel configurations are usually messy and uncertain.

Looking ahead, we are also considering the prospect of directly applying our
training solutions to real hardware in future research endeavors. This goal
presents its own set of challenges, especially in tasks of this nature, where
devising methods that ensure minimal human intervention and safety becomes of
crucial importance.

## References

  * [1] Youngrock Yoon, Guilherme N DeSouza and Avinash C Kak  “Real-time tracking and pose estimation for industrial objects using geometric features”  In _2003 IEEE International Conference on Robotics and Automation (ICRA)_ 3, 2003, pp. 3473–3478  IEEE
  * [2] Menglong Zhu et al.  “Single image 3D object detection and pose estimation for grasping”  In _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , 2014, pp. 3936–3943  IEEE
  * [3] Emilio Frazzoli, Munther A Dahleh and Eric Feron  “Maneuver-based motion planning for nonlinear systems with symmetries”  In _IEEE Transactions on Robotics_ 21.6 IEEE, 2005, pp. 1077–1091
  * [4] Sergey Levine, Chelsea Finn, Trevor Darrell and Pieter Abbeel  “End-to-end training of deep visuomotor policies”  In _The Journal of Machine Learning Research_ 17.1 JMLR. org, 2016, pp. 1334–1373
  * [5] Dmitry Kalashnikov et al.  “Scalable deep reinforcement learning for vision-based robotic manipulation”  In _Conference on Robot Learning (CoRL)_ , 2018, pp. 651–673  PMLR
  * [6] Lucas Manuelli, Wei Gao, Peter Florence and Russ Tedrake  “kpam: Keypoint affordances for category-level robotic manipulation”  In _The International Symposium of Robotics Research_ , 2019, pp. 132–157  Springer
  * [7] Tejas D Kulkarni et al.  “Unsupervised learning of object keypoints for perception and control”  In _Advances in Neural Information Processing Systems_ 32, 2019
  * [8] Philipp Jund, Andreas Eitel, Nichola Abdo and Wolfram Burgard  “Optimization beyond the convolution: Generalizing spatial relations with end-to-end metric learning”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 4510–4516  IEEE
  * [9] Peter R Florence, Lucas Manuelli and Russ Tedrake  “Dense object nets: Learning dense visual object descriptors by and for robotic manipulation”  In _arXiv preprint arXiv:1806.08756_ , 2018
  * [10] Peter Florence, Lucas Manuelli and Russ Tedrake  “Self-supervised correspondence in visuomotor policy learning”  In _IEEE Robotics and Automation Letters_ 5.2 IEEE, 2019, pp. 492–499
  * [11] Bing Liu  “Supervised learning”  In _Web Data Mining_ Springer, 2011, pp. 63–132
  * [12] Andy Zeng et al.  “Transporter Networks: Rearranging the Visual World for Robotic Manipulation”  In _Conference on Robot Learning (CoRL)_ , 2020  PMLR
  * [13] Haojie Huang, Dian Wang, Robin Walters and Robert Platt  “Equivariant Transporter Network”  In _Proceedings of Robotics: Science and Systems_ , 2022
  * [14] Mohit Shridhar, Lucas Manuelli and Dieter Fox  “Cliport: What and where pathways for robotic manipulation”  In _Conference on Robot Learning (CoRL)_ , 2022, pp. 894–906  PMLR
  * [15] Richard S Sutton and Andrew G Barto  “Reinforcement learning: An introduction”  MIT press, 2018
  * [16] Vincent François-Lavet et al.  “An introduction to deep reinforcement learning”  In _Foundations and Trends® in Machine Learning_ 11.3-4 Now Publishers, Inc., 2018, pp. 219–354
  * [17] Volodymyr Mnih et al.  “Playing atari with deep reinforcement learning”  In _arXiv preprint arXiv:1312.5602_ , 2013
  * [18] Erwin Coumans and Yunfei Bai  “Pybullet, a python module for physics simulation for games, robotics and machine learning”, 2016
  * [19] B Scholz-Reiter, W Echelmeyer and E Wellbrock  “Development of a robot-based system for automated unloading of variable packages out of transport units and containers”  In _2008 IEEE International Conference on Automation and Logistics_ , 2008, pp. 2766–2770  IEEE
  * [20] Sunwoo Hwang et al.  “Object detection for cargo unloading system based on fuzzy C means”  In _CMC-Comput. Mater. Contin_ 71, 2022, pp. 4167–4181
  * [21] Juan Medrano et al.  “Box segmentation, position and size estimation for robotic box handling applications”  In _2022 19th International Conference on Ubiquitous Robots (UR)_ , 2022, pp. 194–199  IEEE
  * [22] Todor Stoyanov et al.  “No more heavy lifting: Robotic solutions to the container unloading problem”  In _IEEE Robotics & Automation Magazine_ 23.4 IEEE, 2016, pp. 94–106
  * [23] Paul Doliotis et al.  “A 3D perception-based robotic manipulation system for automated truck unloading”  In _2016 IEEE International Conference on Automation Science and Engineering (CASE)_ , 2016, pp. 262–267  IEEE
  * [24] Fahad Islam et al.  “Planning, learning and reasoning framework for robot truck unloading”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 5011–5017  IEEE
  * [25] Manash Pratim Das et al.  “Learning Optimal Decision Making for an Industrial Truck Unloading Robot using Minimal Simulator Runs”  In _arXiv preprint arXiv:2105.05019_ , 2021
  * [26] Hyeonjun Park et al.  “Reinforcement Learning-based Box Unloading Sequence Planning for Robotic Container-Unloading System”  In _2021 18th International Conference on Ubiquitous Robots (UR)_ , 2021, pp. 371–374  IEEE
  * [27] Richard Szeliski  “Computer vision: algorithms and applications”  Springer Science & Business Media, 2010
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun  “Deep residual learning for image recognition”  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 770–778
  * [29] Hado Van Hasselt, Arthur Guez and David Silver  “Deep reinforcement learning with double q-learning”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 30.1, 2016
  * [30] Sebastian Thrun and Anton Schwartz  “Issues in using function approximation for reinforcement learning”  In _Proceedings of the 1993 Connectionist Models Summer School Hillsdale, NJ. Lawrence Erlbaum_ 6, 1993, pp. 1–9
  * [31] Scott Fujimoto, Herke Hoof and David Meger  “Addressing function approximation error in actor-critic methods”  In _International Conference on Machine Learning_ , 2018, pp. 1587–1596  PMLR

## Appendix

##  VIII Algorithm and hyperparameters

The $K$ critic networks are defined as
$Q_{\bm{\theta}_{k}}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$, and are
parameterized by a $43$-layer encoder-decoder residual network (ResNet) [28]
with $12$ residual blocks. We use three $2$-stride convolutions in the
encoder, which is shared among all the critic networks, and three bilinear-
upsampling layers in each of the $K$ decoders. All the layers, except the
last, are interleaved with ReLU activation functions. For more implementation
details refer to our GitHub repository. Our DRL algorithm is summarized in
Algorithm 2 and the used hyperparameters are in Table II.

Input:

$T_{\text{train}}$, $\zeta$, $B$, $\alpha$, $\gamma$, $K$,
$Q_{\bm{\theta}_{k}}$, $\epsilon$, $\lambda$, Mask, $W_{\mathcal{I}}$,
$\mathcal{U}$, $\sigma$: training steps, target update rate, batch size,
learning rate, discount factor, number of critic networks, critic networks,
exploration probability, $z$-coordinate picking weight, safety bias layer,
robot workspace, uniform distribution, and softmax function.

begin Training

$s_{0}\sim\rho_{0}$

$\bar{\bm{\theta}}_{k}\leftarrow\bm{\theta}_{k}\ \ \ \forall
k\in\\{1,\dots,K\\}$

for  _$t=0,\dots,T_{\text{{train}}}$_ do

$a_{t}\leftarrow\text{ExplorationAction}[s_{t}]$

$s_{t+1}\sim\mathcal{T}(\cdot|s_{t},a_{t})$ and
$r_{t}=\mathcal{R}_{v}(s_{t},a_{t})$

$\mathcal{B}\leftarrow\mathcal{B}\cup(s_{t},a_{t},r_{t},s_{t+1})$

UpdateCritic[$\mathcal{B}$]

$s_{t}\leftarrow s_{t+1}$

begin ExplorationAction[$s$]

$u\sim\mathcal{U}(0,1)$

if  _$u\leq\epsilon$_ then

$a\sim\mathcal{U}\\{a_{W_{\mathcal{I}}}\\}$ where $a_{W_{\mathcal{I}}}$
denotes $a$ such that $\bm{p}^{\text{pick}}_{\mathcal{I}\mathcal{E}}(a)\in
W_{\mathcal{I}}$

else

$a\sim\sigma(\text{Mask}(\min_{k}Q_{\bm{\bar{\theta}}_{k}}(s,a)))$

return _$a$_

begin UpdateCritic[$\mathcal{B}$]

$\\{(s_{t},a_{t},r_{t},s_{t+1})\\}\sim\mathcal{B}$ (sample $B$ transitions)

Update $Q_{\bm{\theta}_{k}}$ to minimize (4)-(5) with learning rate $\alpha$

$\bar{\bm{\theta}}_{k}\leftarrow(1-\zeta)\bar{\bm{\theta}}_{k}+\zeta\bm{\theta}_{k}\
\ \ \forall k\in\\{1,\dots,K\\}$

Algorithm 2 Deep Q-Learning for unloading TABLE II: Hyperparameter values for our experiments. Hyperparameter Name | Value
---|---
Target update rate $(\zeta)$ | $0.005$
Batch size $(B)$ | $64$
Learning rate $(\alpha)$ | $10^{-4}$
Discount factor $(\gamma)$ | $0.99$
Number of critic networks $(K)$ | $2$
Exploration probability $(\epsilon)$ | $0.1$
$z$-coordinate picking weight ($(\lambda)$ in (6)) | $2$
Safety bias ($(b)$ in (7)) | $100$
Image size | $64\times 64$
Optimizer | Adam

##  IX Additional Experiments

(a) Training curves for the seeds with the best final performance.

(b) Box plot summarizing the results in Fig. 7(a) where each evaluation step
is treated as a data sample.

Figure 7: Results for the seeds with the best final performance. For each
version of our algorithm, the best seed is chosen among the seeds averaged in
Fig. 4. The final performance is highlighted with "$\filledstar$". The
experiments are conducted as in Fig. 4 and the labels Mask-on and Mask-off
denote the use of the safety bias layer in (7), while v-reward denotes the use
of $\mathcal{R}_{v}$ rather than $\mathcal{R}_{b}$ in (6).

(a) Training curves for the seeds with the best average performance during the
training process.

(b) Box plot summarizing the results in Fig. 8(a) where each evaluation step
is treated as a data sample.

Figure 8: Results for the seeds with the best average performance during the
training process. For each version of our algorithm, the best seed is chosen
among the seeds averaged in Fig. 4. The experiments are conducted as in Fig. 4
and the labels Mask-on and Mask-off denote the use of the safety bias layer in
(7), while v-reward denotes the use of $\mathcal{R}_{v}$ rather than
$\mathcal{R}_{b}$ in (6).

(a) Training curves for the seeds with the best maximum performance.

(b) Box plot summarizing the results in Fig. 9(a) where each evaluation step
is treated as a data sample.

Figure 9: Results for the seeds with the best maximum performance. For each
version of our algorithm, the best seed is chosen among the seeds averaged in
Fig. 4. The maximum performance is highlighted with "$\filledstar$". The
experiments are conducted as in Fig. 4 and the labels Mask-on and Mask-off
denote the use of the safety bias layer in (7), while v-reward denotes the use
of $\mathcal{R}_{v}$ rather than $\mathcal{R}_{b}$ in (6). These results are
also illustrated in Fig. 5 and 6(b) and are enlarged here for additional
clarity.