# End-to-end deep learning-based framework for path planning and collision
checking: bin picking application

Mehran Ghafarian Tamizi<EMAIL_ADDRESS>Homayoun Honari<EMAIL_ADDRESS>Aleksey Nozdryn-Plotnicki<EMAIL_ADDRESS>Homayoun Najjaran
<EMAIL_ADDRESS>
###### Abstract

Real-time and efficient path planning is critical for all robotic systems. In
particular, it is of greater importance for industrial robots since the
overall planning and execution time directly impact the cycle time and
automation economics in production lines. While the problem may not be complex
in static environments, classical approaches are inefficient in high-
dimensional environments in terms of planning time and optimality. Collision
checking poses another challenge in obtaining a real-time solution for path
planning in complex environments. To address these issues, we propose an end-
to-end learning-based framework viz., Path Planning and Collision checking
Network (PPCNet). The PPCNet generates the path by computing waypoints
sequentially using two networks: the first network generates a waypoint, and
the second one determines whether the waypoint is on a collision-free segment
of the path. The end-to-end training process is based on imitation learning
that uses data aggregation from the experience of an expert planner to train
the two networks, simultaneously. We utilize two approaches for training a
network that efficiently approximates the exact geometrical collision checking
function. Finally, the PPCNet is evaluated in two different simulation
environments and a practical implementation on a robotic arm for a bin-picking
application. Compared to the state-of-the-art path planning methods, our
results show significant improvement in performance by greatly reducing the
planning time with comparable success rates and path lengths.

###### keywords:

Path planning , Artificial Neural Network , Collision Checking , Bin Picking ,
Imitation Learning , Data Aggregation

††journal: arxiv

a4paper, total=170mm,257mm, left=20mm, right=20mm, top = 30mm, bottom = 30mm

[inst1]organization=Department of Electrical and Computer engineering,
University of Victoria,addressline=3800 Finnerty Rd., city=Victoria,
postcode=V8P5C2, state=BC, country=Canada

[inst2]organization=Department of Mechanical engineering, University of
Victoria,addressline=3800 Finnerty Rd., city=Victoria, postcode=V8P5C2,
state=BC, country=Canada [inst3]organization=Apera AI,addressline=Ste. 501-
134 Abbott St., city=Vancouver, postcode=V6B2K4, state=BC, country=Canada

##  1 Introduction

Bin-picking, which includes object detection, motion planning, and grasping,
is a crucial part of automation in industry. Bin picking has been one of the
trending research topics in recent years because of the challenges in image
processing and path planning [1, 2]. Industrial bin-picking consists of a
2D/3D camera, which is used to collect the environmental information (object
and obstacle detection), and a conveyor with the bins. The vision system is
used to detect object positions and obstacle shapes, while the motion planning
module calculates the path to the grasp position based on this information
[3]. An efficient and real-time path for the manipulator can greatly improve
the efficiency of mass production and assembly lines.

In a typical cycle of bin-picking operations, a combination of machine vision,
inverse kinematics, and other algorithms are used to find the best grasping
configuration for the robot. Once a configuration is found, path planning
algorithms generate a path from the robot’s fixed home state to the grasp, and
then to a place state. Finally, the robot physically moves to complete the
task. The serial and critical-path nature of path planning and movement means
that planning time and execution time are essential factors that impact the
overall cycle time and automation economics. It is crucial to ensure that
robot paths are collision-free, considering both obstacles in the environment
and self-collision.

Classical planners can be used for bin-picking, but they often fail to take
advantage of the repetitive nature of the task. In a static environment, where
a robot cell operates for extended periods, path planning is limited to fixed
initial configurations and a fixed subspace of possible end-effector poses
within a bin. This paper proposes a more efficient path planner in terms of
the planning time for bin-picking tasks. We introduce the Path Planning and
Collision checking Network (PPCNet), a deep neural network-based tool
specifically designed for repetitive tasks like bin-picking. The PPCNet
generates waypoints along the path and estimates collisions faster, making it
a reliable solution for autonomous systems such as robotic arms. By using
PPCNet, we can generate a safe and efficient path in a relatively short amount
of time. Our proposed method outperforms conventional approaches in terms of
planning time and stays competitive in path quality and success rate, making
it a viable solution for industrial bin-picking applications.

The following are the major features of this research:

  * 1.

The proposed PPCNet end-to-end framework is an easy-to-implement and efficient
approach in the context of path planning.

  * 2.

The proposed post-processing dataset generation improves the quality of PPCNet
paths.

  * 3.

Collision detection is a critical and time-intensive aspect of path planning.
PPCNet offers a faster solution, reducing planning time significantly through
its ability to quickly detect collisions.

  * 4.

We employ and compare two different methods for training the collision
checking network to determine the best and safest approach for collision
detection.

The paper is structured as follows: In Section 2, we provide a comprehensive
overview of various path planning techniques, discussing their advantages,
disadvantages, and limitations for this problem. Section 3 outlines the
problem definition. We then introduce our framework, PPCNet, in Section 4,
where we also discuss the training process. In Section 5, we evaluate and
compare the performance of our approach against four state-of-the-art path
planners. Finally, we conclude our study in Section 6.

##  2 Related work

###  2.1 Path planning

Generating collision-free paths for robotic arms is a key area of research in
robotics. Path planning methods can broadly be classified into two categories:
classical and learning-based [4]. Classical path planning approaches encompass
a wide range of techniques, including artificial potential field (APF) [5],
bio-inspired heuristic methods [6], and sampling-based path planners [7]. In
contrast, learning-based path planners primarily utilize various machine-
learning techniques to plan for the robot’s path. These methods have been
gaining popularity in recent years due to their ability to handle complex and
dynamic environments.

Sampling-based methods are widely used in path planning. In this regard, they
can be divided into two categories: Single-query algorithms and Multi-query
algorithms. While single-query approaches aim to find a path between a single
pair of initial and goal configurations, the multi-query ones try to be
efficient when there are multiple pairs [8]. The Rapidly-exploring Random Tree
(RRT) [9] and Probabilistic Road Map (PRM) [10] are two of the most well-known
algorithms for single-query and multi-query approaches, respectively. PRM is a
roadmap algorithm which aims to build an exhaustive roadmap of the
configuration space to be able to path plan between any two configurations
given to it. While PRM has been used for path planning of robotic arms before
[11], it has several limitations. Firstly, it generates paths that are far
from the optimal solution. Secondly, generating the roadmap is computationally
expensive [12], making it inefficient for path planning in relatively high-
dimensional environments, even for static tasks such as bin-picking [13].

Furthermore, given an initial and goal configuration pair, RRT grows a tree of
waypoints from the initial configuration in order to connect the two
configurations. An important characteristic of RRT algorithm is its
probabilistically completeness, meaning that it is guaranteed to find a path
if there exists one [8]. Due to RRT’s capability in non-convex high
dimensional spaces, it has been extensively employed in many studies for path
planning of robotic arms [14, 15, 16]. To increase the performance of RRT in
different aspects, such as planning time and optimality, there are many
variants of this method in the literature. In [17], the Bi-directional RRT
(Bi-RRT) is introduced for path planning of the 7-DOF manipulator. Moreover,
in [18] this algorithm used for distributed picking application. This method
simultaneously builds two RRTs; one of them attempts to find the path from the
initial configuration to the final configuration, and the other one attempts
to find the path in the opposite direction and tries to connect these two
trees in each iteration. Although this method is faster than RRT, they both
often produce sub-optimal paths in practice and the quality of the path
depends on the density of samples and the shape of the configuration space.

RRT* is the optimal version of the RRT and finds the asymptotically optimal
solution if one exists [12]. While that is a strong feature of RRT*, it is
inefficient in high-dimensional spaces and has a relatively slow convergence
rate. As a result, it is not a suitable solution for the real-time path
planning of a robotic arm. To overcome the problems mentioned above, biased
sampling is one of the promising methods which can improve the performance of
sampling-based path planners, due to the adaptive sampling of the
configuration space of the robot. Batch-informed trees (BIT*) [19] and
informed RRT* [20] are other significant variants of the RRT* that employ an
informed search strategy and batch-processing approach to find the optimal
path in a more efficient way by focusing the search on promising areas of the
tree. Although this method is able to find the optimal solution, it is not
suitable for real-time path planning since it requires a large amount of
computational resources, which can lead to delays in the system’s response
time.

As mentioned above, since collision-checking is computationally expensive and
the configuration space gets exponentially bigger with the increase of
dimensionality, most of the classical sampling-based methods suffer from high
computation and low convergence rates, making them impractical for use in a
complex environment with high dimensional configuration space and real-time
applications such as bin picking [21]. Ellekilde et al. [13] present a new
algorithm for planning an efficient path in a bin-picking scenario based on
using lookup tables. Although this method is almost instantaneous and faster
than sampling-based planners, it is memory inefficient.

To this end, learning-based path planners have emerged to deal with the
limitations of classical methods. Learning-based techniques can solve the long
run-time problem of sampling-based methods by providing biased sampling nodes,
allowing them to run faster and more efficiently [22]. For instance, Qureshi
et al. [23] proposed a deep neural network-based sampler to generate nodes for
sampling-based path planners. This work shows a significant improvement in the
performance of sampling-based path planners in terms of planning time.
Reinforcement learning (RL) is a subcategory of learning-based techniques for
manipulator path planning in an unknown environment [24, 25]. Although RL-
based approaches illustrate promising performance in path planning research,
they need exhaustive interaction with the environment to acquire experience,
and this is not applicable in many practical cases. Moreover, imitation
learning is an alternative to these kinds of cases. In imitation learning, the
training dataset is provided by an expert during the execution of the task.
For instance, in [26], a recurrent neural network is trained by using
demonstration from the virtual environment to perform a manipulation task.

Neural planners are relatively new methods in the path planning context. These
planners are based on deep neural networks, and their purpose is to solve the
path planning problem as an efficient and fast alternative to sampling-based
methods. Bency et al. [27] proposed a recurrent neural network to generate an
end-to-end near-optimal path iteratively in a static environment. Moreover, in
[28] and [29], motion planning networks (MPnet) is introduced. This method has
a strong performance in unseen environments as the path can be learned
directly.

###  2.2 Collision Approximation

Collision detection is a vital component of path planning in robotics and can
consume a significant amount of computation time. In fact, it has been
reported that it can take up to 90$\%$ of the total path planning time [7].
There are several methods for performing collision checking in path planning.
Analytical methods like GJK (Gilbert-Johnson-Keerthi) algorithm [30] use
mathematical equations and geometric models to determine if two objects will
collide. They are often fast and efficient but can struggle with complex
shapes and interactions. Grid-based Methods like Voxel Grid [31] divide the
environment into a grid of cells and check for collisions by looking at the
occupied cells. This is often faster than analytical methods but can result in
a loss of precision. Another simple method for collision detection is by
approximating objects with overlapping spheres [32]. In this approach, the
number of spheres used to represent each object is a crucial hyperparameter.
Additionally, this method is more instantaneous than other traditional
collision detection algorithms.

Recently, machine learning techniques have been utilized to overcome the
limitations of traditional collision detection methods in robotics. For
example, Support Vector Machines (SVM) [33] have been used to calculate
precise collision boundaries in configuration space. Gaussian Mixture Models
(GMM) were applied in another study [34], resulting in the path being
generated five times faster than Bi-directional RRT. K-Nearest Neighbors (KNN)
is another machine learning technique used in modeling configuration space for
collision detection [35]. Fastron, a machine learning model, was proposed by
Das et al. [36] to model configuration space as a proxy collision detector,
decomposing it into subspaces and training a collision checkers for each
region. The majority of previous studies in this field rely on decomposing the
configuration space, which requires significant engineering and
simplifications.

##  3 Problem definition

In this section, we will define the path-planning problem for the bin-picking
application. Besides, the notation that we use in this paper will be
introduced.

The first important concept in path planning is the configuration space
($C_{space}$). $C_{space}$ includes all positions and configurations of the
robot. We can define $q$ as the specific configuration for an n-joints robotic
arm:

$q=\left[\begin{array}[]{ccc}\theta_{1}&...&\theta_{n}\end{array}\right]^{\text{T}}$
(1)

Where $\theta_{i}$ is the angular position of the i-th joint of the arm.
$C_{free}$ can be defined as the subspace of $C_{space}$ in which the robot is
collision-free. Let $\tau$ be a trajectory and the first and last element of
it be $q_{initial}$ and $q_{target}$, respectively. The path planning problem
is to find all of the elements between $q_{initial}$ and $q_{target}$ in a way
that all of the segments in the trajectory lie entirely in the $C_{free}$
space. In the context of path planning, a segment between two configurations
is the space of the configurations inside the linear combination of the two
configurations. In the other words:

$\Big{\\{}\alpha
q_{i}+(1-\alpha)q_{i+1}~{}|~{}\alpha\in[0,1]\Big{\\}}\subseteq
C_{free},\forall i\in\\{1,...,N-1\\}$ (2)

Where N is the number of waypoints.

Bin-picking includes two different path-planning problems. The first one is
finding the path between the initial configuration of the robot and the grasp
configuration which is located inside the bin ($Q_{pick}$); the second one is
finding the path between the grasp position and the place configuration
($Q_{place}$). Figure 1 shows the pick-and-place operation overview. Now the
problem is to design a planner that can satisfy the following objectives:

  * 1.

generating a safe and obstacle-free path.

  * 2.

answering the queries in real time.

  * 3.

generating a high-quality path with a high probability of success.

Figure 1: Path planning for pick and place operation.

##  4 Path planning and Collision checking framework

In this section, we will delve into the details of the proposed framework for
path planning in bin-picking tasks. As illustrated in Figure 2, the proposed
method is comprised of two consecutive networks, namely the planning network
and the collision checking network.

The first network is a modified version of MPNet [29]. The original version of
the MPNet consists of two networks. The first one is the planning network
which generates the waypoints in a sequential manner by imitating the behavior
of an expert planner. The second one, the encoder network, encodes the
environment to a fixed-length vector in order to give environment information
to the planning network for better planning.

The primary obstacles of concern in this paper, are the bin, other permanent
structures, and the robot itself. These are considered to be fully known and
encoded within the planning environment such that unlike the MPNet, our
proposed planner does not use a depth map or other sensory readings of the
environment to encode information for the planning network. This allows for
faster planning times, which is a key focus of our research. In a full bin-
picking solution, the process that determines the grasp configuration will
select grasps that are at low risk of collision during grasping and robot
path.

In our work, the planning network takes the current and goal configurations
and generates the next joint space configuration to move into. Each
configuration is represented by an n-dimensional column vector. Therefore, the
path planning network’s input is a 2n-dimensional vector which is obtained by
the concatenation of the current and goal configuration. In each iteration of
decision-making, after getting the next configuration from the path planning
network, the feasibility of the segment is checked using the second network.
The second network, namely the collision checking network, gets the next
waypoint and estimates how likely it is that segment between the two
configurations will be inside $C_{free}$. By following this approach, the
process of decision-making (followed by collision-checking) in the
configuration space can be done in a more efficient manner.

Figure 2: Path Planning and collision checking network (PPCNet).

###  4.1 Training and dataset generation

The main purpose of the path planning network is to clone the behavior of an
”expert” planner. In cases like this, where it is feasible for an expert to
demonstrate the desired behavior, imitation learning is a useful approach.
Moreover, while RRT-based path planners are not themselves sequential
decision-makers, they can be modeled as:

$q_{t+1}=f(q_{t},q_{target})$ (3)

The purpose of the neural planner is to learn the function $f$. Behavior
cloning, which focuses on using supervised learning to learn the expert’s
behavior, is the most basic type of imitation learning. The process of
behavioral cloning is rather straightforward. The expert’s demonstrations are
broken down into state-action pairs, which are then used as independent and
identically distributed (i.i.d.) training data for supervised learning. An
important concern when using supervised learning for behavioral cloning is the
risk of encountering out-of-distribution data. During the inference process,
the neural planner may make decisions that were not previously observed in the
dataset. Consequently, even minor inaccuracies in the replicated behavior can
accumulate rapidly, resulting in a substantial failure rate in identifying
viable paths. To mitigate this issue, we propose the data aggregation
algorithm (DAGGER) in our study [37].

The training process of the neural planner and the collision checker is
explained in Algorithm 1. At the beginning of the process, some initial
demonstrations from the expert planner is required. To do so,
$RandomGoalConf(T)$ function samples T goal configurations in the collision-
free space of the bin. The generated goal configurations are given to the
expert planner to generate the paths using $GeneratePath$ function which
requires initial and goal configurations. The expert planner outputs two
datasets: the neural planner dataset ($D$) and the collision checker network
dataset ($C$). $D$ only consists of the final path that the expert planner
generates; however, $C$ includes every segment which was checked by the expert
planner using the geometrical collision checker. In the case of an RRT-based
algorithm, the final path will be stored in $D$ and the whole RRT tree and all
the instances of its corresponding steer function are stored in $C$.

To train the neural planner, the demonstration dataset generated by the expert
planner should be of high quality. To generate a high-quality dataset, post-
processing is applied to the paths generated by the expert planner. As shown
in Figure 4, the expert planner generates a collision-free path for a random
query. Followed by that, the Binary State Contraction (BSC) is utilized to
remove redundant and unnecessary waypoints, resulting in a shorter overall
path. The sparsity of waypoints is desirable as it simplifies the subsequent
processing, such as collision checking, communication with the robot
controller, and trajectory planning. While the output of BSC is similar to the
lazy state contraction presented in [28], BSC is computationally more
efficient and reduces the run time. After BSC, the resampling function is used
to ensure the smoothness and efficiency of the planned path. This involves
discretizing the waypoints at regular intervals, which are guaranteed to be
collision-free as they already exist on the collision-free path. Importantly,
these additional waypoints do not increase the overall length of the path but
do serve to reduce the mean and variance of the target step magnitudes,
$||q_{t+1}-q_{t}||$. While larger steps require fewer neural network forward
passes and collision checks during the inference phase, smaller steps offer
several advantages. Firstly, they minimize the deviation from the planned path
by reducing the magnitude of individual errors $||\hat{q}_{t+1}-q_{t+1}||$.
Secondly, they are more likely to be collision-free as they occupy less
physical space. Finally, they offer a more normalized training target for the
neural network, where the problem is reduced to predicting a direction in the
limit. However, BSC is a crucial step before resampling, as it relies on
dividing large steps into regular steps, and a dense path of short steps
cannot be divided. Figure 3 illustrates this procedure in a 2D environment,
highlighting how this method enhances the smoothness and quality of the path.
The $PostProcess$ procedure in Algorithm 1 presents the use of BSC and
resampling on the generated paths.

Figure 3: Post processing procedure: a) The generated path by the planner, b)
Path after binary state contraction, c) Path after resampling.

####  4.1.1 Path planner training formulation

Once the dataset has been generated, it is time to train the network. The goal
of the network is to predict the next configuration, $\hat{q}_{t+1}$, which
should be as close as possible to the actual next configuration, $q_{t+1}$, in
the training dataset. The network uses the dataset to learn the underlying
patterns and relationships between the input and output data.

To achieve the goal of accurate prediction, the network must minimize the
difference, or loss, between the predicted and actual configurations. This is
done by comparing the predicted output $\hat{q}_{t+1}$ to the actual output
$q_{t+1}$ and adjusting the network’s parameters to minimize the difference.
This process is known as backpropagation and it is typically done using
optimization algorithms such as stochastic gradient descent.

$L_{planner}=\frac{1}{N_{t}}\sum_{j=1}^{N_{d}}\sum_{i=1}^{T}||\hat{q}_{j,i}-q_{j,i}||^{2}$
(4)

In this equation, $N_{t}$ is the number of trajectories and $N_{d}$ is the
number of all individual waypoints in the training dataset.

####  4.1.2 Collision checker training formulation

To ensure the safety of the robot, we experimented with two different
approaches for training the collision checker and evaluated their performance.
As mentioned earlier, the collision checker network takes the current and next
configurations as input and estimates the probability of the segment being in-
collision. Therefore, the training set for this network should include
segments and corresponding labels indicating whether they are in collision or
not. The optimization function utilized to train the collision checker for
both of the approaches is called binary cross-entropy loss:

$L(\hat{p},p_{true})=\hat{p}\log{p_{true}}+(1-\hat{p})\log{(1-p_{true})}$ (5)

where $\hat{p}$ is the probability estimation made by the neural network and
$p_{true}$ is the true probability. The reason for using binary cross entropy
is due to the fact that, in essence, the collision checker is a binary
classification network. Therefore, the standard approach in the literature is
to use cross entropy due to it giving a better probability distribution over
the training data.

Figure 4: End-end-training process of PPCNet: Top row: The imitation learning
and data aggregation processes for training the planner network. Bottom row:
Population-based probability estimation and collision checker network training
processes.

Binary labels: The most basic approach to tackle the optimization of the
network is to use the true labels given by the analytical collision checker.
In that case, $p_{true}$ in Eq. 5 becomes either 0 (collision-free segment) or
1 (in-collision segment). In Figure 4 this approach has been named as
$L_{binary}$. The binary cross-entropy loss function measures the
dissimilarity between the network’s predicted probability distribution and the
true probability distribution in the binary classification problem.

Population-based labels: The sparsity of the binary labels hinders the
performance of the collision checker since it skews the output of the network
towards the more common label. The motivation for this method is to make the
binary labels continuous. In addition to the mentioned reasoning, making the
labels continuous can help the network be better optimized in corner cases.
For example, a collision-free segment inside the bin is mostly surrounded by
in-collision segments. Hence, the collision-free segment will not be of much
help in optimizing the network compared to the massive existence of the in-
collision segments. Making the labels continuous will amplify the effect of
the collision-free segment. To make the binary labels continuous, we give a
regional estimation of the probability of a specific segment being collision-
free by retrieving the segments similar to it whose centers lie within the
hypersphere (with a specific radius) surrounding the center of the segment.
Finally, the population-based probability (denoted as $\hat{p}_{ref}$ to
indicate it being a reference probability) estimates the probability of a
segment being collision-free by dividing the number of collision-free segments
by the total number of segments:

$\hat{p}_{ref}=\frac{\sum_{k=1}^{K}\mathds{1}[p_{true}^{(k)}==0]}{K}$ (6)

where $\mathds{1}$ is the identity-indication function. The advantage of the
proposed approach is that the probability labels generated by this method can
provide a well-behaving continuous function where near the obstacles, the
probability gets near zero and the further it gets from them, it approaches
one. In practice, we utilize KD-Tree method (built with the centers of the
segments) to find similar segments. Moreover, the more data this approach
gets, the better the estimation will be. Therefore, our proposed algorithm
stores every segment that was checked by the analytical collision checker
during the path-generation process with the expert planner. In the case of an
RRT-based expert planner, this corresponds to storing the whole tree and all
of the in-collision segments generated by it. While this approach may have
higher computational cost compared to the binary labels approach, our results
indicate that it can achieve better prediction performance. For this approach,
$\hat{p}_{ref}$ is used instead of $p_{true}$ in Eq. 5 and the corresponding
loss function is called $L_{population}$.

Algorithm 1 Training process of PPCNet

Require expert planner $\pi^{*}$, number of initial demonstrations $T$, number
of rollouts in each iteration $T^{\prime}$,number of state samples $S$,
required policy success rate $\zeta$

Initialize Planner and Collision Dataset: $D,C\leftarrow\emptyset$

Generate initial demonstrations:

$GoalConf\leftarrow RandomGoalConf(T)$

$D,C\leftarrow GeneratePath(\pi^{*},HomeConf,GoalConf)$

Apply post-processing: $PostProcess(D)$

$SuccessRate\leftarrow 0$

for $i=1,...$ do

$\pi_{i}\leftarrow TrainPolicy(D)$ $\triangleright$ Start of DAGGER

$GoalConf\leftarrow RandomGoalConf(T^{\prime})$

$D_{\pi_{i}}\leftarrow GeneratePath(\pi_{i},HomeConf,GoalConf)$

$S_{i}\leftarrow SampleStates(D_{\pi_{i}},S)$

$D_{i},C_{i}\leftarrow GeneratePath(\pi^{*},S_{i},GoalConf)$

$D_{i}\leftarrow PostProcess(D_{i})$

$D\leftarrow D\cup D_{i}$. $\triangleright$ End of DAGGER

$C\leftarrow C\cup C_{i}$.

$\eta_{i}\leftarrow TrainCollisionChecker(C)$

$SuccessRate\leftarrow TestPolicy(\pi_{i},\eta_{i})$

if $SuccessRate>\zeta$ then

return $\pi_{i},\eta_{i}$

####  4.1.3 End-to-end training process

In this section, an overview of the end-to-end process of training the
integrated model which consists of both the planner network and the collision
checker network will be discussed. The training procedure is outlined in
Algorithm 4.1.

Initially, a set of trajectories are sampled using the expert planner as the
initial demonstration, and the planner and collision dataset are filled. In
each iteration, the path planner’s policy is trained using the demonstrations
from an expert planner. Consequently, DAGGER process (as explained in Section
4.1) is executed which results in the expansion of both of the datasets.
Finally, the collision-checking dataset is trained and the model’s performance
is evaluated in the $TestPolicy$ function. The function uses some random goal
configurations and attempts to path plan between them (following the process
in Algorithm 2). If the success rate of the model was satisfying, the path
planner and the collision checker network weights are returned.

Algorithm 2 Path generation process using PPCNet

Require Trained neural planner $\pi$ and collision checker $\eta$, and initial
and goal configurations $q_{start},q_{target}$

$path\leftarrow\\{q_{start}\\}$

$q_{current}\leftarrow q_{start}$

for $i=1,...,s_{max}$ do

if $Steer(q_{current},q_{target},\eta)==q_{target}$ then

$path\leftarrow path\cup q_{target}$

if $IsFeasible(path)$ then

return $path$

else$\triangleright$ Apply Patching

$segments\leftarrow InCollision(path)$

for $(q_{s},q_{e})~{}in~{}segments$ do

$path_{alt}\leftarrow\pi^{*}(q_{s},q_{e})$

if $Failed(path_{alt})$ then

return failure

$path\leftarrow Patch(path,path_{alt})$

return $path$

$q_{next}\leftarrow\pi(q_{current},q_{target})$

if $Steer(q_{current},q_{next},\eta)$ then

$q_{next}\leftarrow Steer(q_{current},q_{next},\eta)$

$path\leftarrow path\cup q_{next}$

$q_{current}\leftarrow q_{next}$

Algorithm 3 Steer Function $(q_{current},q_{next})$

Require collision checking network $\eta$, safety threshold $\bar{\eta}$

$segments\leftarrow Discretize(q_{current},q_{next})$

$q_{final}\leftarrow q_{current}$

for $(q_{i},q_{e})~{}in~{}segments$ do

if $\eta(q_{i},q_{e})>\bar{\eta}$ then

$q_{final}\leftarrow q_{e}$

else

if $q_{final}==q_{current}$ then

return failure

else

return success, $q_{final}$

####  4.1.4 Path planning process

In this section, the end-to-end path planning process of the PPCNet is
explained. As illustrated in Figure 2, the two networks, the planning network
and collision checking network, do the decision-making sequentially. In the
context of the bin-picking application, given the start, pick, and place
configurations, the proposed model picks the object from the bin and places it
in the place configuration. As outlined in Algorithm 2, in each iteration of
the decision-making process the path planning network outputs the next
configuration that the robotic arm must transition into. The current
configuration and the next configuration are then checked with the collision-
checking network. To do so, the $Steer$ function is used. As explained in
Algorithm 3, the path between the two waypoints is discretized into equal
segments with lengths close to the resolution step size of the expert planner.
This is done in order to have the segments be close to the collision checker
training dataset’s distribution. Furthermore, in each segment, the collision-
free probability of the segment is checked. In order to decide whether a
segment is collision-free or not, a safety threshold is needed. If the
probability is more the threshold, the algorithm continues to the next segment
for collision checking. If the probability is less than the threshold, then
the segment is deemed to be in-collision. In that case, if the in-collision
segment is the first segment of the path, then the planner has failed to
output a good waypoint. Otherwise, if the in-collision segment is not the
first segment, the planning network was successful and the tail of the last
collision-free segment will be the output of the $steer$ function. Moving on,
if the $steer$ function outputs success, then the next waypoint toward the
goal configuration is added to the path. In the case that the steer function
outputs failure, the planner network attempts to generate a new waypoint. For
that purpose, stochasticity is needed inside the network. Therefore, dropout
layers are used during inference to have the network be able to recover from
failure cases [28]. Furthermore, in each iteration, if it is possible to
directly connect the current configuration to the target configuration, then
the target configuration is added to the path and it will be the final path
generated by the PPCNet. At this point, the feasibility of the path is checked
with the geometrical collision checker using $IsFeasible$ function. If the
path was collision-free, it will be returned; otherwise, the algorithm will
attempt to patch the in-collision segments using the expert planner. To this
end, the $InCollision$ function outputs the segments in the path that are in-
collision. It is important to note that if some consecutive segments were in-
collision, all of the segments will be counted as one bigger in-collision
segment and the head and tail of the bigger segment will be considered. This
is done in order to have lesser expert planner calls and be more
computationally efficient. Furthermore, for each segment, the expert planner
attempts to generate an alternative path between the two configurations. If
for all of the segments, the patching was done with success, then the proposed
path planner would be successful and the path would be returned.

Figure 5: Experimental environments: a) UR5 scene, b) UR5 scene with a wall as
an obstacle, c) Real-world implementation on Kinova Gen3 . Table 1:
Hyperparameters selection for planners.

Methods Hyperparameters Values Max iterations 1000 Bi-RRT Max planning time 5
s Resolution 0.1 rad Number of batches 500 Resolution 0.05 Informed RRT*
$\gamma$ 500 Goal probability 0.1 Max iterations 1000 Max planning time 10 s
Max iterations 1000 Max planning time 10 s BIT* Number of samples 500 $\eta$
20 Goal probability 0.1 Max iterations 100 learning rate 0.0001 MPNet Expert
planner Bi-RRT Network architecture Fully connected 6 layers, 300 neurons per
layer Post-processing resample step size 0.1745 rad Max iterations 100
Optimizer Adam PPCNet Similarity distance 0.4 rad Dagger iterations 30 Expert
planner Bi-RRT Safety threshold ($\bar{\eta}$) 0.8 Network architecture
(planning and , collision checking network) Fully connected 6 layers, 300
neurons per layer

##  5 Experimental results

In this section, we will discuss the performance of our proposed algorithm,
which employs PyTorch [38] to implement the planner network and collision
checker network. The simulation environment we used is Pybullet planning [39,
40]. To evaluate the effectiveness of our approach compared to the state-of-
the-art path planning techniques, we compared it against Bi-RRT [17], informed
RRT* [20], BIT* [19], and MPNet [29] on three different environments (Figure
5). The first environment consists of a universal robot UR5 and a bin, the
second one involves a UR5 robot and a wall as an obstacle, and the third one
features a Kinova Gen3 robotic arm, which we implemented on both simulation
and practical settings. The system used for training and testing has a 3.700
GHz AMD Ryzen 9 processor with 96 GB RAM and NVIDIA TITAN RTX GPU. Moreover,
Table 1 shows the hyperparameter selection for each of the planners.

Table 2 presents a comparative analysis of the performance of PPCNet against
other planning methods. Our experiments demonstrate that PPCNet exhibits
remarkable performance in terms of planning time in all three environments.
Figure 6 provides a visual illustration of the planning time comparison. The
hyperparameters for the competing RRT-based algorithms were tuned such that
they produce paths of comparable length to PPCNet, while ensuring that they
find a path for each planning task in less than 10 seconds and 1000
iterations. However, the results show that BIT* and Informed RRT* methods take
longer to plan and have lower success rates than PPCNet.

PPCNet has shown its ability to generate paths with comparable length and
success rates compared to other algorithms, which is particularly important in
evaluating the effectiveness of path-planning methods. Additionally, the post-
processing method employed in dataset generation has allowed PPCNet to
outperform its expert planner (Bi-RRT) in terms of path length. The BSC
function plays a critical role in eliminating redundant and unnecessary
waypoints, resulting in a reduction in path length and an improvement in path
quality. The combination of the BSC function and resampling has significantly
enhanced the quality of paths generated by PPCNet. Furthermore, our findings
show that while the addition of the collision checking to the PPCNet has
caused significant improvement compared to the MPNet in the expense of the
marginal reduction in the PPCNet’s success rate.

Finally, the study’s results indicate that the population-based training
method for the collision-checking network performed better than the binary
classification approach, resulting in a more accurate collision-checking
model. This is evident from the decrease in the planning time due to a
reduction in the need for patching. Hence, the trade-off between the
computational cost of training the population-based collision checker against
the accuracy it offers can be observed.

Table 2: Planning time and path length comparison of the proposed method and
BI-RRT for 500 random pick-and-place queries.

Environments Methods Planning time (s) Path length (rad) Success rate ($\%$)
Bi-RRT 0.701 ± 0.246 7.516 ± 2.147 100 Informed RRT* 10.186 ± 3.192 5.304 ±
1.983 79.8 BIT* 8.621 ± 3.594 5.278±2.315 77.6 UR5 MPNet 0.384 ± 0.079 7.305 ±
2.298 94.4 PPCNet Collision: Binary 0.101 ± 0.031 5.887 ± 1.038 90.2 PPCNet
Collision: Population 0.093 ± 0.033 5.679 ± 1.503 93.4 Bi-RRT 0.749 ± 0.182
7.925 ± 3.307 100 Informed RRT* 10.856 ± 3.425 5.813 ± 2.056 73.6 BIT* 9.299 ±
3.919 5.514 ± 2.747 70.4 UR5 with a wall MPNet 0.392 ± 0.106 7.874 ± 3.011 93
PPCNet Collision: Binary 0.106 ± 0.027 6.142 ± 1.979 89.8 PPCNet Collision:
Population 0.099 ± 0.034 6.016 ± 1.749 92.2 Bi-RRT 0.645 ± 0.281 6.812 ± 2.749
100 Informed RRT* 9.535 ± 4.122 5.124 ± 2.205 79.2 BIT* 8.496 ± 3.082 5.099 ±
2.464 79.6 Kinova Gen3 MPNet 0.227 ± 0.151 6.714 ± 2.812 90.6 PPCNet
Collision: Binary 0.090 ± 0.029 5.315 ± 1.482 88.2 PPCNet Collision:
Population 0.085 ± 0.032 5.188 ± 1.335 89.8

Figure 6: Planning time comparison between different methods.

##  6 Conclusions

In this paper, we introduced a novel path-planning framework, PPCNet, and
examined its efficacy for bin-picking applications. The robot path planning
for the bin-picking application is typically a computationally expensive
process because the traditional algorithms need to compute the path in its
entirety every time the target changes. Further, most existing algorithms lack
the ability to learn from past experiences. The proposed PPCNet utilizes deep
neural networks to find a real-time solution for path-planning and collision-
checking problems. Specifically, we propose an end-to-end training algorithm
that leverages the data aggregation algorithm to generate i.i.d. data for
supervised learning. Using an expert planner in the training process, data
aggregation can assist the agent in encountering unseen states. Another novel
component of the proposed PPCNet algorithm includes two variations of the
collision-checking network for approximating the exact geometrical collision
function. We evaluate our proposed framework, PPCNet, by conducting simulation
on two robotic arms namely UR5 and Kinova Gen3 in a bin-picking application.
We have also examined the algorithm in a real-world scenario using Kinova Gen3
robot. Our results indicate that our approach significantly reduced planning
time while producing paths with path lengths and success rates comparable to
those of the state-of-the-art path planning methods.

## Acknowledgment

We would like to acknowledge the financial support of Apera AI and Mathematics
of Information Technology and Complex Systems (MITACS) under IT16412 Mitacs
Accelerate.

## References

  * [1] D. Buchholz, D. Kubus, I. Weidauer, A. Scholz, F. M. Wahl, Combining visual and inertial features for efficient grasping and bin-picking, in: 2014 IEEE international conference on robotics and automation (ICRA), IEEE, 2014, pp. 875–882.
  * [2] Y. Domae, H. Okuda, Y. Taguchi, K. Sumi, T. Hirai, Fast graspability evaluation on single depth maps for bin picking with general grippers, in: 2014 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2014, pp. 1997–2004.
  * [3] D. Buchholz, Bin-picking, Studies in Systems, Decision and Control 44 (2016) 3–12.
  * [4] M. G. Tamizi, M. Yaghoubi, H. Najjaran, A review of recent trend in motion planning of industrial robots, International Journal of Intelligent Robotics and Applications (2023) 1–22.
  * [5] Z. Long, Virtual target point-based obstacle-avoidance method for manipulator systems in a cluttered environment, Engineering Optimization 52 (11) (2020) 1957–1973.
  * [6] L. Tian, C. Collins, An effective robot trajectory planning method using a genetic algorithm, Mechatronics 14 (5) (2004) 455–470.
  * [7] M. Elbanhawi, M. Simic, Sampling-based robot motion planning: A review, Ieee access 2 (2014) 56–77.
  * [8] K. M. Lynch, F. C. Park, Modern robotics, Cambridge University Press, 2017.
  * [9] S. M. LaValle, et al., Rapidly-exploring random trees: A new tool for path planning, Tech. rep. (1998).
  * [10] L. E. K. J.-C. Latombe, Probabilistic roadmaps for robot path planning, Practical motion planning in robotics: current approaches and future challenges, Citeseer (1998) 33–53.
  * [11] C. Rodríguez, A. Montaño, R. Suárez, Planning manipulation movements of a dual-arm system considering obstacle removing, Robotics and Autonomous Systems 62 (12) (2014) 1816–1826.
  * [12] S. Karaman, E. Frazzoli, Sampling-based algorithms for optimal motion planning, The international journal of robotics research 30 (7) (2011) 846–894.
  * [13] L.-P. Ellekilde, H. G. Petersen, Motion planning efficient trajectories for industrial bin-picking, The International Journal of Robotics Research 32 (9-10) (2013) 991–1004.
  * [14] T. Rybus, K. Seweryn, Application of rapidly-exploring random trees (rrt) algorithm for trajectory planning of free-floating space manipulator, in: 2015 10th International Workshop on Robot Motion and Control (RoMoCo), IEEE, 2015, pp. 91–96.
  * [15] X. Wang, X. Luo, B. Han, Y. Chen, G. Liang, K. Zheng, Collision-free path planning method for robots based on an improved rapidly-exploring random tree algorithm, Applied Sciences 10 (4) (2020) 1381.
  * [16] T. Rybus, Point-to-point motion planning of a free-floating space manipulator using the rapidly-exploring random trees (rrt) method, Robotica 38 (6) (2020) 957–982.
  * [17] J. Kuffner, S. L. RRT-Connect, An efficient approach to single-query path planning ieee international conference on robotics and automation, San Francisco (2000) 473–479.
  * [18] M. A. Riedlinger, M. G. Tamizi, J. Tikekar, M. Redeker, Concept for a distributed picking application utilizing robotics and digital twins, 27th IEEE ETFA (2022).
  * [19] J. D. Gammell, S. S. Srinivasa, T. D. Barfoot, Batch informed trees (bit): Sampling-based optimal planning via the heuristically guided search of implicit random geometric graphs, in: 2015 IEEE international conference on robotics and automation (ICRA), IEEE, 2015, pp. 3067–3074.
  * [20] J. D. Gammell, S. S. Srinivasa, T. D. Barfoot, Informed rrt*: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic, in: 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 2014, pp. 2997–3004.
  * [21] T. F. Iversen, L.-P. Ellekilde, Benchmarking motion planning algorithms for bin-picking applications, Industrial Robot: An International Journal (2017).
  * [22] R. Cheng, K. Shankar, J. W. Burdick, Learning an optimal sampling distribution for efficient motion planning, in: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2020, pp. 7485–7492.
  * [23] A. H. Qureshi, M. C. Yip, Deeply informed neural sampling for robot motion planning, in: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2018, pp. 6582–6588.
  * [24] I. Aleo, P. Arena, L. Patané, Sarsa-based reinforcement learning for motion planning in serial manipulators, in: The 2010 International Joint Conference on Neural Networks (IJCNN), IEEE, 2010, pp. 1–6.
  * [25] M. Duguleana, F. G. Barbuceanu, A. Teirelbar, G. Mogan, Obstacle avoidance of redundant manipulators using neural networks based reinforcement learning, Robotics and Computer-Integrated Manufacturing 28 (2) (2012) 132–146.
  * [26] R. Rahmatizadeh, P. Abolghasemi, A. Behal, L. Bölöni, Learning real manipulation tasks from virtual demonstrations using lstm, arXiv preprint arXiv:1603.03833 (2016).
  * [27] M. J. Bency, A. H. Qureshi, M. C. Yip, Neural path planning: Fixed time, near-optimal path generation via oracle imitation, in: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), IEEE, 2019, pp. 3965–3972.
  * [28] A. H. Qureshi, A. Simeonov, M. J. Bency, M. C. Yip, Motion planning networks, in: 2019 International Conference on Robotics and Automation (ICRA), IEEE, 2019, pp. 2118–2124.
  * [29] A. H. Qureshi, Y. Miao, A. Simeonov, M. C. Yip, Motion planning networks: Bridging the gap between learning-based and classical motion planners, IEEE Transactions on Robotics 37 (1) (2020) 48–66.
  * [30] E. G. Gilbert, D. W. Johnson, S. S. Keerthi, A fast procedure for computing the distance between complex objects in three-dimensional space, IEEE Journal on Robotics and Automation 4 (2) (1988) 193–203.
  * [31] O. S. Lawlor, L. V. Kalée, A voxel-based parallel collision detection algorithm, in: Proceedings of the 16th international conference on Supercomputing, 2002, pp. 285–293.
  * [32] P. M. Hubbard, Approximating polyhedra with spheres for time-critical collision detection, ACM Transactions on Graphics (TOG) 15 (3) (1996) 179–210.
  * [33] J. Pan, D. Manocha, Efficient configuration space construction and optimization for motion planning, Engineering 1 (1) (2015) 046–057.
  * [34] J. Huh, D. D. Lee, Learning high-dimensional mixture models for fast collision detection in rapidly-exploring random trees, in: 2016 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2016, pp. 63–69.
  * [35] J. Pan, D. Manocha, Fast probabilistic collision checking for sampling-based motion planning using locality-sensitive hashing, The International Journal of Robotics Research 35 (12) (2016) 1477–1496.
  * [36] N. Das, M. Yip, Learning-based proxy collision detection for robot motion planning applications, IEEE Transactions on Robotics 36 (4) (2020) 1096–1114.
  * [37] S. Ross, G. Gordon, D. Bagnell, A reduction of imitation learning and structured prediction to no-regret online learning, in: Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, 2011, pp. 627–635.
  * [38] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, A. Lerer, Automatic differentiation in pytorch (2017).
  * [39] C. R. Garrett, Pybullet planning (2020).
URL https://pypi.org/project/pybullet-planning/

  * [40] E. Coumans, Y. Bai, Pybullet, a python module for physics simulation for games, robotics and machine learning, http://pybullet.org (2016–2021).


# RUPNet: Residual upsampling network for real-time polyp segmentation
††thanks: Citation: Authors. Title. Pages…. DOI:000000/11111.

Nikhil Kumar Tomar
Machine and Hybrid Intelligence Lab,
Department of Radiology,
Northwestern University,
Chicago
USA
<EMAIL_ADDRESS>
Ulas Bagci
Machine and Hybrid Intelligence Lab,
Department of Radiology,
Northwestern University,
Chicago
USA
<EMAIL_ADDRESS>
Debesh Jha
Machine and Hybrid Intelligence Lab,
Department of Radiology,
Northwestern University,
Chicago
USA
<EMAIL_ADDRESS>

###### Abstract

Colorectal cancer is among the most prevalent cause of cancer-related
mortality worldwide. Detection and removal of polyps at an early stage can
help reduce mortality and even help in spreading over adjacent organs. Early
polyp detection could save the lives of millions of patients over the world as
well as reduce the clinical burden. However, the detection polyp rate varies
significantly among endoscopists. There is numerous deep learning-based method
proposed, however, most of the studies improve accuracy. Here, we propose a
novel architecture, Residual Upsampling Network (RUPNet) for colon polyp
segmentation that can process in real-time and show high recall and precision.
The proposed architecture, RUPNet, is an encoder-decoder network that consists
of three encoders, three decoder blocks, and some additional upsampling blocks
at the end of the network. With an image size of $512\times 512$, the proposed
method achieves an excellent real-time operation speed of 152.60 frames per
second with an average dice coefficient of 0.7658, mean intersection of union
of 0.6553, sensitivity of 0.8049, precision of 0.7995, and accuracy of 0.9361.
The results suggest that RUPNet can give real-time feedback while retaining
high accuracy indicating a good benchmark for early polyp detection.

_K_ eywords Deep learning $\cdot$ Convolutional neural network $\cdot$ Medical
image segmentation $\cdot$ Colon polyps
$\cdot$ Colorectal cancer $\cdot$ Computer aided diagnosis $\cdot$ Polyp
segmentation $\cdot$ Residual network

##  1 Introduction

Colonoscopy is considered the gold standard and is recommended screening tool
for colon cancer diagnosis and follow-up. Early detection and removal of
adenomas is crucial in colonoscopy for removing incidence and mortality. A 1%
increase in adenoma detection rate showed associated with a 3% decrease in
interval colorectal cancer incidence [1]. Still, the adenoma miss rate is
around 6-27% [2]. Therefore, an endoscopist is expected to perform a high-
quality colonoscopy examination. The essential quality indicators for
colonoscopy are adenoma detection rate (ADR) and withdrawal time (WT) [3, 4].
Once the adenomas are detected during examination there are chances that the
endoscopists become less motivated in finding the subsequent adenomas [5, 6,
7].

There are various reasons for high polyp miss-rate such as size, shape,
quality of bowel preparation, and faster colonoscope withdrawal time [2].
Sometimes during a routine examination, the polyp is not recognized and
sometimes despite being recognized they are missed. The polyp detection rate
can be improved by detecting the unrecognized polyps that are on the visual
field [8]. Therefore, an algorithm-based second observer might help to improve
the polyp detection rate during live colonoscopy examination. Additionally, a
real-time computer-aided diagnosis system could highlight the suspicious
polyps that could be clinically relevant for selecting optimal treatment,
ideal colonoscopic resection and reducing overall cost [9].

Deep learning based algorithms can highlight the presence of precancerous
tissue in the colon and have the potential to improve the diagnostic
performance of endoscopists. In clinical practice, precise polyp segmentation
provides important information in the early detection of colorectal cancer.
Despite of success of deep learning algorithms for automatic polyp
segmentation, it is still an unsolved problem. This is because even the most
successful method fails when tested across new datasets obtained from the
other centers mostly because of different imaging protocols, different cohorts
populations and various scanners. The challenges which make polyp segmentation
complicated are the variable nature of polyps in their size, shapes, subtle
appearances (for example, sessile serrated lesions) and high recurrence rate
etc. The human factors such as bowel preparation and skill of the particular
endoscopist can also affect examination procedure [10].

Tomar et al. [11] proposed a feedback attention network (FANet) for improved
biomedical image segmentation, where they showed the state-of-the-art
performance on seven publicly available benchmark datasets. FANet unifies the
mask of the previous epoch with the current training epoch and rectifies the
prediction iteratively during test time for improved performance. Recently,
Biffi et al. [12] proposed an intelligent medical device for real-time optical
characterization of colonoscopy polyps which can be also integrated easily
into clinics. Besides that, there are several recent works on colonoscopy
targeting polyps attributes such as size and count [13], flat, sessile or
diminutive polyps [14], out-of-distribution polyp detection through cross
dataset test [15, 14], and synthetic data generation for improving the
performance of the network [16]. As the study of colonoscopy cancer is
becoming mature more emphasis is being given towards exploring the
possibilities of new technological advancement such as diagnostic
classification, risk stratification, and outcome examination [17].

The main contribution are as follow:

  1. 1.

We propose RUPNet, a novel lightweight encoder decoder based architecture for
real-time polyp segmentation. Our algorithm is based on residual block and the
architecture is designed in a way that requires low computational power for
training.

  2. 2.

We evaluate RUPNet with the popular benchmarks and showed improvement over
those architectures. Most interestingly, our method obtained a real-time speed
of 152.60 which is important for the clinical integration.

The rest of the paper is organized as follows. In Section 2, we present the
method. In Section 3, we present the experimental setup. Then, in Section 4,
we present results and Discussion. Finally, we conclude our work in Section
LABEL:sec:conclusion.

##  2 Method

Figure 1 shows the block diagram of the proposed RUPNet architecture. RUPNet
is an encoder-decoder network that consists of three encoder blocks, three
decoder blocks, and a few additional upsampling blocks at the end of the
network. The input image is passed to the first encoder block consisting of
the residual block, followed by $2\times 2$ max-pooling. The residual block
[18] helps propagate information over layers that allows to build a deeper
neural network for solving the degradation problems in each encoder block. It
helps to improve the channel inter-dependencies and reduces the overall
computational cost. The max-pooling is an operation for computing the maximum
value for patches of feature maps which is used for creating downsampled
feature maps. The output of the last encoder block passes through a residual
block, which acts as the bridge between the encoder network and the decoder
network. The output of the bridge acts as the input for the decoder network.

Figure 1: Overall architecture of the RUPNet.

The decoder begins with a bilinear upsampling with a factor of two. After
that, it is followed by a residual block. The feature map increases in height
and width with every decoder block until it reaches the last decoder. Here, we
take the skip connections, followed by a bilinear upsampling to ensure that
all the skip connections have the same resolution. The skip connection in the
network allows the passing of lower-level semantic information to the latter
layer, which helps preserve the information for better information flow. Next,
we concatenate the upsampled feature maps with the output of the last decoder
block. Finally, we have a $1\times 1$ convolution with the Sigmoid activation
function to get the final mask.

Figure 2: The figure shows examples of Kvasir-SEG [19] used for training the
RUPNet algorithm.

##  3 Experimental setup

We have selected the Kvasir-SEG [19] dataset consisting of 1,000 polyp images
and their corresponding ground truth for training the proposed model. Figure 2
shows the example images from the Kvasir-SEG dataset. We use 880 images and
masks for training our model and the rest 120 images for testing. We have
performed extensive data augmentation, such as flipping, rotation, random
brightness, etc., to increase the number of training samples. Our experiments
are run on an NVIDIA RTX 3090 GPU system. We have used the Adam optimizer
having a learning rate of 1e-4 and a batch size of 8. We have used a
combination of binary cross-entropy and dice loss as a loss function. For the
evaluation, we use standard medical image segmentation based metrics such as
mean intersection over union (mIoU), dice coefficient (mDSC), recall or
sensitivity, precision, accuracy, F2-score and processing speed.

Table 1: Quantitative results on the Kvasir-SEG [19] test dataset. Method | mDSC | mIoU | Recall | Precision | Accuracy | FPS
---|---|---|---|---|---|---
U-Net [20] | 0.5969 | 0.4713 | 0.6171 | 0.6722 | 0.8936 | 22.02
ResUNet [21] | 0.6902 | 0.5721 | 0.7248 | 0.7454 | 0.9169 | 14.82
ResU-Net++ [22] | 0.7143 | 0.6126 | 0.7419 | 0.7836 | 0.9172 | 7.01
RUPNet (Ours) | 0.7658 | 0.6553 | 0.8049 | 0.7995 | 0.9361 | 152.60
Figure 3: The figure shows the example of qualitative results of RUPNet
prediction. From the example, it can be observed that RUPNet can accurately
segment medium and smaller-sized polyps. However, it also shows over-
segmentation when exposed to light lighting conditions.

##  4 Result and Discussion

###  4.1 Comparison with State-of-the-Art:

Table 1 shows the result of the RUPNet. The proposed architecture obtain an
F1-score of 0.7658, mIoU of 0.6553, recall of 0.8049, precision of 0.7995, and
F2-score 0.9361. With the image resolution of $512\times 512$, the proposed
method achieves a real-time operation speed of 152.60 frames per second (FPS).
The main reason behind higher speed is its architectural design, which has
only a few trainable parameters, making the network lightweight for real-time
processing. The most competitive benchmarking method of RUPNet was ResUNet++
with a dice coefficient of 0.7143. However, it has only a speed of 7.0193 FPS.
Therefore, our method is 21.74 times faster than one of the widely used
benchmark in the field of automated polyp segmentation.

Figure 3 shows the qualitative results of RUPNet. The qualitative result shows
that RUPNet can correctly segment smaller and medium-sized polyps that are
commonly missed due to their size. The visual results also show that RUPNet
shows over-segmentation for the lighting conditions, as evidenced last two
examples from Figure 3. Thus, both the qualitative and quantitative results
exhibit an acceptable overall performance.

###  4.2 Limitations and open challenges

Our study has some limitations. We classify images into polyp and non-polyp
regions. However, we do not cover if the polyp is adenoma or non-adenoma and
does not look into diagnostic classes such as hyperplastic or sessile polyps.
For the algorithm to be integrated into the clinical workflow, the proposed
algorithm should perform well on out-of-the-distribution datasets that are
collected from different medical centres. Additionally, the algorithm should
perform well in conditions such as camouflage and noise and should possess
real-time processing speed. Although, we achieve real-time speed our algorithm
is trained and tested on the same distribution datasets. We have not
experimented on multi-center datasets, and our algorithm shows that there are
still some challenges when there is camouflage which is caused by the false
positive. This is also observed through the qualitative analysis. Moreover, we
have not performed statistical tests for any experiments, which is also a way
of interpreting the best model.

##  5 Conclusion

In this work, we proposed the RUPNet architecture that utilizes a simple
residual block to accurately segment polyps with high processing speed. The
experimental results on the colon polyp dataset showed that RUPNet can
accurately give real-time feedback with high accuracy, and it might help
endoscopists in early polyp detection in clinics. In the future, we plan to
extend RUPNet by adding transformer blocks in the encoder and evaluate with
the multi-centre out-of-distribution dataset to study the robustness of our
algorithm. Additionally, we will also research on distinguishing colon polyps
based on their attributes, such as the number of polyps counts (for example,
one or many) and their size (for example, $\leq 5mm$, between $5mm\leq 10mm$,
and $\geq$ than 10 mm) and compare it with the recent state-of-the-art polyp
segmentation benchmark.

## Acknowledgments

This project is supported by the NIH funding: R01-CA246704 and R01-CA240639.

## References

  * [1] Gregor Urban, Priyam Tripathi, Talal Alkayali, Mohit Mittal, Farid Jalali, William Karnes, and Pierre Baldi.  Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy.  Gastroenterology, 155(4):1069–1078, 2018.
  * [2] Sang Bong Ahn, Dong Soo Han, Joong Ho Bae, Tae Jun Byun, Jong Pyo Kim, and Chang Soo Eun.  The miss rate for colorectal adenoma determined by quality-adjusted, back-to-back colonoscopies.  Gut and liver, 6(1):64, 2012.
  * [3] Douglas K Rex, John H Bond, Sidney Winawer, Theodore R Levin, Randall W Burt, David A Johnson, Lynne M Kirk, Scott Litlin, David A Lieberman, Jerome D Waye, et al.  Quality in the technical performance of colonoscopy and the continuous quality improvement process for colonoscopy: recommendations of the us multi-society task force on colorectal cancer.  Official journal of the American College of Gastroenterology| ACG, 97(6):1296–1308, 2002.
  * [4] Michal F Kaminski, Jaroslaw Regula, Ewa Kraszewska, Marcin Polkowski, Urszula Wojciechowska, Joanna Didkowska, Maria Zwierko, Maciej Rupinski, Marek P Nowacki, and Eugeniusz Butruk.  Quality indicators for colonoscopy and the risk of interval cancer.  New England journal of medicine, 362(19):1795–1803, 2010.
  * [5] Edwin J Lai, Audrey H Calderwood, Gheorghe Doros, Oren K Fix, and Brian C Jacobson.  The boston bowel preparation scale: a valid and reliable instrument for colonoscopy-oriented research.  Gastrointestinal endoscopy, 69(3):620–625, 2009.
  * [6] Thomas F Imperiale and Douglas K Rex.  A new quality indicator of colonoscopy: caveat emptor.  Gastrointestinal Endoscopy, 84(3):507–511, 2016.
  * [7] Takahiro Amano, Tsutomu Nishida, Hiromi Shimakoshi, Akiyoshi Shimoda, Naoto Osugi, Aya Sugimoto, Kei Takahashi, Kaori Mukai, Dai Nakamatsu, Tokuhiro Matsubara, et al.  Number of polyps detected is a useful indicator of quality of clinical colonoscopy.  Endoscopy International Open, 6(07):E878–E884, 2018.
  * [8] Nadim Mahmud, Jonah Cohen, Kleovoulos Tsourides, and Tyler M Berzin.  Computer vision and augmented reality in gastrointestinal endoscopy.  Gastroenterology report, 3(3):179–184, 2015.
  * [9] Douglas K Rex, C Richard Boland, Jason A Dominitz, Francis M Giardiello, David A Johnson, Tonya Kaltenbach, Theodore R Levin, David Lieberman, and Douglas J Robertson.  Colorectal cancer screening: recommendations for physicians and patients from the us multi-society task force on colorectal cancer.  Gastroenterology, 153(1):307–323, 2017.
  * [10] Debesh Jha.  Machine learning-based classification, detection, and segmentation of medical images.  PhD thesis, 2022.
  * [11] Nikhil Kumar Tomar, Debesh Jha, Michael A Riegler, Håvard D Johansen, Dag Johansen, Jens Rittscher, Pål Halvorsen, and Sharib Ali.  Fanet: A feedback attention network for improved biomedical image segmentation.  IEEE Transactions on Neural Networks and Learning Systems, 2022\.
  * [12] Carlo Biffi, Pietro Salvagnini, Nhan Ngo Dinh, Cesare Hassan, Prateek Sharma, and Andrea Cherubini.  A novel ai device for real-time optical characterization of colorectal polyps.  NPJ digital medicine, 5(1):1–8, 2022.
  * [13] Nikhil Kumar Tomar, Debesh Jha, Ulas Bagci, and Sharib Ali.  Tganet: Text-guided attention for improved polyp segmentation.  In Proceedings of the International Conference on Medical image computing and computer-assisted intervention (MICCAI), 2022.
  * [14] Debesh Jha, Pia H Smedsrud, Dag Johansen, Thomas de Lange, Håvard D Johansen, Pål Halvorsen, and Michael A Riegler.  A comprehensive study on Colorectal Polyp Segmentation With ResUNet++, Conditional Random Field and Test-Time Augmentation.  IEEE Journal of Biomedical and Health Informatics, 25(6):2029–2040, 2021.
  * [15] Abhishek Srivastava, Sukalpa Chanda, Debesh Jha, Umapada Pal, and Sharib Ali.  GMSRF-Net: An improved generalizability with global multi-scale residual fusion network for polyp segmentation.  In Proceedings of the International Conference on Pattern Recognition (ICPR), pages 4321–4327, 2022.
  * [16] Jan Andre Fagereng, Vajira Thambawita, Andrea M Storås, Sravanthi Parasa, Thomas de Lange, Pål Halvorsen, and Michael A Riegler.  Polypconnect: Image inpainting for generating realistic gastrointestinal tract images with polyps.  arXiv preprint arXiv:2205.15413, 2022.
  * [17] James A Diao and Joseph C Kvedar.  Computer copilots for endoscopic diagnosis, 2022.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 770–778, 2016.
  * [19] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen.  Kvasir-SEG: a segmented polyp dataset.  In Proceedings of the International Conference on Multimedia Modeling (MMM), pages 451–462, 2020.
  * [20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-Net: Convolutional Networks for Biomedical Image Segmentation.  In Proceedings of the International Conference on Medical image computing and computer-assisted intervention (MICCAI), pages 234–241, 2015.
  * [21] Zhengxin Zhang, Qingjie Liu, and Yunhong Wang.  Road extraction by deep residual u-net.  IEEE Geoscience and Remote Sensing Letters, 15(5):749–753, 2018\.
  * [22] Debesh Jha, Pia H Smedsrud, Michael A Riegler, Dag Johansen, Thomas De Lange, Pål Halvorsen, and Håvard D Johansen.  Resunet++: An advanced architecture for medical image segmentation.  In Proceedings of the International Symposium on Multimedia (ISM), pages 225–2255, 2019.