[2]Yura Jargalbaatar

[2]Donghan Kim

1]Department of Artificial Intelligence, College of Software, Kyung Hee
University, Yongin, Republic of Korea

2]Department of Electronic Engineering (AgeTech-Service Convergence Major),
College of Electronics & Information, Kyung Hee University, Yongin, Republic
of Korea

# Transformable Gaussian Reward Function for Socially-Aware Navigation with
Deep Reinforcement Learning

Jinyeob Kim<EMAIL_ADDRESS>Sumin Kang<EMAIL_ADDRESS>Sungwoo Yang
<EMAIL_ADDRESS>Beomjoon Kim<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>[  [

###### Abstract

Robot navigation has transitioned from prioritizing obstacle avoidance to
adopting socially aware navigation strategies that accommodate human presence.
As a result, the recognition of socially aware navigation within dynamic
human-centric environments has gained prominence in the field of robotics.
Although reinforcement learning technique has fostered the advancement of
socially aware navigation, defining appropriate reward functions, especially
in congested environments, has posed a significant challenge. These rewards,
crucial in guiding robot actions, demand intricate human-crafted design due to
their complex nature and inability to be automatically set. The multitude of
manually designed rewards poses issues with hyperparameter redundancy,
imbalance, and inadequate representation of unique object characteristics. To
address these challenges, we introduce a transformable gaussian reward
function (TGRF). The TGRF significantly reduces the burden of hyperparameter
tuning, displays adaptability across various reward functions, and
demonstrates accelerated learning rates, particularly excelling in crowded
environments utilizing deep reinforcement learning (DRL). We introduce and
validate TGRF through sections highlighting its conceptual background,
characteristics, experiments, and real-world application, paving the way for a
more effective and adaptable approach in robotics. The complete source code is
available on https://github.com/JinnnK/TGRF

###### keywords:

Artificial intelligence, Machine learning, Reinforcement learning, Robotic
programming, Robots, Reward shaping

##  1 Introduction

Over the years, the robotics field has shown a persistent interest in robot
navigation. Initially, research efforts focused on basic obstacle avoidance
and random navigation strategies [1-3]. Advances in navigation techniques led
to simultaneous localization and mapping (SLAM) [4-6], where robots estimate
their positions and create maps to move effectively. Strategies expanded to
address dynamic environments [7-17], as robotics consistently pursued
advancements in navigation.

However, despite the integration of robots into human environments,
effectively avoiding collisions with humans remains a significant challenge.
While safety, performance, and navigation in static environments were
previously emphasized [18], the field has now shifted its focus to socially
aware navigation, recognized as crucial [12-14, 16, 17]. Socially aware
navigation integrates perception, intelligence, and behavior to adhere to
social norms [19, 20], necessitating the critical ability to differentiate
between humans and static objects.

Figure 1: The robot’s actions when the adequate reward functions are used or
not. The red shapes represent penalties, and the yellow arrows indicate the
robot’s actions. Penalties are imposed when the robot is in proximity to
humans, within their surroundings, or moving in their direction.

To deal with this challenge, two main research directions emerged: reactive
navigation [9-11, 15] and navigation utilizing reinforcement learning (RL)
[21-24]. Reactive navigation responds to real time sensor data, with
limitations in foreseeing future movements. RL uses markov decision process
(MDP) and deep reinforcement learning (DRL) [12-14, 16, 17, 25], utilizing
deep neural networks for well-informed decisions and enabling robots to
navigate safely in human environments.

However, the challenges in defining these reward functions become particularly
evident in crowded environments where robots navigate [21, 26-28]. These
reward functions essentially serve as the guiding principle steering the
actions of agents by evaluating the potential value of each action. As a
result, human-crafted rewards become indispensable as they can’t be
automatically set. However, as demonstrated in Fig. 1, inadequately designed
reward functions can induce risky behaviors in human-robot interactions.
Moreover, the manual design of numerous such rewards presents several critical
issues.

First, the proliferation of distinct reward functions for various components
in crowded environments necessitates a redundant number of hyperparameters
[12-14, 16, 17, 25, 29-31]. Each reward demands tailored functions that align
with its specific attributes, be it the distance from humans, the direction
towards the goal, or even human intentions. This surplus of reward functions
results in excessive fine-tuning of hyperparameters, leading to imbalances in
the rewards. These imbalances can inadvertently steer robots towards humans,
increasing the risk of collisions [12]. Such disparities in action priorities,
caused by reward imbalances pose a significant challenge in tuning
hyperparameters and identifying inadequate rewards.

Second, the fixed form of each reward function is not temporally and
experimentally inefficient. The utilization of static context specific reward
functions has been observed [12, 14, 16, 17, 32-35]. However, these often fall
short in adequately representing the its unique characteristics. Even when the
same formula is employed, the diverse attributes may not be accurately
captured. This discrepancy demands extensive empirical design and
experimentation with specific reward functions to achieve higher performance.

Third, the hurdle can extend to effective learning [27]. Crafting appropriate
reward functions with the correct number of hyperparameters remains a
significant challenge, leading to collisions and hindering robot learning.

This study proposes a transformable gaussian reward function (TGRF), aiming to
guide robots along safer routes. This approach brings several crucial
contributions. (1) The less number of hyperparameters significantly alleviates
the burden of parameter tuning, expediting the search for an optimal reward
function. (2) TGRF demonstrates adaptability to various reward functions
through dynamic shape adjustments. Such adaptability stands in stark contrast
to previous models [13], which often required extensive redesigns for shape
changes. (3) TGRF shows accelerated learning rates, excelling notably in
crowded environments, effectively harnessing the potential of DRL.

To demonstrate the performance of TGRF, we introduce key points in reward
shaping and relevant papers for comparison in experiments in Section II. In
Section III, we present background knowledge, characteristics of TGRF, and
introduces the reward functions using TGRF. In Section IV, we showcase two
experiments conducted to demonstrate the performance of TGRF, and present the
results of this study’s application in real environments in Section V.
Finally, we concludes the paper in Section VI.

##  2 Related Works

###  2.1 Integration of Prior Knowledge through Human Delivered Reward
Functions

Reinforcement learning is a machine learning approach operating within MDP
[21], where an agent interacts with a specific environment, receiving feedback
in the form of rewards. The primary objective is to achieve the maximum
cumulative reward, driving the learning of policies. The reward function
significantly influences the agent’s decision making process, steering it
towards converging to optimal policies.

However, a prevalent issue arises as agents typically lack complete
information about all aspects of the environment [36]. In vast state spaces,
transitions between states and rewards might be unknown or stochastic.
Moreover, the agent remains unaware of the consequences and outcomes of
actions until they are executed.

As a result, in such scenarios, agents require substantial experience to
converge to optimal policies for complex tasks in the absence of prior
knowledge. To address this challenge, research in reinforcement learning
explores reward shaping, aiming to guide agents towards making better
decisions at appropriate times through suitable reward values [27, 28]. This
approach aims to significantly reduce the learning time by fostering the
convergence to optimal policies without explicit prior knowledge.

Previous studies have extensively explored incorporating prior knowledge into
reward functions [26, 28, 37-40]. However, crafting reward functions
encompassing general prior knowledge, such as apprehension concerning
collision risks based on proximity to humans or progress relative to the
destination, is challenging due to various environmental and psychological
factors. These factors make it impossible to express such knowledge simply
through mathematical formulations.

To address this challenge, recent research has focused on utilizing inverse
reinforcement learning (IRL), where humans intervene at intermediate stages to
provide rewards [41]. In addition, a study utilizing natural language to
communicate intermediate reward functions to agents have emerged [42]. These
studies involve humans evaluating the actions of robots as rewards and
assigning them accordingly. As a result, they have demonstrated the
transmission of rewards imbued with prior knowledge to agents during learning,
thereby accelerating the learning process and enhancing algorithm performance.

However, due to their reliance on human intervention, these approaches are
unsuitable for environments requiring extensive learning or complex tasks
without direct human involvement. Therefore, there is a growing need for
research on reward shaping that can consider prior knowledge and deliver high
performance without direct human intervention, especially in environments
demanding substantial learning.

###  2.2 Reward Function Analysis for Human Avoidance in Robot Navigation

Recent research has been actively exploring methods for robot navigation in
environments where humans and robots coexist [12-14, 16, 17, 25, 29-31]. Their
reward functions commonly employ different formulas based on objectives, all
without direct human intervention, and can broadly be classified into four
types. These include rewards categorized as follows: reaching the destination,
$r_{goal}(s_{t})$ ; collision avoidance with humans, $r_{col}(s_{t})$;
distance from humans, $r_{disc}(s_{t})$; and distance from the destination,
$r_{pot}(s_{t})$.

$r_{goal}(s_{t})$ and $r_{col}(s_{t})$ usually assume consistent values, while
$r_{disc}(s_{t})$ and $r_{pot}(s_{t})$ encompass numerous formulas reflecting
prior knowledge, such as linear, L2 norm, or exponential functions. For
instance, $r_{dist}(s_{t})$ consistently imposes a larger penalty as the
distance between humans and robots diminishes. This design aligns with the
psychological theory of Proxemics [20], which evaluates discomfort based on
interpersonal distances, integrating prior knowledge about the potential
discomfort associated with varying distances between humans and robots.

In addition, $r_{pot}(s_{t})$ incentivizes the robot’s faster arrival at the
destination by applying rewards or penalties based on changes in the L2 norm
distance between the robot and the destination. These approaches reflect
rational strategies by integrating prior knowledge concerning discomfort
levels associated with distances (Proxemics) and apprehensions regarding
collision probabilities based on proximity to humans. Moreover, they encourage
robots to expedite their arrival at the destination through rewards/penalties
based on the changing distance from the destination using the L2 norm.

However, papers related to reward shaping and reinforcement learning argue
that it is crucial to verify if the rewards take appropriate forms and
maintain suitable proportions [27, 28]. If the shapes of rewards are
inadequate for the objectives or overly biased, the robot may steer its
learning process in a direction not intended by the algorithm, potentially
leading to the freezing robot problem [42]. In addition, an excessive number
of hyperparameters might hinder the quest for optimal performance.

The aforementioned prior studies experimentally determined reward functions
and the count of hyperparameters. As a result, some were excessively
simplistic, preventing researchers from intuitively adjusting rewards through
hyperparameters, while others exhibited complex structures that hindered
straightforward modifications of hyperparameters. This resulted in significant
time consumption to achieve optimal performance and limitations in adjusting
inadequate rewards, necessitating redesigns of the reward function.

For instance, in [13], researchers designed simple reward functions with
redundant hyperparameters. This led to a substantial nine fold gap between
$r_{pred}(s_{t})$ and $r_{disc}(s_{t})$. This results in situations in which
the robot favors actions with smaller penalties from $r_{disc}(s_{t})$ over
larger penalties from $r_{pred}(s_{t})$, thereby resulting in intrusions and
collisions.

This directly impacts the learning process, rendering the task of identifying
the appropriate reward function and hyperparameters more challenging and
requiring formula modifications.

Therefore, this study proposes TGRF, allowing for intuitive and versatile
applications with less hyperparameters. By enabling researchers to intuitively
adjust rewards, it reduces the time required for exploring suitable reward
functions and ensures optimal performance by finely tuning reward balances. To
substantiate this claim, we directly compare the reward function utilized in
socially aware navigation (SCAN) [16], decentralized structural-RNN (DS-RNN)
[17], gumbel social transformer + human-human attention (GST+HH Attn) [13,
43], and crowd aware memory based reinforcement learning (CAM-RL) [31].

##  3 Suggested Reward Function

In section III-A, we briefly introduce background knowledge regarding the
model. Subsequently, in section III-B, we elaborate on the proposed TGRF in
this paper. Lastly, in III-B-3, we describe the application of TGRF to the
reward functions within the environment and model of [13].

###  3.1 Preliminaries

####  3.1.1 Markov decision process (MDP) and navigation methods

Figure 2: Diagram of an MDP with the Navigation model applied.

In the crowded environment described by the tuple ⟨S,A,P,R,$\gamma$,$S_{0}$⟩,
robots and humans follow distinct policies during simultaneous movements. It
includes states (S), actions (A), rewards (R), transition probabilities (P),
discount factor ($\gamma$), and initial state ($S_{0}$) [21].

The robot utilizes a learning based model called GST + HH Attn [13], which
employs an Attention mechanism [45] for navigation based on the past and
current positions of humans to predict future positions.

In the GST + HH Attn model, the robot begins at $S_{0}$ and receives as state
information: the robot’s current position ($p_{x}$,$p_{y}$), velocity
($v_{x}$,$v_{y}$), destination ($g_{x}$,$g_{y}$), maximum velocity $v_{max}$ ,
angle $\theta$, and robot radius $\rho$, along with the set of $i-th$ person’s
current position information ($p_{x}^{i}$,$p_{y}^{i}$) at time $t$, denoted as
$u_{i}^{t}$. $i$ can vary from 0 to a maximum number of people, $n_{max}$.
However, due to limitations in LiDAR range, the total number of people n in
the received state by the robot might differ from $n_{max}$ and can vary at
each time t. In addition, the robot remembers the past positions of detected
people up to a constant M steps, using the trajectory prediction algorithm
(GST) to nonlinearly predict $\hat{u}_{i}^{t+1:t+K}$, the predicted positions
of individuals from time t+1 to t+K.

Therefore, the GST + HH Attn model leverages the set of predicted positions
$\hat{U}_{i}^{t+1:t+K}=[\hat{u}_{0}^{t+1:t+K},\hat{u}_{1}^{t+1:t+K},...,\hat{u}_{n}^{t+1:t+K}]$
up to constant K steps and the set of current positions
$U^{t}=[u_{0}^{t},u_{1}^{t},u_{2}^{t},…,u_{n}^{t}]$ of people to predict the
overall dynamics of people through an Attention mechanism. It employs a type
of recurrent neural network (RNN) called GRU to detect and process longterm
dependencies, enabling a broader field of view [46]. The model focuses on
nearby individuals while selecting actions. Furthermore, it receives feedback
and updates its state for the next step based on the chosen actions. Besides,
the study employs five learning based methods using the same model:

  * •

DS-RNN: A model utilizing an RNN. It does not predict trajectories.

  * •

No pred + HH Attn: Attention based model, excluding trajectory prediction
($r_{pred}=0$).

  * •

Const vel + HH Attn: An experimental case assumes that the trajectory
prediction algorithm predicts trajectories to move at a constant velocity.

  * •

Truth + HH Attn: An experiment assuming that the robot predicts the actual
human trajectory.

  * •

GST + HH Attn: Scenarios in which the robot predicts the human trajectory
nonlinearly using the GST.

For a detailed model explanation, please refer to [13, 17].

Each human was randomly generated with a starting location and destination.
Humans exchange their location information
$U^{t}=[u_{0}^{t},u_{1}^{t},u_{2}^{t},…,u_{n}^{t}]$ with each another and
calculate velocities based on the positions, engaging in collision avoidance
by altering their speed and direction using reactive based methods such as
ORCA and SF [9-11].

Similar to robots, humans were designated destinations, and they aim to reach
or modify them with a predetermined probability. Humans also possess
attributes such as a size, speed, and maximum speed.

However, humans were assumed to disregard a robot’s location for two primary
reasons. First, in crowded environments, human reactions to robots lead to
subtle movements resulting in negligible positional changes. Even, rapid
shifts are infeasible in densely populated settings. Second, encapsulating
human dynamics within a mathematical model is challenging and can impede the
learning process of robots. The diversity of the manners in which humans react
to robots, along with associated uncertainties, makes modeling impractical.
Furthermore, even if such modeling were feasible, it would interfere with the
robot’s learning. As a result, humans do not perceive the robot’s location and
ignore it.

Therefore, humans make their next actions based on their own characteristics,
and information about the current positions and velocities of others.

###  3.2 Transformable Gaussian Reward Function (TGRF)

####  3.2.1 Formula and number of hyperparameters

TGRF leverages the characteristics of the normal distribution [47]. The normal
distribution offers the advantage of being able to transform into various
shapes with only two hyperparameters. This flexibility enhances the model’s
adaptability, allowing adjustments to fit diverse prior knowledge and apply
them to reward functions. Furthermore, it alleviates the burden on researchers
in tuning hyperparameters, aiding in the swift identification of appropriate
hyperparameters within a short time frame.

The formula for the normal distribution (N) is as follows:

Figure 3: Normal distribution. The X-axis denotes the X-value, and Y-axis
represents the $N(x;\mu,\sigma)$. In (a), $\mu=0,\sigma=1$. In (b),
$\mu=0,\sigma=2$.

$N(x;\mu,\sigma)=\frac{1}{\sqrt{2\pi\sigma}}\cdot\exp\left(-\frac{(x-\mu)^{2}}{2\sigma^{2}}\right).$
(1)

The normal distribution involves variables x, mean $\mu$, and variance $\rho$.
In Fig.3, the normal distribution represents the shape in which the values of
a random variable are distributed, exhibiting symmetry around $\mu$. It peaks
at $x=\mu$ and decreases as x moves away from $\mu$. $\rho$ determines the
width of the normal distribution, with the length of the width proportional to
$\rho$. Therefore, the normal distribution has the advantage of being highly
versatile in assuming various shapes with just two parameters, $\mu$ and
$\rho$.

Figure 4: TGRF. The X-axis denotes the X-position in meters, the Y-axis
represents the Y-position in meters, and the Z-axis indicates TGRF value when
$w_{TGRF}=1$,$x_{TGRF}=dist(X,Y)$,$\mu_{TGRF}=0$,$\sigma_{TGRF}=2$.

$\begin{gathered}TGRF(w_{TGRF},\mu_{TGRF},\sigma_{TGRF};x_{TGRF})=\frac{w_{TGRF}\cdot
N(x_{TGRF};\mu_{TGRF},\sigma_{TGRF})}{C_{\text{norm}}}\\\
C_{\text{norm}}=\max_{x_{\text{norm}}}N(x_{\text{norm}};\mu_{TGRF},\sigma_{TRGF})\end{gathered}$
(2)

TGRF leverages characteristics of a normal distribution. In Equation (2), TGRF
involves three hyperparameters: $w_{TGRF}$, representing the weight of TGRF;
$\mu_{TGRF}$, denoting the mean; and $\rho_{TGRF}$, indicating the variance.

It incorporates a single variable $x_{TGRF}$. $C_{norm}$ ensures that TGRF
attains a maximum value of 1 irrespective of $\rho_{TGRF}$. This allows the
scaling of TGRF solely by $w_{TGRF}$, enabling researchers to intuitively
control its maximum value, preventing the freezing robot problem [43], and
balancing it with other reward values to attain the desired algorithmic
performance ultimately.

$\rho_{TGRF}$ determines the transformability of TGRF. As
$\lim_{\sigma_{TGRF}\to\infty}\mathcal{N}(x_{TGRF};\mu_{TGRF},\sigma_{TGRF})$,
it takes on a constant form insensitive to changes in $x_{TGRF}$, while as
$\lim_{\sigma_{TGRF}\to 0}\mathcal{N}(x_{TGRF};\mu_{TGRF},\sigma_{TGRF})$, it
resembles an impulse function. This versatility enables the creation of
diverse forms such as constant, linear, non-linear, Gaussian, etc., adaptable
to specific objectives.

$\mu_{TGRF}$ determines the position of TGRF’s maximum value. Thus, it allows
adjustments of the maximum value position according to the objectives
requiring specific values to be reached. The variable $x_{TGRF}$ signifies the
variable in N($x_{TGRF}$; $\mu_{TGRF}$,$\rho_{TGRF}$). As a result, it
ultimately represents a shape similar to Fig. 4.

####  3.2.2 Transformability

Figure 5: Transformability of TGRF. The X-axis denotes the X-position in
meters, the Y-axis represents the Y-position in meters, and the Z-axis
indicates TGRF value. In (a),
$w_{TGRF}=1$,$x_{TGRF}=dist(X,Y)$,$\mu_{TGRF}=0$,$\sigma_{TGRF}=5$. In (b),
$w_{TGRF}=1$,$x_{TGRF}=dist(X,Y)$,$\mu_{TGRF}=0$,$\sigma_{TGRF}=5000$.

The reward functions in Section II-B had different formulas and limited
flexibility. These constraints lead to complexity in the reward functions and
difficulty in determining the optimal performance. Therefore, researchers must
redesign and tune them to achieve better performances. However, the TGRF
offers versatility in generating various shapes.

Fig. 5 illustrates the creation of different shapes using the same TGRF by
simply adjusting $\rho_{TGRF}$. Fig. 5a results in a TGRF that generates a
continuous Gaussian distribution, making it suitable for moving objects or
humans by applying varying penalties. Moreover, Fig. 5b forms a discrete
column like shape. This configuration is suitable for stationary reward
functions or objects. This approach allows various shapes via adjusting only a
hyperparameter ($\rho_{TGRF}$) without requiring an additional function. This
significantly reduces the researchers’ time and effort while enabling
finetuning to match the specific characteristics of objects.

####  3.2.3 Reward function

Reward $r(s_{t},a_{t})$ is categorized into five types. First,
$r_{goal}(s_{t})$ = +10 represents the reward when the robot successfully
reaches its destination. Second, $r_{col}(s_{t})$ = -10 serves as a penalty
incurred upon colliding with another individual. Third, $r_{disc}(s_{t})$
represents a penalty for entering a danger zone. Fourth, $r_{pot}(s_{t})$
corresponds to the reward/penalty contingent on the change in distance to
destination $S_{goal}$. Finally, $r_{pred}(s_{t})$ denotes the penalty invoked
when entering a prediction.

Figure 6: TGRF applied to reward functions. The X-axis denotes the X-position
in meters, the Y-axis represents the Y-position in meters, and the Z-axis
indicates negative reward value. The central cylinder represents $r_{col}$,
and the surrounding distribution represents $r_{disc}(s_{t})$. Beyond
$d_{disc}$, $r_{disc}(s_{t})$ becomes 0 ($s_{t}\notin S_{\text{danger
zone}}$). In (a),
$w_{TGRF}=8$,$x_{TGRF}=dist(X,Y)$,$\mu_{TGRF}=0$,$\sigma_{TGRF}=3$. In (b),
$w_{TGRF}=10$,$x_{TGRF}=dist(X,Y)$, $\mu_{TGRF}=0$, $\sigma_{TGRF}=1000$.

Penalty $r_{disc}(s_{t})$ is imposed when the robot enters the danger zone
($d_{min}$ is within $d_{disc}$) determined by the nearest human distance,
denoted as $d_{min}$. The formula is as follows:

$\begin{gathered}r_{\text{disc}}(s_{t})=\text{TGRF}(w_{\text{disc}},0,\sigma_{\text{disc}};d_{\text{min}})\end{gathered}$
(3)

In (3) and Fig. 6a, $r_{disc}(s_{t})$ is designed to prevent collisions with
humans, following a gaussian distribution with $\mu_{disc}$=0, whereby smaller
values of $d_{min}$ (closer proximity to humans) result in a higher penalty.
This design aligns with prior knowledge and exhibiting a more plausible and
realistic shape.

$r_{disc}(s_{t})$ can be adjusted using only two hyperparameters. By tuning
$\rho_{disc}$, researchers can regulate the breadth of the gaussian penalty
concerning the distance between humans and the robot. This enables the robot
to react more sensitively or less sensitively to the distance from humans.
$w_{disc}$ directly scales the reward function, establishing a linear
correlation with $r_{disc}(s_{t})$, thereby enabling the adjustment of the
overall reward balance to prioritize driving tasks.

Potential reward $r_{pot}(s_{t})$ represents the reward associated with the
potential field, and is defined as follows:

$\begin{gathered}r_{\text{pot}}(s_{t})=\Delta d\cdot
TGRF(1.5,\mu_{\text{pot}},\sigma_{\text{pot}};\mu_{\text{pot}})\\\ \Delta
d=(-d_{\text{goal}}^{t}+d_{\text{goal}}^{t-1})\end{gathered}$ (4)

$r_{pot}(s_{t})$ plays a crucial role in guiding a robot toward its
destination and mitigating the freezing robot problem [21]. However, high
values of $r_{pot}(s_{t})$ can lead to increased collision rates with humans.
Therefore, in equation (4), by setting $x_{pot}=\mu_{pot}$, we aimed to
maintain a constant TGRF regardless of $\mu_{pot}$ and $\rho_{pot}$. In
addition, by applying $w_{pot}$=1.5, we ensured equal weighting for all
$\Delta d$, aiming for uniformity across them.

Penalty $r_{pred}(s_{t})$ is the reward for the prediction and defined as
follows:

$\begin{gathered}r_{\text{pred}}^{i}(s_{t})=\min_{k=1,\ldots,K}\left(\mathbbm{1}_{i}^{t+k}\frac{r_{\text{col}}}{2^{k}}\right)\\\
r_{\text{pred}}(s_{t})=\min_{i=1,\ldots,n}r_{\text{pred}}^{i}(s_{t})\end{gathered}$
(5)

$r_{pred}(s_{t})$ is used only in some models employing trajectory prediction.
$r_{pred}(s_{t})$ denotes the penalty value when the robot is positioned in
the trajectory of the $i-th$ person. $\mathbbm{1}_{i}^{t+k}$ indicates whether
the robot exists at the predicted position of the $i-th$ person at time t+k or
not. Thus, $r_{pred}(s_{t})$ takes the smallest penalty among all individuals’
trajectory penalties that the robot takes. In our experiments, to demonstrate
the performance enhancement even with different reward functions, we adopted
$r_{pred}(s_{t})$ used in the GST + HH Attn from [13]. The final definition of
our reward function is as follows:

$r(s_{t},a_{t})=\begin{cases}+10,&\text{if }s_{t}\in S_{\text{goal}}\\\
-10,&\text{if }s_{t}\in S_{\text{collision}}\\\
r_{\text{pred}}(s_{t})+r_{\text{disc}}(s_{t}),&\text{if }s_{t}\in
S_{\text{danger zone}}\\\
r_{\text{pred}}(s_{t})+r_{\text{pot}}(s_{t}),&\text{otherwise}\end{cases}$ (6)

In summary, the TGRF offers the distinct advantage of intuitively and
efficiently modifying the reward function with fewer hyperparameters. This
enables the robot to make rational decisions and reduces the time consuming
task of fine-tuning.

##  4 Simulation Experiments

This section describes the experimental setup, procedure, and outcomes.

###  4.1 Experimental environment

Figure 7: Comparing a scenario with TGRF and a scenario without TGRF. Yellow
circles represent robots, blue circles represent humans within the sensor
range, red circles represent humans outside the sensor range, and orange
circles in front of the blue circles indicate trajectories predicted by the
GST.

We employed a 2D environmental simulator used in previous studies [13]. This
simulator features a 12 × 12m space, with a 360° field of view and 5m sensor
range for the light detection and ranging (LiDAR). We used a fixed number of
humans (20) to represent a crowded setting.

$\displaystyle p_{x}[t+1]$ $\displaystyle=p_{x}[t]+v_{x}[t]\Delta t$ (7)
$\displaystyle p_{y}[t+1]$ $\displaystyle=p_{y}[t]+v_{y}[t]\Delta t$

Both humans and robot operated using holonomic kinematics to determine their
velocities $(a_{t}=[v_{x},v_{y}]m/s)$ along the x- and y-axes. Holonomic
kinematics refers to a state where degrees of freedom can move independently
without any constraints. This implies that robots and machines can move
without any limitations on their position or orientation. As a result, the
action space of a robot is continuous, allowing both robots and humans to
immediately achieve their desired speed within a time frame of $\Delta t$,
assuming they operate within the maximum speed limit. Therefore, the positions
of the humans and robots were continuously updated according to (7).

The robot has attributions such as a size of $\rho=0.3m$ and maximum speed of
$v_{max}$ = $1.0m/s$. Humans also have characteristics such as a size ranging
from 0.3 to 0.5m and maximum speed varying between 0.5 and 1.5m/s. In
addition, the locations and destinations of the robot were randomized, with
the destinations not set excessively close. If the robot collides with an
individual, reaches its destination, or exceeds the maximum time T, the
episode is terminated, leading to the beginning of the subsequent episode.

Random seeding was applied during training, resulting in varying outcomes for
each training episode. To handle the varying outcomes, multiple training runs
were conducted with a total time step of 2×$10^{7}$ for the DS-RNN and
1×$10^{7}$ for the other algorithms. The learning rate was set to 4×$10^{-5}$
for all policies. Subsequently, we acquired test data from 500 test episodes.
The evaluation metrics applied to the test data included the success rate
(SR), average navigation time (NT) in seconds, path length (PL) in meters for
successful episodes, and intrusion time ratio (ITR).

###  4.2 Results

####  4.2.1 Results in different navigation methods and environment

Table 1: Navigation results using the reward function from [13] and TGRF. Humans follow ORCA. Reward | Navigation Method | Mean(sigma) of SR | SR(%) | NT(s) | PL(m) | ITR(%)
---|---|---|---|---|---|---
| DS-RNN | 35.5 (7.697) | 44.0 | 20.48 | 20.58 | 15.45
| No pred + HH Attn | 59.636 (4.848) | 67.0 | 17.49 | 20.30 | 17.22
Without | Const vel + HH Attn | 65 (8.023) | 81.0 | 17.34 | 21.95 | 6.15
TGRF | Truth + HH Attn | 5.545 (1.616) | 5.0 | 21.60 | 23.89 | 14.83
| GST + HH Attn | 77.1 (5.718) | 88.0 | 14.18 | 20.19 | 7.38
| DS-RNN | 30.5 (4.843) | 40.0 | 27.11 | 25.19 | 12.19
With | No pred + HH Attn | 59.364 (6.692) | 72.0 | 18.17 | 21.92 | 14.16
TGRF | Const vel + HH Attn | 87.909 (3.029) | 92.0 | 16.38 | 22.33 | 5.08
(Ours) | Truth + HH Attn | 84.909 (4.776) | 92.0 | 17.13 | 22.52 | 5.30
| GST + HH Attn | 94.091 (2.843) | 97.0 | 17.63 | 23.81 | 3.92
Table 2: Navigation results using the reward function from [13] and TGRF. Humans follow SF. Reward | Navigation Method | Mean(sigma) of SR | SR(%) | NT(s) | PL(m) | ITR(%)
---|---|---|---|---|---|---
| DS-RNN | 29.8 (5.231) | 36.0 | 23.26 | 27.13 | 13.38
Without | No pred + HH Attn | 12.091 (8.062) | 28.0 | 26.52 | 34.98 | 12.78
TGRF | Const vel + HH Attn | 92.182 (3.588) | 96.0 | 14.74 | 21.49 | 5.24
| GST + HH Attn | 91.636 (2.267) | 95.0 | 13.74 | 20.47 | 5.37
| DS-RNN | 48.6 (9.013) | 62.0 | 22.48 | 25.26 | 10.14
With | No pred + HH Attn | 77.364 (6.079) | 87.0 | 16.19 | 21.95 | 13.43
TGRF | Const vel + HH Attn | 95.273 (1.911) | 98.0 | 17.00 | 23.55 | 5.39
(Ours) | GST + HH Attn | 92.909 (3.579) | 96.0 | 15.37 | 21.91 | 5.81

We compared the performance of rewards (from section III-B-3) when the TGRF
was applied to the performance without its application (reward function from
[13]). The experiment involved the application of five different navigation
methods for robots, as mentioned in Section III-A-1. Table I shows the
performance when individuals adhere to ORCA, while Table II outlines the
performance when adhering to SF. The hyperparameters are set to
$w_{disc}=0.25$, $x_{disc}=d_{min}$, $\mu_{disc}=0$,$\rho_{disc}=0.2$,
$d_{disc}=0.5$, $w_{pot}=1.5$,$x_{pot}=\mu_{pot}$,
$\mu_{pot}=0$,$\rho_{pot}=1000$. Through Table 1, 2, and Fig. 7, we were able
to identify the three impacts of TGRF.

First, the TGRF results in a higher performance. As shown in Tables I and II,
the TGRF outperformed significantly in terms of the SR. Notably, Table I shows
that the TGRF achieves an average SR of 94.091%, whereas the reward function
without TGRF attains only 77.1% in the GST + HH Attn policy, marking a notable
17% increase.

Hence, models employing the TGRF generally exhibit equal or superior
performances in terms of the SR and ITR. This implies that TGRF demonstrates
relatively high performance across various models. This signifies that TGRF
effectively incorporates prior knowledge based on the role of the reward
function, indicating resilience in the freezing robot problem [43]. As a
result, it is evident that the robot demonstrates high SR by taking
appropriate actions according to the situation.

Second, even when combined with different reward functions without TGRF, TGRF
exhibits high performance. In Tables I and II, for models utilizing trajectory
prediction, $r_{pred}(s_{t})$ employed the previous paper’s reward function
without TGRF. Nonetheless, it showcased superior performance compared to [13],
proving its adaptability to other reward functions.

Third, this leads to an enhanced recognition of human intent and collision
avoidance. In Fig. 7a, the behavior of the robot when a reward function
without TGRF was applied. The robot evidently struggles when confronted with a
gathering crowd. Notably, in the test cases, the robot ventured into a crowd,
resulting in unintended collisions with humans while attempting to navigate
the crowd. Similar situations occurred in the other test cases. This means
that the robot select aggressive or impolite behaviors, such as sidestepping
to avoid human and unintentional collisions, or making risky decisions to
reach a destination faster, resulting in collisions. This behavior reflects a
deficiency in understanding the broader intentions of humans and imbalance of
the reward functions.

However, in Fig. 7b, the robot proactively positions itself behind the crowd
before converging at a single point. This means that the values of
$r_{pred}(s_{t})$, $r_{pot}(s_{t})$, and $r_{disc}(s_{t})$ related to human
interactions were well balanced, enabling the robot to navigate effectively
without colliding with individuals.

Figure 8: Comparison of performance among four types of reward functions. The
blue represents the reward function incorporating TGRF, while the orange
corresponds to the reward function in [13], the green reflects [16], and the
red signifies [31]. (a) denotes SR, (b) represents NT, (c) stands for PL, and
(d) signifies ITR.

This signifies that TGRF effectively incorporates prior knowledge, and its
priorities are well integrated into the policy. It suggests that the
algorithm’s performance can be further enhanced when TGRF is applied. Further
evidence of this enhancement is reflected in the results in Tables I and II,
where the average SR and standard deviation show similar or superior
performances compared with previous iterations. In other test cases, we could
observe that the robot selected a more secure and effective route than a
faster and more dangerous route.

####  4.2.2 Performance comparison with other reward functions

In this experiment, we conducted trials applying the reward functions from
studies [13, 16, 31] introduced in Section II-B, along with the reward
function incorporating TGRF described in Section III-B-3. The navigation
method employed GST + HH Attn models. Participants adhered to the ORCA
approach, recording SR, NT, PL, ITR every 2000 episodes. The total episodes
conducted were 20000 for GST + HH Attn.

First, it was observed that applying TGRF to the model led to an overall
performance improvement compared to other models. In Fig. 8a, the reward
function with TGRF was able to drive the algorithm’s performance up to a
maximum of 95% over 16000 total episodes. Conversely, other reward functions
achieved a maximum of 90% SR. This indicates that TGRF harmonizes
appropriately with other reward functions, assuming a shape that aligns with
the role of rewards, thereby eliciting the algorithm’s maximum performance.

In Fig. 8b, it was noticed that in the model applying TGRF, NT decreased with
repeated learning, ultimately confirming the second lowest NT.
Correspondingly, in Fig. 8c, the second lowest PL was also observed. This is
associated with ITR, as lower NT and PL imply the robot tolerating penalties
due to $r_{disc}(s_{t})$ to reach the destination, resulting in higher ITR.
For instance, looking at the orange graph, the highest ITR along with the
lowest NT and PL can be observed. As a result, in Fig. 7a, this leads to
choosing shorter and riskier paths, increasing the likelihood of not
understanding human intentions and a higher possibility of collisions.
However, in Fig. 8d, the model incorporating TGRF maintains the lowest ITR in
most cases. This demonstrates that by applying TGRF, it selects the most
efficient and safe paths compared to other models while maintaining the
highest SR, reflecting the intentions of the algorithm, as shown in the
results of Fig. 7b.

Second, Fig. 8 demonstrated the significant advantage of TGRF in terms of
learning speed compared to the reward function without TGRF. In Fig. 8a, it
was observed that three models reached saturation from a total of 6000
episodes. At this point, the model applying TGRF achieved the highest SR. This
indicates the contribution of TGRF to faster learning speeds.

However, the TGRF has limitation in crowded environments. It does not
inherently enhance the performance of core algorithms. For certain policies,
comparable performances were achieved, as shown in Tables I and II. This
suggests that the TGRF expedites the algorithm to achieve an optimal
performance rather than inherently enhancing the algorithm itself.

##  5 Real-World Experiments

Figure 9: Evasive actions performed by the robot in real-world scenarios and
corresponding renderings with four humans (from left to right). The green
arrows represent the movement path of the human, while the red arrows indicate
the movement path of the robot. In addition, yellow circles indicate robots,
green circles represent humans, and red stars indicate the destinations. These
illustrations showcase the avoidance strategies employed by the robot as it
encounters successive individuals: the (a) first, (b) second, (c) third, and
(d) fourth human.

This study extends beyond simulations to real-world experiments. The model,
trained using the unicycle approach, was applied to a physical robot in a real
environment. Unicycle kinematics present limitations in terms of direction and
position control as compared to holonomic kinematics. Our experimental setup
comprised a host computer equipped with an Intel i5-10600 processor @ 3.30 GHz
and Nvidia RTX 3070 GPU; the robot utilized was a Turtlebot3. A LiDAR sensor,
LDS-01, played a pivotal role in the human detection and robot position
estimation. Robot positioning relied on SLAM localization, and human detection
was accomplished using a 2D people detection algorithm based on 2D LiDAR data
[35]. Detailed information can be accessed from
https://github.com/JinnnK/Tranformable-Gaussian-Reward-sim2real-with-
Turtlebot3.

Although our study assumed an absence of static obstacles other than humans,
our real-world experiments were conducted in a confined space measuring
approximately 3 × 5m with static obstacles. These experiments involved
scenarios in which the robot navigated between predefined start and
destination points, encountering one to four pedestrians along its path. The
maximum speed was approximately 0.6 m/s, and the investigation covered
scenarios involving four moving individuals.

In Fig. 9a, the robot facing diagonally upward as the pedestrian moves from
right to left. In this scenario, the robot rotates to the left, aligning with
the pedestrian’s movement direction, instead of moving behind (to the right
of) the human. This decision appears rational because both the destination and
robot’s current orientation are oriented diagonally upward, making a leftward
maneuver the most efficient choice when considering the human direction,
speed, and destination. Notably, in another experiment involving three
individuals, the robot was observed to halt temporarily instead of moving to
the left.

In Fig. 9b, the robot encounters a pedestrian walking diagonally from left to
right. In response, the robot navigates to the left to avoid obstructing the
path of the pedestrian.

In Fig. 9c, the robot faces a human walking from left to right. Similarly, it
predicts the human trajectory and executes a leftward turn to avoid a
collision while approaching the destination.

In Fig. 9d, the robot encountered a human crossing diagonally from left to
right near the destination. The robot smartly avoided the human by initially
turning to the left, avoiding the pedestrian, and then turning to the right to
reach the destination.

These actions underscore the robot’s ability to make real time decisions based
on a dynamic environment, considering factors such as the human path,
velocity, and proximity to the destination. The robot’s avoidance strategies
prioritize efficiency while maintaining safety, and are influenced by various
factors, including its current orientation and overall context of the
situation. These real-world experiments testify the model’s adaptability in
complex and dynamic environments, where human-robot interactions demand
responsive and context aware behavior. The comprehensive renderings and
additional experimental videos are available on
https://youtu.be/9x24k75Zj5k?si=OtczdVXPUnbGwpv-.

Two primary limitations were encountered during this experiment. (1)
Computation load: the use of DNNs for action and trajectory predictions
significantly increases computational demands. Considering the number of
pedestrians considered, particularly for trajectory prediction, the time
required for the next action was approximately 0.22 s. This resulted in
seemingly irregular robot movements and delayed pedestrian responses. (2)
Physical constraints: the accuracy of human detection and prediction is
affected by sensor noise, limitations in the detection performance, and
challenges in determining human angles. These factors lead to the occasional
misidentification of obstacles as humans or limitations in the precision of
human location information, thereby reducing the accuracy of the trajectory
prediction. In addition, noise from the LiDAR sensor and location information
errors caused by the robot’s movement accumulated over time, resulting in
inaccuracies in location values as the experiment progressed.

##  6 Conclusion and Further Works

This study proposed the TGRF tailored for robots navigating crowded
environments. This reward function has three advantages: a good performance
with few hyperparameters, versatility for various objectives, and faster
learning and stabilization. These statement were improved by the success rates
and algorithm’s ability to discern human intentions compared with previous
reward functions.

However, challenges emerged in both the simulations and real-world
experiments. In the simulations, these challenges involved sensitivity to
hyperparameters, algorithmic limitations, a trade-off correlation between the
SR and NT, and absence of static obstacles. In real-world tests, challenges
include the sensor noise, and physical constraints.

Hence, in future research, (1) we will apply TGRF in various environments and
with different objects. In this study, we demonstrated its performance limited
to rewards concerning humans. Therefore, we plan to conduct experiments by
applying TGRF to diverse objects such as walls, obstacles, drones, and more.
(2) we intend to devise TGRF considering physical limitations. While TGRF
performs exceptionally well in ideal conditions, its performance decreases in
reality due to computational load and physical issues. Hence, we aim to
implement a dynamically adaptive TGRF that adjusts based on the situation by
incorporating knowledge regarding these physical limitations.

## Funding

This research was partly supported by the MSIT(Ministry of Science and ICT),
Korea, under the Convergence security core talent training business support
program(IITP-2023-RS-2023-00266615) supervised by the IITP(Institute for
Information & Communications Technology Planning & Evaluation), and the BK21
plus program ”AgeTech-Service Convergence Major” through the National Research
Foundation (NRF) funded by the Ministry of Education of Korea[5120200313836],
and Institute of Information & communications Technology Planning & Evaluation
(IITP) grant funded by the Korea government(MSIT) (No.RS-2022-00155911,
Artificial Intelligence Convergence Innovation Human Resources Development
(Kyung Hee University)), and the Ministry of Trade, Industry and Energy
(MOTIE), South Korea, under Industrial Technology Innovation Program under
Grant 20015440, 20025094

## References

  1. 1.

Nourbakhsh, I.R., et al., Mobile robot obstacle avoidance via depth from
focus. Robotics and Autonomous Systems, 1997. 22(2): p. 151-158.

  2. 2.

Ulrich, I. and J. Borenstein. VFH+: Reliable obstacle avoidance for fast
mobile robots. in Proceedings. 1998 IEEE international conference on robotics
and automation (Cat. No. 98CH36146). 1998. IEEE.

  3. 3.

Nalpantidis, L. and A. Gasteratos, Stereovision-based fuzzy obstacle avoidance
method. International Journal of Humanoid Robotics, 2011. 8(01): p. 169-183.

  4. 4.

Nalpantidis, L., G.C. Sirakoulis, and A. Gasteratos, Non-probabilistic
cellular automata-enhanced stereo vision simultaneous localization and
mapping. Measurement Science and Technology, 2011. 22(11): p. 114027.

  5. 5.

Pritsker, A.A.B., Introduction to Simulation and SLAM II. 1984: Halsted Press.

  6. 6.

Grisetti, G., et al., A tutorial on graph-based SLAM. IEEE Intelligent
Transportation Systems Magazine, 2010. 2(4): p. 31-43.

  7. 7.

Ai, Y., et al., DDL-SLAM: A robust RGB-D SLAM in dynamic environments combined
with deep learning. IEEE Access, 2020. 8: p. 162335-162342.

  8. 8.

Cui, L. and C. Ma, SDF-SLAM: Semantic depth filter SLAM for dynamic
environments. IEEE Access, 2020. 8: p. 95301-95311.

  9. 9.

Borenstein, J. and Y. Koren, Real-time obstacle avoidance for fast mobile
robots. IEEE Transactions on systems, Man, and Cybernetics, 1989. 19(5): p.
1179-1187.

  10. 10.

Van Den Berg, J., et al. Reciprocal n-body collision avoidance. in Robotics
Research: The 14th International Symposium ISRR. 2011. Springer.

  11. 11.

Helbing, D. and P. Molnar, Social force model for pedestrian dynamics.
Physical review E, 1995. 51(5): p. 4282.

  12. 12.

Patel, U., et al., DWA-RL: Dynamically Feasible Deep Reinforcement Learning
Policy for Robot Navigation among Mobile Obstacles, in 2021 IEEE International
Conference on Robotics and Automation (ICRA). 2021. p. 6057-6063.

  13. 13.

Shuijing Liu, P.C., Zhe Huang, Neeloy Chakraborty, Kaiwen Hong, and D.L.M.
Weihang Liang, Junyi Geng, and Kathe-rine Driggs-Campbell. Intention Aware
Robot Crowd Navigation with Attention-Based Interaction Graph. 2023.

  14. 14.

Chen, C., et al. Crowd-robot interaction: Crowd-aware robot navigation with
attention-based deep reinforcement learning. in 2019 international conference
on robotics and automation (ICRA). 2019. IEEE.

  15. 15.

Van den Berg, J., M. Lin, and D. Manocha. Reciprocal velocity obstacles for
real-time multi-agent navigation. in 2008 IEEE international conference on
robotics and automation. 2008. Ieee.

  16. 16.

Oh, J., et al., SCAN: Socially-Aware Navigation Using Monte Carlo Tree Search.
2022.

  17. 17.

Liu, S., et al. Decentralized structural-rnn for robot crowd navigation with
deep reinforcement learning. in 2021 IEEE International Conference on Robotics
and Automation (ICRA). 2021. IEEE.

  18. 18.

Kretzschmar, H., G. Grisetti, and C. Stachniss, Lifelong map learning for
graph-based slam in static environments. KI-Künstliche Intelligenz, 2010. 24:
p. 199-206.

  19. 19.

Brown, N., Edward T. Hall: Proxemic Theory, 1966. Center for Spatially
Integrated Social Science. University of Cali-fornia, Santa Barbara.
http://www.csiss.org/classics/content/13 Read, 2001. 18: p. 2007.

  20. 20.

Rios-Martinez, J., A. Spalanzani, and C. Laugier, From proxemics theory to
socially-aware navigation: A survey. Inter-national Journal of Social
Robotics, 2015. 7: p. 137-153.

  21. 21.

Bellman, R., A Markovian decision process. Journal of mathematics and
mechanics, 1957: p. 679-684.

  22. 22.

Hastings, W.K., Monte Carlo sampling methods using Markov chains and their
applications. 1970.

  23. 23.

Mnih, V., et al., Playing atari with deep reinforcement learning. arXiv
preprint arXiv:1312.5602, 2013.

  24. 24.

Sutton, R.S., Learning to predict by the methods of temporal differences.
Machine learning, 1988. 3: p. 9-44.

  25. 25.

Jeong, H., et al., Deep Reinforcement Learning for Active Target Tracking, in
2021 IEEE International Conference on Robotics and Automation (ICRA). 2021. p.
1825-1831.

  26. 26.

Gleave, A., et al., Quantifying differences in reward functions. arXiv
preprint arXiv:2006.13900, 2020.

  27. 27.

Mataric, M.J., Reward functions for accelerated learning, in Machine learning
proceedings 1994. 1994, Elsevier. p. 181-189.

  28. 28.

Laud, A.D., Theory and application of reward shaping in reinforcement
learning. 2004: University of Illinois at Urba-na-Champaign.

  29. 29.

Montero, E.E., et al., Dynamic warning zone and a short-distance goal for
autonomous robot navigation using deep reinforcement learning. Complex &
Intelligent Systems, 2023: p. 1-18.

  30. 30.

Samsani, S.S. and M.S. Muhammad, Socially compliant robot navigation in
crowded environment by human behavior resemblance using deep reinforcement
learning. IEEE Robotics and Automation Letters, 2021. 6(3): p. 5223-5230.

  31. 31.

Samsani, S.S., H. Mutahira, and M.S. Muhammad, Memory-based crowd-aware robot
navigation using deep rein-forcement learning. Complex & Intelligent Systems,
2023. 9(2): p. 2147-2158.

  32. 32.

Choi, J., G. Lee, and C. Lee, Reinforcement learning-based dynamic obstacle
avoidance and integration of path plan-ning. Intelligent Service Robotics,
2021. 14: p. 663-677.

  33. 33.

Liu, S., et al., Socially aware robot crowd navigation with interaction graphs
and human trajectory prediction. arXiv preprint arXiv:2203.01821, 2022.

  34. 34.

Pérez-D’Arpino, C., et al. Robot navigation in constrained pedestrian
environments using reinforcement learning. in 2021 IEEE International
Conference on Robotics and Automation (ICRA). 2021. IEEE.

  35. 35.

Scholz, J., et al. Navigation Among Movable Obstacles with learned dynamic
constraints. in 2016 IEEE/RSJ Interna-tional Conference on Intelligent Robots
and Systems (IROS). 2016. IEEE.

  36. 36.

Cassandra, A.R. A survey of POMDP applications. in Working notes of AAAI 1998
fall symposium on planning with partially observable Markov decision
processes. 1998.

  37. 37.

Hu, Y., et al., Learning to utilize shaping rewards: A new approach of reward
shaping. Advances in Neural Infor-mation Processing Systems, 2020. 33: p.
15931-15941.

  38. 38.

Icarte, R.T., et al., Reward machines: Exploiting reward function structure in
reinforcement learning. Journal of Arti-ficial Intelligence Research, 2022.
73: p. 173-208.

  39. 39.

Yuan, M., et al., Automatic Intrinsic Reward Shaping for Exploration in Deep
Reinforcement Learning. arXiv preprint arXiv:2301.10886, 2023.

  40. 40.

Zhang, S., et al. Average-reward off-policy policy evaluation with function
approximation. in international confer-ence on machine learning. 2021. PMLR.

  41. 41.

Rucker, M.A., et al., Reward Shaping for Human Learning via Inverse
Reinforcement Learning. arXiv preprint arXiv:2002.10904, 2020.

  42. 42.

Goyal, P., S. Niekum, and R.J. Mooney, Using natural language for reward
shaping in reinforcement learning. arXiv preprint arXiv:1903.02020, 2019.

  43. 43.

Trautman, P. and A. Krause. Unfreezing the robot: Navigation in dense,
interacting crowds. in 2010 IEEE/RSJ Interna-tional Conference on Intelligent
Robots and Systems. 2010. IEEE.

  44. 44.

Huang, Z., et al., Learning sparse interaction graphs of partially detected
pedestrians for trajectory prediction. IEEE Robotics and Automation Letters,
2021. 7(2): p. 1198-1205.

  45. 45.

Niu, Z., G. Zhong, and H. Yu, A review on the attention mechanism of deep
learning. Neurocomputing, 2021. 452: p. 48-62.

  46. 46.

Fu, R., Z. Zhang, and L. Li. Using LSTM and GRU neural network methods for
traffic flow prediction. in 2016 31st Youth academic annual conference of
Chinese association of automation (YAC). 2016. IEEE.

  47. 47.

Goodman, N.R., Statistical analysis based on a certain multivariate complex
Gaussian distribution (an introduction). The Annals of mathematical
statistics, 1963. 34(1): p. 152-177.