# A Fully Controllable Agent in the Path Planning using Goal-Conditioned
Reinforcement Learning

GyeongTaek Lee GyeongTaek Lee is with Department of Industrial and Systems
Engineering, Rutgers University, The State University of New Jersey, 96
Frelinghuysen Road, Piscataway, 08854, NJ, USA
E-mail<EMAIL_ADDRESS>Manuscript received May 19, 2022; revised
August 26, 2022.

###### Abstract

The aim of path planning is to reach the goal from starting point by searching
for an agent’s route. In the path planning, the routes may vary depending on
the number of variables such that it is important for the agent to reach
various goals. Numerous studies, however, have dealt with a single goal that
is predefined by the user. In the present study, I propose a novel
reinforcement learning framework for a fully controllable agent in the path
planning. To do this, I propose a bi-directional memory editing to obtain
various bi-directional trajectories of the agent, in which the agent’s
behavior and sub-goals are trained on the goal-conditioned RL. As for the
agent’s to move in various directions, I utilize the sub-goals dedicated
network, separated from a policy network. Lastly, I present the reward shaping
to shorten the number of steps for the agent to reach the goal. In the
experimental result, the agent was able to reach the various goals that have
never been visited by the agent in the training. We confirmed that the agent
could perform difficult missions such as a round trip and the agent used the
shorter route with the reward shaping.

###### Index Terms:

Controllable agent, path planning, goal-conditioned reinforcement learning,
bi-directional memory editing.

##  1 Introduction

Path planning is a method to find an optimal route from the starting point to
the target point. It has been widely used in various fields such as robotics
[1, 2, 3], drone [4, 5, 6, 7, 8, 9], military service [10, 11], and self-
driving car [12, 13]. Recently, reinforcement learning (RL) has been mainly
studied for the path planning [10, 7, 14, 15, 9, 16, 17, 3, 11]. To get an
optimal solution, it is essential to give enough reward for an agent to reach
the goal and to set up a specific environment. Several studies on learning the
RL model have proposed to make an agent be robust in a complicated or an
unknown environment for the path planning [18, 19, 20]. However, existing
studies have defined one single goal before the learning. That is, the agent’s
ability to search the path when completed learning can be limited. To make the
agent reach a number of goals in a dynamic environment, learning a
controllable agent is needed. One of the recent approaches for controlling the
agent has a limitation in that the agent can only learn the behavior from
trajectories that have been directly experienced [21]. Therefore, the agent
can only be under control in the area visited by the agent. In this paper, I
focus on learning a fully controllable agent in the path planning using a
goal-conditioned RL. Especially, I apply, to the goal-conditioned RL, a bi-
directional memory editing and a sub-goals dedicated network to improve the
ability to search the path of the agent.

Figure 1: Examples of training and test environment in this study. The
experiment was set as a simple version of the two-dimensional grid
environment. The starting point and the target point were fixed at the
location in the training environment. The policy of the agent is to search the
path from the starting point to the target point. (a), The training
environment was set for the agent to easily get an optimal route. (b), The
test environment was set for the agent to get an optimal route in a difficult
way. Several sub-goals including the final goal were given to the agent.

In the goal-conditioned RL, the agent learns the sub-goals, part of the
trajectory of the agent [22, 23, 24, 25, 26]. By learning the sub-goals,
eventually, the agent can reach the final goal. This method showed good
performance mainly in robotics. However, previous studies have focused on
reaching a complicated single goal. In addition, the multi-goal RL model, in
which the agent can perform many goals, has been proposed [27, 28]. However,
the multi-goals should also be defined before the learning. Unlike these
studies, in this present study, I propose an RL framework in which the agent
can perform a number of sub-goals in various scenarios. An important point
here is that the sub-goals are not defined in advance. In other words, the
agent completed the learning reaches the goals, which have never been visited
by the agent in the learning.

Fig 1 shows the examples of the training and the test environments of this
study. The training environment has been simply set for the agent to reach the
final goal. But the test environments have been set in a difficult way. The
agent should go through the sub-goals and reach the target point, even if the
starting point and the target point are set in reverse. The difficult missions
were also given to the agent such as a round trip. In the test environments,
an important thing is that the agent has to reach the goal never been visited
in the learning. To perform the goals, the agent must be fully controlled by
the sub-goals that are customized by the user.

Meanwhile, memory editing occurs in our daily lives. [29, 30, 31, 32]. The
memory editing helps us to get away from mental illnesses such as trauma [32].
We can also get precise information by editing untidy memories [31]. A recent
study used the concept of the memory editing on goal-conditioned RL for the
agent to reach sub-goals so that the user can control the agent [21]. However,
the agent cannot move the sub-goals in difficult environments. This is because
the memory of the RL model is edited only based on one direction of the paths
that are moved by the agent. The first purpose of the RL is to achieve the
final goal. Despite the sub-goals, the agent can ignore the sub-goals if the
sub-goals get in the way of the final goal. That is, the agent does not need
to go round and round to reach the final goal. In this study, I developed the
concept of the memory editing for the fully controllable agent in the path
planning.

Let’s assume that we walk to the destination (See Fig 2). From the route that
we walked, we can learn how we reach the destination. We can remember
intermediate stops in the route and know that the total path is the overall
set, which consists of the intermediate stops. Further, if we recall our
memories backward, we can also find the route back to the starting point. For
example, in the Fig 2, when we want to return to the starting point, we know
which action we have to perform. However, it is difficult to find an inverse
action in a dynamic RL environment. Thus, an inverse module to predict the
inverse action is necessary to obtain the exact knowledge to come back to the
starting point.

Based on the processes, in which we recall our memories and obtain knowledge,
I utilize a bi-directional memory reminiscence and editing (bi-directional
memory editing) to obtain various trajectories. Using these trajectories, the
agent can learn various behaviors. Furthermore, I use the dedicated network
for learning the sub-goals to improve learning efficiency. Finally, I present
a reward shaping for the shorter path of the agent. Using these techniques,
the agent can achieve various sub-tasks as well as the final goal in the path
planning environment. The fully controllable agent can be useful in the
environments where we have to consider a number of variables and we have to
assume various scenarios. The main contributions of this article are as
follows:

  * •

Using the bi-directional memory editing, we can obtain various trajectories,
and can learn the agent to perform various tasks, based on the trajectories.
The agent can be fully controlled so that the agent can reach any point in the
path planning environment.

  * •

I employ the sub-goals dedicated network to improve the efficiency of learning
the sub-goals. To distinguish the network from the policy network, the agent
can focus on performing the various sub-goals.

  * •

I propose the reward shaping for the shorter path of the agent. In the path
planning, it is important for the agent not only to reach the final goal but
also to reach it within a limited time. By applying the reward shaping in the
bi-directional memory editing, the agent can reach the final goal within a
shorter time.

  * •

To the best of our knowledge, this study is the first RL methodology for the
path planning in that the agent is fully under control. Therefore, the agent
achieves the user-defined sub-goals such as a round trip in a difficult test
environment. Moreover, the agent can move to the point that has never been
reached by the agent in the training. By using this methodology, we can
suppose and conduct various scenarios in the path planning.

The rest of the paper proceeds as follows. Section 2 presents the background
of this study. In Section 3, learning a fully controllable agent is proposed.
To do this, I introduce the bi-directional memory editing and the sub-goals
dedicated network. Furthermore, I propose the reward shaping for the shorter
path of the agent. Section 4 provides the results of the experiment. I
confirmed whether the agent can successfully perform the sub-goals in various
scenarios. Section 5 concludes this paper and calls for future research.

Figure 2:  Illustration of the concept of the bi-directional memory editing.
After we arrive at the destination, if we recall our memory, we can find out
the route for returning to the starting point. Further, we can also obtain
knowledge about which action to perform to reach the starting point.

##  2 Background

###  2.1 Path planning

To get an optimal solution to reach the target point from the starting point,
traditionally, optimization methods have been used in the path planning.
Several studies using an A* [33, 34], a genetic algorithm [35], and a particle
swarm optimization [36, 37, 38] have been proposed. Combining two optimization
methods also has been studied [39].

Figure 3: Illustration of the proposed method. The red line indicates the
route visited by the agent. The blue line indicates the inverse direction of
the route. The inverse module predicts the action when $s_{t}$ and $s_{t+1}$
are given, and the predicted action is used to collect the reverse
trajectories. In the bi-directional memory editing, various sub-goals are
generated and stored in the separated replay memory $(D_{s})$. The transition
samples from $D_{s}$ are trained by the sub-goals dedicated network.

Recently, with the development of deep learning, studies on the path planning
using the RL have mainly been proposed [40, 10, 41, 7, 14, 42, 6, 15, 9, 16,
17, 3, 11]. They have supposed the specific scenario and set an environment to
apply the agent in the path planning. Especially, they have applied their
study to robotics [7, 3, 16], drone [4, 5, 42, 6, 8, 9], and ship [43, 44].
Also, they have focused on the one single goal of the agent to reach the
target point, avoiding obstacles. After learning is completed, the agent could
reach the goal; however, the agent cannot be under control in the previous
studies. That is, the agent can only perform the predefined goal.

In addition, learning user-defined sub-goals has been proposed [21]. However,
the agent was partially under control. The agent could not perform the round
trip and only moved the area that was visited by the agent. In this study, I
focus on learning the fully controllable agent so that the agent can perform
various trips, as shown in Fig 1.b.

###  2.2 Goal-conditioned RL

Hindsight experience replay (HER) used the sub-goals and the pseudo rewards to
make the RL model converge to the final policy [45]. The reason why learning
the sub-goals improves the performance of the RL is that the sub-goals are
located on the route to reaching the final goal. Several studies developed the
HER for an exploration of the agent [46, 47, 48].

The goal-conditional RL models excel to reach the goal through the
intermediate sub-goals and show excellent performance on the robotics problem
[49, 28, 50, 51, 52, 27, 53, 22, 23, 24, 25, 26]. They have focused on
searching for meaningful sub-goals and improving the performance of the main
policy network (high-level policy network). To reach the desired policy, it is
important to induce the agent to reach the landmark of the sub-goals [51, 49,
27, 24, 54] and to improve sample efficiency [55, 52, 56]. The existing
studies with learning the sub-goals are similar to this study in terms of
generating the sub-goals and relabeling the rewards.

However, in this study, the sample efficiency and searching for the valid sub-
goals are not essential. By performing the bi-directional memory editing, we
can get trajectories two times more than when bi-directional memory editing is
not performed. Then, enough sub-goals can be collected for learning the sub-
goals. Also, it does not matter the time when the sub-goals dedicated network
is trained, which is separated from the original policy network. Whether the
sub-goals are trained after learning the policy network or during learning the
policy network, if sufficient trajectories are gathered, the agent can learn
various behaviors and sub-goals. The main purpose of this study is to make the
agent under control for performing various tasks which are not defined in the
training environment. After learning is completed, the agent can achieve
various user-defined sub-goals as well as the final goal.

I consider a discounted, finite-horizon, goal-conditioned Markov decision
process (MDP) defined as a tuple
($\mathcal{S},\mathcal{G},\mathcal{A},p,R,\gamma,H$), where $\mathcal{S}$ is
the set of state, $\mathcal{G}$ is the set of goals, $\mathcal{A}$ is set of
actions, $p(s_{t+1}|s_{t},a_{t})$ is a dynamics function, $R$ is the reward
function, $\gamma\in[0,1)$ is the discounted factor, and $H$ is the horizon.
In the goal-conditioned RL, the agent learns to maximize the expected
discounted cumulative reward $\mathbb{E}[\sum_{t=1}^{\infty}R(s_{t},g,t)]$,
where $g$ is the goal. The objective is to obtain a policy
$\pi(a_{t}|s_{t},g,t)$.

##  3 Proposed method

Fig 3 shows the illustration of a summary of the proposed method. For the
fully controllable agent in the path planning, I introduce three simple
techniques. First, I propose a bi-directional memory editing to generate
various behaviors and sub-goals of the agent. Here, to secure reverse
trajectories, an inverse module to predict actions is used. Second, to improve
the efficiency of learning, I utilize the sub-goals dedicated network
separated from the policy network. Finally, I present a reward shaping for the
shorter path of the agent.

###  3.1 Bi-directional memory editing

The memory editing is performed to generate sub-goals of the agent. The sub-
goals are generated from the trajectories of the agent, and additional rewards
are given to the agent. As the agent begins to recognize the sub-goals, the
agent can achieve the sub-goals and greedily reach the final goal in the
previous studies.

In the path planning, it is important to allow the agent to visit a wider area
such that the agent can visit various locations and learn the optimal route to
reach the goal. Thus, if we can get various trajectories more than the
trajectories that are actually visited by the agent, it can bring greater
benefit to learning the agent’s ability for searching the path. In addition to
a forward route, the reverse route from the goal to the starting point can be
a useful ingredient for the agent to learn various behaviors and sub-goals. To
do this, I employ the reverse trajectory to generate various sub-goals and to
learn the robust RL model by performing the bi-directional memory editing.

First, a forward memory editing is performed, which is described in line 21-24
in Algorithm 1. The sub-goals ($g$) are generated from the forward route, and
the state of the sub-goals ($s_{t+1}\|g$) are made. After that, a backward
memory editing is performed, which is described in line 25-29 in Algorithm 1.
Reverse transition is $\\{(s_{t+1},a_{t+1}^{\prime},r_{t},s_{t})\\}$ whereas
original transition is $\\{(s_{t},a_{t},r_{t},s_{t+1})\\}$. Here, it is
difficult to obtain $a_{t+1}’$. The reason is that it is not simple to find an
action to derive the $s_{t}$, when $s_{t+1}$ is given. I propose to use the
inverse module to obtain $\hat{a}_{t+1}$ by predicting $a_{t+1}’$ when $s_{t}$
and $s_{t+1}$ are given. Like the forward memory editing, the sub-goals ($g$)
and the state of the sub-goals ($s_{t+1}\|g$) are generated. Finally, the
edited memories are stored in the replay memory for the sub-goals
($\mathcal{D_{s}}$).

Using the bi-directional memory editing, we can obtain two routes from the one
single route moved by the agent. Beyond being two times the trajectories, it
means that the agent can learn various relationships between the actions and
the sub-goals. In the path planning, the agent can almost only learn the route
that is visited by the agent. For instance, as shown in Fig 1.(a), the agent
can almost learn the leftward and upward directions because of the location of
the goal. However, using the bi-directional memory editing, the agent can
learn all directions in various locations. Therefore, by learning a number of
sub-goals and behaviors, the agent can reach the goals that were never visited
in the training.

1:Initialize policy network parameters $\theta_{p}$

2:Initialize sub-goals dedicated network parameters $\theta_{g}$

3:Initialize inverse module parameters $\theta_{iv}$

4:Initialize replay buffer for original goal $\mathcal{D}\leftarrow\emptyset$

5:Initialize replay buffer for sub-goals $\mathcal{D_{s}}\leftarrow\emptyset$

6:procedure Learning the sub-goals

7: for episode = 1, M do

8: `\\` Simulation stage.

9: for each step do

10: Execute an action $s_{t},a_{t},e_{t},s_{t+1}\approx\pi_{\theta}(a_{t}\mid
s_{t})$

11: Store transition
$\mathcal{E}\leftarrow\mathcal{E}\cup\\{(s_{t},a_{t},r_{t})\\}$

12: Optimize inverse module $\theta_{iv}$

13: end for

14: if $s_{t+1}$ is terminal then

15: Compute returns $R_{t}=\Sigma^{\infty}_{k}\gamma^{k-t}{r}_{k}$ in
$\mathcal{E}$

16: $\mathcal{D}\leftarrow\mathcal{D}\cup\\{(s_{t},a_{t},R_{t})\\}$

17: Clear episode buffer $\mathcal{E}\leftarrow\emptyset$

18: end if

19:

20: `\\`Bi-directional memory editing

21: Generate sub-goals $g$

22: Generate state of sub-goals $s_{t}\|g$ $\triangleright$ $\|$ denotes
concatenation

23: Set additional rewards $r_{t}^{\prime}$

24:
$\mathcal{D_{s}}\leftarrow\mathcal{D_{s}}\cup\\{s_{t}\|g,a_{t},r_{t}^{\prime})\\}$

25: Predict the inverse action
$\hat{a_{t+1}}\leftarrow\theta_{iv}(s_{t+1},s_{t})$

26: Generate reverse transition $\\{(s_{t+1},\hat{a}_{t+1},r_{t},s_{t})\\}$

27: Generate sub-goals $g$

28: Generate state of sub-goals $s_{t+1}\|g$

29: Set additional rewards $r_{t}^{\prime\prime}$

30: $s_{t}\leftarrow s_{t+1}$,$a_{t}\leftarrow\hat{a}_{t+1}$,$r_{t}\leftarrow
r_{t}^{\prime\prime}$

31:
$\mathcal{D_{s}}\leftarrow\mathcal{D_{s}}\cup\\{s_{t}\|g,a_{t},r_{t}^{\prime})\\}$

32:

33: `\\`Learning stage.

34: for k= 1, N do

35: Sample a minibatch $\\{(s,a,R)\\}$ from $\mathcal{D}$

36: $\triangleright$ Optimize policy network $\theta_{p}$

37: end for

38: for k= 1, P do

39: Sample a minibatch $\\{(s\|g,a,r^{\prime})\\}$ from $\mathcal{D_{s}}$

40: $\triangleright$ Optimize sub-goals dedicated network $\theta_{g}$

41: end for

42: end for

43:end procedure

Algorithm 1 Learning the sub-goals using bi-directional memory editing

###  3.2 The sub-goals dedicated network

Using the bi-directional memory editing, the agent can learn various behaviors
and sub-goals. However, the agent at the middle of the route can be confused
about where the agent has to go. Because the agent is forced to learn how to
go in both directions at one point due to the bi-directionally edited
memories. For example, in the Fig 2, if the agent is located near the tree,
the agent learns to move both upwards and leftwards. If the agent is trained
using the policy network, the agent can be confused about where the agent has
to go if the agent is located near the tree. Because the purpose of the policy
network is only to train the agent to reach the final goal. Actually, the
agent has been hovering around the middle point of the environment in the
experiment. Therefore, I employ the network for the sub-goals separately from
the network for the final goal.

The sub-goals dedicated network only learns the sub-goals, and the original
policy network only learns the final goal. To do this, I also employ the
replay memory for the sub-goals ($\mathcal{D_{s}}$) in addition to replay
memory for the final goal ($\mathcal{D}$). Moreover, using the sub-goals
dedicated network can improve the sample efficiency. In general, the capacity
of the replay memory is limited and the replay memory is updated with the last
transition of the agent. Thus, the agent mainly learns the recent transitions
and gradually reaches the goal. However, as previously mentioned, using the
bi-directional memory editing, we can obtain a number of sub-goals and
behavior of the agent. Therefore, using the separated network and the replay
memory, we can learn the agent to reach the sub-goals whenever, during or
after the policy network learning.

Although the agent of the policy network cannot reach the final policy, the
agent of the sub-goals dedicated network can reach the various sub-goals as
well as the final goal. The users can fully control the agent by collecting
various sub-goals from the bi-directional memory editing and by learning the
agent on the sub-goals dedicated network.

###  3.3 Reward shaping for the shorter path

In the path planning, one of the important factors is to make the agent reach
the destination within a specific period. To improve the agent’s ability, it
is necessary to give enough rewards according to the steps to reach the target
point. However, in this study, I focus on learning the sub-goals, and
additional rewards are given to the agent to reach the sub-goals, regardless
of the shortest path. Furthermore, I want to confirm that the agent can reach
various sub-points in the environment. Thus, I assume that the environment of
this study is a sparse reward environment so that the agent does not need to
reach the target point in the shortest path. Rather, due to the exploration
bonus, it is likely for the agent to delay the one’s arrival. Therefore, we
propose the reward shaping when the bi-directional editing is performed for
the shorter path of the agent in the path planning. When the bi-directional
editing is performed, the additional rewards ($r_{t}^{\prime}$) are given with
the corresponding sub-goals. Here, the rewards are reshaped as follows:

$\displaystyle{r_{tg}}$ $\displaystyle=$ $\displaystyle
r_{t}^{\prime}+(dist_{short}–dist_{s_{t}})$ (1)

: where $dist_{short}$ indicates the number of the shortest steps that can be
possibly reached by the agent from $s_{t}$ to the current sub-goal ($g_{s}$),
and $dist_{st}$ indicates the number of steps of the path that is moved by the
agent from $s_{t}$ to the current sub-goal ($g_{s}$). That is, in each sub-
goal, the agent gets a penalty by the number of steps to reach the sub-goal
point. If the agent reaches the sub-goal in the shortest path, the agent does
not receive any penalty. Otherwise, the agent receives a penalty depending on
the number of steps to reach the sub-goal points.

###  3.4 Learning a fully controllable agent in the path planning

Fig 3 shows the summary of the proposed method. The inverse module predicts
the action of when $s_{t}$ and $s_{t+1}$ are given in the bi-directional
memory editing. The sub-goals from two trajectories are generated. At this
time, the reward shaping for the shorter path is performed. Then, the
transitions are stored in the replay memory for the sub-goals
($\mathcal{D_{s}}$). The sub-goals dedicated network is trained independently
with the policy network, whereas the policy network is only trained for the
final policy. After learning is completed, in the various scenarios, the agent
gets the sub-goals that are defined by the users and tried to achieve the sub-
goals as well as the final goal.

Algorithm 1 shows the procedure in the proposed method in detail. The
simulation stage is similar to the other RL methods except for the inverse
module to obtain $\hat{a}_{t+1}$ by predict $a_{t+1}’$ when $s_{t}$ and
$s_{t+1}$ are given. In the bi-directional memory editing, the reverse
transition $\\{(s_{t+1},\hat{a}_{t+1},r_{t},s_{t})\\}$ is obtained using the
inverse module and various sub-goals are generated and stored in the memory
$(D_{s})$. In the learning stage, optimize the policy network $(\theta_{p})$
for the final goal and optimize the sub-goals dedicated network $(\theta_{g})$
for the sub-goals. In fact, it does not matter to learn the goals dedicated
network separately after learning the policy network is completed, if the
number of the edited memories is enough.

##  4 Experiments

###  4.1 Experimental setting

In the experiments, I wanted to confirm that the agent can be fully controlled
by the sub-goals and that the agent can achieve various sub-goals, which were
never visited by the agent in the learning. Thus, the training environment was
constructed in a simpler way, while the test environment in a difficult way. I
assumed a number of scenarios to test the agent’s ability of the path
searching. Furthermore, I wanted to show the effect of the reward shaping for
the shorter path.

Fig 1 shows the first environment. The goal of the agent was set to reach the
target point in a simple two-dimensional (2D) grid environment. The reward was
0 except for the agent reaching the target point (+30). The RL model was
performed for a total of 10,000 episodes. In the test environment, I set a
total of 26 scenarios. In each scenario, several sub-goals were given to the
agent. At first, a sub-goal nearest the current location of the agent was
given to the agent. Then, if the agent reaches the sub-goal, the next sub-goal
nearest the current location was given. Also, if the agent could not reach the
given sub-goal, the next sub-goal nearest the current location was given. In
various scenarios, I observed whether the agent reaches various sub-goals and
successfully performs difficult missions such as a round trip. Moreover, I
compared with and without the reward shaping for the shorter path in the
proposed method. In each scenario, I calculated the number of steps to reach
the target point.

Fig 4 shows the second environment. The environment is the ‘key-door domain’.
The environment has a total of 4 stages and the agent should go through the
bonus point (key) to clear the stage (door). Even though the agent reaches the
target point (door), if the agent could not pass the bonus point (key), the
agent cannot jump up to the next stage. The reward was set +10 for a bonus
point, -10 for a penalty point, and +100 for the goal, when the agent goes
through these points, respectively. This environment was difficult because of
the condition that the agent must pass the bonus point to clear each stage and
that the environment was defined as the sparse reward setting. The RL model
performed for a total of 50,000 episodes. In the test environment, I set the
two scenarios. In each stage, two sub-goals, defined by the user, the bonus
point, and the goal, were given to the agent as the sub-goals in an order.

Figure 4: The key-door domain environment. In each stage, the factors (a
starting point, a bonus point, a wall, and a target point) were set
differently. The agent must go through the bonus point to clear the stage.
Even if the agent reaches the target point if the agent did not go through the
bonus point, the agent cannot pass by the next stage.

###  4.2 Base architecture of the RL model

Self-imitation learning (SIL) is an on-policy algorithm to exploit valuable
past decisions from the replay memory [57]. In the learning stage, the
transitions are sampled, and they are trained by the policy network. At this
time, if the transitions of the past are not valuable compared to the current
value, the transitions are not exploited. That is, the SIL imitates valuable
behaviors of the agent in the past. The authors combined the SIL and the on-
policy RL model [58, 59] and proposed the following off-policy actor-critic
loss:

$\displaystyle\mathcal{L}$ $\displaystyle=$
$\displaystyle\mathbb{E}_{s,a,R\in\mathcal{D}}[\mathcal{L}_{policy}+\beta\mathcal{L}_{value}],$
(2) $\displaystyle\mathcal{L}_{policy}$ $\displaystyle=$ $\displaystyle-
log\pi_{\theta}(a|s)(R-V_{\theta}(S))_{+},$ (3)
$\displaystyle\mathcal{L}_{value}$ $\displaystyle=$
$\displaystyle{{1}\over{2}}\parallel(R-V_{\theta}(S))_{+}\parallel^{2},$ (4)

where $(\cdot)_{+}=max(\cdot,0)$; $\pi_{\theta}$ and $V_{\theta}(s)$ are the
policy network and the value network, respectively, parameterized by $\theta$.
The value loss is controlled by $\beta\in\mathbb{R}^{+}$. From the
$(\cdot)_{+}$ operator in the loss, the transition, in which the current value
is larger than the past return, is trained by the policy network and the value
network.

The reason why that the SIL is used in the study is that the exploitation of
valuable transition is needed. In fact, in the study, the off-policy RL model
is necessary to utilize the replay memory [60, 61, 62]. However, in the path
planning, to reach the target point, various routes can be obtained.
Accordingly, I utilize the off-policy actor-critic RL model to get an effect
of both the on-policy and the off-policy. The final RL architecture in this
study is the combination of the SIL and the actor-critic network (ASIL).

In addition, I utilized the random network distillation (RND), which is widely
used as an exploration bonus method [63]. The RND uses two networks: a fixed
and random initialized network (target network), and a predictor network
trained using the output of the target network. The exploration bonus is given
as a difference between the outputs of the two networks. If the agent visits a
specific point continually, the exploration bonus is gradually decreased.
Otherwise, if the agent visits a novel space, a large exploration bonus will
be given to the agent.

Figure 5: The visualization of the path of the agent for the first
environment. A color change from blue to red indicates that the agent has
visited more often. a, The path of the agent of the policy network without
learning the sub-goals. The agent easily reached the target point. b, The path
of the agent of policy network with learning the sub-goals using the bi-
directional memory editing. The agent was confused by the sub-goals, so that
the agent could not reach the final goal.

###  4.3 Simple 2D grid environment

Fig 5 shows the visualization of the path that is moved by the agent of the
policy network without (a) and with (b) learning the sub-goals. A color change
from blue to red means that the agent has visited more often. When the policy
network did not learn the sub-goals, the agent of the policy network easily
reached the target point, and the agent mostly moved only between the starting
point and the target point. That is, the agent mostly moved the left and top
areas in the environment because of the location of the target point.

Figure 6: a, The visualization of the path of the agent of the policy network
with learning the sub-goals. The agent could not learn the sub-goals so the
agent had a tendency to just move to the right. b, The visualization of the
path of the agent of the sub-dedicated network with the forward directional
memory editing. The agent was trained using a one-directional route.
Therefore, the agent tried leftward, even though the given sub-goals were
located on the right of the agent.  Figure 7: The visualization of the path of
the agent when the sub-goals were given in the test environment according to
the reward shaping. The top and the bottom of the figure indicate the route of
the agent without and with the reward shaping, respectively. (a), The agent
successfully went through the sub-goals and reached the final goal. (b), In a
round-trip environment, the agent also passed all the sub-goals and came back
to the starting point. With the reward shaping (bottom), the agent arrived at
each sub-goal point and the final goal faster than without the reward shaping
(top).

In addition, if the policy network was trained on the sub-goals using the bi-
directional memory editing, the agent of the policy network could not reach
the target point, as shown in Fig 5.b. At a specific point, if the agent is
trained to go in various directions by the sub-goals, the agent of the policy
network was confused about where it has to go. Then, the agent would fail to
reach the final goal. In the experiment, the agent just moved to the right
area in the environment. This case was repeated every time the network was
trained. Notably, in the experiment, the agent of the policy network without
learning the sub-goals always could reach the final goal. In the test
environment, the agent of the policy network, which learns the sub-goals using
bi-directional editing, also failed to reach the sub-goals, as shown in Fig
6.a. The agent even left the environment as soon as the agent departed, as
shown in Fig 6.a.(left). This is because the agent of the policy network
mainly moved to the right area in the environment due to the confusion.

When the sub-goals dedicated network was trained using the forward directional
memory editing only, the agent failed to reach the sub-goals, as shown in Fig
6.b. It was observed that the agent tried to move in the left direction. The
reason is that the agent of the policy network mainly moved to the leftward
and the upward direction, and the agent is trained using the one-directional
trajectories.

Fig 7 shows the result of learning the sub-goals in the test environments
without (top) and with (bottom) the reward shaping for the shorter path. The
agent was trained from the sub-goals dedicated network using the trajectories,
which were collected from the agent of the policy network. In the test
environment, I set the sub-goals difficultly, even though all the points
including the starting point and the target point were set inversely. The
agent successfully reached all the sub-goals and the target points in the
experiment, as shown in Fig 7.a. That is, the agent was able to be fully
controlled by the sub-goals, in various scenarios. The agent reached the sub-
goals that had never been visited by the agent of the policy network.
Interestingly, in extremely hard environments (round trip tasks), as shown in
Fig 7.b, the agent departed from the starting point and went through the sub-
goals, and then the agent turned halfway point and came back to the starting
point.

Figure 8: The visualization of the path of the agent without and with the
reward shaping for the shorter path in 20 scenarios. The number, located in
the upper of each figure, indicates the number of steps to reach the target
point. The agent of the sub-goals dedicated network arrived at the target
point in all scenarios. With a visual inspection, the path of the agent with
the reward shaping was shorter than without the reward shaping. In the
comparison of the number of steps, it can be seen that the agent with the
reward shaping reaches the target point within a short time than the agent
without the reward shaping.

In addition, with the reward shaping for the shorter path, the agent reached
the target point faster than without the reward shaping, as shown in Fig 7.
With a visual inspection, we can observe that the agent with the reward
shaping reached the sub-goals located diagonally with a shorter distance from
the current location. Moreover, the difference between the number of steps to
reach the target point with and without the reward shaping was significant.

To strongly confirm the performance difference between without and with the
reward shaping of the proposed method, I additionally constructed 20
scenarios. Fig 8 shows the route of the agent with and without the reward
shaping in each scenario. The number in each figure indicates the number of
steps to reach the target point. In all scenarios, the agent went through the
sub-goals and reached the final goal. It can be seen that the agent was able
to be fully under control by the sub-goals. Moreover, except for three
scenarios, the scenario 14, 15, and 16, the agent with the reward shaping
reached the target point much faster in all scenarios. This characteristic was
salient in complex environments such as the scenario 1 and 2. The average of
the steps with the reward shaping was 338.25 and the average of the steps
without the reward shaping was 411.45. It was confirmed that the rewards
shaping can shorten the number of steps to arrive at the destination by 21.6.

###  4.4 Key-door domain

The agent of the (ASIL + RND) reached the final stage within 50,000 episodes
only one time out of 10 trials in the ‘key-door domain’ environment. It was a
very difficult environment to clear. This was because of the condition to
clear each stage and the sparseness of the reward. I assumed the two scenarios
in the test environment. Two sub-goals were imposed on the agent differently
in each stage. The bonus point and the goal point are also given as the sub-
goals. The agent was enforced to reach the sub-goals at first and after that,
the agent was encouraged to reach the bonus point and the target point.

Figure 9: The visualization of the path of the agent in the key-door domain
environment. a, The path of the agent of the policy network in each stage,
when the agent cleared all stages. The agent of the policy network cleared
only one time out of 10 trials in total. b, The path of the agent of the
policy network in each stage, when the agent did not clear all stages. c
$\sim$ d, The path of the agent of the sub-goals dedicated network, when the
sub-goals were given. The sub-goals dedicated network was used, when the agent
of the policy network failed to clear all stages.

Fig 9.a shows the visualization of the path of the agent, success case of
learning the desired policy. If the policy network converges to the desired
policy, the agent of the network showed a clear path to the goal of Stage 4.
However, in the experiment, the agent of the policy network almost failed to
reach the final stage in the training environment within 50,000 episodes. Fig
9.b shows the visualization of the path of the agent, when the agent of the
policy network failed to learn the desired policy. The agent could reach Stage
4, but the agent failed to clear the stage. In contrast, the agent of the sub-
goals dedicated network was able to reach the goal in the final stage, going
through the sub-goals and the bonus point in the two scenarios as shown in Fig
9.c $\sim$ d, even though the agent of the policy network failed, as shown in
Fig 9.b. This result means that learning the sub-goals can improve the ability
of the agent to reach the final goal, just like in previous studies. Unlike
the previous study, I set the mission for the agent’s performance to the sub-
goals dedicated network in order to apply in the path planning problem, in
which the ability to move in various directions is necessary. Indeed, in the
experiments, the last 300 episodes of the policy network were enough to learn
the sub-goals dedicated network so that the agent could be under control. We
do not need to collect a number of trajectories to learn the agent in the path
planning, if we utilize the proposed method.

However, the agent of the sub-goals dedicated network did not show the
shortest path whereas the agent of the policy network showed the almost
shortest path. Furthermore, like a previous study [21], sometimes, the agent
that learned the sub-goals was confused when the bonus point was near the
current location of the agent. The reason is that the agent was trained to go
through the bonus point, at first, to clear each stage. That is, the agent was
enforced to move to the sub-goals in learning the sub-goals, and the agent was
also encouraged to move to the bonus point and the target point. When the
agent was near the bonus point, the agent could get a larger reward, although
the agent did not reach the sub-goals. The agent was partially able to be
controllable in a complex environment with various variables that have a great
deal of influence on the agent. These remaining issues call for further
studies.

##  5 Conclusion

In this paper, I propose a novel RL framework within which the agent can be
under control in the path planning environment so that the agent can reach
various sub-goals. The agent that completed the learning can perform the
difficult missions such as a round trip, even the agent can reach an unknown
area. Therefore, the bi-directional memory editing and the sub-goals dedicated
network were presented on the goal-conditioned RL. From the bi-directional
memory editing, we can obtain various sub-goals and behaviors of the agent
such that the agent can be more robust in the test environment. In addition,
using the sub-goals dedicated network, the agent can perform several behaviors
that are directed by different sub-goals at one point. It was confirmed that
the agent can be fully controlled and can achieve various sub-goals that are
customized by the users in the test environment. Furthermore, the proposed
reward shaping for the shorter path can improve the ability of the path
planning.

However, in a complex environment with various variables such as the key-door
domain, the agent was confused about whether to select the sub-goals or the
bonus point. Although a fully controllable agent is useful in the path
planning, various variables such as an obstacle and the limited number of
steps in an environmental setting should be considered. Further, the reward
shaping cannot guarantee the optimal path. Future studies are required for a
fully controllable agent for the path planning. I expect that this study would
be applied and studied in a variety of domains with a fully controllable agent
in various scenarios.

## Acknowledgments

## References

  * [1] J. Yu, Y. Su, and Y. Liao, “The path planning of mobile robot by neural networks and hierarchical reinforcement learning,” _Frontiers in Neurorobotics_ , p. 63, 2020.
  * [2] R. Kumar, L. Singh, and R. Tiwari, “Comparison of two meta–heuristic algorithms for path planning in robotics,” in _2020 International Conference on Contemporary Computing and Applications (IC3A)_. IEEE, 2020, pp. 159–162.
  * [3] J. Wöhlke, F. Schmitt, and H. van Hoof, “Hierarchies of planning and reinforcement learning for robot navigation,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 10 682–10 688.
  * [4] V. Jeauneau, L. Jouanneau, and A. Kotenkoff, “Path planner methods for uavs in real environment,” _IFAC-PapersOnLine_ , vol. 51, no. 22, pp. 292–297, 2018\.
  * [5] H. Qie, D. Shi, T. Shen, X. Xu, Y. Li, and L. Wang, “Joint optimization of multi-uav target assignment and path planning based on multi-agent reinforcement learning,” _IEEE access_ , vol. 7, pp. 146 264–146 272, 2019\.
  * [6] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, and L. Hanzo, “Multi-agent deep reinforcement learning-based trajectory planning for multi-uav assisted mobile edge computing,” _IEEE Transactions on Cognitive Communications and Networking_ , vol. 7, no. 1, pp. 73–84, 2020.
  * [7] B. Wang, Z. Liu, Q. Li, and A. Prorok, “Mobile robot path planning in dynamic environments through globally guided reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 6932–6939, 2020.
  * [8] S. Hayat, E. Yanmaz, C. Bettstetter, and T. X. Brown, “Multi-objective drone path planning for search and rescue with quality-of-service requirements,” _Autonomous Robots_ , vol. 44, no. 7, pp. 1183–1198, 2020.
  * [9] H. Bayerlein, M. Theile, M. Caccamo, and D. Gesbert, “Multi-uav path planning for wireless data harvesting with deep reinforcement learning,” _IEEE Open Journal of the Communications Society_ , vol. 2, pp. 1171–1187, 2021.
  * [10] G. T. Lee and C. O. Kim, “Autonomous control of combat unmanned aerial vehicles to evade surface-to-air missiles using deep reinforcement learning,” _IEEE Access_ , vol. 8, pp. 226 724–226 736, 2020.
  * [11] J. Hu, L. Wang, T. Hu, C. Guo, and Y. Wang, “Autonomous maneuver decision making of dual-uav cooperative air combat based on deep reinforcement learning,” _Electronics_ , vol. 11, no. 3, p. 467, 2022.
  * [12] P. Wang, S. Gao, L. Li, B. Sun, and S. Cheng, “Obstacle avoidance path planning design for autonomous driving vehicles based on an improved artificial potential field algorithm,” _Energies_ , vol. 12, no. 12, p. 2342, 2019.
  * [13] M. Tammvee and G. Anbarjafari, “Human activity recognition-based path planning for autonomous vehicles,” _Signal, Image and Video Processing_ , vol. 15, no. 4, pp. 809–816, 2021.
  * [14] Q. Yao, Z. Zheng, L. Qi, H. Yuan, X. Guo, M. Zhao, Z. Liu, and T. Yang, “Path planning method with improved artificial potential field—a reinforcement learning perspective,” _IEEE Access_ , vol. 8, pp. 135 513–135 523, 2020\.
  * [15] Z. Qiao, Z. Tyree, P. Mudalige, J. Schneider, and J. M. Dolan, “Hierarchical reinforcement learning method for autonomous vehicle behavior planning,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 6084–6089.
  * [16] G. Lin, L. Zhu, J. Li, X. Zou, and Y. Tang, “Collision-free path planning for a guava-harvesting robot based on recurrent deep reinforcement learning,” _Computers and Electronics in Agriculture_ , vol. 188, p. 106350, 2021.
  * [17] Z. Cao, S. Xu, H. Peng, D. Yang, and R. Zidek, “Confidence-aware reinforcement learning for self-driving cars,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021.
  * [18] H. Li, Q. Zhang, and D. Zhao, “Deep reinforcement learning-based automatic exploration for navigation in unknown environment,” _IEEE transactions on neural networks and learning systems_ , vol. 31, no. 6, pp. 2064–2076, 2019\.
  * [19] Y. Yang, M. A. Bevan, and B. Li, “Efficient navigation of colloidal robots in an unknown environment via deep reinforcement learning,” _Advanced Intelligent Systems_ , vol. 2, no. 1, p. 1900106, 2020.
  * [20] J. Hu, H. Niu, J. Carrasco, B. Lennox, and F. Arvin, “Voronoi-based multi-robot autonomous exploration in unknown environments via deep reinforcement learning,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 12, pp. 14 413–14 423, 2020.
  * [21] G. Lee, “Learning user-defined sub-goals using memory editing in reinforcement learning,” _arXiv preprint arXiv:2205.00399_ , 2022.
  * [22] L. Lee, B. Eysenbach, R. R. Salakhutdinov, S. S. Gu, and C. Finn, “Weakly-supervised reinforcement learning for controllable behavior,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 2661–2673, 2020.
  * [23] T. Okudo and S. Yamada, “Subgoal-based reward shaping to improve efficiency in reinforcement learning,” _IEEE Access_ , vol. 9, pp. 97 557–97 568, 2021\.
  * [24] L. Zhang, G. Yang, and B. C. Stadie, “World model as a graph: Learning latent landmarks for planning,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 12 611–12 620.
  * [25] E. Chane-Sane, C. Schmid, and I. Laptev, “Goal-conditioned reinforcement learning with imagined subgoals,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 1430–1440.
  * [26] R. Yang, Y. Lu, W. Li, H. Sun, M. Fang, Y. Du, X. Li, L. Han, and C. Zhang, “Rethinking goal-conditioned supervised learning and its connection to offline rl,” _arXiv preprint arXiv:2202.04478_ , 2022.
  * [27] C. Bai, P. Liu, W. Zhao, and X. Tang, “Guided goal generation for hindsight multi-goal reinforcement learning,” _Neurocomputing_ , vol. 359, pp. 353–367, 2019.
  * [28] R. Zhao, X. Sun, and V. Tresp, “Maximum entropy-regularized multi-goal reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 7553–7562.
  * [29] D. M. Bernstein and E. F. Loftus, “How to tell if a particular memory is true or false,” _Perspectives on Psychological Science_ , vol. 4, no. 4, pp. 370–374, 2009.
  * [30] D. L. Schacter, S. A. Guerin, and P. L. S. Jacques, “Memory distortion: An adaptive perspective,” _Trends in cognitive sciences_ , vol. 15, no. 10, pp. 467–474, 2011.
  * [31] J. Fernández, “What are the benefits of memory distortion?” 2015.
  * [32] E. A. Phelps and S. G. Hofmann, “Memory editing from science fiction to clinical practice,” _Nature_ , vol. 572, no. 7767, pp. 43–50, 2019.
  * [33] C. Yan and X. Xiang, “A path planning algorithm for uav based on improved q-learning,” in _2018 2nd International Conference on Robotics and Automation Sciences (ICRAS)_. IEEE, 2018, pp. 1–5.
  * [34] J. Chen, M. Li, Z. Yuan, and Q. Gu, “An improved a* algorithm for uav path planning problems,” in _2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC)_ , vol. 1. IEEE, 2020, pp. 958–962.
  * [35] H. Zhou, H.-L. Xiong, Y. Liu, N.-D. Tan, and L. Chen, “Trajectory planning algorithm of uav based on system positioning accuracy constraints,” _Electronics_ , vol. 9, no. 2, p. 250, 2020.
  * [36] C. Huang and J. Fei, “Uav path planning based on particle swarm optimization with global best path competition,” _International Journal of Pattern Recognition and Artificial Intelligence_ , vol. 32, no. 06, p. 1859008, 2018.
  * [37] J. Chen, H. Zhao, and L. Wang, “Three dimensional path planning of uav based on adaptive particle swarm optimization algorithm,” in _Journal of Physics: Conference Series_ , vol. 1846, no. 1. IOP Publishing, 2021, p. 012007.
  * [38] X. Wang, C. Huang, and F. Chen, “An improved particle swarm optimization algorithm for unmanned aerial vehicle route planning,” in _Journal of Physics: Conference Series_ , vol. 2245, no. 1. IOP Publishing, 2022, p. 012013.
  * [39] V. Jamshidi, V. Nekoukar, and M. H. Refan, “Analysis of parallel genetic algorithm and parallel particle swarm optimization algorithm uav path planning on controller area network,” _Journal of Control, Automation and Electrical Systems_ , vol. 31, no. 1, pp. 129–140, 2020.
  * [40] X. Lei, Z. Zhang, and P. Dong, “Dynamic path planning of unknown environment based on deep reinforcement learning,” _Journal of Robotics_ , vol. 2018, 2018.
  * [41] X. Liu, D. Zhang, T. Zhang, Y. Cui, L. Chen, and S. Liu, “Novel best path selection approach based on hybrid improved a* algorithm and reinforcement learning,” _Applied Intelligence_ , vol. 51, no. 12, pp. 9015–9029, 2021\.
  * [42] C. Yan, X. Xiang, and C. Wang, “Towards real-time path planning through deep reinforcement learning for a uav in dynamic environments,” _Journal of Intelligent & Robotic Systems_, vol. 98, no. 2, pp. 297–309, 2020.
  * [43] C. Chen, X.-Q. Chen, F. Ma, X.-J. Zeng, and J. Wang, “A knowledge-free path planning approach for smart ships based on reinforcement learning,” _Ocean Engineering_ , vol. 189, p. 106299, 2019.
  * [44] S. Guo, X. Zhang, Y. Zheng, and Y. Du, “An autonomous path planning model for unmanned ships based on deep reinforcement learning,” _Sensors_ , vol. 20, no. 2, p. 426, 2020.
  * [45] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight experience replay,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [46] H. Nguyen, H. M. La, and M. Deans, “Hindsight experience replay with experience ranking,” in _2019 Joint IEEE 9th International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob)_. IEEE, 2019, pp. 1–6.
  * [47] M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang, “Curriculum-guided hindsight experience replay,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [48] Y. Lai, W. Wang, Y. Yang, J. Zhu, and M. Kuang, “Hindsight planner,” in _Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems_ , 2020, pp. 690–698.
  * [49] S. Nasiriany, V. Pong, S. Lin, and S. Levine, “Planning with goal-conditioned policies,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [50] D. Ghosh, A. Gupta, A. Reddy, J. Fu, C. Devin, B. Eysenbach, and S. Levine, “Learning to reach goals via iterated supervised learning,” _arXiv preprint arXiv:1912.06088_ , 2019.
  * [51] A. V. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine, “Visual reinforcement learning with imagined goals,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [52] B. Eysenbach, R. R. Salakhutdinov, and S. Levine, “Search on the replay buffer: Bridging planning and reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [53] B. Eysenbach, R. Salakhutdinov, and S. Levine, “C-learning: Learning to achieve goals via recursive classification,” _arXiv preprint arXiv:2011.08909_ , 2020.
  * [54] J. Kim, Y. Seo, and J. Shin, “Landmark-guided subgoal generation in hierarchical reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [55] O. Nachum, S. S. Gu, H. Lee, and S. Levine, “Data-efficient hierarchical reinforcement learning,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [56] N. Gürtler, D. Büchler, and G. Martius, “Hierarchical reinforcement learning with timed subgoals,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [57] J. Oh, Y. Guo, S. Singh, and H. Lee, “Self-imitation learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 3878–3887.
  * [58] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _International conference on machine learning_. PMLR, 2016, pp. 1928–1937.
  * [59] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [60] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski _et al._ , “Human-level control through deep reinforcement learning,” _nature_ , vol. 518, no. 7540, pp. 529–533, 2015.
  * [61] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” _arXiv preprint arXiv:1511.05952_ , 2015.
  * [62] H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in _Proceedings of the AAAI conference on artificial intelligence_ , vol. 30, no. 1, 2016.
  * [63] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” _arXiv preprint arXiv:1810.12894_ , 2018.

|  GyeongTaek Lee received a Ph.D degree in industrial engineering from Yonsei
University, South Korea, in 2022. He is a visiting researcher in Department of
Industrial and Systems Engineering from Rutgers University, The State
University of New Jersey, NJ, USA. His current research interests include
reinforcement learning and manufacturing data science.
---|---


# Chaos Removal in the $R+qR^{2}$ gravity: the Mixmaster model

Riccardo Moriconi<EMAIL_ADDRESS>Dipartimento di Fisica, Università di
Napoli “Federico II”, Compl. Univ. di Monte S. Angelo, Edificio G, Via
Cinthia, I-80126, Napoli, Italy  INFN Sezione di Napoli, Compl. Univ. di Monte
S. Angelo, Edificio G, Via Cinthia, I-80126, Napoli, Italy  Giovanni Montani
<EMAIL_ADDRESS>Dipartimento di Fisica, Università degli
studi di Roma ”La Sapienza”, P.le A. Moro 5 (00185) Roma, Italy  ENEA, Unità
Tecnica Fusione, ENEA C. R. Frascati, via E. Fermi 45, 00044 Frascati (Roma),
Italy  Salvatore Capozziello<EMAIL_ADDRESS>Dipartimento di Fisica,
Università di Napoli “Federico II”, Compl. Univ. di Monte S. Angelo, Edificio
G, Via Cinthia, I-80126, Napoli, Italy  INFN Sezione di Napoli, Compl. Univ.
di Monte S. Angelo, Edificio G, Via Cinthia, I-80126, Napoli, Italy  Gran
Sasso Science Institute (INFN), Via F. Crispi 7, I-67100, L’ Aquila, Italy

###### Abstract

We study the asymptotic dynamics of the Mixmaster Universe, near the
cosmological singularity, considering $f(R)$ gravity up to a quadratic
corrections in the Ricci scalar $R$. The analysis is performed in the scalar-
tensor framework and adopting Misner-Chitré-like variables to describe the
Mixmaster Universe, whose dynamics resembles asymptotically a billiard-ball in
a given domain of the half-Poincaré space. The form of the potential well
depends on the spatial curvature of the model and on the particular form of
the self-interacting scalar field potential. We demonstrate that the potential
walls determine an open domain in the configuration region, allowing the
point-Universe to reach the absolute of the considered Lobachevsky space. In
other words, we outline the existence of a stable final Kasner regime in the
Mixmaster evolution, implying the chaos removal near the cosmological
singularity. The relevance of the present issue relies both on the general
nature of the considered dynamics, allowing its direct extension to the BKL
conjecture too, as well as the possibility to regard the considered modified
theory of gravity as the first correction to the Einstein-Hilbert action as a
Taylor expansion of a generic function $f(R)$ (as soon as a cut-off on the
space-time curvature takes place).

###### pacs:

04.50.Kd, 04.60.Kz

## Introduction

The chaotic dynamics of the Mixmaster Universe BKLref1 ,mixmastermisner
,misner  is a basic prototype of the local (sub-horizon) behaviour of the
generic cosmological solution (the so-called BKL conjectureBKLref2 ).
Investigating the stability of such a chaotic picture with respect to the
presence of matter scalar field ,montanireview ,primordial  and space-time
dimensions numberhennaux1 ,hennaux2 ,del1  has seen a great effort over the
last four decades and the most significant issue was the proof of the chaos
removal when a massless scalar field is involved in the dynamics scalar field
. Such a result is a consequence of the capability manifested by the scalar
field kinetic energy of affecting the second (quadratic) Kasner condition,
easily restated in the Hamiltonian picture, as shown in berger . This property
of the massless scalar field acquires intriguing perspectives when $f(R)$
modified theory of gravity are considered capfirst ,capozzielloreview
,odintsovreview ,odintsov ,guo . In fact, these alternative formulation of the
gravitational field dynamics can be represented by an equivalent scalar-tensor
picture: the scalar degree of freedom associated to the form of the function
$f$ is expressed via a self-interacting scalar field, coupled to the ordinary
General Relativity bar1 ,bar2 ,bar3 ,del2 . When implementing this scalar-
tensor scheme to the Mixmaster Universe dynamics, a natural question arise:
the kinetic term of the scalar field removes the chaotic behaviour, but the
presence of a potential term could restore it? Thus we can study, for specific
modified theories of gravity, if the Mixmaster chaos survives or not, simply
characterizing the corresponding scalar field potential. Here we analyse the
modified gravity theory corresponding to a quadratic correction in the Ricci
scalar to the ordinary Einstein-Hilbert Lagrangian, both because it is the
simplest viable deviation from General Relativity (apart from a cosmological
constant term), as well as the first correction emerging from a Taylor
expansion of a $f(R)$ theory for very small values of the space-time Ricci
scalar, i.e. for very law curvatures, like we observe today in the Solar
Systemstaro1 . The quadratic term in the Ricci scalar provides an exponential-
like potential term for the self-interacting scalar field, when a scalar-
tensor reformulation of the model is considered. This case is particularly
appropriate to the analysis we pursue of the Mixmaster dynamics in terms of
the Misner-Chitré-like variables primordial ,montani-kirillov ,imponente-
montani ,MCthesis . In fact, the kinetic term of the scalar field is on the
same footing of the anisotropy term contribution and, for the considered
Lagrangian, also the potential term is isomorphic to the spatial curvature of
the model, i.e. the total potential term is constituted by equivalent
exponential profile. In the asymptotic limit toward the initial singularity
the total potential takes the form of four potential walls, whose morphology
determines if the configuration domain is closed or not. Indeed, we
demonstrate how the whole domain, available in principle, is a constant
negative curvature space (half-Poincarè space). We first analyse the case of
the Mixmaster Universe in the presence of a massless scalar field,
demonstrating the open nature of its configuration space and the implied
existence of a stable Kasner regime to the initial singularity. Then, we face
a detailed study of the dynamics in the presence of the total potential and
the still open structure of the configuration domain. Thus, we demonstrate the
non-chaotic nature of the Mixmaster Universe behaviour, as it is described by
the scalar-tensor version of the $R^{2}$-gravity.

##  $f(R)$ gravity

The $f(R)$ theories of gravity are a direct generalization of the Einstein-
Hilbert Lagrangian consisting in a replacement of the Ricci Scalar $R$ by a
general function $f(R)$capozzielloreview ,odintsovreview ,odintsov ,odintsov2
:

$S=\frac{1}{16\pi}\int d^{4}x\sqrt{-g}f(R)$ (1)

where $g$ is the determinant of the metric111We use the $(-,+,+,+)$ signature
of the metric and the geometric unit system $(c=G=\hbar=1)$.. The introduction
of the additional degree of freedom, related to the presence of the $f(R)$
term, can be translated into a dynamics of a self-interacting scalar field
coupled with the Einstein-Hilbert Action, the so-called Scalar-Tensor
framework. In this approach, a new auxiliary field $\chi$ is introduced to get
the following equivalent version of the action (1):

$S=\frac{1}{16\pi}\int d^{4}x\sqrt{-g}[f(\chi)-f^{\prime}(\chi)(R-\chi)].$ (2)

The variation of the action (2) with respect to $\chi$ provides
$f^{\prime\prime}(\chi)(R-\chi)=0$, implying $\chi=R$ if
$f^{\prime\prime}(\chi)\neq 0$. By a redefinition of the auxiliary field
$\chi$ in the form $\varphi=f^{\prime}(\chi)$ the action becomes

$S=\frac{1}{16\pi}\int d^{4}x\sqrt{-g}[\varphi
R-\chi(\varphi)\varphi+f(\chi(\varphi))].$ (3)

It is now possible to perform a conformal transformation on the metric
$g_{\mu\nu}\rightarrow\tilde{g_{\mu\nu}}=\varphi g_{\mu\nu}$ and a scalar
field redefinition $\varphi\equiv
f^{\prime}(R)\rightarrow\phi=\sqrt{\frac{3}{16\pi}}\ln f^{\prime}(R)$ in order
to obtain

$S=\int
d^{4}x\sqrt{-\tilde{g}}\left[\frac{\tilde{R}}{16\pi}-\frac{1}{2}\partial^{\alpha}\phi\partial_{\alpha}\phi-U(\phi)\right],$
(4)

where the potential term $U(\phi)$ has the form:

$U(\phi)=\frac{Rf^{\prime}(R)-f(R)}{16\pi(f^{\prime}(R))^{2}}.$ (5)

For small values of the Ricci scalar, the first order correction to the
Einstein-Hilbert Lagrangian, is represented by a quadratic correction, i.e.

$f(R)=R+qR^{2}.$ (6)

By this choice, the potential term (5) takes the form

$U(\phi)=\frac{1}{64\pi
q}\left(1-2\exp^{-4\sqrt{\frac{\pi}{3}}\phi}+\exp^{-8\sqrt{\frac{\pi}{3}}\phi}\right).$
(7)

This is the effective potential that emerges in the so called Starobinsky-
inflation modelstaro1 . Such a model ensures a ”slow-rolling” face and it is
an inflationary model passing the latest inflation constraintplanck .

## The Mixmaster model dynamics

Following the standard representation of the Bianchi IX modelgravitation  in
the Misner variablesmixmastermisner ,misner  the Einstein-Hilbert action takes
the form:

$S_{g}=\int
dt\left[p_{\alpha}\dot{\alpha}+p_{+}\dot{\beta_{+}}+p_{-}\dot{\beta_{-}}-\frac{Ne^{-3\alpha}}{24\pi}\mathcal{H}_{IX}\right],$
(8)

where the dynamics of the model implies the superHamiltonian constraint

$\mathcal{H}_{IX}\equiv-
p_{\alpha}^{2}+p_{+}^{2}+p_{-}^{2}+12\pi^{2}e^{4\alpha}V_{IX}(\beta_{\pm})=0.$
(9)

Here $\alpha$ expresses the isotropic component of the Universe(i.e. the
volume of the universe) and the initial singularity is reached for
$\alpha\rightarrow-\infty$, while the traceless matrix
$\beta_{ab}=diag(\beta_{+}+\sqrt{3}\beta_{-},\beta_{+}-\sqrt{3}\beta_{-},-2\beta_{+})$
accounts for the anisotropy of this model. Furthermore, $p_{\alpha},p_{\pm}$
are the conjugated momenta to $\alpha,\beta_{\pm}$ respectively and
$V_{IX}(\beta_{\pm})$ is the potential term depending only on $\beta_{\pm}$,
corresponding to the spatial curvature. If we execute an ADM reduction of the
dynamicsADM , the Bianchi IX model resembles the behaviour of a two-
dimensional particle, evolving with respect to the time-like variable $\alpha$
in the $\beta_{+},\beta_{-}$ plane. By other words, the system dynamics is
summarized by the time-dependent hamiltonian $H_{IX}$:

$-p_{\alpha}=H_{IX}\equiv\sqrt{p_{+}^{2}+p_{-}^{2}+12\pi^{2}e^{4\alpha}V_{IX}(\beta_{\pm})}.$
(10)

Looking at the form of the potential term $V_{IX}(\beta_{\pm})$, it is
possible, taking into account the three leading terms, to parametrize it as an
infinitely steep potential wellmisner . This way, the point-Universe lives
inside the triangular region of the configuration space where the potential
term is negligible; such a region it is individuate when the following three
conditions hold:

$\begin{split}&\frac{1}{3}+\frac{\beta_{+}+\sqrt{3}\beta_{-}}{3\alpha}>0,\\\
&\frac{1}{3}+\frac{\beta_{+}-\sqrt{3}\beta_{-}}{3\alpha}>0,\\\
&\frac{1}{3}-\frac{2\beta_{+}}{3\alpha}>0.\end{split}$ (11)

The presence of the “time” variable $\alpha$ in the relations
(LABEL:anisotropyMisner) causes the outside motion of the potential walls and
the corresponding time-dependence of the domain allowed to the point-Universe
motion. Such a dependence can be removed in the framework of the Misner-Cithrè
variables $\tau,\zeta,\theta$gravitation ,MCthesis  as standing in the
Poincare Half-Plane:

$\begin{split}&\alpha-\alpha_{0}=-e^{\tau}\frac{1+u+u^{2}+v^{2}}{\sqrt{3}v},\\\
&\beta_{+}=e^{\tau}\frac{-1+2u+2u^{2}+2v^{2}}{2\sqrt{3}v}\\\
&\beta_{-}=e^{\tau}\frac{-1-2u}{2v}.\end{split},$ (12)

where $-\infty<\tau<\infty$, $-\infty<u<+\infty$, $0<v<+\infty$. In this
scheme the role of the hamiltonian time is assigned to $\tau$ and the
singularity is approach for $\tau\rightarrow\infty$. The transformations
(LABEL:MCtrans) permit to rewrite the conditions (LABEL:anisotropyMisner) as
independent of the variable $\tau$ and thus the domain within which the
particle lives is fixed respect to the time variable. Making use of the
transformations (LABEL:MCtrans), the Hamiltonian (10 in the free-potential
case rewrites as

$-p_{\tau}=H_{IX}\equiv\sqrt{v^{2}(p_{u}^{2}+p_{v}^{2})},$ (13)

and the point-Universe lives in the $u,v$ plane inside the region individuate
when the following three conditions hold:

$\begin{split}&\frac{-u}{1+u+u^{2}+v^{2}}>0,\\\
&\frac{1+u}{1+u+u^{2}+v^{2}}>0,\\\
&\frac{u(u+1)+v^{2}}{1+u+u^{2}+v^{2}}>0.\end{split}$ (14)

As shown by imponente-montani , the asymptotic evolution towards the
singularity is covariantly chaotic because it is isomorphic to a billiard on
the Lobachevsky plane. This demonstration is based on three points:i)the
Jacobi metric in the $u,v$ plane has a negative constant curvature; ii)the
Lyapunov exponent, defined as in pesin , are greater than zero; iii)the
configuration space is (dinamically) compact. The occurrence of the these
three properties ensures that the geodesic trajectories cover the whole
configuration space.

## Mixmaster Universe in the $R^{2}$-gravity

Now we analyse the case of the gravitational Lagrangian (6) when the Bianchi
IX model is considered. As starting point we consider the modified gravity
model (4) in terms of the variables $\alpha,\beta_{+},\beta_{-},\phi$.
Following the same procedure of the previous section we get the generalized
reduced hamiltonian $-p_{\alpha}=H$ of the form

$H\equiv\sqrt{p_{+}^{2}+p_{-}^{2}+p_{\phi}^{2}+12\pi^{2}e^{4\alpha}V_{IX}+4e^{6\alpha}U}.$
(15)

In Eq.(15), we rescaled the zero point of
$\alpha\rightarrow\alpha-\alpha_{0}$, so that the spatial metric factor
$e^{3\alpha}\rightarrow\frac{1}{(6\pi)}e^{3\alpha}$, and a redefinition of the
scalar field amplitude $\phi\rightarrow\sqrt{2}(6\pi)\phi$ is considered too.
A natural parametrization, in the Misner-Cithrè - Poincare Half-Plane scheme,
that reduces to the relations (LABEL:MCtrans) if the scalar field is turned-
off, reads as follows

$\begin{split}&\alpha-\alpha_{0}=-e^{\tau}\frac{1+u+u^{2}+v^{2}}{\sqrt{3}v},\\\
&\beta_{+}=e^{\tau}\frac{-1+2u+2u^{2}+2v^{2}}{2\sqrt{3}v},\\\
&\beta_{-}=e^{\tau}\frac{-1-2u}{2v}\cos\delta,\\\
&\phi=e^{\tau}\frac{-1-2u}{2v}\sin\delta,\\\ \end{split}$ (16)

where $-\infty<\tau<\infty$, $-\infty<u<+\infty$ , $0<v<+\infty$ and
$0<\delta<2\pi$. In this new system of variables the reduced Hamiltonian takes
the form:

$-p_{\tau}=H\equiv\sqrt{v^{2}\left[p_{u}^{2}+p_{v}^{2}+4\frac{p_{\delta}^{2}}{(1+2u)^{2}}\right]+e^{2\tau}\mathcal{V}}.$
(17)

The introduction of the degree of freedom related to the scalar field implies
that the point-Universe lives inside a 3-dimensional domain in the
configuration space $u,v,\delta$ determined by the potential term:

$e^{2\tau}\mathcal{V}=e^{2\tau}[12\pi^{2}e^{-4e^{\tau}\xi(u,v)}V_{IX}(u,v,\delta,\tau)+4e^{-6e^{\tau}\xi(u,v)}U(u,v,\delta,\tau)]=\\\
=12\pi^{2}e^{2\tau}\left(e^{-\frac{12e^{\tau}}{\sqrt{3}v}(u+u^{2}+v^{2})}+e^{-\frac{6e^{\tau}}{\sqrt{3}v}(1+(1+2u)\cos\delta)}+e^{-\frac{6e^{\tau}}{\sqrt{3}v}(1-(1+2u)\cos\delta)}\right)+\\\
+\frac{e^{2\tau}}{8\pi
q}\left(e^{-\frac{12e^{\tau}}{\sqrt{3}v}(1+u+u^{2}+v^{2})}-2e^{-\frac{6e^{\tau}}{\sqrt{3}v}(1+u+u^{2}+v^{2}-2\sqrt{2\pi^{3}}(1+2u)\sin\delta)}+e^{-\frac{6e^{\tau}}{\sqrt{3}v}(1+u+u^{2}+v^{2}-4\sqrt{2\pi^{3}}(1+2u)\sin\delta)}\right),$
(18)

where $\xi(u,v)=\frac{1+u+u^{2}+v^{2}}{\sqrt{3}v}$. Due to the exponential
forms of the terms in Eq.(18), when the singularity is approached
($\tau\rightarrow\infty$) the point-Universe is confined to live inside a
3-dimensional domain defined as the region where all the exponents of the six
terms are simultaneously greater than zero. Looking the Eq.(18), the potential
term $\mathcal{V}$ behaves as an infinitely steep potential well as in the
Poincarè variables (LABEL:anisotropyUV). So for the evolution of the point-
Universe it is possible to neglect the potential everywhere in a suitable
domain. As first step we study the case in absence of all the potential terms
($\mathcal{V}=0$), i.e. we deal with the Hamiltonian problem:

$H=v\sqrt{p_{u}^{2}+p_{v}^{2}+4\frac{p_{\delta}^{2}}{(1+2u)^{2}}}.$ (19)

The hamiltonian equations for this potential-free system (Bianchi I model with
the massless scalar field) are

$\begin{split}&\dot{u}=\frac{\partial H}{\partial
p_{u}}=\frac{v^{2}}{\epsilon}p_{u}\quad,\quad\dot{p_{u}}=-\frac{\partial
H}{\partial u}=\frac{8v^{2}}{\epsilon}\frac{p_{\delta}^{2}}{(1+2u)^{3}}\\\
&\dot{v}=\frac{\partial H}{\partial
p_{v}}=\frac{v^{2}}{\epsilon}p_{v}\quad,\quad\dot{p_{v}}=-\frac{\partial
H}{\partial v}=-\frac{\epsilon}{v}\\\ &\dot{\delta}=\frac{\partial H}{\partial
p_{\delta}}=\frac{4v^{2}}{\epsilon}\frac{p_{\delta}}{(1+2u)^{2}}\quad,\quad\dot{p_{\delta}}=-\frac{\partial
H}{\partial\delta}=0.\end{split}$ (20)

It is possible to demonstrate, as we approach the singularity, that $H$ is a
constant of motion with respect to the “time” variable $\tau$, following
imponente-montani . Thus, we perform the substitution $H\simeq\epsilon=const.$
inside Eq’s(LABEL:MCphitrans). It is now possible, by following the Jacobi
procedurearnold  and using the equations of motion (LABEL:MCphitrans), to
write down the line element for the three-dimensional Jacobi metric in terms
of the configuration variables, i.e.

$ds^{2}=\frac{\epsilon}{v^{2}}\left[du^{2}+dv^{2}+\frac{(1+2u)^{2}}{4}d\delta^{2}\right].$
(21)

By a direct calculation we see that this metric has a negative constant
curvature (the associated Ricci scalar is $R=-\frac{6}{\epsilon}$) and then
the point-Universe moves over a negatively curved three-dimensional space.
Furthermore, we can find two singular values for the metric in correspondence
to $u=-\frac{1}{2}$, $v=0$. This feature allows us to restrict the domain of
the configuration space in which we will study the trajectories of the point-
Universe to the fundamental one identified by the inequalities
$-\frac{1}{2}<u<+\infty$, $0<v<+\infty$, $0<\delta<2\pi$. Indeed there is no
way for the point-Universe trajectories to cross over the two planes
$u=-\frac{1}{2}$, $v=0$ (each choice of the Lobachevsky “half-space” is
equivalent respect to the other one). The intermediate step toward the general
case of the potential (18), corresponding to the ordinary Mixmaster model in
the presence of a massless scalar field, takes place when we retain only the
exponential terms due to the spatial curvature, namely $\mathcal{V}\simeq
12\pi^{2}e^{-4e^{\tau}\xi(u,v)}V_{IX}(u,v,\delta,\tau)$. Then, the point-
Universe lives in the region where are simultaneously satisfied the three
following conditions

$\begin{cases}1+(1+2u)\cos\delta>0,\\\ 1-(1+2u)\cos\delta>0,\\\
u(u+1)+v^{2}>0.\end{cases}$ (22)

Figure 1:  The black lines represent the trajectories associated to a points-
Universe that bounce against the walls. Instead, the red lines describe the
points-Universe witch directly approach the singularity.

We now implement a numerical integration of the system (LABEL:MCphitrans) in
order to analyse the behaviour of the trajectories in the potential free
region and then use this result for interpreting the effect of the scalar
curvature. As we can see in the Fig.1 an opening of the domain emerges due to
the presence of the scalar field and it is possible to individuate two
families of trajectories: those ones corresponding to a point-Universe that
bounces against the walls and turn back inside the domain (the black ones) and
those corresponding to a particle that approach the so called
“absolute”kirillov1997  (the red ones), for values $v\rightarrow 0,\infty$,
with no other bounces until the singularity. The presence of the trajectories
of the second family shows the removal of the oscillatory behavior of the
Mixmaster model coupled with a massless scalar field BKLref2 ,berger . Let us
see what happen if we consider the complete potential term(18). This time the
restrictions on the dynamics imply that the particle is confined inside a
region where all the six exponential terms in Eq.(18) are simultaneously
greater than zero. We can immediately remove one of the six conditions because
the first exponent related to the potential of the scalar field
$1+u+u^{2}+v^{2}$ is always greater than zero for any values of $u,v$ taking
in consideration. Thus, the five conditions that identify the domain are

$\begin{cases}1+(1+2u)\cos\delta>0\\\ 1-(1+2u)\cos\delta>0\\\
u(u+1)+v^{2}>0\\\ 1+u+u^{2}+v^{2}-2\sqrt{2\pi^{3}}(1+2u)\sin\delta>0\\\
1+u+u^{2}+v^{2}-4\sqrt{2\pi^{3}}(1+2u)\sin\delta>0\end{cases}$ (23)

We observe that the last of the conditions above naturally implies the
validity of the fourth one too. Thus, we indeed deal with four potential walls
only. As we can see in the Fig.2, taking into account also the potential term
$U(u,v,\delta,\tau)$ implies that the available configuration space for the
point-Universe is clearly reduced with respect to the case $U=0$ (see Fig.1).
However, trajectories yet exist(the red lines in the Fig.2) corresponding to a
point-Universe that is able to reach the absolute for $v\rightarrow 0,\infty$.
For this reason we can firmly conclude that a quadratic correction in the
Ricci scalar to the Einstein-Hilbert action, that in the Scalar-Tensor theory
is equivalent to the dynamics of a self-interacting scalar field (with
potential terms of the form (7)), is able to remove the never-ending bounces
of the point-Universe against the walls. As a result of the bounces against
the infinite potential walls (which can be described by a reflection
rulemontani-kirillov ,indicikasner ), soon or later the point-Universe reach a
trajectory connected with the absolute. It is worth noting that the analysis
above is referred to the choice $q>0$, in which case the sign of the scalar
field potential is the same one of the scalar curvature. This choice is forced
by the request that the additional scalar mode, associated to the quadratic
modification, be a real (non-tachyonic) massive one, accordingly to the
original Starobinsky approach in staro1  and demonstrated also in segnoq .
However, in the case $q<0$, the scalar field potential would not contribute an
infinite positive wall, but an infinite depression. Since in the region of
zero potential, the point-Universe has always positive “energy”, we can easily
conclude that such a case overlaps the non-chaotic potential-free one. We now
observe that, in correspondence to the configuration region $v=0,\infty$ and
$\delta=\frac{3\pi}{2}$, the scalar field acquires negative diverging values
and its potential terms manifests a diverging behaviour. Such a profile of the
scalar field is typical of a Bianchi I solution near the singularity scalar
field  and the diverging character of the potential term means that General
Relativity can not be asymptotically recovered. Rigorously speaking, the
present result on the chaos structure applies to a quadratic correction in the
Ricci scalar only, because it is the first terms of the Taylor expansion of
the function $f(R)$ working nearby the singularity. Nonetheless, our analysis
has a general validity, as soon as, we take into account a physical cut-off at
the Planck time, where classical theory starts to fail and a quantum treatment
is required. In fact, the Planckian cut-off would remove the $\phi$ and
$U(\phi)$ divergences, allowing the Taylor expansion for
$q\lesssim\frac{(ct_{cut})^{2}}{l_{P}^{2}}$, where $t_{cut}$ being the cut-off
time and $l_{P}$ the Planck length. Since $t_{cut}>\frac{l_{p}}{c}$, we deal
with the (non-severe) restriction $q\lesssim 1$ for preserving the general
nature of our result. This estimation follows requiring $R>qR^{2}$ and
remembering that for the case of a Kasner solution, in the presence of a
potential-free scalar field, the Ricci scalar behaves as
$R\sim\frac{1}{t_{s}^{2}}$, where $t_{s}$ is the synchronous time. We stress
that qualitatively, a similar argument is at the ground of the non-chaotic
nature of the Bianchi IX Loop Quantum dynamics in the semi-classical limit boj
. However, the field $\phi(t_{s})$ admits, both for $v\rightarrow 0,\infty$
and $\delta=\pi/2$, trajectories implying its positive divergence. For such
behaviours, corresponding to an open region in the initial condition, the
potential $U(\phi)$ approaches a constant value and $\phi$ is effectively
massless. It is just the existence of these diverging profiles at the ground
of the chaos removal in the present model. The massless nature of the
potential along specific trajectories is a good criterion for determining the
chaotic properties of the Mixmaster Universe in a specific non-expanded $f(R)$
model. In fact, The behavior of the free scalar field reads
$\phi_{f}(t_{s})\propto\ln t_{s}$ and the corresponding kinetic energy density
stands as $1/t_{s}^{2}$. Then, for a given $f(R)$ model, fixing the potential
$U(\phi)$, the chaos removal is ensured by the validity of the condition
$\lim_{t_{s}\rightarrow 0}U(\phi_{f}(t_{s}))t_{s}^{2}=0$. Clearly, the non-
chaoticity is ensured if such a limit holds for a non-zero measure set of
trajectories.

Figure 2:  The point-Universe lives inside the region marked by the walls,
where the conditions (23) are verified. We also sketch the trajectories
reaching the absolute.

## Conclusions

The analysis above demonstrated how including a quadratic correction in the
Ricci scalar to the Einstein-Hilbert Lagrangian of the gravitational field
gives a deep insight on the nature of the Mixmaster singularity: the evolution
of the scale factors is no longer chaotic and a stable Kasner regime emerges
as the final approach to the singular point.

The relevance of this result is in its generality with respect to the
behaviour of the cosmological gravitational field. In fact, on one hand, the
result we derived in the homogeneous cosmological setting, can be naturally
extended to a generic inhomogeneous Universe, simply following the line of
investigation discussed in BKLref2 ,scalar field .

The basic statement, at the ground of the BKL conjecture, is the space point
decoupling in the asymptotic dynamics toward the cosmological singularity.
Such a dynamical property of a generic inhomogeneous cosmological model allows
to reduce the behavior of a sub-horizon spatial region BKLref2 ,M95  to the
prototype offered by the homogeneous Mixmaster Universe. We are actually
stating that the time derivative of the dynamical variables asymptotically
dominate their spatial gradients, limiting the presence of the spatial
coordinates in the Einstein equation to a pure parametrical role. We are
speaking of a conjecture because the chaotic features of the point-like
dynamics induce a corresponding stochastic behaviour of the spatial dependence
and the statement above requires a non-trivial treatment for its proof.
Nonetheless a valuable estimation of the spatial gradient behaviour, when the
space-time takes the morphology of a foam, is provided in K93 . When a scalar
field is present the situation is even more simple, because, after a certain
number of iterations of the BKL map, in each space point, a stable Kasner
regime takes place K87  and the validity of the solution is rigorously
determined LK63 . Thus, we can extend our result to a generic inhomogeneous
cosmological model simply considering the dynamical variables as space-time
functions $u=u(\tau,x^{i})$, $v=v(\tau,x^{i})$ and
$\delta=\delta(\tau,x^{i})$, which, in each space point, live in a half-
Poincar space and are governed by an independent and morphologically
equivalent dynamics. On the other hand, the extension of General Relativity we
considered here is the most simple and natural one, widely studied in
literatures in view of its implications on the primordial Universe features.
Since the classical evolution is expected to be predictive up to a finite
value of the Universe volume, i.e. up to a given amplitude of the space-time
curvature, for sufficiently small coupling constant $q$ values, the present
model can be considered as the quadratic Taylor expansion of a generic $f(R)$
theory and we can then guess that the non-chaotic feature is a very general
dynamical property, at least within the classical domain of validity of the
$f(R)$ theory. In this sense we traced a very general and reliable properties
of the cosmological gravitational field in modified theories of gravity of
significant impact on the so-called billiard representation of the generic
primordial Universehennaux1 ,montani-kirillov ,indicikasner .

## References

  * (1) V.A. Belinskii, I.M. Khalatnikov, E.M. Lifshttz, Oscillatory approach to a singular point in the relativistic cosmology, Adv. Physics, 19(80), 525-573 (1970).
  * (2) C. W. Misner, Mixmaster Universe, Phys. Rev. Letters 22, 1071, 1969).
  * (3) C. W. Misner, Quantum Cosmology I, (PhysRev. 186, 1319, 1969).
  * (4) V. A. Belinski, I. M. Khalatnikov and E. M. Lifshitz, A General Solution of the Einstein Equations with a Time Singularity, (Adv.Phys. 31, 639, 1982).
  * (5) V. A. Belinskii, I. M. Khalatnikov, Effect of scalar and vector fields on the nature of the cosmological singularity, (Sov. Phys. JETP 36(4), 591-597 , 1973).
  * (6) G. Montani, M. V. Battisti, R. Benini, G. Imponente, Classical and quantum features of the Mixmaster singularity, (International Journal of Modern Physics A 23, pp. 23532503, 2008).
  * (7) G. Montani, M. V. Battisti, G. Imponente, R. Benini, Primordial cosmology, (World Scientific, 2011).
  * (8) J. Demaret, M. Henneaux and P. Spindel, Phys. Lett., 164B, 27, (1985).
  * (9) Y. Elskens and M. Henneaux, Nucl. Phys., B290, 111 (1987).
  * (10) N. Deruelle and P. Spindel, Classical Quantum Gravity 7, 1599 (1990).
  * (11) B. K. Berger, Influence of scalar fields on the approach to a cosmological singularity, (Physical Review D 61, 023508 , 1999).
  * (12) Capozziello, S., Curvature Quintessence, Int. J. Mod. Phys. D 11 (2002) 483.
  * (13) Capozziello, S., De Laurentis, M., Extended Theories of Gravity, Physics Reports 509 (2011) 167.
  * (14) Nojiri, S., Odintsov, S.D., Unified cosmic history in modified gravity: from F(R) theory to Lorentz non-invariant models , Physics Reports 505 (2011) 59.
  * (15) S. Nojiri, S. D. Odintsov, Introduction to Modified Gravity and Gravitational Alternative for Dark Energy, Int. J. Geom. Methods Mod. Phys. 04, 115 (2007)
  * (16) Nojiri, S. and Odintsov, S. D. (2008). Dark energy, inflation and dark matter from modified f(r) gravity, arXiv:0807.0685
  * (17) J.Q. Guo, D. Wang, and A. V. Frolov, Phys. Rev. D 90, 024017 (2014).
  * (18) J. D. Barrow and S. Cotsakis, Phys. Lett. B, 232, 172-176 (1989).
  * (19) J. D. Barrow and H. Sirousse-Zia, Phys. Rev. D, 39, 2187-92 (1989).
  * (20) J. D. Barrow and S. Cotsakis, Phys. Lett. B, 214, 515-518 (1988).
  * (21) N. Deruelle, Nucl. Phys. B327 253 (1989)
  * (22) Starobinsky, A. A. (1980). A new type of isotropic cosmological models without singularity. Physics Letters B 91 99-102.
  * (23) P. A. R. Ade et al. [Planck Collaboration] ,Planck 2013 results. XXII. Constraints on inflation, arXiv:1303.5082 [astro-ph.CO]
  * (24) Misner, C. W., K. S. Thorne, and J. A. Wheeler, 1973, Gravitation (Freeman Press, San Francisco).
  * (25) G. Montani, A. A. Kirillov, Origin of a classical space in quantum inhomogeneous models, (Zh. ksp. Teor. Fiz. 66, No. 7, 449-453 ,1997).
  * (26) G. P. Imponente, G. Montani, On the covariance of the mixmaster chaoticity, (Physical Review D 63, p. 103501, 2001).
  * (27) R. Arnowitt, S. Deser, C. W. Misner, Canonical variables for general relativity, (Physical Review 117, 6, pp. 1595-1602, 1959).
  * (28) Pesin Ya B (1977) UMN (Russian Mathematical Surveys), Lyapunov Characteristic Numbers and Smooth Ergodic Theory 32, n.4 55-112
  * (29) V. I. Arnold, Mathematical Methods of Classical Mechanics, Springer-Verlag (1989)
  * (30) Chitré D. M. (1972) Ph.D. Thesis, University of Maryland
  * (31) A. A. Kirillov, G. Montani (1997) Phys. Rev. D, 56, n.10, 6225.
  * (32) I. M. Khalatnikov and E. M. Lifshitz, Phys. Rev. Lett. 24, 76 (1970).
  * (33) T. Damour, O. M. Lecian, Statistical Properties of Cosmological Billiards, Phys. Rev. D, 83 044038, (2011).
  * (34) S. Capozziello and S. Vignolo, The Cauchy problem for metric-affine f(R)-gravity in presence of perfect-fluid matter, Class. Quant. Grav. 26 (2009) 175013
  * (35) Bojowald, M., Date, G. and Hossain, G. M. (2004). The Bianchi IX model in loop quantum cosmology, Class.Quant.Grav. 21, pp. 35413570
  * (36) S. Capozziello and S. Vignolo, The Cauchy problem for metric-affine f(R)-gravity in presence of perfect-fluid matter, Class. Quant. Grav. 26 (2009) 175013
  * (37) G.Montani On the General Behaviour of the Universe Near Cosmological Singularity, Classical and Quantum Gravity, 12, 2503, (1995).
  * (38) Kirillov, A. A. On the question of the characteristics of the spatial distribution of metric inhomogeneities in a general solution to einstein equations in the vicinity of a cosmological singularity, Soviet Physics JETP 76, p. 355., (1993).
  * (39) Kirillov, A. A. and Kochnev, A. A. (1987). Cellular structure of space in the vicinity of a time singularity in the einstein equations, Pis ma Zhurnal Eksperimental noi i Teoreticheskoi Fiziki 46, pp. 345?348.
  * (40) E.M. Lifshitz, I.M. Khalatnikov, Investigations in relativistic cosmology, Adv. Physics, 12(46), 185-249 (1963