# MTLight: Efficient Multi-Task Reinforcement Learning for Traffic Signal
Control

Liwen Zhu
Peking University
<EMAIL_ADDRESS>
&Peixi Peng
Peking University
<EMAIL_ADDRESS>
&Zongqing Lu
Peking University
<EMAIL_ADDRESS>
&Yonghong Tian
Peking University
<EMAIL_ADDRESS>

###### Abstract

Traffic signal control has a great impact on alleviating traffic congestion in
modern cities. Deep reinforcement learning (RL) has been widely used for this
task in recent years, demonstrating promising performance but also facing many
challenges such as limited performances and sample inefficiency. To handle
these challenges, MTLight is proposed to enhance the agent observation with a
latent state, which is learned from numerous traffic indicators. Meanwhile,
multiple auxiliary and supervisory tasks are constructed to learn the latent
state, and two types of embedding latent features, the task-specific feature
and task-shared feature, are used to make the latent state more abundant.
Extensive experiments conducted on CityFlow demonstrate that MTLight has
leading convergence speed and asymptotic performance. We further simulate
under peak-hour pattern in all scenarios with increasing control difficulty
and the results indicate that MTLight is highly adaptable.

##  1 introduction

Traffic signal control aims to coordinate traffic signals across intersections
to improve the traffic efficiency of a district or a city, which plays an
important role in efficient transportation. Most conventional methods aim to
control traffic signals by fixed-time Koonce & Rodegerdts (2008) or hand-
crafted heuristics Kouvelas et al. (2014), which heavily rely on expert
knowledge and in-depth excavation of regional historical traffic, making it
difficult to migrate. Recently, deep reinforcement learning (DRL) based
methods Guo et al. (2021); Jintao et al. (2020); Pan et al. (2020); He & Shin
(2020); Tong et al. (2021); Wang et al. (2020); Gu et al. (2020); Liu et al.
(2021); Xu et al. (2021); Zhang et al. (2021) employ a deep neural network to
control an intersection where the network is learned by directly interacting
with the environment. However, due to the plenty of traffic indicators (number
of vehicles, queue length, waiting time, speed, etc.), complex observation and
the dynamic environment, the problem is challenging and remains unsolved.

Since the observation, reward and dynamics of each traffic signal are closely
related to others, hence optimizing traffic signal control in a large-scale
road network is naturally modeled as a multi-agent reinforcement learning
(MARL) problem. Most exiting works Wei et al. (2019a); Zhang et al. (2020b);
Chen et al. (2020); Zheng et al. (2019a) are proposed to learn the policy of
each agent only conditioned on the raw observations of the intersection, while
ignoring the help of the global state, which is accessible in smart city. As
stated in Zheng et al. (2019b), different metrics have a considerable impact
on the traffic signal control task. Hence, the observation design of agent
should not only involve the raw observations of the intersection, but also the
global state. A good agent observation design could make full use of samples,
and improves not only the policy performance but also the sample efficiency.
However, there are a huge amount of traffic indicators or metrics in the
global state, and it is hard to subjectively design suited and non-redundant
agent observation among these indicators. On one hand, an overly concise
observation design could not adequately and comprehensively represent the
state characteristics and therefore affects the accuracy of the estimation of
state transition and as well as influencing action selection. In contrast, if
an overly complex combination of metrics is used as an observation, the
weights of different metrics are difficult to precisely define, and it may
cause data redundancy and dimension explosion, which will not only increase
the computational consumption, but also make the agent hard to learn.

Figure 1: Multi-Task module forms task-shared and task-specific latent states
to enhance the agent observation.

In order to provide an adequate representation of the traffic signal control
task, the latent state is introduced. Specifically, the raw observation is
identical to the intersection, which consists of several variables with
concrete semantic meanings (i.e., the number of vehicles on each incoming lane
and current signal phase). Then, the raw observation is enhanced by the latent
space. To learn the latent space from the global state, multiple auxiliary and
supervisory tasks are constructed, which are related to traffic signal
control. That is, several statistics of global state history are taken as
inputs, a RNN-based network is employed firstly, and several branches are
introduced subsequently to predict multiple types of statistics of the global
state, such as the flow distribution and the travel time distribution,
respectively. To make the latent space more abundant, two types of embedding
features are extracted: the task-specific feature and task-shared feature. The
former is extracted by the task-specific branch and represents the task-driven
information, while the later is from the task-shared layer and could express
more general underlying characteristics. Hence, they are complementary to each
other and are both used to enhance the raw observation. Finally, conditioned
on the enhanced observation, the policy is learned by DRL Mnih et al. (2015).
Note that the multiple tasks are learned simultaneously with the DRL, which
makes the latent space more adaptive to the policy learning.

##  2 Problem Statement

###  2.1 Problem Definition

We consider a multi-agent traffic signal control problem, the task is modeled
as a Markov Game Littman (1994), which can be denoted by a tuple
$\mathcal{G}=<\mathcal{N},\mathcal{S},\mathcal{A},\mathcal{O},\mathcal{P},\mathcal{R},\mathcal{H},\gamma>$.
$\mathcal{N}\equiv\\{1,\ldots,n\\}$ is a finite set of agents, and each
intersection in the scenario is controlled by an agent. $\mathcal{S}$ is a
finite set of global state space. $\mathcal{A}$ denotes the action space for
an individual agent. The joint action
$\bm{a}\in\mathbf{A}\equiv\mathcal{A}^{n}$ is a collection of individual
actions $\left[a_{i}\right]_{i=1}^{n}$. At each timestep, each agent $i$
receives an observation $o_{i}\in\mathcal{O}$, selects an action $a_{i}$,
results in the next state $s^{\prime}$ according to the transition function
$\mathcal{P}\left(s^{\prime}\mid s,\bm{a}\right)$ and a reward
$r=\mathcal{R}(s,\mathbf{a})$ for each agent. $\mathcal{H}$ is the time
horizon and $\gamma\in[0,1)$ is the discount factor.

###  2.2 Agent Design

Each intersection in the system is controlled by an agent. In the following,
we introduce the state design, action design and reward design of the RL
agent.

  * •

Observation. Our primitive observation consists of two parts: (1) the number
of vehicles on each incoming lane $\mathbf{f}_{t}^{v}$; (2) current signal
phase $\mathbf{f}_{t}^{s}$. Both of them can be obtained directly from the
simulator, the concepts are described in detail in Section B.4. The raw
observation of agent $i$ is defined by

$\displaystyle o_{i}=\\{\mathbf{f}_{t}^{v},\mathbf{f}_{t}^{s}\\},$ (1)

where
$\mathbf{f}_{t}^{v}=\\{{V}_{l_{1}^{in}},{V}_{l_{2}^{in}},\ldots,{V}_{l_{m}^{in}}\\}$
and ${l}^{in}=\\{l_{1}^{in},\ldots,l_{m}^{in}\\}$ is a finite set of incoming
lanes in the intersection. Current signal phase
$\mathbf{f}_{t}^{s}={p}_{k},k\in{1,\ldots,K}$, and $K$ is the total number of
phases. Each phase $p$ is represented as a one-hot vector. Our goal is to
learn latent space to enhance the raw observation to make better use of the
sample.

  * •

Action. The action of each agent is to choose the phase for the next time
interval. Note that the phases may organize in a sequential way in reality,
while directly selecting a phase makes the traffic control plan more flexible.
Action of agent $i$ is defined by

$\displaystyle a_{i}=\\{\mathbf{f}_{t}^{s}\\},$ (2)

where $\mathbf{f}_{t}^{s}={p}_{k},k\in{1,\ldots,K}$.

  * •

Reward. We define the reward as the negative of the queue length on incoming
lanes, which is generally accepted and reasonable in previous work Zheng et
al. (2019b); Huang et al. (2021); Zang et al. (2020); Zheng et al. (2019a);
Wei et al. (2019b). Reward of agent $i$ is defined by

$\displaystyle r_{i}=-\sum^{M}_{m}q_{l^{in}_{m}},$ (3)

where $q_{l^{in}_{m}}$ is the queue length on incoming lane $l^{in}_{m}$.

##  3 method

In this section, we will introduce the main modules of our proposed method
MTLight, which focuses on learning task-related task-shared latent state and
task-specific latent state by introducing an auxiliary Multi-Task network to
help policy learning. The whole process of MTLight is described in Algorithm
1, and the framework of MTLight is shown in Fig. 2.

MTLight consists of a Multi-Task network and an agent network. For the latter,
Deep Q-Network (DQN) Mnih et al. (2015) is employed as function approximator
to estimate the Q-value function, which is consistent with the previous
methods Chen et al. (2020); Wei et al. (2019b; a); Zheng et al. (2019a); Wei
et al. (2018). The Multi-Task module adopts a hard parameter sharing paradigm
Caruana (1997), which generally applied by sharing the hidden layers between
all tasks, while keeping several task-specific output layers.

###  3.1 Multi-Task Learning for Latent State

Figure 2: MTLight consists of a multi-task network and a policy network. RL
agent is augmented with a task-shared latent state
$\mathrm{\mathbf{o}_{t}^{shr}}$ and a task-specific latent state
$\mathrm{\mathbf{o}_{t}^{spe}}$.

For each agent, its raw observation includes the number of vehicles
$\mathbf{f}_{t}^{v}$ and the current signal phase $\mathbf{f}_{t}^{s}$.
Besides, several information from the global state is given, such as: the
number of incoming cars in the last $\tau$ steps, denoted as
$\mathbf{f}_{t-\tau:t}^{c}=[\mathbf{f}_{t-\tau}^{c},\mathbf{f}_{t-\tau+1}^{c},\ldots,\mathbf{f}_{t}^{c}]$,
the average travel time during the past $\tau$ steps, denoted as
$\mathbf{f}_{t-\tau:t}^{tr}=[\mathbf{f}_{t-\tau}^{tr},\mathbf{f}_{t-\tau+1}^{tr},\ldots,\mathbf{f}_{t}^{tr}]$,
the queue length during the past $\tau$ steps, denoted as
$\mathbf{f}_{t-\tau:t}^{q}=[\mathbf{f}_{t-\tau}^{q},\mathbf{f}_{t-\tau+1}^{q},\ldots,\mathbf{f}_{t}^{q}]$,
the current vehicles during the past $\tau$ steps, which is denoted as
$\mathbf{f}_{t-\tau:t}^{vr}=[\mathbf{f}_{t-\tau}^{vr},\mathbf{f}_{t-\tau+1}^{vr},\ldots,\mathbf{f}_{t}^{vr}]$.

The Multi-Task module includes the following four tasks:

  1. 1.

Flow distribution approximation. We use $\mathcal{T}_{flow}$ to denote the
traffic distribution estimation task, i.e., to predict the mean $\mu_{f}$ and
variance $\sigma_{f}^{2}$ of flow arrival rate from start up to the time step
$t$. The task could be denoted as:

$\displaystyle(\mu_{f},\sigma_{f}^{2})\leftarrow[\mathbf{f}_{t}^{v},\mathbf{f}_{t}^{s},\mathbf{f}_{t-\tau:t}^{c},\mathbf{f}_{t-\tau:t}^{tr},\mathbf{f}_{t-\tau:t}^{q},\mathbf{f}_{t-\tau:t}^{vr}].$
(4)

  2. 2.

Travel time distribution approximation. We use $\mathcal{T}_{travel}$ to
denote the travel distribution estimation task, i.e., to predict the mean
$\mu_{tr}$ and variance $\sigma_{tr}^{2}$ of average travel time of vehicles
that have completed the trip from start up to the time step $t$:

$\displaystyle(\mu_{tr},\sigma_{tr}^{2})\leftarrow[\mathbf{f}_{t}^{v},\mathbf{f}_{t}^{s},\mathbf{f}_{t-\tau:t}^{c},\mathbf{f}_{t-\tau:t}^{tr},\mathbf{f}_{t-\tau:t}^{q},\mathbf{f}_{t-\tau:t}^{vr}].$
(5)

  3. 3.

Next queue length approximation. We use $\mathcal{T}_{queue}$ to denote the
next queue length estimation task, i.e., to predict the average number $q$ of
vehicles in queue at the next step:

$\displaystyle
q\leftarrow[\mathbf{f}_{t}^{v},\mathbf{f}_{t}^{s},\mathbf{f}_{t-\tau:t}^{c},\mathbf{f}_{t-\tau:t}^{tr},\mathbf{f}_{t-\tau:t}^{q},\mathbf{f}_{t-\tau:t}^{vr}].$
(6)

  4. 4.

Vehicles on the road approximation. We use $\mathcal{T}_{vehicles}$ to denote
the vehicles on the road approximation task, i.e., to predict the number of
vehicles $V^{r}$ existing in the system:

$\displaystyle
V^{r}\leftarrow[\mathbf{f}_{t}^{v},\mathbf{f}_{t}^{s},\mathbf{f}_{t-\tau:t}^{c},\mathbf{f}_{t-\tau:t}^{tr},\mathbf{f}_{t-\tau:t}^{q},\mathbf{f}_{t-\tau:t}^{vr}].$
(7)

Note that vehicles that have completed the trips or have not yet entered the
road network do not belong to these.

The above tasks act auxiliary tasks to learn the latent space. Since the
numbers of $\mathbf{f}_{t-\tau:t}^{c}$, $\mathbf{f}_{t-\tau:t}^{tr}$,
$\mathbf{f}_{t-\tau:t}^{q}$, $\mathbf{f}_{t-\tau:t}^{vr}$ have different
scales and their dimensions are different with $\mathbf{f}_{t}^{v}$ and
$\mathbf{f}_{t}^{s}$, four independent linear layers and ReLU functions are
employed firstly to scale them respectively:

$\displaystyle\mathbf{h}^{c}={ReLU}(\mathbf{W}_{1}\mathbf{f}_{t-\tau:t}^{c}+\mathbf{b}_{1}),\
\mathbf{h}^{tr}={ReLU}(\mathbf{W}_{2}\mathbf{f}_{t-\tau:t}^{tr}+\mathbf{b}_{2}),$
(8)
$\displaystyle\mathbf{h}^{q}={ReLU}(\mathbf{W}_{3}\mathbf{f}_{t-\tau:t}^{q}+\mathbf{b}_{3}),\
\mathbf{h}^{vr}={ReLU}(\mathbf{W}_{4}\mathbf{f}_{t-\tau:t}^{vr}+\mathbf{b}_{4}).$
(9)

Then a linear layer and ReLU function is used to calculate the hidden state
after concatenating all embedded inputs:

$\displaystyle\mathbf{H}_{t}={ReLU}(\mathbf{W}(\mathbf{f}_{t}^{v},\mathbf{f}_{t}^{s},\mathbf{h}^{c},\mathbf{h}^{tr},\mathbf{h}^{q},\mathbf{h}^{vr})+\mathbf{b}).$
(10)

Based on $\mathbf{H}_{t}$, a task-shared network module is used to generate
its task-shared latent feature ($\mathrm{\mathbf{o}_{t}^{shr}}$, also called
apparent state). Then, 4 independent branches are introduced for each task and
calculate task-specific latent feature ($\mathrm{\mathbf{o}_{t}^{spe}}$, also
called mental state) from $\mathrm{\mathbf{o}_{t}^{shr}}$. The specific
implementation of network architecture is listed in the supplementary.

We use a single latent variable model to extract hierarchical latent features,
which follows insights by Zhao et al. (2017). That is, the mental state is
output of the shared-layer after GRU in Multi-Task network and could express
more general underlying characteristics. In contrast, the apparent state is
the the concatenation of the output of the task-specific layer and represents
the task-driven information. In other words, the mental state is more coarse-
grained, while apparent state is more fine-grained. Hence, they are
complementary to each other and both used in our method.

###  3.2 Policy with Latent State

With the help of latent state, the agent observation is enhanced from
$\mathrm{\mathbf{o}_{t}}$ to
$(\mathrm{\mathbf{o}_{t}},\mathrm{\mathbf{o}_{t}^{shr}},\mathrm{\mathbf{o}_{t}^{spe}})$.
For the policy $\pi^{\theta}$, the objective is to maximize the cumulative
reward:

$\displaystyle\max\limits_{\theta}J(\theta)=\mathbb{E}_{\begin{subarray}{c}a_{t}\sim\pi^{\theta}(a_{t}\mid\mathrm{\mathbf{o}_{t}},\mathrm{\mathbf{o}_{t}^{shr}}.\mathrm{\mathbf{o}_{t}^{spe}})\end{subarray}}\sum\limits_{t=0}^{\mathcal{H}-1}\gamma^{t}r_{t+1}.$
(11)

An agent that maximises Eq. 11 acts optimally under uncertainty and is called
Bayes-optimal Ghavamzadeh et al. (2015), assuming we treat the knowledge over
related tasks as our epistemic prior about the environment. Multi-Task module
minimizes the complexity of the model and give informative priors to the
model. Besides, it can minimize the representation bias in a way that push the
learning algorithm to find a solution on a smaller area of representations on
the intersection rather than on a large area of a single task. This
incentivises a faster and better convergence.

##  4 experiment

We conduct the experiments on CityFlow Zhang et al. (2019), an city-level
open-source simulation platform for traffic signal control. The simulator is
used as the environment to provide state for traffic signal control, the
agents execute actions by changing the phase of traffic lights, and the
simulator returns feedback.

Please refer to Appendix D.1 and Appendix D.2 for the detailed settings of
road network and traffic flow configuration. Baselines are described in detail
in Appendix F.

###  4.1 Performance Comparison

Figure 3: Illustration of strategies for all RL methods under Real configuration in Hangzhou. Table 1: Overall performance comparison on Hangzhou, Jinan, New York and Shenzhen under Real and Synthetic configurations. Average travel time is reported in the unit of second. ”Mean” in the last column shows the average performance of the scenarios shown in the previous 8 columns. Model | Hangzhou | Jinan | Newyork | Shenzhen | Mean
---|---|---|---|---|---
real | syn_peak | real | syn_peak | real | syn_peak | real | syn_peak
MaxPressure | 416.82 | 2320.65 | 355.12 | 1218.13 | 380.42 | 1481.48 | 389.45 | 1387.87 | 1387.87
Fixedtime | 718.29 | 1787.58 | 814.09 | 1739.69 | 1849.78 | 2086.59 | 786.54 | 1845.03 | 1453.45
SOTL | 1209.26 | 2062.49 | 1453.97 | 1991.03 | 1890.55 | 2140.15 | 1376.52 | 2098.09 | 1777.76
Individual RL | 743.00 | 1819.57 | 843.63 | 1745.07 | 1867.86 | 2100.68 | 769.47 | 1845.34 | 1466.83
MetaLight | 480.77 | 1576.32 | 784.98 | 1854.38 | 261.34 | 2145.49 | 694.83 | 2083.26 | 1235.17
PressLight | 529.64 | 1754.09 | 809.87 | 1930.98 | 302.87 | 1846.76 | 639.04 | 1832.76 | 1205.75
CoLight | 297.89 | 1077.29 | 511.43 | 1217.17 | 159.81 | 1457.56 | 438.45 | 1367.38 | 815.87
GeneraLight | 335.18 | 1574.93 | 585.89 | 1616.28 | 1208.73 | 1686.49 | 792.22 | 1574.10 | 1171.73
Base | 705.85 | 1718.37 | 808.28 | 1703.21 | 903.82 | 2097.84 | 728.49 | 1937.45 | 1325.41
Base+raw | 684.34 | 1845.92 | 623.94 | 1835.45 | 592.34 | 1934.04 | 703.56 | 1845.32 | 1258.11
Base+shr | 313.28 | 1146.79 | 499.88 | 1325.27 | 463.15 | 1416.65 | 438.69 | 1371.53 | 871.91
Base+spe | 431.55 | 1446.63 | 517.09 | 1430.96 | 431.65 | 1669.61 | 684.83 | 1442.35 | 1006.83
MTLight | 161.24 | 1011.67 | 346.93 | 1176.02 | 209.46 | 1394.15 | 402.57 | 1284.93 | 748.37
Figure 4: Performance of RL methods under real configurations.

Tab. 1 lists the comparative results, and it is evident that: 1) In general,
RL methods perform better than conventional methods, and it indicates the
advantage of the RL. Moreover, MTLight is outperforms other methods in almost
all cities and flow configurations, which demonstrates the effectiveness of
the method. 2) MTLight shows good generalization for different scenarios and
configurations. For example, MaxPressure performs well in
$\mathcal{D}_{Hangzhou}$ with the Real, while under the Synthetic traffic
conditions, MaxPressure shows significantly worse than other methods. In
contrast, MTLight can not only achieve good performance under diverse
configurations of $\mathcal{D}_{Hangzhou}$, but also shows great stability. 3)
MTLight outperforms Individual RL, MetaLight and PressLight with 693.46,
461.80 and 432.38, respectively. The reason is that they learn the traffic
light’s policy only using its observation and ignore the influence of the
neighbors, while MTLight considers the neighbors as the latent part of the
environment to help learning. 4) The neighbor’s information is modeled in
CoLight and GeneraLight can adapt to a variety of flows, they both perform
well. While results of MTLight is superiors to them in multiple scenarios,
resulting mean 42.5 and 398 improvement. Compared to them, MTLight benefits
from prior knowledge learned from Multi-Task network to make more accurate
decisions.

Fig. 4 shows the performances of all RL methods of $\mathcal{D}_{Hangzhou}$
under Real traffic pattern, and it is obvious that MTLight converges faster
and has better asymptotic performance. Fig. 5 shows the performances of all RL
methods of $\mathcal{D}_{Hangzhou}$ under Synthetic traffic pattern, we can
conclude that MTLight converges quickly and learns effectively during the peak
hour, while the other method have only a weak boost during training.

Figure 5: Performance of RL methods under synthetic peak configurations.

Fig. 8 and Tab. 5 illustrates the turning statistics of vehicle routes. Take
$\mathcal{D}_{Hangzhou}$ Real as an example, the frequency of turning left and
going straight is 14% and 86% respectively (turning right are not considered
because they are free from the control by lights). Fig. 3 shows the percentage
of each phase of RL methods, we can find: 1) The total left-turn phase of
MTLight accounts for 15.3%, which is highly consistent with the left-turn
frequency of 14%, which indicates that the strategy is interpretable. 2) The
GeneraLight left-turn ratio of 10.9% is also close, but because it has an
excessive proportion of straight phases, it may cause left-turn vehicles to be
stranded, resulting in increased travel time. 3) Individual RL tends to
consider phase 1 and 2, which account for as much as 65.9%, MetaLight prefers
to go straight, PressLight is eccentric to phase 1, and CoLight assigns a
relatively even distribution to each phase, rather than aligning with the
traffic flow direction. These all demonstrate the limitations of other RL
methods in multi-agent environments, while MTLight can learn more stable
strategies by introducing task-shared and task-specific latent states.

###  4.2 Ablations

To better validate the contribution of each component, three variants of
MTLight are evaluated under a variety of scenarios, as shown in Tab. 1.

  * •

Base only keeps the policy network and removes the Multi-Task network.

  * •

Base+raw only keeps the policy network and discards Multi-Task network, but
directly uses the original input of Multi-Task module as part of the
observation.

  * •

Base+shr retains the Multi-Task network and the policy, but only has task-
shared latent state and removes task-specific latent state.

  * •

Base+spe retains the Multi-Task network and the policy. In contrast to
Base+shr, Base+spe has only the task-specific latent state and removes the
task-shared latent state.

Note that MTLight contains the whole modules: policy network, Multi-Task
network with both task-specific latent state and task-shared latent state.

The quantitative evaluation results are presented in Tab. 1. We can obtain the
following findings: 1) Among these 4 models, the performance of Base is the
worst. The reason is that it is hard to learn the effective policy
independently in the multi-agent traffic signal control task, where the
surrounding environment is changing dynamically, but Base has no sense of it.
2) Compared with the Base and Base+raw, the improvement of Base+shr and
Base+spe demonstrate the effectiveness of the task-shared latent state
$\mathrm{\mathbf{o}_{t}^{shr}}$ and task-specific latent state
$\mathrm{\mathbf{o}_{t}^{spe}}$ respectively. $\mathrm{\mathbf{o}_{t}^{shr}}$
reflects prior information that is constant over time with multiple related
tasks , $\mathrm{\mathbf{o}_{t}^{spe}}$ reflects prior information that is
align with the latest changing trends, both of them help policy to make
Bayesian optimal decisions. 3) The $\mathrm{\mathbf{o}_{t}^{shr}}$ and
$\mathrm{\mathbf{o}_{t}^{spe}}$ are both effective because each of them is an
efficient representations of environmental features. Compared to them, the
superiority of MTLight indicates $\mathrm{\mathbf{o}_{t}^{shr}}$ and
$\mathrm{\mathbf{o}_{t}^{spe}}$ are complementary to each other. Overall, all
of the proposed components contribute positively to the final results.

##  5 conclusion

We introduced MTLight, an efficient Multi-Task reinforcement learning method
for traffic signal control that can be scaled to complex multi-agent urban
road networks of different scale. We showed that MTLight’s latent structure
learns a hierarchical latent representations of related tasks, separating the
task-shared and task-specific latent states. On several cities’ datasets we
demonstrated that this latent representation inspired from related multiple
tasks, and conditioning the policy on it, allows an agent to adapt to the
complex environment. We conclude that maintaining prior approximations over
related tasks helps compared to model-free approaches, especially when there
is too much information in the environment and it cannot be fully expressed by
artificial state design.

For the future, the latent prior could be learned from expert data prepared in
advance using imitation learning techniques Song et al. (2018), or by using
existing multi-agent algorithms to pre-train Multi-Task network.

## References

  * Abdoos et al. (2011) Monireh Abdoos, Nasser Mozayani, and Ana LC Bazzan.  Traffic light control in non-stationary environments based on multi agent q-learning.  In _ITSC_. IEEE, 2011.
  * Abdoos et al. (2013) Monireh Abdoos, Nasser Mozayani, and Ana LC Bazzan.  Holonic multi-agent system for traffic signals control.  _Engineering Applications of Artificial Intelligence_ , 2013.
  * Arel et al. (2010) Itamar Arel, Cong Liu, Tom Urbanik, and Airton G Kohls.  Reinforcement learning-based multi-agent system for network traffic signal control.  _IET Intelligent Transport Systems_ , 2010.
  * Bellemare et al. (2019) Marc Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, and Clare Lyle.  A geometric perspective on optimal representations for reinforcement learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 1997.
  * Chen et al. (2020) Chacha Chen, Hua Wei, Nan Xu, Guanjie Zheng, Ming Yang, Yuanhao Xiong, Kai Xu, and Zhenhui Li.  Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control.  In _AAAI_ , 2020.
  * Chiu (1992) Stephen Chiu.  Adaptive traffic signal control using fuzzy logic.  In _Proceedings of the Intelligent Vehicles92 Symposium_. IEEE, 1992\.
  * Chiu & Chand (1993) Stephen Chiu and Sujeet Chand.  Self-organizing traffic control via fuzzy logic.  In _IEEE Conference on Decision and Control_. IEEE, 1993.
  * Chu et al. (2019) Tianshu Chu, Jie Wang, Lara Codecà, and Zhaojian Li.  Multi-agent deep reinforcement learning for large-scale traffic signal control.  _ITS_ , 2019.
  * Cools et al. (2013) Seung-Bae Cools, Carlos Gershenson, and Bart D’Hooghe.  Self-organizing traffic lights: A realistic simulation.  In _Advances in applied self-organizing systems_. Springer, 2013\.
  * Dusparic & Cahill (2009) Ivana Dusparic and Vinny Cahill.  Distributed w-learning: Multi-policy optimization in self-organizing systems.  In _self-adaptive and self-organizing systems_. IEEE, 2009.
  * El-Tantawy et al. (2013) Samah El-Tantawy, Baher Abdulhai, and Hossam Abdelgawad.  Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (marlin-atsc): methodology and large-scale application on downtown toronto.  _IEEE TITS_ , 2013.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _ICML_. PMLR, 2017.
  * Ghavamzadeh et al. (2015) Mohammad Ghavamzadeh, Shie Mannor, Joelle Pineau, Aviv Tamar, et al.  Bayesian reinforcement learning: A survey.  _Foundations and Trends® in Machine Learning_ , 2015\.
  * Gu et al. (2020) Jingjing Gu, Qiang Zhou, Jingyuan Yang, Yanchi Liu, Fuzhen Zhuang, Yanchao Zhao, and Hui Xiong.  Exploiting interpretable patterns for flow prediction in dockless bike sharing systems.  _IEEE Transactions on Knowledge and Data Engineering_ , 2020.
  * Guo et al. (2021) Xin Guo, Zhengxu Yu, Pengfei Wang, Zhongming Jin, Jianqiang Huang, Deng Cai, Xiaofei He, and Xiansheng Hua.  Urban traffic light control via active multi-agent communication and supply-demand modeling.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * He & Shin (2020) Suining He and Kang G Shin.  Spatio-temporal capsule-based reinforcement learning for mobility-on-demand coordination.  _IEEE Transactions on Knowledge and Data Engineering_ , 2020.
  * Huang et al. (2021) Xingshuai Huang, Di Wu, Michael Jenkin, and Benoit Boulet.  Modellight: Model-based meta-reinforcement learning for traffic signal control.  _arXiv preprint arXiv:2111.08067_ , 2021.
  * Hunt et al. (1981) PB Hunt, DI Robertson, RD Bretherton, and RI Winton.  Scoot-a traffic responsive method of coordinating signals.  Technical report, 1981.
  * Jaderberg et al. (2016) Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu.  Reinforcement learning with unsupervised auxiliary tasks.  _arXiv preprint arXiv:1611.05397_ , 2016.
  * Jiang et al. (2021) Qize Jiang, Jingze Li, Weiwei Sun SUN, and Baihua Zheng.  Dynamic lane traffic signal control with group attention and multi-timescale reinforcement learning.  IJCAI, 2021.
  * Jintao et al. (2020) KE Jintao, Hai Yang, Jieping Ye, et al.  Learning to delay in ride-sourcing systems: a multi-agent deep reinforcement learning framework.  _IEEE Transactions on Knowledge and Data Engineering_ , 2020.
  * Koonce & Rodegerdts (2008) Peter Koonce and Lee Rodegerdts.  Traffic signal timing manual.  Technical report, United States. Federal Highway Administration, 2008\.
  * Kouvelas et al. (2014) Anastasios Kouvelas, Jennie Lioris, S Alireza Fayazi, and Pravin Varaiya.  Maximum pressure controller for stabilizing queues in signalized arterial networks.  _Transportation Research Record_ , 2014.
  * Kuyer et al. (2008) Lior Kuyer, Shimon Whiteson, Bram Bakker, and Nikos Vlassis.  Multiagent reinforcement learning for urban traffic control using coordination graphs.  In _ECML-PKDD_. Springer, 2008.
  * Lin et al. (2019) Xingyu Lin, Harjatin Baweja, George Kantor, and David Held.  Adaptive auxiliary task weighting for reinforcement learning.  _Advances in neural information processing systems_ , 2019.
  * Littman (1994) Michael L Littman.  Markov games as a framework for multi-agent reinforcement learning.  In _Machine learning proceedings_. Elsevier, 1994.
  * Liu et al. (2021) Jia Liu, Tianrui Li, Shenggong Ji, Peng Xie, Shengdong Du, Fei Teng, and Junbo Zhang.  Urban flow pattern mining based on multi-source heterogeneous data fusion and knowledge graph embedding.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Lowrie (1990) PR Lowrie.  Scats, sydney co-ordinated adaptive traffic system: A traffic responsive method of controlling urban traffic.  1990\.
  * Lyle et al. (2021) Clare Lyle, Mark Rowland, Georg Ostrovski, and Will Dabney.  On the effect of auxiliary tasks on representation dynamics.  In _International Conference on Artificial Intelligence and Statistics_. PMLR, 2021.
  * Mannion et al. (2016) Patrick Mannion, Jim Duggan, and Enda Howley.  An experimental review of reinforcement learning algorithms for adaptive traffic signal control.  In _Autonomic road transport support systems_. Springer, 2016.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _Nature_ , 2015.
  * Ndirango & Lee (2019) Anthony Ndirango and Tyler Lee.  Generalization in multitask deep neural classifiers: a statistical physics approach.  _Advances in Neural Information Processing Systems_ , 2019.
  * Nishi et al. (2018) Tomoki Nishi, Keisuke Otaki, Keiichiro Hayakawa, and Takayoshi Yoshimura.  Traffic signal control based on reinforcement learning with graph convolutional neural nets.  In _ITSC_. IEEE, 2018.
  * Oh et al. (2017) Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli.  Zero-shot task generalization with multi-task deep reinforcement learning.  In _ICML_. PMLR, 2017.
  * Oroojlooy et al. (2020) Afshin Oroojlooy, Mohammadreza Nazari, Davood Hajinezhad, and Jorge Silva.  Attendlight: Universal attention-based reinforcement learning model for traffic signal control.  _arXiv preprint arXiv:2010.05772_ , 2020.
  * Pan et al. (2020) Zheyi Pan, Wentao Zhang, Yuxuan Liang, Weinan Zhang, Yong Yu, Junbo Zhang, and Yu Zheng.  Spatio-temporal meta learning for urban traffic prediction.  _IEEE Transactions on Knowledge and Data Engineering_ , 2020.
  * Rizzo et al. (2019) Stefano Giovanni Rizzo, Giovanna Vantini, and Sanjay Chawla.  Time critic policy gradient methods for traffic signal control in complex and congested scenarios.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2019.
  * Roess et al. (2004) Roger P Roess, Elena S Prassas, and William R McShane.  _Traffic engineering_.  Pearson/Prentice Hall, 2004.
  * Ruder (2017) Sebastian Ruder.  An overview of multi-task learning in deep neural networks.  _arXiv preprint arXiv:1706.05098_ , 2017.
  * Song et al. (2018) Jiaming Song, Hongyu Ren, Dorsa Sadigh, and Stefano Ermon.  Multi-agent generative adversarial imitation learning.  _Advances in neural information processing systems_ , 2018.
  * Svanes & Delaney (1981) Torgny Svanes and James R Delaney.  Scat: System control analysis and training simulator.  In _Human Detection and Diagnosis of System Failures_. Springer, 1981\.
  * Tong et al. (2021) Yongxin Tong, Dingyuan Shi, Yi Xu, Weifeng Lv, Zhiwei Qin, and Xiaocheng Tang.  Combinatorial optimization meets reinforcement learning: Effective taxi order dispatching at large-scale.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Tongloy et al. (2017) T Tongloy, S Chuwongin, K Jaksukam, C Chousangsuntorn, and S Boonsang.  Asynchronous deep reinforcement learning for the mobile robot navigation with supervised auxiliary tasks.  In _International Conference on Robotics and Automation Engineering (ICRAE)_ , pp. 68–72. IEEE, 2017.
  * Van der Pol & Oliehoek (2016) Elise Van der Pol and Frans A Oliehoek.  Coordinated deep reinforcement learners for traffic light control.  _NeurIPS_ , 2016.
  * Varaiya (2013) Pravin Varaiya.  The max-pressure controller for arbitrary networks of signalized intersections.  In _Advances in Dynamic Network Modeling in Complex Transportation Systems_. Springer, 2013.
  * Wang et al. (2020) Senzhang Wang, Jiannong Cao, and Philip Yu.  Deep learning for spatio-temporal data mining: A survey.  _IEEE transactions on knowledge and data engineering_ , 2020.
  * Webster (1958) Fo Vo Webster.  Traffic signal settings.  Technical report, 1958.
  * Webster (1966) FV Webster.  Traffic signals.  _Road research technical paper_ , 1966.
  * Wei et al. (2018) Hua Wei, Guanjie Zheng, Huaxiu Yao, and Zhenhui Li.  Intellilight: A reinforcement learning approach for intelligent traffic light control.  In _SIGKDD_ , 2018.
  * Wei et al. (2019a) Hua Wei, Chacha Chen, Guanjie Zheng, Kan Wu, Vikash Gayah, Kai Xu, and Zhenhui Li.  Presslight: Learning max pressure control to coordinate traffic signals in arterial network.  In _SIGKDD_ , 2019a.
  * Wei et al. (2019b) Hua Wei, Nan Xu, Huichu Zhang, Guanjie Zheng, Xinshi Zang, Chacha Chen, Weinan Zhang, Yanmin Zhu, Kai Xu, and Zhenhui Li.  Colight: Learning network-level cooperation for traffic signal control.  In _CIKM_ , 2019b.
  * Xiong et al. (2019) Yuanhao Xiong, Guanjie Zheng, Kai Xu, and Zhenhui Li.  Learning traffic signal control from demonstrations.  In _CIKM_ , 2019.
  * Xu et al. (2021) Bingyu Xu, Yaowei Wang, Zhaozhi Wang, Huizhu Jia, and Zongqing Lu.  Hierarchically and cooperatively learning traffic signal control.  In _AAAI_ , 2021.
  * Yu et al. (2020) Zhengxu Yu, Shuxian Liang, Long Wei, Zhongming Jin, Jianqiang Huang, Deng Cai, Xiaofei He, and Xian-Sheng Hua.  Macar: Urban traffic light control via active multi-agent communication and action rectification.  In _IJCAI_ , 2020.
  * Zang et al. (2020) Xinshi Zang, Huaxiu Yao, Guanjie Zheng, Nan Xu, Kai Xu, and Zhenhui Li.  Metalight: Value-based meta-reinforcement learning for traffic signal control.  In _AAAI_ , 2020.
  * Zhang et al. (2021) Feng Zhang, Yani Liu, Ningxuan Feng, Cheng Yang, Jidong Zhai, Shuhao Zhang, Bingsheng He, Jiazao Lin, Xiao Zhang, and Xiaoyong Du.  Periodic weather-aware lstm with event mechanism for parking behavior prediction.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Zhang et al. (2019) Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li.  Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario.  In _WWW_ , 2019.
  * Zhang et al. (2020a) Huichu Zhang, Markos Kafouros, and Yong Yu.  Planlight: Learning to optimize traffic signal control with planning and iterative policy improvement.  _IEEE Access_ , 2020a.
  * Zhang et al. (2020b) Huichu Zhang, Chang Liu, Weinan Zhang, Guanjie Zheng, and Yong Yu.  Generalight: Improving environment generalization of traffic signal control via meta reinforcement learning.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, 2020b.
  * Zhang & Yang (2021) Yu Zhang and Qiang Yang.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Zhao et al. (2017) Shengjia Zhao, Jiaming Song, and Stefano Ermon.  Learning hierarchical features from deep generative models.  In _ICML_. PMLR, 2017.
  * Zheng et al. (2019a) Guanjie Zheng, Yuanhao Xiong, Xinshi Zang, Jie Feng, Hua Wei, Huichu Zhang, Yong Li, Kai Xu, and Zhenhui Li.  Learning phase competition for traffic signal control.  In _CIKM_ , 2019a.
  * Zheng et al. (2019b) Guanjie Zheng, Xinshi Zang, Nan Xu, Hua Wei, Zhengyao Yu, Vikash Gayah, Kai Xu, and Zhenhui Li.  Diagnosing reinforcement learning for traffic signal control.  _arXiv_ , 2019b.

##  Appendix A Appendix

You may include other additional sections here.

Table 2: Implementation details of MTLight Items | Details
---|---
Number of policy steps | 3600
Discount factor $\gamma$ | 0.95
Policy $\epsilon$ | 0.1 $\rightarrow$ 0.01
$\epsilon$ decay rate | 0.995
Policy Learning rate | 0.005
Policy minibatch | 32
task-shared latent space dim | 5
task-specific latent space dim | 5
task-shared latent state coef | 10
task-specific latent state coef | 10
Policy network | 2 hidden layers,
architecture | 20 nodes each,
| ReLU activations
Policy network | RMSprop with learning rate 0.001
optimizer | and MSE loss
| 5 MLP embedding layers ,
| 2 shared FC layers before GRU,
| GRU with hidden size 64,
Multi-Task architecture | 1 shared FC layer after GRU,
| 4 task-specific FC layers,
| 4 output task layers
| ReLU activations
Multi-Task optimizer | Adam with learning rate 0.01
| and MSE loss

##  Appendix B related work

###  B.1 Conventional and Adaptive Traffic Signal Control

Most conventional traffic signal control methods are designed based on fixed-
time signal control Webster (1958), actuated control Chiu (1992) or self-
organizing traffic signal control Chiu & Chand (1993); Cools et al. (2013);
Lowrie (1990); Svanes & Delaney (1981); Hunt et al. (1981). These approaches
rely on expert knowledge and often perform unsatisfactorily in complicated
real-world situations. To solve this problem, several optimization-based
methods Roess et al. (2004); Varaiya (2013); Kouvelas et al. (2014) have been
proposed to optimize average travel time, throughput, etc., which decide the
traffic signal plans according to the observed data instead of the human
prior. However, these approaches typically rely on strict assumptions which
might not hold in the real-world cases Webster (1966). Furthermore, the
optimization problems are usually hard to tract and require significant
computing power in complex scenarios.

###  B.2 RL-based Traffic Signal Control

RL-based traffic signal control methods aim to learn the policy from
interactions with the environment. Earlier studies use tabular Q-learning El-
Tantawy et al. (2013); Abdoos et al. (2013); Dusparic & Cahill (2009); Abdoos
et al. (2011) where the states in an environment are required to be
discretized and low-dimensional. To address the unmanageable large or
continuous state space, recent advances employ deep RL with more complex
continuous state representations (like images or feature vectors) to map the
high-dimensional states into actions.

Efforts have been made to design strategies that formulate the task as a
single agent Wei et al. (2018); Mannion et al. (2016); Huang et al. (2021);
Zang et al. (2020); Oroojlooy et al. (2020); Jiang et al. (2021); Rizzo et al.
(2019) or some isolated intersections Zheng et al. (2019b; a); Xiong et al.
(2019); Wei et al. (2019a); Chen et al. (2020); Oroojlooy et al. (2020); Zhang
et al. (2020b; a), i.e., each agent makes decision for its own. The above
methods are usually easy to scale, but they may have difficulty achieving
globally optimal performance due to a lack of collaboration. To solve the
problem, another way is to consider jointly modeling the action between
learning agents with centralized optimization Van der Pol & Oliehoek (2016);
Kuyer et al. (2008). However, as the number of agents increases, joint
optimization usually leads to dimensional explosion, which has inhibited the
widespread adoption of such methods to a large-scale traffic signal control.
To overcome the difficulty, another type of methods are implemented in a
decentralized manner, taking into account the collaboration between neighbors
with appropriate reward and state design Arel et al. (2010); Nishi et al.
(2018); Wei et al. (2019b); Xu et al. (2021). Methods such as El-Tantawy et
al. (2013); Chu et al. (2019) add neighboring information into states, Nishi
et al. (2018); Wei et al. (2019b); Yu et al. (2020); Guo et al. (2021) add
neighbors’ hidden features into states, and Xu et al. (2021) optimizes
neighborhood travel time as an additional reward. However, simple
concatenation of neighboring information is not reasonable enough because the
influence of neighboring intersections is not balanced. Unlike the above
methods that add neighbor information to the state, our method learns task-
shared and task-specific latent states by constructing Multi-Task network.

###  B.3 Multi-Task Learning

Multi-Task Learning(MTL) Caruana (1997) is a learning paradigm aims to jointly
learn multiple related tasks so that the knowledge contained in a task can be
leveraged by other tasks. Past works Oh et al. (2017); Zhang & Yang (2021);
Ruder (2017); Ndirango & Lee (2019) have found that, by sharing a
representation among related tasks and jointly learning all the tasks, better
generalization can be achieved over independently learning each task.
Constructing auxiliary tasks to help the main task is a branch of Multi-Task
Learning. Reinforcement learning is known to be sample inefficient,
transferring knowledge from other auxiliary tasks is a powerful tool for
improving the learning efficiency Jaderberg et al. (2016); Lin et al. (2019);
Lyle et al. (2021); Tongloy et al. (2017); Bellemare et al. (2019). Lin et al.
(2019) combines different auxiliary tasks which provide gradient directions to
speed up the training of the main reinforcement learning task. In comparison,
our work aims to transfer knowledge from the task-related auxiliary tasks as a
prior to the main reinforcement learning task, to ultimately boost the
performance. Specifically, we model the Multi-Task network as a latent
structure where the task-shared latent state is generated from early layers
and the task-specific latent state is generated from deeper layers. This
incentivies the policy to learn the Bayers-optimal behaviours: the policy can
take into account its uncertainty over the comprehensive information when
choosing actions.

###  B.4 Preliminaries

In this section, we first introduce some basic concepts related to traffic
signal control (TSC) that have been widely recognized in previous work Wei et
al. (2019b); Zheng et al. (2019a); Zhang et al. (2020b); Wei et al. (2019a);
Chen et al. (2020); Zang et al. (2020). Note that the concepts can be easily
generalized to other intersections with different structures.

Figure 6: Illustration of phase.

  * •

Incoming/Outgoing Lanes. The incoming lanes refer to the lanes where the
vehicles are about to enter the intersection. It usually contains three basic
types: ”left-turn”, ”straight” and ”right-turn” from inner to outer. The
outgoing lanes refer to the lanes where the vehicles are about to leave the
intersection.

  * •

Roadnet. A roadnet is a part of a dataset that represents an area of a city. A
roadnet consists of signalized intersections, unsignalized intersections, and
lanes connecting the intersections. Generally, the lane lengths, number of
lanes and relative locations of intersections vary from one roadnet to
another.

  * •

Phase. Phase is a controller timing unit associated with the control of one or
more movements, representing the permulation and combination of different
traffic flows. The 4-phase setting is the most common configuration in
reality, illustrated in Fig. 6, but the number of phases can vary due to
different intersection topologies (3-way, 5-way intersections, etc.).

  * •

Queue Length. Queue length is the number of vehicles waiting at an
intersection due to a red light. Vehicles on the incoming lane with a speed of
less than 0.1m/s are considered to be waiting.

  * •

Average Travel Time. The travel time of a vehicle is the time discrepancy
between entering and leaving a particular area. Average travel time of all
vehicles in a road network is the most frequently used measure to evaluate the
performance of traffic signal control Wei et al. (2019b; a); Zhang et al.
(2020b); Chen et al. (2020); Zheng et al. (2019a).

  * •

Flow Distribution. Flow distribution is the distribution of traffic entering
the road network, which is generally expressed by the arrival rate of
vehicles, i.e., the volume of traffic entering the road network per unit time.

  * •

Vehicles on Road. Vehicles on road indicate the running vehicle, i.e.,
vehicles that have entered the road network and have not reached the end
point. Vehicles on road can represent the real-time load on the road network.

##  Appendix C Algorithm

The algorithm is shown in Alg. 1.

Input: Roadnet file; traffic flow file; number of training episodes $E$;
frequency of updating policy $t_{p}$; frequency of updating multi-task network
$t_{m}$; total simulate time $T$

Output: Set of optimized parameters for the intersections; optimized parameter
for the multi-task network

1 Initialize task-shared and task-specific latent state
$\mathrm{\mathbf{o}_{t}^{shr}}$, $\mathrm{\mathbf{o}_{t}^{spe}}$

2 Initialize policy replay buffer $\mathcal{B}^{\pi}$

3 Initialize policy $\pi^{\theta}$ and multi-task network $\mathbf{M}^{\phi}$

4 Initialize reward of each agent $\\{r_{i}\mid i\in 1,\ldots,n\\}$

5 for  _episode $\longleftarrow$ 1, 2, …, $E$_ do

6 for  _step t $\longleftarrow$ 1, 2, …, $T$_ do

7 Collect original observations for all agents

8 Add task-shared $\mathrm{\mathbf{o}_{t}^{shr}}$ and task-specific
$\mathrm{\mathbf{o}_{t}^{spe}}$ latent state to the observations

9 for  _agent i $\longleftarrow$ 1, 2, …, n_ do

10 Select action according to $\pi^{\theta}$

11

12 Employ joint action $\bm{a}$ to the environment

13 Get new observations and environmental reward

14 Collect trajectories to replay buffer $\mathcal{B}^{\pi}$

15 Get multi-task network input
$\mathbf{f}_{t}^{v},\mathbf{f}_{t}^{s},\mathbf{f}_{t}^{c},\mathbf{f}_{t}^{tr},\mathbf{f}_{t}^{q},\mathbf{f}_{t}^{vr}$
from the environment

16 Predict results using multi-task network $\mathbf{M}^{\phi}$

17 Get task-shared $\mathrm{\mathbf{o}_{t}^{shr}}$ and task-specific
$\mathrm{\mathbf{o}_{t}^{spe}}$ latent state from $\mathbf{M}^{\phi}$

18 Calculate statistics from 0 up to $t$ as supervised signal

19 if  _t = $t_{p}$_ then

20 Train policy $\pi^{\theta}$ by maximizing reward in Eq. 11

21 Clean up $\mathcal{B}^{\pi}$

22 if  _t = $t_{m}$_ then

23 Calculate loss from the results of step 1 and step 1

24 Train multi-task network $\mathbf{M}^{\phi}$

25

26 if  _t = $T$_ then

27 Collect the average total travel time of all vehicles as criteria

28

29

30

Algorithm 1 Training Process of MTLight

##  Appendix D Datasets

###  D.1 Road Networks

The evaluation scenarios come from four real road network maps of different
scales, including Hangzhou (China), Jinan (China), New York (USA) and Shenzhen
(China), illustrated in Fig. 7. The road networks and data of Hangzhou, Jinan
and New York are from the public datasets111https://traffic-signal-
control.github.io/. The road network map of Shenzhen is made by ourselves
which is derived from OpenStreetMap222The road network map and data of
Shenzhen will be released to facilitate the future research.. The road
networks of Jinan and Hangzhou contain 12 and 16 intersections in $4\times 3$
and $4\times 4$ grids, respectively. The road network of New York includes 48
intersections in $16\times 3$ grid. The road network of Shenzhen contains 33
intersections, which is not grid compared to other three maps.

Figure 7: The illustration of the road networks. The figures from left to right represent the road network of Jinan(China), Hangzhou(China), New York(USA) and Shenzhen(China), containing 12 ($4\times 3$), 16 ($4\times 4$), 48 ($16\times 3$) and 33 (Non-grid) traffic signals respectively. Figure 8: Turning statistics of vehicle routes. Table 3: Arrival rate of real-world traffic dataset Dataset | # Intersections | Arrival rate (vehicles/300s)
---|---|---
Mean | Std | Max | Min
$\mathcal{D}_{Hangzhou}$ | 16 (4 $\times$ 4) | 248.58 | 42.25 | 333 | 212
$\mathcal{D}_{Jinan}$ | 12 (4$\times$3) | 524.58 | 102.91 | 672 | 256
$\mathcal{D}_{NewYork}$ | 48 (16$\times$3) | 235.33 | 5.84 | 244 | 224
$\mathcal{D}_{Shenzhen}$ | 33 (Non-grid) | 147.92 | 79.35 | 255 | 22
Table 4: Data statistics of synthetic traffic dataset Dataset | Time |  | Arrival rate
---
(vehicles/s)
| Incoming
---
vehicles
| Accumulated
---
vehicles
$\mathcal{D}_{Hangzhou}$/ $\mathcal{D}_{Jinan}$/ $\mathcal{D}_{NewYork}$/ $\mathcal{D}_{Shenzhen}$ | 0-600 | 1.00 | 600 | 600
600-1200 | 0.25 | 150 | 750
1200-1800 | 4.00 | 2400 | 3150
1800-2400 | 2.00 | 1200 | 4350
2400-3000 | 0.2 | 120 | 4470
3000-3600 | 0.5 | 150 | 4770
Table 5: Statistics of turning frequency at intersections in all routes. Model | Hangzhou | Jinan | Newyork
---|---|---|---
real | syn_peak | real | syn_peak | real | syn_peak
turn left | 1093 (14%) | 5175 (24%) | 3044 (20%) | 5833 (30%) | 3886 (18%) | 7169 (20%)
go straight | 6620 (86%) | 16293 (76%) | 12175 (80%) | 13704 (70%) | 17498 (82%) | 27976 (80%)
turn right | 3184 | 8752 | 5972 | 8747 | 4021 | 7421

###  D.2 Flow configurations

We run the experiments under two traffic flow configurations: real traffic
flow and synthetic traffic flow. The real traffic flow is real-world hourly
statistical data with slight variance in vehicle arrival rates, as shown in
Tab. 3. Since the real-world strategies tend to break down during bottleneck
period (peak hour), to better evaluate the performances of traffic light
control methods in the flat-peak-flat scenario, we use synthetic datasets,
which have a more dramatic variance in vehicle arrival rates, as shown in Tab.
4. A detailed description of traffic flow configurations is:

  * •

Real. The traffic flows of Hangzhou (China), Jinan (China) and New York (USA)
are from the public datasets, which are processed from multiple sources. The
traffic flow of Shenzhen (China) is made by ourselves generated based on the
traffic trajectories collected from 80 red-light cameras and 16 monitoring
cameras in a hour. The data statistics are listed in Tab. 3.

  * •

Synthetic. The Synthetic is a mixed traffic flow with a total flow of 4770 in
one hour, to simulate a heavy peak. The arrival rate changes every 10 minutes,
which is used to simulate the uneven traffic flow distribution in the real
world, the details of the vehicle arrival rate and cumulative traffic flow are
shown in Tab. 4.

##  Appendix E Evaluation Criteria

Following existing studies Wei et al. (2019b; a); Xiong et al. (2019); Chen et
al. (2020); Zang et al. (2020), we use the average travel time to evaluate the
performance of different methods for traffic signal control. The average
travel time indicates the overall traffic situation in an area over a period
of time. For a detailed definition of average travel time, see Section B.4.
Since the number of vehicles and the origin-destination (OD) positions are
fixed, better traffic signal control strategies result in less average travel
time.

##  Appendix F Baselines

Our method is compared with the following two categories of methods:
conventional transportation methods and RL methods333Some existing RL based
traffic signal control methods, such as AttendLight Oroojlooy et al. (2020)
and SD-MaCAR Guo et al. (2021), evaluate their method under different
experimental settings (e.g., road network or traffic flow), and the source
codes are not available yet. Therefore, they are not compared in our
experiments.. Note that for a fair comparison all the RL methods are learned
without any pre-trained parameters and the methods are evaluated under the
same settings. The results are obtained by running the source
codes444https://github.com/traffic-signal-control/RL_signals. All the
baselines are run with three random seeds, and the mean is taken as the final
result. The action interval is five seconds for each method, and the horizon
is 3600 seconds for each episode. Specifically, the compared methods contain:

###  F.1 Conventional methods

  * •

MaxPressure Varaiya (2013) is a leading conventional method, which greedily
chooses the phase with the maximum pressure. The pressure is defined as the
difference of vehicle density between the incoming lane and the outgoing lane,
and the vehicle density means the actual number of vehicles divided by the
maximum permissible vehicle number.

  * •

Fixedtime Koonce & Rodegerdts (2008) with random offset Roess et al. (2004)
executes each phase in a phase loop with a pre-defined span of phase duration,
which is widely used for steady traffic.

  * •

SOTL Cools et al. (2013) specifies a pre-defined threshold for the number of
waiting vehicles on approaching lanes. Once the waiting vehicles exceeds the
threshold, it will switch to the next phase.

###  F.2 RL-based methods

  * •

Individual RL. Wei et al. (2018) Independent control is performed for each
agent in multi-agent environment, each intersection is controlled by one
agent. The replay buffer and network parameters are not shared, and the model
update is independent. There is no information transfer between agents, and no
neighbor information is considered.

  * •

MetaLight Zang et al. (2020) is a value-based meta reinforcement learning
method via parameter initialization, which is based on MAML Finn et al.
(2017). MetaLight is originally a single-agent approach for meta-learning on
multiple separate tasks. Here we extend it to a multi-agent scenario without
considering neighbor information.

  * •

PressLight Wei et al. (2019a) combines the traditional traffic method
MaxPressure Varaiya (2013) with RL technology together. PressLight is a RL
method that optimizes the pressure of each intersection.

  * •

CoLight Wei et al. (2019b) uses graph convolution and attention mechanism to
model the neighbor information, and then further uses this neighbor
information to optimize the queue length.

  * •

GeneraLight Zhang et al. (2020b) is a meta reinforcement learning method which
uses generative adversarial network to generate diverse traffic flows and uses
them to build training environments.


00footnotetext: $\bf{2020\ Mathematics\ Subject\ Classification\>}.$ 53B20;
53B30; 53B50; 83D05; 83C05.
Key words and phrases: Spacetime of quasi-constant sectional curvature;
Perfect fluids; $f(\mathcal{R})$-gravity.

# Characterizations of a spacetime of quasi-constant sectional curvature and
$\mathcal{F}(\mathcal{R})$-gravity

Uday Chand De, Krishnendu De∗, Fusun OZEN ZENGIN and Sezgin ALTAY DEMIRBAG
Department of Pure Mathematics, University of Calcutta, West Bengal, India.
ORCID iD: https://orcid.org/0000-0002-8990-4609<EMAIL_ADDRESS>∗ Department
of Mathematics, Kabi Sukanta Mahavidyalaya, The University of Burdwan.
Bhadreswar, P.O.-Angus, Hooghly, Pin 712221, West Bengal, India. ORCID iD:
https://orcid.org/0000-0001-6520-4520<EMAIL_ADDRESS>Department of
Mathematics, Istanbul Technical University, 34469, Istanbul, Turkey. ORCID iD:
https://orcid.org/ 0000-0002-5468-5100<EMAIL_ADDRESS>Department of
Mathematics, Istanbul Technical University, 34469, Istanbul, Turkey. ORCID iD:
https://orcid.org/0000-0002-6643-6267<EMAIL_ADDRESS>
###### Abstract.

The main aim of this article is to investigate a spacetime of quasi-constant
sectional curvature. At first, the existence of such a spacetime is
established by several examples. We have shown that a spacetime of quasi-
constant sectional curvature agrees with the present state of the universe and
it represents a Robertson Walker spacetime. Moreover, if the spacetime is
Ricci semi-symmetric or Ricci symmetric, then either the spacetime represents
a spacetime of constant sectional curvature, or the spacetime represents
phantom era. Also, we prove that a Ricci symmetric spacetime of quasi-constant
sectional curvature represents a static spacetime and the spacetime under
consideration is of Petrov type I, D or O. Finally, we concentrate on a quasi-
constant sectional curvature spacetime solution in
$\mathcal{F}(\mathcal{R})$-gravity. As a result, various energy conditions are
studied and analysed our obtained outcomes in terms of a
$\mathcal{F}(\mathcal{R})$-gravity model.

∗ Corresponding author

##  1\. Introduction

To study a conformally flat hypersurfaces of a Euclidean space the authors
[10] acquire the ensuing expression of the curvature tensor

(1.1) $\displaystyle R_{hijk}$ $\displaystyle=$
$\displaystyle\gamma(g_{hk}g_{ij}-g_{hj}g_{ik})$
$\displaystyle+\mu\\{g_{hk}A_{i}A_{j}+g_{ij}A_{h}A_{k}-g_{hj}A_{i}A_{k}-g_{ik}A_{h}A_{j}\\},$

in which $\gamma,\mu$ are scalars and $A_{i}$ is a unit vector, called the
generator. An n-dimensional conformally flat space obeying (1.1) is named a
space of quasi-constant sectional curvature and denoted by $(QC)_{n}$.

However, if the equation (1.1) of the curvature tensor holds, then it can be
easily verified that the space is conformally flat. So in the definition
conformally flatness is not required. A space of quasi-constant sectional
curvature has been studied by several authors, such as ([4], [10] [22], [37])
and many others.

Spacetime is a time-oriented 4-dimensional Lorentzian manifold $\mathcal{M}$
which is a specific class of semi-Riemannian manifold endowed with a
Lorentzian metric $g$ of signature $(-,+,+,+)$. A Lorentzian manifold is named
a spacetime of quasi-constant sectional curvature if the curvature tensor
$R_{hijk}$ fulfills the condition (1.1). Here, we assume the generator $A_{i}$
is a unit time-like vector field, that is, $A_{i}A^{i}=-1$,
$A^{i}=g^{ij}A_{j}$.

From the foregoing definition we can easily acquire

(1.2) $R_{ij}=\\{(n-1)\gamma+\mu\\}g_{ij}+(n-2)\mu
A_{i}A_{j};\;\;\;\;A_{i}A^{i}=-1,$

where $R_{hk}=R^{i}_{hki}$ is the Ricci tensor.

Multiplying the above equation with $g^{ij}$ yields

(1.3) $\mathcal{R}=n(n-1)\gamma-2(n-1)\mu,$

where $R=g^{hk}R_{hk}$ is the Ricci scalar. Again multiplying (1.2) by
$A^{i}A^{j}$, we get

(1.4) $R_{ij}A^{i}A^{j}=-(n-1)\gamma+(n-3)\mu.$

Let us denote the Ricci curvature $R_{ij}A^{i}A^{j}$ by $\nu$.

Hence from equations (1.3) and (1.4), we obtain

(1.5)
$\gamma=\frac{(n-3)\mathcal{R}-2\nu}{(n-1)\\{n(n-3)+2\\}};\;\;\;\mu=\frac{\mathcal{R}+n\nu}{n(n-3)+2}.$

Therefore, the associated scalars $\gamma$ and $\mu$ are related with the
Ricci scalar $\mathcal{R}$ and Ricci curvature $\nu$.

In a spacetime $\mathcal{M}^{n}$, for a smooth function $\psi>0$ (also called
scale factor, or warping function), if
$\mathcal{M}=-I\times_{\psi},\mathrm{M}$, where $I$ is the open interval of
$\mathbb{R}$ , $\mathrm{M}^{n-1}$ denotes the Riemannian manifold, then
$\mathcal{M}$ is named a generalized Robertson Walker (briefly, GRW) spacetime
[1]. If $\mathcal{M}^{3}$ is of constant sectional curvature, then the
spacetime represents a Robertson Walker (briefly, RW) spacetime.

Throughout this paper we consider 4-dimensional spacetime of quasi-constant
sectional curvature, denoted by $(QC)_{4}$\- spacetime.

Due to the absence of a stress tensor and heat conduction terms corresponding
to viscosity, the fluid is referred to as perfect and the energy momentum
tensor $T_{hk}$ is written by

(1.6) $T_{hk}=(\sigma+p)u_{h}u_{k}+pg_{hk},$

in which $p$ and $\sigma$ stand for the perfect fluid’s isotropic pressure and
energy density, respectively [29]. In the last equation, $g$ is the Lorentzian
metric and $u_{h}$, the velocity vector is defined by $g_{hk}u^{h}u^{k}=-1$
and $u_{h}=g_{hk}u^{k}$.

For a gravitational constant $\kappa$, the Einstein’s field equations
(briefly, EFEs) without a cosmological constant is described by

(1.7) $R_{hk}-\frac{R}{2}g_{hk}=\kappa T_{hk},$

where $R_{hk}=R^{i}_{hki}$ and $R=g^{hk}R_{hk}$ indicate the Ricci tensor and
the Ricci scalar, respectively.

A spacetime $\mathcal{M}$ is named a perfect fluid (briefly PF) spacetime if
the non-vanishing Ricci tensor $R_{hk}$ fulfills

(1.8) $R_{hk}=\alpha g_{hk}+\beta u_{h}u_{k},$

in which $\alpha$ and $\beta$ are smooth functions. The foregoing equation is
obtained from the equations (1.6) and (1.7).

Combining the equations (1.6), (1.7) and (1.8), we acquire

(1.9) $\beta=k^{2}(p+\sigma),\,\,\alpha=\frac{k^{2}(p-\sigma)}{2-n}.$

In [29], O’Neill established that a RW- spacetime is a PF- spacetime. Every
GRW- spacetime of dimension four is a PF- spacetime if it is a RW- spacetime
and vice-versa [24]. For additional insights about the PF-spacetimes and GRW
spacetimes, we refer( [3], [11], [18], [27]) and its references.

Additionally, a state equation with the form $p=p(\sigma)$ connects $p$ and
$\sigma$, and the PF-spacetime is known as isentropic. Furthermore, if
$p=\sigma$, the PF- spacetime is referred to as stiff matter. The PF-
spacetime is referred to as the dark matter era if $p+\sigma=0$, the dust
matter fluid if $p=0$, and the radiation era if $p=\frac{\sigma}{3}$ [9]. The
universe is represented as accelerating phase when
$\frac{p}{\sigma}<{-\frac{1}{3}}$. It covers the quintessence phase if
$-1<\frac{p}{\sigma}<0$ and phantom era if $\frac{p}{\sigma}<-1$.

In this paper, we consider a $(QC)_{4}$\- spacetime and prove the following:

######  Theorem 1.1.

A spacetime of quasi-constant sectional curvature represents a perfect fluid
spacetime and agrees with the present state of the universe.

As a consequence, we also show that the $(QC)_{4}$\- spacetime represents a
RW- spacetime.

The Ricci tensor is said to be of Codazzi type [23] if it fulfills the
subsequent relation

$\nabla_{l}R_{hk}=\nabla_{k}R_{hl}.$

Here we discuss a $(QC)_{4}$-spacetime fulfilling the Ricci tensor is of
Codazzi type and establish the subsequent theorem:

######  Theorem 1.2.

A $(QC)_{4}$-spacetime obeying $\gamma^{h}A_{h}=0$ endowed with Codazzi type
of Ricci tensor represents a spacetime of constant sectional curvature, or the
vorticity vanishes.

A spacetime is named semi-symmetric [33] if it obeys the relation

(1.10) $\nabla_{l}\nabla_{m}R^{h}_{ijk}-\nabla_{m}\nabla_{l}R^{h}_{ijk}=0,$

in which $\nabla$ indicates the covariant differentiation. It is to be noted
that the class of locally symmetric spaces ($\nabla_{l}R^{h}_{ijk}=0$) due to
Cartan is a proper subset of semi-symmetric spaces.

A spacetime is named Ricci semi-symmetric [28] if it fulfills the relation

(1.11) $\nabla_{l}\nabla_{m}R_{ij}-\nabla_{m}\nabla_{l}R_{ij}=0.$

In this article we consider a Ricci semi-symmetric $(QC)_{4}$-spacetime and
establish the following:

######  Theorem 1.3.

If a $(QC)_{4}$-spacetime is Ricci semi-symmetric, then either the spacetime
represents a spacetime of constant sectional curvature, or the spacetime
represents phantom era.

Again, the class of Ricci symmetric spaces ($\nabla_{l}R_{ij}=0$) is a proper
subset of Ricci semi-symmetric spaces. Every semi-symmetric space is known to
be Ricci semi-symmetric, but the converse is usually not true. In a Riemannian
space they are equivalent for dimension three. In [36], it has been
established that for $n\geq 3$, the above stated relations are equivalent for
hypersurfaces having non negative scalar curvature in a Euclidean space
$E^{n+1}$.

Inspired by the above studies we prove the subsequent result:

######  Corollary 1.1.

A Ricci symmetric $(QC)_{4}$-spacetime either represents a spacetime of
constant sectional curvature, or the phantom era.

If a Lorentzian manifold permits a timelike Killing vector field $\rho$, it is
referred to as a stationary spacetime and static ([30], [31], p. 283) if,
additionally, $\rho$ is irrotational. We will refer to $\rho$ in this context
as the static vector field, where it is assumed that spacetime is time-
oriented. The product $\mathbb{R}\times S$ is called a static spacetime if it
is equipped with the metric

(1.12) $g[(t,y)]=-\beta(y)dt^{2}+g_{S}[y],$

in which $g_{S}$ denotes a Riemannian metric on S. Any static spacetime
behaves like a standard one locally, with $\rho$ recognisable to $\partial t$.

Here we also consider a Ricci symmetric $(QC)_{4}$-spacetime and prove the
following:

######  Theorem 1.4.

If a $(QC)_{4}$-spacetime with $\mu\neq 0$ is Ricci symmetric, then the
spacetime represents a static spacetime and the spacetime is of Petrov type I,
D or O.

EFEs are incapable to answer for the universe’s late-time inflation without
accepting the existence of dark energy. This inspired other scholars to expand
it in order to obtain higher order gravity field equations. One of the
aforementioned modified theories of gravity was developed by substituting an
arbitrary function $\mathcal{F}(\mathcal{R})$ for the Ricci scalar
$\mathcal{R}$ in the Einstein-Hilbert action, which was first suggested in
1970 by H.A. Buchdahl [5]. Of course, various theoretical scalar-tensor
findings and observational data, constrain the feasibility of such functions.
Many useful functional versions of $\mathcal{F}(\mathcal{R})$ theory have been
developed recently; for additional information, see ([6], [12], [14], [16],
[17]). By the motivation of these investigations, in this article we
concentrate on a $(QC)_{4}$\- spacetime solution in
$\mathcal{F}(\mathcal{R})$-gravity and prove the following main results.

######  Theorem 1.5.

For $\mathcal{R}=$ constant, in any $(QC)_{4}$ spacetime solution of the
$\mathcal{F}(\mathcal{R})$-gravity, the Ricci tensor $R_{ij}$ is semi-
symmetric if and only if the energy momentum tensor $T_{ij}$ is semi-
symmetric.

######  Theorem 1.6.

For $\mathcal{R}=$ constant, in any $(QC)_{4}$ spacetime solution of the
$\mathcal{F}(\mathcal{R})$-gravity, if the Ricci tensor $R_{ij}$ is symmetric
then either the spacetime represents dark matter, or the energy density is
constant.

######  Theorem 1.7.

For $\mathcal{R}=$ constant, in any $(QC)_{4}$ spacetime solution of the
$\mathcal{F}(\mathcal{R})$-gravity, the isotropic pressure $p$ and energy
density $\sigma$ obey the subsequent equations

$p=\frac{(3\gamma-\mu)\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{\kappa^{2}}-\frac{\mathcal{F}(\mathcal{R})}{2\kappa^{2}},$
$\sigma=\frac{3(\mu-\gamma)\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{\kappa^{2}}+\frac{\mathcal{F}(\mathcal{R})}{2\kappa^{2}}.$

The energy conditions (ECs), as is well known in GR, are essential tools for
studying wormholes and black holes in various modified gravities ([2], [25]).
The ECs are carefully constructed using the Raychaudhuri equations [32], which
reflect the attractiveness of gravity through the positive condition
$\mathcal{R}_{lk}v^{l}v^{k}\geq 0,$ where $v^{l}$ is a null vector. This
condition on geometry in GR is comparable to the null energy condition (NEC)
$\mathcal{T}_{lk}v^{l}v^{k}\geq 0$ on matter through the EFE. In particular,
the weak energy condition (WEC), states that $\mathcal{T}_{lk}u^{l}u^{k}\geq
0,$ for every timelike vector $u^{l}$ and under the assumption of a positive
local energy density.

ECs are coordinate-invariant constraints on the energy-momentum tensor
(effective) that are helpful when considering the probability of different
matter origins, not just a PF continuum, and which fulfill the theories of
modified gravity while maintaining the notion that energy has to be positive.
There are various ECs; some of them, like the trace EC, are no longer relevant
today, while others are weaker and combined with others. However, the main
idea is to construct various scalar fields by contracting the energy momentum
tensor with any lightlike ,or timelike vectors. For our topic, we derive
several important ECs using our obtained results. Finally, we demonstrate some
ECs by considering a $\mathcal{F}(\mathcal{R})$ model.

##  2\. Examples of $(QC)_{4}$\- spacetime

###  2.1. Example 1 :

In [27], Mantica and Molinari established that every RW-spacetime is a
$(QC)_{4}$\- spacetime.

###  2.2. Example 2 :

The pseudo-symmetric manifold established by Chaki [8], whose Riemann
curvature fulfills the condition

$\nabla_{l}R_{hijk}=2A_{l}R_{hijk}+A_{h}R_{lijk}+A_{i}R_{hljk}+A_{j}R_{hilk}+A_{k}R_{hijl},$

in which $A_{i}$ indicates the non-zero 1-form. In [35], Tarafdar established
that a conformally flat pseudo-symmetric manifold is a space of quasi constant
curvature.

###  2.3. Example 3 :

In a 4-dimensional conformally flat spacetime, $R_{hijk}$ (curvature tensor)
is given by

(2.1) $\displaystyle R_{hijk}$ $\displaystyle=$
$\displaystyle\frac{1}{2}(g_{hk}R_{ij}-g_{hj}R_{ik}+g_{ij}R_{hk}-g_{ik}R_{hj})$
$\displaystyle+\frac{\mathcal{R}}{6}\\{g_{hk}g_{ij}-g_{hj}g_{ik}\\},$

A semi-Riemannian manifold $M$ is said to be Ricci simple [13] if

$R_{ij}=-\mathcal{R}A_{i}A_{j},$

$A_{i}$ is a unit time-like vector. The physical interpretation of Ricci
simple spacetime is given by Mantica and Molinari [27] in which they prove
that it represents stiff matter fluid. Now using the above relation in 2.1 we
obtain that the curvature tensor is of the form of a $(QC)_{4}$. Hence, every
Ricci simple conformally flat spacetime is a $(QC)_{4}$\- spacetime.

###  2.4. Example 4 :

A weakly Ricci symmetric manifold invented by Tamassy and Binh [34], whose
Ricci tensor $R_{ij}$ is not identically zero and obeys the condition

$\nabla_{l}R_{ij}=A_{l}R_{ij}+B_{i}R_{lj}+C_{j}R_{il},$

in which $A,B,C$ indicate the non-zero 1-forms.

In a $(WRS)_{4}$ (weakly Ricci symmetric spacetime) [15],
$R_{ij}=-\mathcal{R}u_{i}u_{j}.$ Using this in 2.1 we arrive at the same
result. Therefore, every conformally flat $(WRS)_{4}$ spacetime is a
$(QC)_{4}$\- spacetime.

##  3\. Spacetime of quasi-constant sectional curvature

Proof of the Theorem 1.1 :

Multiplying (1.1) with $g^{ij}$, we acquire

(3.1) $\displaystyle R_{hk}$ $\displaystyle=$
$\displaystyle\gamma(4g_{hk}-g_{hk})+\mu\\{-g_{hk}+4A_{h}A_{k}-A_{h}A_{k}-A_{h}A_{k}\\}$
$\displaystyle=(3\gamma-\mu)g_{hk}+2\mu A_{h}A_{k},$

which represents a PF- spacetime.

Multiplying (3.1) with $g^{hk}$, we obtain

(3.2) $\mathcal{R}=6(2\gamma-\mu).$

Comparing the previous equation with (1.9), we get

(3.3) $2\mu=k^{2}(p+\sigma),$

and

(3.4) $3\gamma-\mu=-\frac{k^{2}(p-\sigma)}{2}.$

From the last two equations, we infer

(3.5)
$p=\frac{1}{\kappa^{2}}(-3\gamma+2\mu)\;\;\;\sigma=\frac{1}{\kappa^{2}}3\gamma.$

Since $p$ and $\sigma$ are not constant, we can say the $(QC)_{4}$\- spacetime
agrees with the present state of the universe.

Hence, the theorem is proved.

If $3\gamma=2\mu$, the equation (3.5) gives $p=0$. Hence, we state

######  Corollary 3.1.

A spacetime of quasi-constant sectional curvature with $3\gamma=2\mu$
represents dust era.

From equation (3.5), we acquire $p+\sigma=\frac{2\mu}{\kappa^{2}}\neq 0$.
Also, we know that the space of quasi-constant sectional curvature is
conformally flat and hence $divC=0$ (‘$div$’ denotes the divergence), where C
is the Weyl curvature tensor. In [27], it is established that a PF- spacetime
with $p+\sigma\neq 0$ and $div\,C=0$, is a GRW- spacetime.

Again, for $n=4$, every GRW- spacetime is a PF- spacetime if and only if it is
a RW- spacetime [24] and thus the spacetime becomes RW- spacetime.

######  Corollary 3.2.

A spacetime of quasi-constant sectional curvature represents a RW- spacetime.

Proof of the Theorem 1.2 :

Let the Ricci tensor be of Codazzi type, which entails

(3.6) $\nabla_{l}R_{hk}=\nabla_{k}R_{hl}.$

Differentiating (3.1) covariantly yields

(3.7)
$\nabla_{l}R_{hk}=(3\gamma_{l}-\mu_{l})g_{hk}+2\mu_{l}A_{h}A_{k}+2\mu(\nabla_{l}A_{h}A_{k}-A_{h}\nabla_{l}A_{k}).$

Similarly, we get

(3.8)
$\nabla_{k}R_{hl}=(3\gamma_{k}-\mu_{k})g_{hl}+2\mu_{k}A_{h}A_{l}+2\mu(\nabla_{k}A_{h}A_{l}-A_{h}\nabla_{k}A_{l}).$

Hence using the foregoing equations in (3.6), we acquire

(3.9) $\displaystyle 0$ $\displaystyle=$
$\displaystyle\nabla_{l}R_{hk}-\nabla_{k}R_{hl}$
$\displaystyle=(3\gamma_{l}-\mu_{l})g_{hk}-(3\gamma_{k}-\mu_{k})g_{hl}$
$\displaystyle+2\mu_{l}A_{h}A_{k}-2\mu_{k}A_{h}A_{l}+2\mu(\nabla_{l}A_{h}A_{k}-A_{h}\nabla_{l}A_{k})-2\mu(\nabla_{k}A_{h}A_{l}-A_{h}\nabla_{k}A_{l}).$

Multiplying the above equation by $g^{hk}$ gives

$\displaystyle 0$ $\displaystyle=$ $\displaystyle
4(3\gamma_{l}-\mu_{l})-(3\gamma_{l}-\mu_{l})$
$\displaystyle-2\mu_{l}-2\mu^{h}A_{h}A_{l}-2\mu(\nabla_{h}A^{h}A_{l}+A^{k}\nabla_{k}A_{l}),$

which implies

(3.10) $\displaystyle 0$ $\displaystyle=$ $\displaystyle
9\gamma_{l}-5\mu_{l}-2\mu^{h}A_{h}A_{l}$
$\displaystyle-2\mu(\nabla_{h}A^{h}A_{l}+A^{k}\nabla_{k}A_{l}).$

Equation (3.6) implies $\mathcal{R}=$ constant and hence from (3.2), we
acquire $2\gamma_{l}=\mu_{l}$. Using this in the above equation we infer

(3.11)
$0=-\gamma_{l}-4\gamma^{h}A_{h}A_{l}-2\mu(A^{h}_{h}A_{l}+A^{k}\nabla_{k}A_{l}).$

Multiplying (3.12) by $A^{l}$, we obtain

$0=-\gamma_{l}A^{l}+4\gamma^{h}A_{h}-2\mu A^{h}_{h},$

which implies

(3.12) $0=3\gamma^{h}A_{h}-2\mu A^{h}_{h}.$

Let us consider $\gamma^{h}A_{h}=0$. Then the previous equation entails that
either $\mu=0$, or $divA^{h}=0$.

If $\mu=0$, the spacetime of quasi constant curvature reduces to the spacetime
of constant sectional curvature.

If $divA^{h}=0$, then the velocity vector field is conservative. A
conservative vector field is always irrotational, thus we conclude that the PF
has zero vorticity.

This ends the proof.

Proof of the Theorem 1.3 :

Let a $(QC)_{4}$-spacetime be Ricci semi-symmetric, that is

(3.13) $\nabla_{l}\nabla_{m}R_{ij}-\nabla_{m}\nabla_{l}R_{ij}=0.$

Hence using (3.1) in the above equation, we acquire

(3.14)
$\mu\\{\nabla_{l}\nabla_{m}A_{h}A_{k}+A_{h}\nabla_{l}\nabla_{m}A_{k}-\nabla_{m}\nabla_{l}A_{h}A_{k}-A_{h}\nabla_{m}\nabla_{l}A_{k}\\}=0,$

This implies either $\mu=0$, or $\mu\neq 0$.

Case (i): If $\mu=0$, the spacetime of quasi constant curvature reduces to the
spacetime of constant curvature.

Case (ii): If $\mu\neq 0$, then
$(\nabla_{l}\nabla_{m}A_{h}-\nabla_{m}\nabla_{l}A_{h})A_{k}+A_{h}(\nabla_{l}\nabla_{m}A_{k}-\nabla_{m}\nabla_{l}A_{k})=0$.
Hence applying Ricci identity, we get

(3.15) $\\{A_{p}R^{p}_{hlm}A_{k}+A_{h}A_{p}R^{p}_{klm}\\}=0.$

Multiplying with $A^{k}$ the foregoing equation yields

(3.16) $\\{-A_{p}R^{p}_{hlm}+A^{p}A^{k}R_{pklm}\\}=0.$

Since $R_{pkml}=-R_{kpml}$, therefore $A^{p}A^{k}R_{pklm}=0.$

Hence the above equation gives

(3.17) $A_{p}R^{p}_{hlm}=0,$

which implies

(3.18) $A_{p}R^{p}_{h}=0.$

Now from (3.1), we obtain

(3.19) $A^{k}R_{hk}=(3\gamma-\mu)A_{h}-2\mu A_{h}.$

Using (3.18) we can write

(3.20) $(3\gamma-\mu)A_{h}-2\mu A_{h}=0,$

which implies $\mu=\gamma$.

Finally, we get either $\mu=0$, or $\mu=\gamma$.

If $\mu=\gamma$, then using this relation in (3.5), we infer
$\frac{p}{\sigma}=-\frac{1}{3}$ which means the spacetime represents phantom
era.

This finishes the proof.

Proof of the Corollary 1.1 :

Since Ricci symmetry ($\nabla_{l}R_{ij}=0$) implies Ricci semi-symmetry
($\nabla_{l}\nabla_{m}R_{ij}-\nabla_{m}\nabla_{l}R_{ij}=0$), therefore from
the above result for a Ricci symmetric $(QC)_{4}$-spacetime we can state
either the spacetime represents a spacetime of constant sectional curvature,
or the spacetime represents phantom era.

Therefore the proof is completed.

Proof of the Theorem 1.4 :

Let $\mu\neq 0$ and $\mu=\gamma$.

Using $\mu=\gamma$ in (3.1), we acquire

(3.21) $R_{ij}=2\mu(g_{ij}+A_{i}A_{j}).$

Covariant differentiation of the foregoing equation yields

(3.22)
$\nabla_{l}R_{ij}=2\nabla_{l}\mu(g_{ij}+A_{i}A_{j})+2\mu(\nabla_{l}A_{i}A_{j}+A_{i}\nabla_{l}A_{j}).$

By hypothesis $\nabla_{l}R_{ij}=0$. Hence from the above we infer

(3.23)
$\nabla_{l}\mu(g_{ij}+A_{i}A_{j})+\mu(\nabla_{l}A_{i}A_{j}+A_{i}\nabla_{l}A_{j})=0.$

Multiplying by $A^{i}$ gives

(3.24) $\mu\nabla_{l}A_{j}=0,$

since $A^{i}\nabla_{l}A_{i}=0$.

Therefore, we obtain $\nabla_{l}A_{j}=0$, since $\mu\neq 0$.

Now $\nabla_{l}A_{j}=0$ implies that $A_{j}$ is a Killing vector and also
irrotational. Hence the spacetime becomes static [31].

It is well-known ([20], Section 10.7) that any static spacetime is everywhere
of Petrov type I, D or O. As a result, the spacetime under consideration is of
Petrov type I, D or O.

This ends the proof.

##  4\.  $\mathcal{F}(\mathcal{R})$-Gravity

Taking the Einstein-Hilbert action term into account

$\displaystyle
S=\frac{1}{2\kappa^{2}}\int{{\sqrt{-g}}\mathcal{F}(\mathcal{R})d^{4}x}+\int{{\sqrt{-g}}L_{m}d^{4}x}$

where $k=\sqrt{8{\pi}G}$, $G$ is Newton’s constant and $L_{m}$ is referred as
the matter Lagrangian density and described by

$\displaystyle T_{ij}=-\frac{2}{\sqrt{-g}}\frac{\delta(\sqrt{-g}L_{m})}{\delta
g^{ij}}$

The field equation by utilizing the variation with regard to $g^{ij}$, can
have the form

(4.1)
$\displaystyle\mathcal{F}_{\mathcal{R}}(\mathcal{R})R_{ij}-(\nabla_{i}\nabla_{j}\mathcal{F}_{\mathcal{R}}(\mathcal{R})-g_{ij}{\square})-\frac{\mathcal{F}(\mathcal{R})}{2}g_{ij}=k^{2}T_{ij}$

where the D’Alembert operator is represented by $\square$ and the
differentiation with respect to $\mathcal{R}$ is denoted by
”$\mathcal{F}_{\mathcal{R}}(\mathcal{R})$”. If
$\frac{\mathcal{R}F_{\mathcal{R}}(\mathcal{R})}{2}g_{ij}$ is subtracted from
both sides of (4.1), then the result is

(4.2)
$\displaystyle\mathcal{F}_{\mathcal{R}}(\mathcal{R})R_{ij}-\frac{\mathcal{R}F_{\mathcal{R}}(\mathcal{R})}{2}g_{ij}=k^{2}T_{ij}+k^{2}T^{(curve)}_{ij}$

such that

(4.3) $\displaystyle T^{(eff)}_{ij}=T_{ij}+T^{(curve)}_{ij}$

where

(4.4) $\displaystyle
T^{(curve)}_{ij}=\frac{1}{k^{2}}\big{[}(\nabla_{i}\nabla_{j}-g_{ij}{\square})\mathcal{F}_{\mathcal{R}}(\mathcal{R})+\frac{(\mathcal{F}(\mathcal{R})-\mathcal{R}\mathcal{F}_{\mathcal{R}}(\mathcal{R}))}{2}g_{ij}\big{]}.$

$T^{(eff)}_{ij}$ represents an efficient voltage-energy tensor.

With the help of (4.2) and (4.4) in case of constant $\mathcal{R}$, we acquire

(4.5) $\displaystyle
R_{ij}-\frac{\mathcal{R}}{2}g_{ij}=\frac{\kappa^{2}}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}T_{ij}+\big{[}\frac{(\mathcal{F}(\mathcal{R})-\mathcal{R}\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{2\mathcal{F}_{\mathcal{R}}(\mathcal{R})}g_{ij}\big{]}.$

######  Lemma 4.1.

[26] For $\mathcal{R}=$ constant, in a PF- spacetime solution of the
$\mathcal{F}(\mathcal{R})$-gravity, the effective pressure $p^{(eff)}$ and the
effective energy density $\sigma^{(eff)}$ obey the subsequent equations

$p^{(eff)}=p+\frac{1}{2\kappa^{2}}\big{[}(\mathcal{F}(\mathcal{R})-\mathcal{R}\mathcal{F}_{\mathcal{R}}(\mathcal{R})\big{]},$
$\sigma^{(eff)}=\sigma-\frac{1}{2\kappa^{2}}\big{[}\mathcal{F}(\mathcal{R})-\mathcal{R}\mathcal{F}_{\mathcal{R}}(\mathcal{R})\big{]}.$

In Theorem 1.1, we establish that every $(QC)_{4}$ spacetime is a PF-
spacetime. Hence, the energy momentum tensor is of the form (1.6). Here we
assume that the velocity vector field $u_{i}$ of the PF-spacetime is identical
with the associated vector $A_{i}$ of the $(QC)_{4}$ spacetime.

Now, using (1.6) in (4.5), we acquire

(4.6) $\displaystyle
R_{ij}-\frac{\mathcal{R}}{2}g_{ij}=\frac{\kappa^{2}}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}\\{(p+\sigma)A_{i}A_{j}+pg_{ij}\\}+\big{[}\frac{(\mathcal{F}(\mathcal{R})-\mathcal{R}\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{2\mathcal{F}_{\mathcal{R}}(\mathcal{R})}g_{ij}\big{]}.$

Multiplying both sides of the previous equation by $g^{ij}$ gives

(4.7)
$\displaystyle\mathcal{R}-2\mathcal{R}=\frac{\kappa^{2}}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}\\{-(p+\sigma)+4p\\}+2\big{[}\frac{(\mathcal{F}(\mathcal{R})-\mathcal{R}\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}\big{]},$

which implies

(4.8)
$\displaystyle\mathcal{R}=\frac{1}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}\\{\kappa^{2}(3p-\sigma)+2\mathcal{F}(\mathcal{R})\\}.$

Putting the value of $\mathcal{R}$ in (4.6) yields

(4.9) $\displaystyle
R_{ij}=\frac{2\kappa^{2}p+\mathcal{F}(\mathcal{R})}{2\mathcal{F}_{\mathcal{R}}(\mathcal{R})}g_{ij}+\frac{\kappa^{2}(p+\sigma)}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}A_{i}A_{j}.$

######  Theorem 4.1.

For $\mathcal{R}=$ constant, in any $(QC)_{4}$ spacetime solution of the
$\mathcal{F}(\mathcal{R})$-gravity theory, the Ricci tensor $R_{ij}$ is of the
form (4.9).

Proof of the Theorem 1.5 :

From the field equation (4.6), we can easily acquire

(4.10)
$\displaystyle\nabla_{l}\nabla_{m}T_{ij}-\nabla_{m}\nabla_{l}T_{ij}=0\Leftrightarrow\nabla_{l}\nabla_{m}R_{ij}-\nabla_{m}\nabla_{l}R_{ij}=0.$

Hence, the theorem is proved.

Proof of the Theorem 1.6 :

Covariant differentiation of the equation (4.9) yields

(4.11)
$\displaystyle\nabla_{l}\nabla_{m}R_{ij}=\nabla_{l}\nabla_{m}ag_{ij}+\nabla_{l}\nabla_{m}bA_{i}A_{j}+b\\{\nabla_{l}\nabla_{m}A_{i}A_{j}+A_{i}\nabla_{l}\nabla_{m}A_{j}\\}.$

where
$a=\frac{2\kappa^{2}p+\mathcal{F}(\mathcal{R})}{2\mathcal{F}_{\mathcal{R}}(\mathcal{R})}$
and $b=\frac{\kappa^{2}(p+\sigma)}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}$.
The foregoing equation immediately gives

(4.12)
$\displaystyle\nabla_{l}\nabla_{m}R_{ij}-\nabla_{m}\nabla_{l}R_{ij}=b\\{\nabla_{l}\nabla_{m}A_{i}A_{j}-\nabla_{m}\nabla_{l}A_{i}A_{j}+A_{i}\nabla_{l}\nabla_{m}A_{j}-A_{i}\nabla_{m}\nabla_{l}A_{j}\\}.$

Let the Ricci tensor be semi-symmetric, that is,
$R^{h}_{ijk,lm}-R^{h}_{ijk,ml}=0$. Then using Ricci identity the above
equation infers that

(4.13) $\displaystyle 0=b\\{A_{j}A_{h}R^{h}_{ilm}+A_{i}A_{h}R^{h}_{jlm}\\},$

which implies either $b=0$, or $b\neq 0.$

Case (i): If $b=0$, then we have $p+\sigma=0$. Therefore, the spacetime
represents the dark matter era.

Case (ii): If $b\neq 0$, then $A_{j}A_{h}R^{h}_{ilm}+A_{i}A_{h}R^{h}_{jlm}=0.$
Therefore multiplying the previous relation by $g^{lm}$ we have

(4.14) $\displaystyle A_{j}A_{h}R^{h}_{i}+A_{i}A_{h}R^{h}_{j}=0.$

Also, from (4.9), we acquire

(4.15) $\displaystyle A^{j}R_{ij}=aA_{i}-bA_{i}=(a-b)A_{i}.$

Using (4.15) in (4.14), we get

(4.16) $\displaystyle(a-b)A_{i}A_{j}=0.$

From the previous equation we conclude that $a=b$ which implies
$\sigma=\frac{\mathcal{F}(\mathcal{R})}{2\kappa^{2}}=$ constant.

This ends the proof.

Proof of the Theorem 1.7 :

Using (3.1) the field equation turns into

(4.17) $\displaystyle(3\gamma-\mu)g_{ij}+2\mu
A_{i}A_{j}-\frac{\mathcal{R}}{2}g_{ij}=\frac{\kappa^{2}}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}T_{ij}+\big{[}\frac{(\mathcal{F}(\mathcal{R})-\mathcal{R}\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{2\mathcal{F}_{\mathcal{R}}(\mathcal{R})}g_{ij}\big{]}.$

Multiplying both sides of the foregoing equation by $g^{ij}$ yields

(4.18) $\displaystyle
4(3\gamma-\mu)-2\mu=\frac{\kappa^{2}}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}\big{[}3p-\sigma\big{]}+\frac{2\mathcal{F}(\mathcal{R})}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}.$

Again multiplying (4.17) by $A^{i}A^{j}$, we obtain

(4.19)
$\displaystyle-(3\gamma-\mu)+2\mu=\frac{\kappa^{2}\sigma}{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}-\frac{\mathcal{F}(\mathcal{R})}{2\mathcal{F}_{\mathcal{R}}(\mathcal{R})},$

which implies

(4.20)
$\displaystyle\sigma=\frac{3(\mu-\gamma)\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{\kappa^{2}}+\frac{\mathcal{F}(\mathcal{R})}{2\kappa^{2}},$

Using the value of $\sigma$, we get from (4.18) that

(4.21) $\displaystyle
p=\frac{(3\gamma-\mu)\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{\kappa^{2}}-\frac{\mathcal{F}(\mathcal{R})}{2\kappa^{2}},$

This completes the proof.

Using Theorem 1.7 in Lemma 4.1, we get the subsequent result:

######  Corollary 4.1.

For $\mathcal{R}=$ constant, in any $(QC)_{4}$ spacetime solution of the
$\mathcal{F}(\mathcal{R})$-gravity, the effective pressure and density are
described by

$\sigma^{(eff)}=(3\mu-3\gamma+\mathcal{R})\frac{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{2\kappa^{2}},$
$p^{(eff)}=(6\gamma-2\mu-\mathcal{R})\frac{\mathcal{F}_{\mathcal{R}}(\mathcal{R})}{2\kappa^{2}}.$

##  5\. Energy Conditions

Now, we are dealing with a PF matter distribution and in accordance with the
research by Capozziello et al. [7], the ECs retrieved from standard GR are

(5.1) $\displaystyle NEC$
$\displaystyle\;\;\;if\;\;\;\sigma^{(eff)}+p^{(eff)}\geq 0,$ (5.2)
$\displaystyle WEC$ $\displaystyle\;\;\;if\;\;\;\sigma^{(eff)}\geq
0\quad\mathrm{and}\quad\sigma^{(eff)}+p^{(eff)}\geq 0,$ (5.3) $\displaystyle
DEC$ $\displaystyle\;\;\;if\;\;\;\sigma^{(eff)}\geq
0\quad\mathrm{and}\quad\sigma^{(eff)}\pm p^{(eff)}\geq 0,$ (5.4)
$\displaystyle SEC$ $\displaystyle\;\;\;if\;\;\;\sigma^{(eff)}+3p^{(eff)}\geq
0.$

We now examine the ECs for a $\mathcal{F}\left(\mathcal{R}\right)$-gravity
model in the following subsection.

A. Model:
$\mathcal{F}\left(\mathcal{R}\right)=e^{\mathcal{R}}\log\mathcal{R}-\log\mathcal{R}-\mathcal{R}-\sum\limits_{l=2}^{\infty}\dfrac{\mathcal{R}^{l}}{l\cdot
l!}$

For this model, with the help of equation (3.2) and corollary 4.1, the
effective energy density and pressure are presented as

(5.5)
$\displaystyle\sigma^{(eff)}=\dfrac{1}{2\kappa^{2}}\left(9\gamma-3\mu\right)e^{\mathcal{R}}\log\mathcal{R},$
(5.6)
$p^{(eff)}=\dfrac{1}{2\kappa^{2}}\left(4\mu-6\gamma\right)e^{\mathcal{R}}\log\mathcal{R}.$

The ECs for this setup can now be discussed using (5.5) and (5.6).

Fig. 1: Evolution of energy density with respect to $\mu$ and $\gamma$
---
Fig. 2: Evolution of WEC with respect to $\mu$ and $\gamma$
---

Figs. $1$ and $2$, respectively, show the profiles of $\sigma^{eff}$ and WEC.
The effective energy density is positive throughout the evolution of Ricci
scalar $\mathcal{R}>1$, $\mu>0$ and $\gamma>\dfrac{\mu}{3}$ . One can see from
Fig. $1$ that for higher values of $\mu$, the effective energy density is
high. Fig. 2 shows the WEC profile, which has a positive range for its value.
As NEC is a part of WEC. Consequently, NEC and WEC are satisfied.  Fig. 3:
Evolution of DEC with respect to $\mu$ and $\gamma$ Fig. 4: Evolution of SEC
with respect to $\mu$ and $\gamma$ To finish up our discussions, we have drawn
DEC and SEC in the Figs. $3$ and $4$ for the above model. From these figures
it is obvious that DEC does not hold but SEC does.

##  6\. Discussion

Spacetime, a torsion-free, time-oriented Lorentzian manifold, is the stage on
which the physical world is now being modelled. According to GR theory, the
universe’s matter content may be determined by choosing the appropriate energy
momentum tensor, and is accepted to act like a PF- spacetime in the
cosmological models.

Here, we study a spacetime of quasi- constant sectional curvature, and we have
demonstrated that this spacetime is consistent with the universe’s current
condition and represents a PF- spacetime. Additionally, if the spacetime is
Ricci symmetric or Ricci semisymmetric, it either represents a spacetime of
constant sectional curvature or phantom era. We further establish that a Ricci
symmetric $(QC)_{4}$-spacetime implies a static spacetime and is of Petrov
type I, D, or O.

The investigation of $(QC)_{4}$-spacetime within the context of
$\mathcal{F}\left(\mathcal{\mathcal{R}}\right)$-gravity has been the foremost
concern of this article. Here, both analytic and graphical analysis of our
investigations have been done. For a better understanding, we have applied the
analytical method to develop our formulation and graphical analysis has been
done to assess the stability of one cosmological toy model, such as
$\mathcal{F}\left(\mathcal{R}\right)=e^{\mathcal{R}}\log\mathcal{R}-\log\mathcal{R}-\mathcal{R}-\sum\limits_{l=2}^{\infty}\dfrac{\mathcal{R}^{l}}{l\cdot
l!}$.

Additionally, we looked at the cosmological models’ stability analysis using
ECs. Figures 1, 2, 3, and 4 show the profiles of ECs for our model. The
evolution of the energy density for parameters $\mu>1$ and $\gamma>0.5$ has
been found to be positive. However, DEC broke the agreement, whereas SEC, NEC
and WEC have been satisfied. The accelerated expansion of the cosmos is
consistent with each of the aforementioned energy condition profiles. This
agrees with the findings from the prior investigation of ECs in
$\mathcal{F}\left(\mathcal{\mathcal{R}}\right)$-gravity.

##  7\. Declarations

###  7.1. Funding

Not applicable.

###  7.2. Conflicts of interest/Competing interests

The authors declare that they have no conflict of interest and all authors
contributed equally to this work.

###  7.3. Availability of data and material

Not applicable.

###  7.4. Code availability

Not applicable.

## References

  * [1] L. Alias, A. Romero and M. Sanchez, Uniqueness of complete spacelike hypersurfaces of constant mean curvature in generalized Robertson-Walker spacetimes, Gen. Relativ. Gravit. 27 (1995), 71-84.
  * [2] K. Bamba, M. Ilyas, M.Z. Bhatti and Z. Yousaf, Energy conditions in modified f (G) gravity, Gen. Relativ. Gravit., 49 (2017), Article ID 112.
  * [3] A. M. Blaga, Solitons and geometrical structures in a perfect fluid spacetime, Rocky Mountain J. Math. 50(1) (2020), 41-53.
  * [4] V. Boju and M. Popescu, Espaces a courbure quasi-constante, J. Diff. Geom. 13 (1978), 373-383.
  * [5] H.A. Buchdahl, Non-linear Lagrangians and cosmological theory, Mon. Not. Roy. Astron. Soc., 150 1, (1970).
  * [6] S. Capozziello, C.A. Mantica and L.G. Molinari, Cosmological perfect fluids in higher-order gravity, Gen. Relativ. Gravit. 52, 36(2020).
  * [7] S. Capozzeillo, S. Nojiri, S. D. Odinstov, The role of energy conditions in f(R) cosmology, Phys. Lett. B, 781 (2018) 99.
  * [8] M.C. Chaki, On pseudo symmetric manifolds, An. Ştiinţ. Univ. Al. I. Cuza Iaşi. Mat. (N.S.), 33(1), (1987), 53-58.
  * [9] P.H. Chavanis, Cosmology with a stiff matter era, Phys. Rev. D 92, 103004 (2015).
  * [10] B.Y. Chen and K. Yano, Hypersurfaces of a conformally flat space, Teensor, N.S., 26 (1972), 318-322.
  * [11] B. Y. Chen, A simple characterization of generalized Robertson-Walker spacetimes, Gen. Relativ. Gravit. 46 (2014), 1833 (5 pages).
  * [12] G.B. Christian, T. Harko and S.N.L. Francisco Lobob,d, Dark matter as a geometric effect in $f(R)$ gravity, Astroparticle Physics 29 (2008) 386–392.
  * [13] Deprez, J., Roter, W. and Verstraelen, L., Conditions on the projective curvature tensor of conformally flat Riemannian manifolds, Kyungpook Math. J., 29 (1989), 153–166.
  * [14] A. De and T.H. Loo, Almost pseudo-Ricci symmetric spacetime solutions in $f(R)$-gravity, Gen. Relativ. Gravit. 53, 5(2021).
  * [15] A. De, Weakly Ricci symmetric spacetimes, Int. J. Geom. Methods Mod. Phys. 15 (2018), 1850007. https://doi.org/10.1142/S021988781850007X
  * [16] K. De and U.C. De, Investigation of generalized $\mathcal{Z}$\- recurrent spacetimes and $f(\mathcal{R},T)$-gravity, Adv. Appl. Clifford Algebras 31, 38 (2021).
  * [17] K. De and U.C. De, Investigations on solitons in $f(\mathcal{R})$-gravity, Eur. Phys. J. Plus (2022) 137:180. https://doi.org/10.1140/epjp/s13360-022-02399-y
  * [18] U.C. De, S.K. Chaubey and S. Shenawy, Perfect fluid spacetimes and Yamabe solitons, J Math Phys. 62, 032501 (2021); https://doi.org/10.1063/5.0033967
  * [19] U.C. De, Y.J. Suh and S.K. Chaubey, Semi-symmetric curvature properties of Robertson-Walker spacetimes, accepted in Journal of Mathematical Physics, Analysis, Geometry.
  * [20] J. Ehlers and W. Kundt, Exact solutions of the gravitational field equations, in Gravitation: An Introduction to Current Research, ed. L. Witten, John Wiley and Sons, Inc., New York, 1962, 49-101.
  * [21] G.F.R. Ellis and R.K. Sachs, Relativistic Cosmology in General Relativity and Cosmology, Academic Press, London, 1971, 104-179.
  * [22] G. Ganchev and V. Mihova, Riemannian manifolds of quasi-constant sectional curvatures, vol. 2000, no. 522, 2000, pp. 119-141. https://doi.org/10.1515/crll.2000.034
  * [23] A. Gray,Einstein-like manifolds which are not Einstein, Geom. Dedicata 7 (1978), 259-280.
  * [24] M. Guti$\acute{e}$rrez and B. Olea, Global decomposition of a Lorentzian manifold as a generalized Robertson-Walker space, Differ. Geom. Appl. 27 (2009), 146-156.
  * [25] S.W. Hawking and G.F.R. Ellis, The Large Scale Structure of Space-Time, Cambridge University Press, London, 1973.
  * [26] T.H. Loo, A. De, S. Mandal and P. K. Sahoo, How a projectively flat geometry regulates F(R)-gravity theory?, Phys. Scr. 96 (2021), 125034. https://doi.org/10.1088/1402-4896/ac3a51
  * [27] C.A. Mantica and L.G. Molinari, Generalized Robertson-Walker spacetimes-A survey, Int. J. Geom. Methods Mod. Phys. 14 (2017), 1730001 (27 pages).
  * [28] V. A. Mirzoyan, Structure theorems for Riemannian Ric-semisymmetric spaces, Russian Math. (Iz. VUZ), 36 (1992), 75–83.
  * [29] B. O’Neill, Semi-Riemannian Geometry with Applications to the Relativity, Academic Press, New York-London, 1983.
  * [30] M. Sánchez, On the geometry of static spacetimes, Nonlinear Analysis: Theory, Methods and Applications, 63 5–7, (2005), 455-463. https://doi.org/10.1016/j.na.2004.09.009.
  * [31] H. Stephani, D. Kramer, M. Mac-Callum, C. Hoenselaers and E. Hertl, Exact Solutions of Einstein’s Field Equations, 2nd edn. Cambridge Monographs on Mathematical Physics, Cambridge University Press, 2009.
  * [32] A.K. Raychaudhuri, S. Banerji and A. Banerjee, General relativity, astrophysics, and cosmology, Springer-Verlag New York, Inc. 1992.
  * [33] Z.I. Szabo, Structure theorems on Riemannian spaces satisfying $R(X,Y).R=0$, J. Diff. Geom. 17 (1982), 531-582.
  * [34] L.Tamassy and T.Q. Binh, On weak symmetries of Einstein and Sasakian manifolds, Tensor, N.S. 53 (1993), 140–148.
  * [35] M. Tarafdar, On conformally flat pseudosymmetric manifolds, An. Ştiinţ. Univ. Al. I. Cuza Iaşi. Mat. (N.S.) 41 (1995), 237-241.
  * [36] S. Tanno, Hypersurfaces satisfying a certain condition on the Ricci tensor,  Tohoku Math. J. 21 (1969), 297-303.
  * [37] Y. Wang, On some properties of Riemannian spaces of quasi-constant curvature, Teensor, N.S., 35 (1981), 173-176.