# Deep Reinforcement Learning-Assisted Age-optimal Transmission Policy for
HARQ-aided NOMA Networks

Kunpeng Liu ID , Aimin Li ID , _Graduate Student Member, IEEE,_ Shaohua Wu ID
, _Member, IEEE_ This work has been supported in part by the Guangdong Basic
and Applied Basic Research Foundation under Grant no. 2022B1515120002, and in
part by the National Natural Science Foundation of China under Grant no.
61871147. K. Liu and A. Li are with the Department of Electronics and
Information Engineering, Harbin Institute of Technology (Shenzhen), Shenzhen
518055, China (e-mail<EMAIL_ADDRESS>liaimin@stu.hit.edu.cn). S.
Wu is with the Guangdong Provincial Key Laboratory of Aerospace Communication
and Networking Technology, Harbin Institute of Technology (Shenzhen), Shenzhen
518055, China, and also with the Peng Cheng Laboratory, Shenzhen 518055, China
(e-mail: hitwush@hit.edu.cn).

###### Abstract

The recent interweaving of AI-6G technologies has sparked extensive research
interest in further enhancing reliable and timely communications. _Age of
Information_ (AoI), as a novel and integrated metric implying the intricate
trade-offs among reliability, latency, and update frequency, has been well-
researched since its conception. This paper contributes new results in this
area by employing a Deep Reinforcement Learning (DRL) approach to
intelligently decide how to allocate power resources and when to retransmit in
a _freshness-sensitive_ downlink multi-user Hybrid Automatic Repeat reQuest
with Chase Combining (HARQ-CC) aided Non-Orthogonal Multiple Access (NOMA)
network. Specifically, an AoI minimization problem is formulated as a Markov
Decision Process (MDP) problem. Then, to achieve deterministic, age-optimal,
and intelligent power allocations and retransmission decisions, the Double-
Dueling-Deep Q Network (DQN) is adopted. Furthermore, a more flexible
retransmission scheme, referred to as Retransmit-At-Will scheme, is proposed
to further facilitate the timeliness of the HARQ-aided NOMA network.
Simulation results verify the superiority of the proposed intelligent scheme
and demonstrate the threshold structure of the retransmission policy. Also,
answers to whether user pairing is necessary are discussed by extensive
simulation results.

###### Index Terms:

Age of information, HARQ, NOMA, reinforcement learning, power allocation.

##  I Introduction

With the rapid development of the 5th Generation Mobile Communication (5G),
more and more Internet of Things (IoT) devices are accessed to the networks
[1]. In the IoT application scenarios, such as automatic driving, smart
factory, and smart healthcare, etc., the timeliness of information plays a
significant role in enabling timely, accurate, and effective decision-making.
In this regard, it is imperative to deliver _fresh_ status updates in a timely
and reliable manner, since the _stale_ one usually contains little value.

To measure the timeliness of information, _Age of Information_ (AoI) is first
proposed in [2] for automated driving applications. AoI is defined as the time
elapsed since the generation of the latest status which is received
successfully. Different from the traditional metric in the 5G ultra-reliable-
low-latency (URLLC), such as latency and throughput, AoI contemplates an
intricate trade-off among reliability, latency, and update frequency. As such,
AoI is also known as a type of semantic metrics, and has attracted massive
interest from government agencies, industry, and academia.

With the massive access of IoT devices, the research of high timeliness access
technique becomes a hot topic. Non-orthogonal multiple access (NOMA) is
considered as a promising access technique for the future network due to its
high spectral efficiency [3]. In [4] and [5], the authors have shed light on
the potential superior AoI performance in the high signal-to-noise ratio (SNR)
regime. However, in the low SNR regime, the age performance of NOMA system
inexorably faces a bottleneck.

To this end, HARQ is introduced in [6] to ensure reliability in the low SNR
regime by achieving retransmission-driven time diversity. Up to this point,
the HARQ-aided NOMA network has demonstrated its superiority in both
throughput and latency [7] and [8], while the age performance of HARQ-aided
NOMA network is seldom investigated. An exception is our previous work in [9],
which analyzes the average blocklength error ratio (BLER) of users in the
HARQ-aided NOMA network, and provides an optimal transmission policy by
formulating an MDP problem. However, the theory and analysis in [9] limited
the number of users and the maximum transmission round to 2 only, which
hinders flexible and timely system design.

In the multi-user NOMA network, user paring is a commonly-implemented
technique to simplify the transmission mode. [10] has demonstrated that in
terms of throughput performance, the optimal pairing policy manifests a
strongest-weakest symmetrical structure, i.e., the user with the best channel
condition is paired with the user with the worst channel condition
sequentially, so as to accomplish the pairing of all users. Nevertheless, the
insight on whether user pairing will benefit the timeliness performance of the
NOMA-based network remains an open issue.

Besides, DRL algorithms are widely used in wireless communication to solve
high-complexity problems. Reinforcement on Federated (RoF) scheme, based on
deep multi-agent reinforcement learning is proposed in [11]. In [12], authors
adopt TD3 algorithm to allocate the available resources for IoT.

Motivated by the above, we aim to investigate the optimal policy to minimize
the average AoI in an HARQ-aided NOMA network with multiple users and
unrestricted transmission rounds. Specifically, this paper achieves three-fold
contributions:

  * •

We optimize the average AoI of an HARQ-aided NOMA network with multiple users
and unrestricted transmission rounds. A Retransmit-At-Will policy that allows
BS to retransmit flexibly is proposed, which manifests superiority in terms of
AoI compared to existing works.

  * •

We reformulate the age-optimal problem as an MDP. A Double-Dueling-DQN is
trained to intelligently allocate power resources and retransmit at will. The
solutions reveal the threshold structure of the retransmission policy.

  * •

We investigate the user pairing issue in terms of AoI under the HARQ-aided
NOMA network. The simulation results illustrate that user pairing can only
enhance the AoI at low SNR.

##  II SYSTEM MODEL AND PROBLEM FORMULATION

###  II-A Transmission Modle of HAQR-aided NOMA Network

Figure 1: HARQ-aided NOMA Network Model.

We consider a NOMA downlink network with a BS and $N$ users
$\left(u_{i},i\in\\{1,\cdots,N\\}\right)$ as Fig. 1, the distances from users
to the BS are ordered as $d_{1}<d_{2}<d_{3}<\cdots<d_{N}$. The BS sends
freshness critical control commands or status updates to the users on the same
shared channel. In a NOMA downlink network, the BS sends out multiple user
signals by superposition coding (SC) technique in the power domain, and HARQ-
CC is applied as a key technique to enhance transmission reliability by
multiple retransmissions. Then, the signal sent by the BS in the $k-$th
transmission can be expressed as

$s_{k}=\sum_{i=1}^{N}{\sqrt{\alpha_{i,k}P}x_{i,k}}\,,$ (1)

where $P$ denotes the total power of a single transmission, $\alpha_{i,k}$ and
$x_{i,k}$ denotes the power allocation coefficient and the unit energy signal
of user $i$ in the $k-$th transmission. Since the receiver end of the NOMA
network needs to decode superposed signals through successive interference
cancellation (SIC), that is, a strong user with high channel quality needs to
decode and eliminate weak users’ signals iteratively, and then decodes its own
signal. Generally, in order to improve the reliability of SIC, the power
allocation coefficients satisfy
$\alpha_{1,k}<\alpha_{2,k}<\cdots<\alpha_{N,k}$,
$\sum_{i=1}^{N}{\alpha_{i,k}}=1$, which indicates that weaker users are
allocated with higher power.

We consider a wireless fading channel where the signal will experience quasi-
static Rayleigh fading. The perfect channel state information (CSI) is assumed
to be shared with the BS. Then the channel coefficient in the $k-$th
transmission from user $u_{i}$ to BS can be expressed as

$h_{i,k}=\sqrt{{d_{i}}^{-\tau}}g\,,$ (2)

where $g\sim\mathcal{C}\mathcal{N}\left(0,1\right)$ denotes the unit Rayleigh
channel coefficient and $\tau$ denotes the path loss exponent. In such a case,
the signal received by $u_{i}$ in the $k-$th transmission is given by:

$y_{i,k}=h_{i,k}\sum_{i=1}^{N}{\sqrt{\alpha_{i,k}P}x_{i,k}}+n_{i,k}\,,$ (3)

where $n_{i,k}$ denotes the unit complex additive white Gaussian noise (AWGN)
with zero mean and variance $\sigma^{2}=1$.

###  II-B AoI Evolution of HARQ-aided NOMA Network

Noticing that as the number of NOMA users increases, SIC decoding becomes more
complex, which makes it harder for users to decode successfully. To this end,
we combine NOMA with HARQ-CC to ensure reliable reception of updates through
retransmitting the former package. In HARQ-CC protocol, if $u_{i}$ decodes its
message successfully, a positive acknowledgment (ACK) will be sent to the BS,
otherwise, a negative acknowledgment (NACK) will be sent. Different from [9],
where $i$) the BS can choose whether retransmit or not only when it receives
an NACK; $ii$) the BS can only transmit a new package if it receives an ACK.
We propose a Retransmit-At-Will policy, which further relaxes these
constraints, that is the BS can choose whether retransmit or not no matter
what it receives.

We consider a time-slotted system where the continuous time is discretized
into time slots111This is an extensively-implemented operation, which is also
conducted in [13, 14]. The BS serves all the users through the generate-at-
will model, which means the BS can generate packages for all the users at the
beginning of every time slot. We adopt AoI as a metric to measure the
timeliness of the information received by users. Let
$\varDelta_{i}\left(t\right)$ denote the instantaneous AoI of $u_{i}$ at the
time slot $t$, then if $u_{i}$ receives a status update successfully,
$\varDelta_{i}\left(t\right)$ will decrease to the instantaneous AoI of this
package, while $\varDelta_{i}\left(t\right)$ will increase by $1$ if $u_{i}$
fails to decode its message. We denote
$t_{i},t_{i}^{{}^{\prime}}\in\\{1,2,\cdots,T_{max}\\}$ as the transmission
round of $u_{i}$’s package at the time slot $t,t+1$ respectively, which can
also represent the instantaneous AoI of $u_{i}$’s package, and $T_{max}$
represents the maximum transmission rounds. Then the AoI evolution of $u_{i}$
between two adjacent time slots can be expressed as

$\varDelta_{i}\left(t+1\right)=\begin{cases}\varDelta_{i}\left(t\right)+1,&w.p.\,\,\varepsilon_{i}\\\
t_{i}^{{}^{\prime}},&w.p.\,\,1-\varepsilon_{i}\\\ \end{cases}\,,$ (4)

where $\varepsilon_{i}$ denotes the BLER of $u_{i}$.

Noticing that if the BS retransmits the $u_{i}$’s package, the transmission
round $t_{i}$ will increase by 1, and if the BS transmits a new package for
$u_{i}$, $t_{i}$ will be set as 1. Therefore, we can obtain:

$t_{i}^{{}^{\prime}}=\begin{cases}t_{i}+1,&\chi_{i}=1\\\ 1,&\chi_{i}=0\\\
\end{cases},$ (5)

where $\chi_{i}=1$ represents retransmitting the former $u_{i}$’s package, and
$\chi_{i}=0$ represents transmitting a new package for $u_{i}$.

Fig. 2 gives the AoI evolution of the HARQ-aided NOMA network with two users
as an example.

Figure 2: A two-user example of instantaneous AoI evolution.

###  II-C BLER analysis

In the $k-$th transmission, only the weakest user $u_{N}$ can decode its
message $x_{N,k}$ from $y_{N,k}$ directly. Other users $u_{i}(i\neq N)$ must
use SIC technique to decode their message, which means $u_{i}$ should decode
and eliminate $x_{j,k}(i\leqslant j\leqslant N)$ from $y_{i,k}$ first, and
then decode its own message $x_{i,k}$. In this case, the signal-to-
interference plus noise ratio (SINR) at $u_{i}$ for decoding $x_{j}$ in the
$k-$th transmission can be given as:

$\gamma_{ij}^{(k)}=\frac{\alpha_{j,k}\left|h_{i,k}\right|^{2}}{\sum_{n=1}^{j-1}{\alpha_{n,k}\left|h_{i,k}\right|^{2}+1/\rho}}\,\,(i\leqslant
j\leqslant N)\,,$ (6)

where $\rho=P/\sigma^{2}$ denotes the SNR.

In the HARQ-aided NOMA network, users can store the transmitted packages. When
these packages are retransmitted, the stored information can be used to
achieve maximal ratio combining (MRC) decoding to improve transmission
reliability. By this means, the SINR of $u_{i}$ for decoding the message of
$u_{j}$ in the $t_{j}$-th transmission round can be given as:

$\gamma_{ij}\left(t_{j}\right)=\sum_{k=1}^{t_{j}}{\gamma_{ij}^{(k)}}\,,$ (7)

where $t_{j}\in{\\{1,2,\cdots,T_{max}\\}}$.

In [15], the BLER of $u_{i}$ for decoding $u_{j}$’s message which has been
transmitted for $t_{j}$ times can be approximated as:

$\varepsilon_{ij}\left(t_{j}\right)\approx
Q\left(\frac{C\left(\gamma_{ij}\left(t_{j}\right)\right)-\frac{N}{m}}{\sqrt{V\left(\gamma_{ij}\left(t_{j}\right)\right)/m}}\right)\,,$
(8)

where $N$ denotes the length of message sent to $u_{j}$, $m$ denotes the
number of encoded symbols,
$C\left(\gamma_{ij}\left(t_{j}\right)\right)=\log_{2}\left(1+\rho\right)$ is
the channel capacity, and $V\left(\gamma_{ij}\left(t_{j}\right)\right)$ is the
channel dispersion, given as
$V\left(\gamma_{ij}\left(t_{j}\right)\right)=(1-1/(1+\gamma_{ij}\left(t_{j}\right))^{2})(\log_{2}e)^{2}$.

Denote a vector
$\boldsymbol{\overset{\sim}{\alpha_{k}}}\triangleq\left(\alpha_{1,k},\alpha_{2,k},\cdots,\alpha_{N,k}\right)$
as the allocated power coefficients for $N$ users at the $k-$th transmission.
According to equation (6)(7)(8), when given a fixed $N$ and $m$,
$\varepsilon_{ij}\left(t_{j}\right)$ is equivalent to a function of $t_{j}$
vectors. In this case, $\varepsilon_{ij}(t_{j})$ can be expressed as:

$\varepsilon_{ij}\left(t_{j}\right)\approx\psi_{ij}\left(\boldsymbol{\overset{\sim}{\alpha_{1}}},\boldsymbol{\overset{\sim}{\alpha_{2}}},\cdots,\boldsymbol{\overset{\sim}{\alpha_{t_{j}}}}\right)\,,$
(9)

According to the principle of SIC, only when the user successfully decodes and
eliminates other interference signals, and then successfully decodes its own
signal, it is a successful decoding process. In this case, the BLER of $u_{i}$
can be given as:

$\varepsilon_{i}=1-\prod_{j=i}^{N}{\left(1-\varepsilon_{ij}\left(t_{j}\right)\right)}\,,$
(10)

which means the BLER of $u_{i}$ is associated with the transmission round of
$u_{j}$’s signal $t_{j}(i\leqslant j\leqslant N)$ and the corresponding power
allocation coefficient vector $\boldsymbol{\overset{\sim}{\alpha_{k}}}$.

###  II-D Problem Formulation

In the HARQ-aided NOMA network, retransmissions can enhance the reliability
but cause information aging. Besides, how to allocate power resources has a
significant impact on the system timeliness. Therefore, it’s necessary for the
BS to adjust transmission policy $\pi$ according to the state of the network
to enhance the timeliness.

We use the expected weighted sum AoI of all the users to measure the AoI
performance of the HARQ-aided NOMA network. To minimize the expected weighted
sum AoI, the problem can be formulated as follows:

$\mathop{\arg\min}\limits_{\pi}\bar{\varDelta}=\lim_{T_{slot}\rightarrow\infty}\frac{1}{T_{slot}}\sum_{t=1}^{T_{slot}}{\sum_{i=1}^{N}{\mathbb{E}\left[\omega_{i}\varDelta_{i}^{\pi}\left(t\right)\right]}}\,,$
(11)

where $\omega_{i}$ denotes the weight of $u_{i}$. The transmission policy
$\pi$ contains two aspects: $i$) the power allocation scheme
$\boldsymbol{\overset{\sim}{\alpha}}$ at every time slot; $ii$) when and
whether to retransmit the former package.

##  III MDP FORMULATION

To obtain the age-optimal transmission policy for the HARQ-aided NOMA network,
we reformulate the problem (11) as an MDP problem, which is characterized by a
quaternary tuple $\left\\{\mathcal{S},\mathcal{A},\mathcal{P},r\right\\}$,
where $\mathcal{S},\mathcal{A},\mathcal{P},r$ denotes the state space, action
space, state-action transition probability matrix, and the reward
respectively. The elements of the MDP for HARQ-aided NOMA network are
described specifically as follows:

  * •

State Space: We denote $s_{t}$ as the state of the network at the time slot
$t$, which is characterized by a tuple
$s_{t}\triangleq\left(\boldsymbol{\overset{\sim}{\alpha_{1}}},\boldsymbol{\overset{\sim}{\alpha_{2}}},\cdots,\boldsymbol{\overset{\sim}{\alpha_{T_{max}-1}}},\boldsymbol{\mathcal{T}},\boldsymbol{\varDelta}\right)$,
where

    1. 1.

The coefficient vector
$\boldsymbol{\overset{\sim}{\alpha_{i}}}\triangleq(\alpha_{1,i},\alpha_{2,i},\cdots,\alpha_{N,i})$
denotes the $i-$th power allocation coefficient for all the $N$ users which is
stored in the buffer. We discretize the power into $M$ levels, and the power
coefficient can only take value from a discrete set, that is
$\alpha_{i,k}\in\\{\frac{1}{M},\frac{2}{M},...,\frac{M-1}{M}\\}$. Due to the
computational complexity, the case $\alpha_{i,k}=1$ which means BS serves only
one user is not taken into consideration.

    2. 2.

The vector $\boldsymbol{\mathcal{T}}\triangleq(t_{1},t_{2},\cdots,t_{N})$,
where $t_{j}$ denotes the transmission round of $u_{j}$’s signal at the time
slot $t$, with $t_{j}\in{\\{1,2,\cdots,T_{max}\\}}$.

    3. 3.

The AoI vector
$\boldsymbol{\varDelta}\triangleq(\varDelta_{1},\varDelta_{2},\cdots,\varDelta_{N})$,
where $\varDelta_{i}$ denotes the instantaneous AoI of $u_{i}$ at the time
slot $t$.

  * •

Action Space: The action $a_{t}\in\mathcal{A}$ contains two aspects: $i$): the
power allocation policy: denote
$\boldsymbol{\overset{\sim}{\alpha}}\triangleq(\alpha_{1},\alpha_{2},\cdots,\alpha_{N})$
as the power coefficient allocated by the BS to all the $N$ users at the time
slot $t$, where
$\alpha_{i}\in\\{\frac{1}{M},\frac{2}{M},...,\frac{M-1}{M}\\}$; $ii$): the
retransmission policy $\chi_{i}$. Noticing that when $t_{i}=T_{max}$, the BS
will only transmit a new package for $u_{i}$, that is $\chi_{i}=0$.

  * •

Reward: We denote the instantaneous expected weighted sum AoI as the reward
$r_{t}$ at the time slot $t$, which can be expressed as
$r_{t}=\sum_{i=1}^{N}{\omega_{i}\varDelta_{i}}$.

  * •

Transition Probability: We denote $P\left(s_{t+1}\mid s_{t},a_{t}\right)$ as
the transition probability from state $s_{t}$ to state $s_{t+1}$ when taking
the action $a_{t}$. The transition probability can be expressed as the
following three parts:

    1. 1.

Power allocation coefficient:

$\begin{cases}\boldsymbol{\overset{\sim}{\alpha_{i}}}^{\prime}=\boldsymbol{\overset{\sim}{\alpha_{i+1}}}\,,1\leq
i\leq T_{max}-2\\\
\boldsymbol{\overset{\sim}{\alpha_{T_{max}-1}}}^{\prime}=\boldsymbol{\overset{\sim}{\alpha}}\end{cases},$
(12)

where $\boldsymbol{\overset{\sim}{\alpha_{i}}}^{\prime}$ denotes the $i-$th
power coefficient vector which is stored in the buffer at the time slot $t+1$.
At each iteration, we store the BS’s power allocation scheme
$\boldsymbol{\overset{\sim}{\alpha}}$ in the buffer as
$\boldsymbol{\overset{\sim}{\alpha_{T_{max}-1}}}$, and then update
$\boldsymbol{\overset{\sim}{\alpha_{i}}}$ into
$\boldsymbol{\overset{\sim}{\alpha_{i+1}}}$. In this way, the buffer can
always maintain $T_{max}-1$ freshest power coefficient vectors for MRC
decoding.

    2. 2.

AoI evolution: the transition of AoI is given as (4). To get $\varepsilon_{i}$
at the time slot $t+1$, we should first obtain
$\varepsilon_{ij}(t_{j}^{{}^{\prime}})$, according to (10). Noticing that the
MRC decoding for a $t_{j}^{{}^{\prime}}$ times transmitted signal will use the
power coefficient scheme $\overset{\sim}{\alpha}$ and the freshest
$t_{j}^{{}^{\prime}}-1$ power coefficient vectors in the buffer. Combined with
(9), we can get:

$\varepsilon_{ij}\left(t_{j}^{{}^{\prime}}\right)=\begin{cases}\psi_{ij}\left(\boldsymbol{\overset{\sim}{\alpha}}\right),&t_{j}^{{}^{\prime}}=1\\\
\psi_{ij}\left(\boldsymbol{\overset{\sim}{\alpha_{T_{max}-1}}},\boldsymbol{\overset{\sim}{\alpha}}\right),&t_{j}^{{}^{\prime}}=2\\\
\cdots\cdots&\\\
\psi_{ij}\left(\boldsymbol{\overset{\sim}{\alpha_{1}}},\cdots,\boldsymbol{\overset{\sim}{\alpha_{T_{max}-1}}},\boldsymbol{\overset{\sim}{\alpha}}\right),&t_{j}^{{}^{\prime}}=T_{max}\\\
\end{cases}.$ (13)

    3. 3.

Transmission round: the transition of transmission round is given as (5).

##  IV PROPOSED DOUBLE-DUELING-DQN ALGORITHM

In this section, we first describe the idea of DQN, and then utilize the
Double-Dueling-DQN algorithm to solve the formulated MDP.

###  IV-A Preliminaries of DQN

The main idea of reinforcement learning (RL) is to find the optimal decision
policy by maximizing the long-term discounted return. We introduce the action-
value function (Q-function) to measure the value under policy $\pi$ by taking
action $a_{t}$ in a given state $s_{t}$, which can be expressed as:

$Q^{\pi}\left(s_{t},a_{t}\right)=\mathbb{E}_{\pi}\left[\sum_{k=0}^{\infty}{\gamma^{k}\cdot
r_{t+k}|\pi}\right]\,,$ (14)

where $\gamma\in[0,1]$ represents the discounted factor. Then, the value
function which measures the value starting from the state $s_{t}$ under policy
$\pi$ can be expressed as:

$V^{\pi}\left(s_{t}\right)=\underset{a_{t}}{\mathbb{E}}\left[Q^{\pi}\left(s_{t},a_{t}\right)\right]\,.$
(15)

Noticing that the Q-function evaluates the value of all actions, if we obtain
$Q^{*}$ which is the Q-function under the optimal policy $\pi^{*}$, the
optimal action can be selected based on the Q-value, therefore, $\pi^{*}$ can
be obtained by:

$\pi^{*}=\mathop{\arg\max}\limits_{a_{t}\in\mathcal{A}}Q^{*}(s_{t},a_{t})\,.$
(16)

As a commonly used RL algorithm, Q-learning is precisely designed to obtain
$Q^{*}$. Q-learning updates $Q^{\pi}$ iteratively in the following way until
it converges to $Q^{*}$.

$Q^{\pi}(s_{t},a_{t})=Q^{\pi}(s_{t},a_{t})+\lambda(y_{t}-Q^{\pi}(s_{t},a_{t}))\,,$
(17)

where $y_{t}=r_{t}+\gamma\underset{a_{t+1}}{\max}Q^{\pi}(s_{t+1},a_{t+1})$ is
the temporal difference target (TD target), and $\lambda$ denotes the learning
rate.

However, Q-learning is difficult to converge when solving high-complexity
problems. To this end, DQN algorithm combines neural networks with Q-learning
to solve the high computational complexity problem. DQN is updated in a
similar way to Q-learning, except that a neural network
$Q^{\pi}(s_{t},a_{t};\theta)$ is used to approximate the Q-function, and the
network parameter $\theta$ is updated by stochastic gradient descent method
for each update.

###  IV-B Double-Dueling-DQN Algorithm

We adopt the Double-Dueling-DQN algorithm which is an improved version of DQN
to deal with the problem (11).

In an interaction, the agent observes the current transmission state $s_{t}$
of the HARQ-aided NOMA network and selects an action $a_{t}$ by
$\epsilon$-Greedy Policy. After the NOMA network takes action $a_{t}$, the
users’ AoI and the transmission state will change. Then, the NOMA network
returns a reward $r_{t}$ and the next state $s_{t+1}$, which are in terms of
$a_{t}$. The $\epsilon$-Greedy Policy is given as:

$a_{t}=\begin{cases}random\,\,action,&w.p.\,\,\epsilon\\\
\mathop{\arg\max}\limits_{a}Q^{\pi}(s_{t},a;\theta),&w.p.\,\,1-\epsilon\end{cases},$
(18)

where $\epsilon$ denotes the exploration probability. This policy makes the
agent inclined to explore more actions in the early stage of the algorithm,
which can avoid the algorithm from converging to a local optimum. With
continuous iterations, we gradually set $\epsilon\rightarrow 1$ to obtain a
stable optimal solution.

After each round of interaction, we put $(s_{t},a_{t},r_{t},s_{t+1})$ into the
experience replay buffer $\mathcal{D}$. If the size of experiences in
$\mathcal{D}$ exceeds a certain amount $\sigma$, a mini-batch experiences
$\mathcal{B}=\\{(s_{t},a_{t},r_{t},s_{t+1})\\}$ will be sampled from
$\mathcal{{D}}$ for updating the network parameter during the training stage.
This technique is called experience replay, which can improve the utilization
of training data and reduce the correlation of adjacent samples.

In the training stage, to solve the overestimation problem of traditional DQN,
two networks are used for training in our algorithm. The main network
parameter is $\theta$, which is used to approximate the Q-function, and the
target network parameter is $\theta^{-}$, which is used to calculate the TD
target to update the main network. The TD target can be expressed as:

$y_{t}^{target}=r_{t}+\gamma\underset{{a_{t}}_{+1}}{\max}Q^{\pi}(s_{t+1},a_{t+1};\theta^{-})\,.$
(19)

The main network updates the parameters $\theta$ by minimizing the loss
function as follows at each interaction:

$L(\theta)=\frac{1}{|\mathcal{B}|}\sum_{({}s_{t},a_{t},r_{t},s_{t+1})\in\mathcal{B}}{\left[y_{t}^{target}-Q\left(s_{t},a_{t};\theta\right)\right]^{2}}\,.$
(20)

The target network parameter $\theta^{-}$ is synchronized to the main network
parameter $\theta$ every $l$ updates.

Besides, we use dueling network architecture to improve the algorithm
performance, which adopts two networks to approximate the Q-function. We
approximate the value function $V^{\pi}(s_{t})$ and the advantage function
$A^{\pi}(s_{t},a_{t})$ with the same parameter, where
$A^{\pi}\left(s_{t},a_{t}\right)=Q^{\pi}\left(s_{t},a_{t}\right)-V^{\pi}\left(s_{t}\right)$.
Then, The Q-function can be obtained by these two networks, which can be
expressed as:

$Q^{\pi}(s_{t},a_{t};\theta)=V^{\pi}(s_{t};\theta)-\underset{a_{t}}{mean}A^{\pi}(s_{t},a_{t};\theta)\,.$
(21)

Details of the training process are given in Algorithm 1.

Input: $E,T_{slot},\epsilon,\eta,\sigma,\lambda,k,l$;

Output: The optimal Q-network $Q^{*}(s,a;\theta)$;

1 Initialize the network parameter $\theta$ and $\theta-$;

2Initialize the state of HARQ-aided NOMA system $s_{0}$;

3for  _$episode=1$ to $E$_ do

4 Reset the system state $s_{t}=s_{0}$;

5 for  _$t=1$ to $T_{slot}$_ do

6 Observe the system state $s_{t}$ and calculate the Q value of all actions
$Q(s_{t},\cdot;\theta)$;

7 Select a action $a_{t}$ according to the $\epsilon-$greedy policy (18);

8 Update $\epsilon=\eta\epsilon$;

9 The BS selects action $a_{t}$, and gets the next state $s_{t+1}$ and the
reward $r_{t}$;

10 Store $(s_{t},a_{t},r_{t},s_{t+1})$ in the replay buffer $\mathcal{D}$;

11 if  _size of experiences $>\sigma$ and $t\,\%\,k=0$_ then

12 Sample a mini-batch $\mathcal{B}$ from $\mathcal{D}$ randomly;

13 Calculate the TD target according to (19);

14 Training the main network with stochastic gradient descent method, and
update $\theta$ with learning rate $\lambda$;

15

16 if  _$t\,\%\,l=0$_ then

17 Update the target network as $\theta^{-}=\theta$;

18

19

return _The optimal Q-network $Q^{*}(s,a;\theta)$_

Algorithm 1 Double-Dueling-DQN Algorithm

##  V SIMULATION RESULTS

In this section, we take a 4-users ($N=4$) scenario as an example to analyze
the timeliness performance of the HARQ-aided NOMA network. We set the distance
from the user to the BS as $d_{1}=1.5,\,d_{2}=2,\,d_{3}=2.5,\,d_{4}=3$, the
path loss exponent as $\tau=2$, the message length as $N=160$, the number of
symbols as $m=200$, and the power level as $M=20$. The training parameters are
set as
$E=500,\,T_{slot}=10^{3},\,\epsilon=0.1,\,\eta=0.99,\,\sigma=200,\,\lambda=10^{-3},\,k=4,\,l=100$.

Fig. 3 compares the Retransmit-At-Will policy with the policy in [9]. Also, a
non-optimal policy is given as a benchmark where the BS only transmits new
packages with a fixed power allocation scheme. Retransmit-At-Will policy
achieves the best AoI performance at low SNR, while two optimal policies
perform similarly at high SNR. This is because retransmission is permitted
when the BS received an ACK in the Retansmit-At-Will policy, which can enhance
the transmission reliability at low SNR. However, since transmission is
reliable enough at high SNR, the system will not choose this retransmission
because it will cause stale delivery.

Figure 3: The average AoI under different transmission policy with maximum
retransmissions $T_{max}=2$.

Fig. 4 illustrates the AoI performance under maximum retransmissions
$T_{max}=2$ and $T_{max}=3$. The network realizes better AoI performance at
low SNR when $T_{max}=3$, but there is almost no difference under different
$T_{max}$ at high SNR. To explain this, we demonstrate the retransmission
policy when $T_{max}=3$ with the ratio of the retransmission rounds $T$. From
the ratio, policy $T=3$ accounts for a larger proportion at low SNR, which
means the BS prefers to take more retransmissions to ensure transmission
reliability. When the SNR exceeds 16 dB, the BS prefers fewer retransmissions
to ensure the timeliness of transmission, and the policy structure of
$T_{max}=3$ is more like that of $T_{max}=2$, so they perform similarly at
high SNR. According to this, we can find that retransmission can only improve
system timeliness at low SNR. When the SNR exceeds a certain threshold, the
system prefers to always transmit new packages, and retransmission is
meaningless for system timeliness.

Figure 4: The average AoI under different maximum retransmissions $T_{max}$
and the policy structure when $T_{max}=3$.

To investigate the impact of user pairing on the system AoI, we set the
optimal pairing scheme in [10] as a comparing benchmark. In this case, $u_{1}$
is paired with $u_{4}$, and $u_{2}$ is paired with $u_{3}$. The BS only
transmits packages for one user pair at a time slot, and the other user pair
can only wait at this time. Fig. 5 compares the performance of the Retransmit-
At-Will policy with and without user pairing. We can find that transmission
with pairing has a better AoI performance at low SNR, while transmission
without pairing performs better at high SNR. That is because the user pairing
technique reduces the users served simultaneously, which can reduce the
complexity of SIC and ensure transmission reliability at low SNR. While
reliability can be assured at high SNR, thus transmission without pairing
performs better because it can serve more users simultaneously and save
waiting time. As a result, user pairing can be used to enhance the timeliness
at low SNR.

Figure 5: The average AoI performance comparison of transmission with pairing
and without pairing when $T_{max}=2$.

##  VI CONCLUSION

In this paper, we optimize the AoI of the HARQ-aided NOMA network with
multiple users and unrestricted retransmissions, which can intelligently
decide how to allocate power resources and when to retransmit. First, a more
flexible retransmission scheme with better AoI performance, referred to as
Retransmit-At-Will is proposed. Then, we reformulate the age-optimal problem
as an MDP, and the Double-Dueling-DQN is adopted to obtain the optimal policy.
At last, a threshold structure of the retransmission policy is demonstrated,
and user pairing is proved to be a technique enhancing the system freshness at
low SNR.

## References

  * [1] T. Xia, M. M. Wang, J. Zhang, and L. Wang, “Maritime internet of things: Challenges and solutions,” _IEEE Wireless Communications_ , vol. 27, no. 2, pp. 188–196, 2020.
  * [2] S. Kaul, R. Yates, and M. Gruteser, “Real-time status: How often should one update?” in _2012 Proceedings IEEE INFOCOM_ , 2012, pp. 2731–2735.
  * [3] Z. Ding, Y. Liu, J. Choi, Q. Sun, M. Elkashlan, I. Chih-Lin, and H. V. Poor, “Application of Non-Orthogonal Multiple Access in LTE and 5G Networks,” _IEEE Communications Magazine_ , vol. 55, no. 2, pp. 185–191, 2017.
  * [4] J. Chen, Y. Liu, Q. Chen, X. Lan, G. Cheng, Y. Fu, and Z. Zhang, “On the Adaptive AoI-aware Buffer-aided Transmission Scheme for NOMA Networks,” in _2021 IEEE Wireless Communications and Networking Conference (WCNC)_ , 2021, pp. 1–6.
  * [5] Q. Wang, H. Chen, Y. Li, and B. Vucetic, “Minimizing age of information via hybrid NOMA/OMA,” in _2020 IEEE International Symposium on Information Theory (ISIT)_. IEEE, 2020, pp. 1753–1758.
  * [6] S. Lin, D. J. Costello, and M. J. Miller, “Automatic-repeat-request error-control schemes,” _IEEE Communications Magazine_ , vol. 22, no. 12, pp. 5–17, 1984.
  * [7] F. Ghanami, G. A. Hodtani, B. Vucetic, and M. Shirvanimoghaddam, “Performance analysis and optimization of NOMA with HARQ for short packet communications in massive IoT,” _IEEE Internet of Things Journal_ , vol. 8, no. 6, pp. 4736–4748, 2020.
  * [8] D. Marasinghe, N. Rajatheva, and M. Latva-Aho, “Block error performance of NOMA with HARQ-CC in finite blocklength,” in _2020 IEEE International Conference on Communications Workshops (ICC Workshops)_. IEEE, 2020, pp. 1–6.
  * [9] S. Wu, Z. Deng, A. Li, J. Jiao, N. Zhang, and Q. Zhang, “Minimizing Age-of-Information in HARQ-CC Aided NOMA Systems,” _IEEE Transactions on Wireless Communications_ , 2022.
  * [10] L. Zhu, J. Zhang, Z. Xiao, X. Cao, and D. O. Wu, “Optimal User Pairing for Downlink Non-Orthogonal Multiple Access (NOMA),” _IEEE Wireless Communications Letters_ , vol. 8, no. 2, pp. 328–331, 2019.
  * [11] W. Zhang, D. Yang, W. Wu, H. Peng, N. Zhang, H. Zhang, and X. Shen, “Optimizing Federated Learning in Distributed Industrial IoT: A Multi-Agent Approach,” _IEEE Journal on Selected Areas in Communications_ , vol. 39, no. 12, pp. 3688–3703, 2021.
  * [12] W. Zhang, D. Yang, H. Peng, W. Wu, W. Quan, H. Zhang, and X. Shen, “Deep Reinforcement Learning Based Resource Management for DNN Inference in Industrial IoT,” _IEEE Transactions on Vehicular Technology_ , vol. 70, no. 8, pp. 7605–7618, 2021.
  * [13] J. P. Champati, M. H. Mamduhi, K. H. Johansson, and J. Gross, “Performance characterization using aoi in a single-loop networked control system,” in _IEEE INFOCOM 2019-IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS)_. IEEE, 2019, pp. 197–203.
  * [14] I. Krikidis, “Average age of information in wireless powered sensor networks,” _IEEE Wireless Communications Letters_ , vol. 8, no. 2, pp. 628–631, 2019.
  * [15] Y. Yu, H. Chen, Y. Li, Z. Ding, and B. Vucetic, “On the performance of non-orthogonal multiple access in short-packet communications,” _IEEE Communications Letters_ , vol. 22, no. 3, pp. 590–593, 2017.