Deep Learning for Wireless Networked Systems: a joint Estimation-Control-Scheduling Approach
Zihuai Zhao, Wanchun Liu*, Member, IEEE, Daniel E. Quevedo, Fellow, IEEE,
Yonghui Li, Fellow, IEEE, and Branka Vucetic Fellow, IEEE
Z.  Zhao, W. Liu, Y. Li, and B. Vucetic are with School of Electrical and Information Engineering, The University of Sydney, Australia.
Emails:	{zihuai.zhao, wanchun.liu, yonghui.li<EMAIL_ADDRESS>D. Quevedo is with the School of Electrical Engineering and Robotics, Queensland University of Technology (QUT), Brisbane, Australia.	Email<EMAIL_ADDRESS>W. Liu is the corresponding author.

Wireless networked control system (WNCS) connecting sensors, controllers, and actuators via wireless communications is a key enabling technology for highly scalable and low-cost deployment of control systems in the Industry 4.0 era.
Despite the tight interaction of control and communications in WNCSs, most existing works adopt separative design approaches.
This is mainly because the co-design of control-communication policies requires large and hybrid state and action spaces, making the optimal problem mathematically intractable and difficult to be solved effectively by classic algorithms.
In this paper, we systematically investigate deep learning (DL)-based estimator-control-scheduler co-design for a model-unknown nonlinear WNCS over wireless fading channels.
In particular, we propose a co-design framework with the awareness of the sensor's age-of-information (AoI) states and dynamic channel states.
We propose a novel deep reinforcement learning (DRL)-based algorithm for controller and scheduler optimization utilizing both model-free and model-based data. An AoI-based importance sampling algorithm that takes into account the data accuracy is proposed for enhancing learning efficiency.
We also develop novel schemes for enhancing the stability of joint training.
Extensive experiments demonstrate that the proposed joint training algorithm can effectively solve the estimation-control-scheduling co-design problem in various scenarios and provide significant performance gain compared to separative design and some benchmark policies.

Wireless networked control systems, control-communications co-design, age of information, deep reinforcement learning, task-oriented communications.

§ INTRODUCTION

Under the rapid development of industrial applications in the Fourth Industrial Revolution, such as smart manufacturing, smart city, smart grids, e-commerce warehouses and industrial automation systems,  wireless networked control system (WNCS) has been considered as a key solution to the  high-scalable and low-cost deployment of ubiquitous automatic control systems [1]. A typical WNCS consisting of plants, sensors, actuators, and a controller is illustrated in Fig. <ref>. In the feedback control loop of the WNCS, the sensors measure plant states and send them to the controller for processing and generating control signals via uplink channels, which will then be sent to the actuators for execution via downlink channels.

In principle, the nature of WNCS design is highly interdisciplinary, which involves signal processing for plant state estimation, control theory for optimally regulating the plant behavior, and communication theory for reliably transmitting the sensor and controller signals under limited communication resources.
Since both estimation and control rely on the information delivered by the communication system, the WNCS design for achieving the optimal control performance should jointly take into account the estimation, control, and communication algorithms that tightly interact with each other. Ideally, those algorithms should be jointly designed to optimize the control performance of WNCSs under resource constraints.

Although the concept of control-communication co-design in WNCSs was proposed decades ago (see [1] and references therein), most related works from different research societies were built on the separative design principle.
The communications society focuses solely on improving the communications performance such as data rate, latency, and reliability, without taking into account the WNCS system dynamics, or performance [2, 3]. Although in the 5G era, ultra-reliable low latency communications have been proposed for mission-critical control applications, the prevailing design principle is standalone and not tailored to any control applications, where the control performance is not treated as a design objective [4].
On the other hand, the control system society's effort on WNCSs mainly focuses on the control (and estimation) algorithm design with predetermined communications policies (see [5] and its follow-up works).
The recent works on control-communication co-design for WNCSs can be categorized into two streams: control-aware communication design and control-communication policy co-design.

A wireless networked control system (WNCS).

In Stream 1, communication protocols are optimized to achieve the best control performance or under  certain control-related constraints.
In [6, 7], transmission scheduling and power allocation problems of WNCSs were investigated for achieving the minimum overall transmission power consumption while guaranteeing certain control performance.
In [8], a control-aware scheduler design problem was considered based on the communications protocol of IEEE 802.15.4.
In [9], a communication protocol with variable packet length was proposed and optimized for achieving the best control performance.
In [10], a transmission power allocation problem of a WNCS with a coding-free communication protocol was investigated, aiming to achieve optimal overall control performance.
In [11], a novel framework was developed for jointly optimizing the communication design parameters to achieve the best control performance.
Note that all those works are restricted to linear dynamical systems with linear control laws.
For remote state estimation of linear WNCSs, transmission scheduling problems have drawn significant attention. In [12, 13, 14, 15, 16, 17], optimal scheduling policies were investigated for various system setups to minimize average estimation errors.

In Stream 2, both the control and communication policies are jointly optimized to achieve the overall control performance.
Stream 2 is more  challenging due to the fact that the joint policy has very large combined state and action spaces when taking into account both control and communications domains.
In a nutshell, most co-design problems can be formulated as dynamic decision-making ones. However, considering large state and action spaces, conventional solutions such as the Markov decision process cannot be applied due to the curse-of-dimensionality.
To solve this issue, most works in this stream rely on deep-learning (DL) approaches with artificial neural networks (NNs) for function approximations.
In [18], a deep reinforcement learning (DRL) approach was adopted to learn both the control and the transmission scheduling signals.
In particular, DRL combines artificial NNs with a framework of reinforcement learning that helps software agents learn how to solve decision-making problems and reach their goals.
In [19], both the control policy and the dynamic transmission power allocation policy were jointly optimized based on DRL.
It is worth noting that those DRL-based algorithms are model-free and are applied to the practical WNCS scenario that does not require accurate knowledge of the nonlinear system (plant) models, while the conventional solutions are purely model-based.

There are still many open problems in the area of control-communications policy co-design with unknown nonlinear system models. Many existing works, such as [18, 19], assume that the sensor measurements are perfect and the (uplink) communications between sensor-controller are error-free. Under such an assumption, the controller has an accurate plant state in real time for generating control signals.
When considering a practical uplink channel, the controller does not always know the plant state and thus needs state estimation. This requires estimation-control co-design.
A key aspect is that the estimation quality significantly depends on the age of the sensor's information available to the estimator, which measures the time duration since the controller's last packet received from the sensor.
Due to system dynamics and uncertainties, a larger age-of-information (AoI) of the sensor indicates a less reliable state estimate.
For real-time control applications, an estimate with a small AoI is more important than the one with a large AoI. Such information about the data importance needs to be taken into account for the controller's training.
We note that the analysis and optimization of AoI in different communication networks have drawn significant attention during the past five years [20]. However, how to leverage the AoI of sensor data for effectively training a controller has not been considered before.
Furthermore, when considering DL-based estimator-control-communication co-design, one needs to systematically design a joint training algorithm for achieving time and performance efficiency, rather than training the three modules one by one. Otherwise, the resulting estimation, control and communication policies may not converge to desired ones, leading to poor overall control performance of the WNCS.
Due to aforementioned difficulties, joint estimator-control-communication policy learning for WNCSs has not been investigated in the open literature.

In this work, we systematically investigate a DL-based estimator-control-scheduler co-design framework for a model-unknown WNCS with nonlinear dynamic systems. We consider fading channels between sensor-controller and controller-actuator. The major contributions are summarized as follows.

* We propose a novel DL-based WNCS over fading channels with time correlations. In particular, the AoI states of the sensor's information are utilized in the three modules of estimator, controller, and scheduler; both the controller and the scheduler leverage the fading channel states for decision-making. The instantaneous and historical states are utilized in each module.
Co-design frameworks for WNCSs with the awareness of AoI and channel states have not been considered in the open literature.

* We develop a joint estimator-controller-scheduler training algorithm. In particular, we propose a DRL-based algorithm for controller and scheduler optimization utilizing both the model-free data that are received from the sensor directly and the model-based data that are generated by the estimator, when packet dropout occurs. An AoI-based importance sampling algorithm that takes into account the data accuracy is proposed for enhancing learning efficiency.
Moreover, we develop novel schemes for enhancing the stability of joint training.

* Extensive experiments building on the OpenAI Gym platform demonstrate that the proposed joint training algorithm can effectively solve the estimation-control-scheduling co-design problem in various scenarios. Remarkable performance gains have been achieved compared to the separative design and some benchmark policies.

Outline: The system model of a general WNCS over fading channels is described in Section <ref>. The estimation and control co-design problems of a low-mobility and a high-mobility WNCS were investigated in Sections <ref> and <ref>, respectively. The numerical results are demonstrated and discussed in Section <ref>, followed by conclusions in Section <ref>.

§ SYSTEM MODEL

§.§ WNCS Model

We consider a wireless networked control system  as shown in Fig. <ref>. The plant is a discrete-time  nonlinear system as

\begin{align}
s_{t+1} &=  f(s_t,u_t) + \nu_t \label{plant dynamics}\\
o_t &= s_t +  v_t\label{sensor dynamics}
\end{align}

where $s_t\in\mathbb{R}^{n_s}$ and $u_t\in\mathbb{R}^{n_u}$ are the plant state and the control input from the actuator at time $t$, respectively. In particular, the nonlinear dynamics $f:\mathbb{R}^{n_s}\times{\mathbb{R}^{n_u}}\to{\mathbb{R}^{n_s}}$ is unknown to the remote  controller, and $\nu_t$ is the plant disturbance.
$o_t\in\mathbb{R}^{n_s}$ is the sensor measurement of the plant state $s_t$ affected by the measurement noise $v_t\in\mathbb{R}^{n_s}$.

Estimator-controller co-design of the low-mobility WNCS.

We model the uplink channel and the downlink channel as $m$-state Markov fading channels
[21]. The channel states of the uplink and the downlink are denoted by $b_t^{\mathrm{U}}\in\mathcal{W}^{\mathrm{U}}\triangleq\{w_1^{\mathrm{U}},\dots,w_m^{\mathrm{U}}\}$ and $b_t^{\mathrm{D}}\in\mathcal{W}^{\mathrm{D}}\triangleq\{w_1^{\mathrm{D}},\dots,w_m^{\mathrm{D}}\}$, respectively.

Let $p_{i,j}^{\mathrm{U}}$ and $p_{i,j}^{\mathrm{D}}$ denote the channel state transition probabilities from state $i$ to $j$ of the uplink channel and the downlink channel, respectively, i.e.,

\begin{equation}
\begin{aligned}
\end{aligned}
\end{equation}

Then, the channel state transition probability matrices of the channels are

\begin{equation}\label{matrix 1}
\begin{bmatrix}
p_{1,1}^{\mathrm{U}} & \dots  & p_{m,1}^{\mathrm{U}}\\
\vdots & \ddots & \vdots\\
p_{1,m}^{\mathrm{U}} & \dots  & p_{m,m}^{\mathrm{U}}
\end{bmatrix}
\end{equation}

\begin{equation}\label{matrix 2}
\begin{bmatrix}
p_{1,1}^{\mathrm{D}} & \dots  & p_{m,1}^{\mathrm{D}}\\
\vdots & \ddots & \vdots\\
p_{1,m}^{\mathrm{D}} & \dots  & p_{m,m}^{\mathrm{D}}
\end{bmatrix}.
\end{equation}

We assume that the instantaneous channel states, i.e., $b^{\mathrm{U}}_t$ and $b^{\mathrm{D}}_t$ are known to the controller by classical channel estimation schemes [22], while the dynamic channel models, i.e., $M^{\mathrm{U}}$ and $M^{\mathrm{D}}$ are not available.

Let the binary variables $\rho^{\mathrm{U}}_{t}\in\{0,1\}$ and $\rho^{\mathrm{D}}_{t}\in\{0,1\}$ denote transmission failure and success of the uplink channel and the downlink channel at time $t$, respectively. The packet error probabilities at different channel states are

\begin{equation}\label{SC dropout rate}
d_i^{\mathrm{U}}\triangleq\textsf{Prob}[\rho^{\mathrm{U}}_{t}=0|b_t^{\mathrm{U}}=w_i^{\mathrm{U}}], \forall{i}\in\{1,\dots,m\}
\end{equation}

\begin{equation}\label{CA dropout rate}
d_i^{\mathrm{D}}\triangleq\textsf{Prob}[\rho^{\mathrm{D}}_{t}=0|b_t^{\mathrm{D}}=w_i^{\mathrm{D}}], \forall{i}\in\{1,\dots,m\}.
\end{equation}

We assume that the actuator sends the one-bit acknowledge information $\rho^{\mathrm{D}}_t$ to the controller via a perfect feedback channel. This is a widely adopted assumption in wireless communications.

§.§ Control and transmission schedule

We consider both a low-mobility scenario (e.g., process control systems in factories) and a high-mobility scenario (e.g., unmanned aerial vehicles) of the WNCS. In the former scenario, the channel coherence time is much longer than each control time slot, and thus channel state is static. We note that the system is stochastic in this scenario. Therefore, the Markov fading channels in (<ref>) and (<ref>) degrade to additive white Gaussian noise (AWGN) channels with constant channel states (i.e., $m=1$ in (<ref>) and (<ref>)). Due to the low mobility, sensors are often able to be connected to power grids, and the transmission power consumption is not a major concern. For the latter scenario, sensors are commonly powered by batteries. Due to the costly battery replacement operations, it is of significant interest to reduce the uplink transmission rate while guaranteeing a certain level of desired control quality.
Therefore, an uplink transmission scheduler implemented at the controller will schedule the sensor's transmissions only when it is necessary. Let $a^{\mathrm{Tx}}_t \in \{0,1\}$ denote the scheduling action and  $\mathcal{S}(\cdot)$ denote the scheduling function mapping from input states to $a^{\mathrm{Tx}}_t$. We will discuss the input states in the following section.

The control function $\mathcal{C}(\cdot)$ maps input states into the control signal $a^{\mathrm{C}}_t \in \mathbb{R}^{n_u}$.
Considering the packet dropouts of the downlink channel, the control input is (see Fig. <ref>)

\begin{equation}
u_t =\rho^{\mathrm{D}}_{t} a_t^{\mathrm{C}},
\end{equation}

and thus (<ref>) can be rewritten as

\begin{equation}\label{dynamics1}
s_{t+1} = f(s_t,\rho^{\mathrm{D}}_{t}a_t^{\mathrm{C}})+ \nu_t.
\end{equation}

§.§ Design Objectives and Challenges

In the low-mobility scenario, the control reward function of the WNCS at time $t$ depends on both the plant state and the control input as

\begin{equation}\label{reward function}
r_t \triangleq \mathcal{R}(s_t,u_t).
\end{equation}

Usually, the reward is large if $s_t$ is close to the desired plant state and the control input $u_t$ is small.
Note that in the classical linear quadratic control scenario, the reward is $s_t^\top \Xi_s  s_t + u_t^\top \Xi_u  u_t$, where $\Xi_s$ and $\Xi_u$ are constant positive semi-definite matrices [5].
Then, the long-term average performance of the WNCS is defined as

\begin{equation}\label{eq:problem1}
J \triangleq  \lim\limits_{T\rightarrow \infty} \sum_{t=1}^{T} \gamma^{(t-1)} r_t,
\end{equation}

where $\gamma\in (0,1)$ is a discount factor. A smaller $\gamma$ indicates that the future reward is less important.
Thus, the optimal control problem is $\max_{\mathcal{C}(\cdot)} J$.

In the high-mobility scenario, the uplink transmission energy consumption and the scheduling policy should be taken into account. Let $\bar{e}$ denote the sensor transmission energy consumption each time.
The reward function of the WNCS is

\begin{equation}
r'_t \triangleq \mathcal{R}(s_t,u_t) - a^{\mathrm{Tx}}_t \bar{e}.
\end{equation}

Thus, the control-schedule co-design problem is

\begin{equation}\label{eq:problem2}
\max_{\mathcal{C}(\cdot), \mathcal{S}(\cdot)} J' \triangleq  \lim\limits_{T\rightarrow \infty} \sum_{t=1}^T \gamma^{(t-1)} r'_t.
\end{equation}

For the dynamic decision making problems (<ref>) and (<ref>), there are several challenges.
First, the conventional decision-making problems require explicit model knowledge to apply existing dynamic programming algorithms, however, both the plant and channel dynamics are unknown.
Second, existing decision-making problems, including both model-free and model-based ones, commonly assume that instantaneous rewards are available. However, due to the transmission scheduling and the packet dropouts of the uplink channel, the plant state may not be received by the remote controller each time. Thus, the instantaneous reward depending on the plant state is not always available. Note that classic partially observable Markov decision process (POMDP) problems assume that the instantaneous system states may not be obtained at all times, but the rewards are. Thus, the POMDP solutions cannot solve our problems.
Last, the co-design problem (<ref>) involves both the control policy $\mathcal{C}(\cdot)$ with continuous actions and the transmission schedule policy $\mathcal{S}(\cdot)$ with discrete actions, while most dynamic programming algorithms can only handle either discrete or continuous actions, not both.

We will develop a novel DL-based framework to tackle the challenges above.
The low-mobility and the high-mobility scenarios will be investigated in the sequel.

§ ESTIMATION AND CONTROL CO-DESIGN OVER STATIC CHANNELS

In this section, we consider a low-mobility WNCS with static uplink and downlink channels.
Considering the feature of the WNCS with uplink packet dropouts, we use a hybrid model-based and model-free (MB-MF) data generation algorithm for controller design:
if a sensor packet is received, the measurement state is used directly by the controller for generating a control signal – the model-free part; for time slots without sensor packets, the plant state is predicted by a DL-based model approximator, i.e.,  the estimator – the model-based part.
The controller is trained by a DRL algorithm, which takes into account the different accuracy levels of data generated by the MB-MF method.
To enhance time efficiency, we propose to train both the DL-based estimator and the DRL-based controller together, rather than one agent after the other. Although the state estimation is inaccurate at the beginning, the DRL controller can take advantage of it to explore a wider range of actions.
The system architecture of the estimator and the controller is shown in Fig. <ref>.

We also note that a simpler model-free DRL method adopted in [23, 24], which has no model-knowledge nor model approximator. Therefore, zero plant state is assumed when the sensor packet is unavailable at the controller, and the zero state is used for calculating the instantaneous control reward and is also sent to the controller for generating the control signal. Due to the poor estimation of the plant state and hence the reward, the method in [23, 24] cannot guarantee acceptable training performance in our scenario. We will provide numerical results to compare it with our proposed method in Section <ref>.

The detailed design of the estimator and the controller are given below.

§.§ DL-based Remote State Estimator

If the sensor measurement $o_t$ is received by the remote controller, we can use $o_t$ to approximate the current plant state $s_t$ in (<ref>); otherwise, a model-based approach is needed, and we use a state estimator $\mathcal{E}(\cdot)$ with historical data for state prediction. Therefore, the estimated state can be written as

\begin{equation}\label{observation}
\tilde{o}_t =
\begin{cases}
o_t, &\text{ if } a^{\mathrm{Tx}}_t \rho^\mathrm{U}_t = 1 \\
\hat{o}_t\triangleq \mathcal{E}(h_t^{\mathrm{E}}), &\text{ otherwise}
\end{cases}
\end{equation}

where $h_t^{\mathrm{E}}$ is an $\ell$-length historical data before time slot $t$ consisted of the estimated plant states and the control actions as

\begin{equation}
(\tilde{o}_{t-\ell},u_{t-\ell},\dots,\tilde{o}_{t-1},u_{t-1}) & \parbox[t]{.1\textwidth}{if $t>\ell$}\\
(0,0,\dots,\tilde{o}_{1},u_{1},\dots,\tilde{o}_{t-1},u_{t-1}) & \parbox[t]{.1\textwidth}{if $\ell\geq{t}>1$}\\
(0,0,\dots,0,0) & \parbox[t]{.1\textwidth}{otherwise.}
\end{cases}
\end{equation}

In particular, the estimator uses the previous $\ell$ estimates for current state prediction once a packet dropout occurs. We note that the history length $\ell$ is a hyper-parameter, and a larger $\ell$ can provide better performance in principle at the cost of higher estimator's computation complexity.

The estimator is approximated by a deep neural network (DNN) with parameter $\mu$, and is then denoted as $\mathcal{E}_\mu(\cdot)$.
Compared with a vanilla feed-forward neural network (FFNN) with memory-less fully connected (FC) layers,
a recurrent neural network (RNN) has connections between neurons that form a graph along a temporal sequence, allowing it to exhibit temporal dynamic behavior. Therefore, RNNs are more suitable for extracting long-term dependency features of multi-dimensional sequential inputs [25, 26, 27].
We use an RNN structure illustrated in Fig. <ref>.

Network structure of the estimator.

The design problem of the estimator network is to minimize the difference between the real sensor measurements and the predicted states. Thus, the received observations $o_{t}$ and the corresponding histories $h^{\mathrm{E}}_{t}$ are used to train the estimator in a supervised learning fashion. Recall that the ground truth state $s_t$ is unavailable, and $o_t$ is an approximate of $s_t$.
For a batch of $N$ estimator training data $\{\mathcal{T}_i^{\mathrm{E}}\}_{i=1}^N\triangleq\{(h_{(i)}^{\mathrm{E}}, o_{(i)})\}_{i=1}^N$, where $i$ denotes the $i$th data in the batch, the loss function related to the expected estimation error is given by

\begin{equation}\label{eq:est_loss}
J_\mu = \frac{1}{N}\sum_{i=1}^N\|o_{(i)} - \mathcal{E}_\mu(h_{(i)}^{\mathrm{E}})\|^2.
\end{equation}

We use a gradient descent method to update the estimator NN for minimizing the loss function (<ref>).

§.§ DRL-based Controller with Hybrid Model-based and Model-free (MB-MF) Data

The controller generates the control signal $a^{\mathrm{C}}_t$ based on the instantaneous MB-MF data $\tilde{o}_t$ and the history $h^{\mathrm{E}}_t$, i.e.,

\begin{equation}
a^{\mathrm{C}}_t = \mathcal{C}(o_{t}^{\mathrm{C}} ,h^{\mathrm{C}}_t).
\end{equation}

\begin{equation}\label{controller input 1}
o_{t}^{\mathrm{C}} \triangleq (\tilde{o}_t, n_t^{\mathrm{AoI}}),
\end{equation}

and $n_t^{\mathrm{AoI}}\geq{0}$ denotes the AoI of $\tilde{o}_t$, which measures the time
elapsed since the latest sensor packet was received by the controller.
$h_t^{\mathrm{C}}$ is an $\ell$-length historical data including the past plant state estimates, the AoI states, and  the control inputs, which is given by

\begin{equation}
(o_{t-\ell}^{\mathrm{C}},u_{t-\ell},\dots,o_{t-1}^{\mathrm{C}},u_{t-1}) & \parbox[t]{.1\textwidth}{if $t>\ell$}\\
(0,0,\dots,o_{1}^{\mathrm{C}},u_{1},\dots,o_{t-1}^{\mathrm{C}},u_{t-1}) & \parbox[t]{.1\textwidth}{if $\ell\geq{t}>1$}\\
(0,0,\dots,0,0) & \parbox[t]{.1\textwidth}{otherwise.}
\end{cases}
\label{controller history}
\end{equation}

Note that the AoI state indicates the accuracy of the current state estimate $\tilde{o}_t$. Such information is critical to the controller design. For example, when the AoI is large, the estimation is very inaccurate, and the controller may generate a zero control signal due to the largely unknown plant state.

We consider the Twin Delayed Deep Deterministic Policy Gradient (TD3) [28], a policy-based DRL algorithm for generating actions with a continuous space, to approximate and optimize the controller $\mathcal{C}(\cdot)$.
In general, TD3 is a modification of the Deep Deterministic Policy Gradient (DDPG) algorithm [29] to address the overestimation of the value estimate in actor-critic methods (e.g., DDPG). That is, since the actor is updated with respect to the maximization of the value estimate given by the approximated critic, the overestimated value estimate will cause a sub-optimal action to be highly rated by a sub-optimal critic. This leads to the sub-optimal action being reinforced in the next policy update, which creates a problematic feedback loop. In particular, TD3 introduces twin critic networks to reduce the overestimation.
It has shown significantly improved learning speed and performance compared to existing deterministic DRL algorithms with continuous action spaces over a variant of tasks [28].
To obtain a tailored TD3 algorithm to our problem, we make two primary changes to the original one:
First, we introduce RNNs into both the actor and the critic NNs for effectively processing the time-correlated historical data.
Second, we propose a novel AoI-based importance sampling method by properly taking into account the data accuracy to enhance the sampling efficiency.

§.§.§ Network Structure of the DRL-based Controller

The controller's actor and critic NNs have an identical network structure as illustrated in Fig. <ref>. Note that the current input and the historical data are processed first separately, and an RNN is applied for processing the historical data with time correlations.

Network structure of the actor (and also the critic).

Actor network: given the state input $(o_t^{\mathrm{C}},h_t^{\mathrm{C}})$, the output $a_t^{\mathrm{C}}$, and the network parameter set $\theta$, the actor NN approximates the control function as

\begin{equation}\label{actor}
\end{equation}

Twin critic networks: given the state input $(o_t^{\mathrm{C}},h_t^{\mathrm{C}})$ and the action input $a_t^{\mathrm{C}}$, the twin networks both approximate the Q-value, i.e., the expected long-term cost under policy $\pi_\theta$:

\begin{equation}
\begin{aligned}
&Q^{\theta}(o_t^{\mathrm{C}},h_t^{\mathrm{C}},a_t^{\mathrm{C}}) \\
&= \mathbb{E}\!\left[\!\sum_{t'=t}^{\infty}\!\gamma^{t'-t}r_{t'} \Big| o_t^{\mathrm{C}},h_t^{\mathrm{C}},a_t^{\mathrm{C}}, a_{t'}^{\mathrm{C}}=\pi_\theta(o_{t'}^{\mathrm{C}},h_{t'}^{\mathrm{C}}),\! \forall t'\!>\!\!t\!\right].
\end{aligned}
\end{equation}

The Q-value represents the long-term average performance of the current state-action pair, and a larger Q-value indicates a better control action. Thus, the Q-values will be used for optimizing the policy $\pi_\theta$.
Note that the twin critic NNs are only used for training. The trained actor NN is the only network used for deployment.

Let $\varphi_1$ and $\varphi_2$ denote the network parameters of the twin critic NNs.
The approximated Q-values are  $Q_{\varphi_1}( o_{t}^{\mathrm{C}}, h_{t}^{\mathrm{C}},a_{t}^{\mathrm{C}})$ and $ Q_{\varphi_2}(o_{t}^{\mathrm{C}}, h_{t}^{\mathrm{C}},a_{t}^{\mathrm{C}})$, respectively. To avoid overestimation of Q-values, where bad states are estimated as high values that can result in suboptimal policy updates and divergent behavior, only the smaller Q-value from the twin critic NNs is accepted for training the actor NN.

§.§.§ Training of Intelligent Controller

We consider an off-policy training scheme for the actor-critic NNs, where transitions consisting of the current state, history, action, reward, and next state are collected and stored in a $N_{\mathrm{C}}$-length experience replay buffer, $\mathcal{D}_{\mathrm{C}}$.

During the policy update, transitions will be sampled from the replay buffer to simultaneously update the actor and critic NNs.

Transitions in the replay buffer.
From the reward definition (<ref>), the immediate reward depends on both the plant state $s_t$ and the control input $u_t$.
We use the estimated plant state $\tilde{o}_t$ to approximate $s_t$.
Since the downlink (control) packet dropouts introduce randomness of the immediate reward, we use the expected reward for enhancing the smoothness and stability of the  controller training. Therefore, we have

\begin{equation}\label{avg reward}
\bar{r}_t\triangleq
\mathbb{E}_{\rho_t^{\mathrm{D}}}\left[\mathcal{R}(\tilde{o}_t,u_t)\right]
= (1-d^{\mathrm{D}})\mathcal{R}(\tilde{o}_t,a_t^{\mathrm{C}})+d^{\mathrm{D}}\mathcal{R}(\tilde{o}_t,0).
\end{equation}

Recall that $d^{\mathrm{D}}$ is the packet error probability of the downlink, and the control input is zero when a packet dropout occurs.
Thus, a transition is denoted as $<\!h_{t}^{\mathrm{C}}, o_{t}^{\mathrm{C}}, a^{\mathrm{C}}_{t}, \bar{r}_{t}, o_{t+1}^{\mathrm{C}}\!>$.
Then, the $i$th transition in the replay buffer is given as

\begin{equation}\label{controller transition}
\mathcal{T}_i^{\mathrm{C}}\triangleq <\!h_{(i)}^{\mathrm{C}}, o_{(i)}^{\mathrm{C}}, a^{\mathrm{C}}_{(i)}, \bar{r}_{(i)}, o_{(i')}^{\mathrm{C}}\!>\in{\mathcal{D}_{\mathrm{C}}},
\end{equation}

where $o_{(i')}^{\mathrm{C}}$ denotes the next sampled state given $o_{(i)}^{\mathrm{C}}$ and $a^{\mathrm{C}}_{(i)}$.

Actor and critic NN update with sampled transitions.
For each round of actor and critic NN update, the controller samples a batch of $N$ transitions from the reply buffer.

By taking into account the importance of different transitions, the optimal sampling probabilities are different. Let $P(i)$ denote the sampling probability of $\mathcal{T}^{\mathrm{C}}_i$, where $\sum_{i=1}^{N_{\mathrm{C}}} P(i) = 1$.
The importance sampling scheme will be discussed later in this section.

Let $\mathcal{T}_{\tilde{i}}^{\mathrm{C}}$ denote the $\tilde{i}$th sampled transition, which was the $\hat{i}$th transition in $\mathcal{D}_{\mathrm{C}}$ prior to the sampling. We then define the loss function for updating the actor NN as

\begin{equation}
J_\theta \triangleq \frac{1}{N}\sum_{{\tilde{i}}=1}^N w_{({\tilde{i}})}^{\mathrm{C}} Q_{\varphi_1}(h_{({\tilde{i}})}^{\mathrm{C}}, o_{({\tilde{i}})}^{\mathrm{C}}, a_{({\tilde{i}})}^{\mathrm{C}}),
\end{equation}

is the importance-sampling weight of $\mathcal{T}_{\hat{i}}$.
Note that the weights aim to correct the bias introduced by the importance sampling, which changes the distribution of the sampled data [30].

Then, we have the gradient for policy update as

\begin{equation}\label{actor update}
\nabla J_\theta = \frac{1}N\sum_{i=1}^{N} w_{(i)}^{\mathrm{C}} \nabla_\theta Q_{\varphi_1}(h_{(i)}^{\mathrm{C}}, o_{(i)}^{\mathrm{C}}, a_{(i)}^{\mathrm{C}})\nabla_\theta\pi_\theta(h_{(i)}^{\mathrm{C}}, o_{(i)}^{\mathrm{C}}).
\end{equation}

The loss function for updating the $k$th critic NN is based on the temporal-difference (TD) error, which is the difference of Q-value estimated by the current state-action pair and the next step pair.
In particular, the next step Q-value estimate is achieved by the Bellman equation as

\begin{equation}\label{target y1}
y_{(i)}^{\mathrm{C}} = \bar{r}_{(i)} + \gamma \min_{k=1,2}Q_{\varphi_k}(h_{(i')}^{\mathrm{C}}, o_{(i')}^{\mathrm{C}}, \pi_{\theta}(h_{(i')}^{\mathrm{C}}, o_{(i')}^{\mathrm{C}})).
\end{equation}

Recall that the smaller estimated Q-value in the next step is adopted to avoid overestimation.
Then, the TD-error of the $k$th critic NN is

\begin{equation}
\mathsf{TD}_{k,i} \triangleq y_{(i)}^{\mathrm{C}} - Q_{\varphi_k}(h_{(i)}^{\mathrm{C}}, o_{(i)}^{\mathrm{C}}, a_{(i)}^{\mathrm{C}}), k \in \{1,2\}.
\end{equation}

Therefore, the loss function and the gradient of the $k$th critic NN are

\begin{equation}\label{eq:loss_TD}
J_{\varphi_k}= \frac{1}{N}\sum_{i=1}^{N} w_{(i)}^{\mathrm{C}} \left(\mathsf{TD}_{k,i}\right)^2,
\end{equation}

\begin{equation}\label{critic update}
\nabla J_{\varphi_k} = - \frac{2}{N} \sum_{i=1}^{N} \omega_{(i)}^{\mathrm{C}} \mathsf{TD}_{k,i} \nabla_{\varphi_k} Q_{\varphi_k}(h_{(i)}^{\mathrm{C}}, o_{(i)}^{\mathrm{C}}, a_{(i)}^{\mathrm{C}}),
\end{equation}

Note that the loss functions $J_\theta$ and $J_{\phi_k}$ need to be maximized and minimized, respectively, by the gradient decent method.

Novel importance sampling scheme.
Different from existing importance sampling schemes, the importance of each transition in our problem depends on three aspects: 1) the AoI indicating the accuracy of a transition, i.e., the one with small AoI should be sampled more often; 2) the freshness of the transition, i.e., an old transition should be sampled less and less often in the training; 3) the TD error, i.e., a transition with larger TD error should be sampled more often, as it is a more dominant term in (<ref>) determining how the critic NNs should be updated.
By properly taking into account the above factors, we propose a novel importance sampling scheme as below to enhance the sampling efficiency.

* Sampling probability. The sampling probability of the $i$th transition $\mathcal{T}_i^{\mathrm{C}}\in{\mathcal{D}_{\mathrm{C}}}$ is given by

\begin{equation}\label{IS prob}
P(i)=\frac{\mathrm{rank}(i)^{-\alpha}}{\sum_{n=1}^{N_{\mathrm{C}}}\mathrm{rank}(n)^{-\alpha}}, 0<\alpha\leq1,
\end{equation}

where $N_{\mathrm{C}}$ is the size of the experience replay buffer $\mathcal{D}_{\mathrm{C}}$ and $\mathrm{rank}(i)$ denotes the rank of the transition $\mathcal{T}_i^{\mathrm{C}}$ in $\mathcal{D}_{\mathrm{C}}$, and the first transition in $\mathcal{D}_{\mathrm{C}}$ has the highest ranking. Thus, the highly ranked transitions (with small $\mathrm{rank}(i)$) are sampled with higher probabilities.

* New data insertion. The most recently generated transition is inserted as the first one in $\mathcal{D}_{\mathrm{C}}$, and the rest of the buffer is shifted by one step. In this way, the new transitions are more likely to be highly ranked and sampled frequently.

* AoI and TD error-based periodic resorting. The replay buffer is sorted every $n_{\mathrm{sort}}$ time step based on the ranking values of each transition.
Let $V_i^{\mathrm{rank}}$ denote the ranking value of $\mathcal{T}_i^{\mathrm{C}}$. It should take into account both the accuracy of the data and the TD error.
In particular, since each transition contains two estimated plant states, the transition AoI is defined as

\begin{equation}\label{transition AoI}
I_i^{\mathrm{AoI}} = n_{(i)}^{\mathrm{AoI}} + n_{(i')}^{\mathrm{AoI}}.
\end{equation}

Since different transitions can have the same transition AoI, we should sort them based on the TD errors. Thus, the ranking value is defined as
\begin{equation}\label{ranking value}
V_i^{\mathrm{rank}}=-I_i^{\mathrm{AoI}}+2\left(\mathrm{sigmoid}\left((\mathsf{TD}_{1,i})^2 \right)-1/2\right),
\end{equation}

where the sigmoid-based function $2(\mathrm{sigmoid}(\cdot)-1/2)$ is to normalize the squared TD error in the range of $[0,1)$.
A higher ranking value indicates an accurate transition with a large TD error, and the corresponding transition will be sorted with a smaller $\mathrm{rank}(i)$ and sampled more often.
To reduce the computation complexity, we do not update all transitions' ranking values at each time step but only update the sampled ones.

The detailed DL-based algorithm for the estimation-control co-design of the low-mobility WNCS is presented in Algorithm <ref>.

Deep learning-based estimation-control co-design for the low-mobility WNCS.

[1]

Initialize estimator network $\mathcal{E}_{\mu}$, actor network $\pi_{\theta}$, critic networks $Q_{\varphi_1},Q_{\varphi_2}$ with random parameters $\mu,\theta,\varphi_1,\varphi_2$
Initialize target networks $\theta'\gets{\theta}, \varphi_1'\gets{\varphi_1}, \varphi_2'\gets{\varphi_2}$
Initialize experience replay buffers $\mathcal{D}_{\mathrm{E}}$ and $\mathcal{D}_{\mathrm{C}}$ with sizes $N_{\mathrm{E}}$ and $N_{\mathrm{C}}$

Receive observation $\tilde{o}_t=o_t$
Predict observation with estimator $\tilde{o}_t=\mathcal{E}_\mu(h_t^{\mathrm{E}})$
Update AoI $n_t^{\mathrm{AoI}}$ to form actor input $o_t^{\mathrm{C}}\triangleq[\tilde{o}_t, n_t^{\mathrm{AoI}}]$

Select action with exploration noise $a_t^{\mathrm{C}}=\pi_\theta(h_t^{\mathrm{C}},o_t^{\mathrm{C}})+\epsilon,\epsilon\sim{\mathcal{N}(0,{\sigma_{\mathrm{expl}}}^2)}$
Calculate expected reward $\bar{r}_t$ according to (<ref>)

Store transition $\mathcal{T}_{i=t}^{\mathrm{C}}$ labeled with ranking value $V_i^{\mathrm{rank}} = -I_{i=t}^{\mathrm{AoI}}$ according to (<ref>) as the first one in $\mathcal{D}_{\mathrm{C}}$
Store transition $\mathcal{T}_{j=t}^{\mathrm{E}}$ in $\mathcal{D}_{\mathrm{E}}$

$t$ mod $N_{\mathrm{C}}$
Sort $\mathcal{D}_{\mathrm{C}}$ according to (<ref>)

Sample a mini-batch of $M$ transitions $\{\mathcal{T}_m^{\mathrm{E}}\}_{m=1}^M$ uniformly at random from $\mathcal{D}_{\mathrm{E}}$
Update estimator $\mathcal{E}_\mu$ by $\mu \gets \mu -
{\alpha_\mu} {\nabla J_\mu}$ according to (<ref>) with learning rate $\alpha_\mu$

Sample a mini-batch of $N$ transitions $\{\mathcal{T}_n^{\mathrm{C}}\}_{n=1}^N$ from $\mathcal{D}_{\mathrm{C}}$ according to (<ref>)
Calculate the TD-errors $\{\mathsf{TD}_{1,n}\}_{n=1}^N$ of sampled transitions and update their ranking values in $\mathcal{D}_{\mathrm{C}}$ according to (<ref>)
Update actor $\pi_\theta$ by $\theta\gets\theta + \alpha_\theta {\nabla J_\theta}$ according to (<ref>) with learning rate $\alpha_\theta$
Update twin critics $Q_{\varphi_k} (k \in \{1,2\})$ by
${\varphi_k} \gets {\varphi_k} - \alpha_\varphi{\nabla J_{\varphi_k}}$ according to (<ref>) with learning rate $\alpha_\varphi$
$t$ mod $n_\mathrm{target}$
Update target networks:
$\theta'\gets{\tau\theta + (1-\tau)\theta'}$
$\varphi_1'\gets{\tau\varphi_1 + (1-\tau)\varphi_1'}$
$\varphi_2'\gets{\tau\varphi_2 + (1-\tau)\varphi_2'}$

§ ESTIMATION-CONTROL-SCHEDULING CO-DESIGN OVER DYNAMIC FADING CHANNELS

In this section, we investigate the high-mobility WNCS over fading channels. Both the control performance and the sensor's transmission energy consumption are considered in the estimation-control-scheduler co-design, as illustrated in Fig. <ref>.
We note that since the estimator is independent of the fading channel states and the scheduling policy, it is identical to the low-mobility scenario in Section <ref>.
However, the controller is different due to the additional channel states.
Since the scheduler solves a dynamic decision-making problem, it is DRL-based, similar to the controller.
Due to the fact that the control and scheduling have continuous and discrete action spaces, respectively, they should be designed by different DRL algorithms.
Most importantly, the joint training of the two DRL agents brings new challenges. For example, the controller needs frequent sensor transmissions to have more accurate state estimation for high-quality control; while the scheduler might tend to transmit as less often as possible to reduce the transmission energy consumption.
It is critical to have a stable training process to achieve the desired tradeoff between them.

In the following, we first present the details of the controller and the scheduler models, and then propose a novel DRL method for joint training.

§.§ DRL-based Controller and Scheduler over Fading Channels

Considering the state estimation, transmission scheduling, control signal generation, and uplink and downlink transmissions, each time slot can be divided into multiple sub-slots as shown in Fig. <ref> including the available signals after each sub-slot. Note that the scheduler and the controller have different plant state input (i.e., $\hat{o}_t$ and $\tilde{o}_t$, respectively) for decision making, since the scheduler operates before receiving the sensor's packet and can only utilize the state estimation $\hat{o}_t$ based on the previous sensor measurements.

Estimator-controller-scheduler co-design of the high-mobility WNCS.

The sub-slots and the signals obtained after each sub-slot.

The controller takes into account the estimated plant state, the AoI state, and the uplink and the downlink channel states for generating the control signal. The extended input of the controller is

\begin{equation}\label{controller input 2}
o_{t}^{\mathrm{C}} \triangleq [\tilde{o}_t, b_t^{\mathrm{U}}, b_t^{\mathrm{D}}, n_t^{\mathrm{AoI}}].
\end{equation}

The action space and the actor-critic framework of the controller are identical to the low-mobility scenario.

The transmission scheduler makes decision based on the estimated plant state $\hat{o}_t$, the uplink channel state $b_t^{\mathrm{U}}$, and the AoI state  $n_{t-1}^{\mathrm{AoI}}$ from the previous time slot. Thus, the current state for the transmission schedule is defined as

\begin{equation}\label{scheduler input}
o_{t}^{\mathrm{Tx}}\triangleq[\hat{o}_t, b_t^{\mathrm{U}}, n_{t-1}^{\mathrm{AoI}}].
\end{equation}

Let  $h_t^{\mathrm{Tx}}$ denote an $\ell$-length state history of the scheduler as

\begin{equation}
(o_{t-\ell}^{\mathrm{Tx}},a_{t-\ell}^{\mathrm{Tx}},\dots,o_{t-1}^{\mathrm{Tx}},a_{t-1}^{\mathrm{Tx}}) & \parbox[t]{.1\textwidth}{if $t>\ell$}\\
(0,0,\dots,o_{1}^{\mathrm{Tx}},a_{1}^{\mathrm{Tx}},\dots,o_{t-1}^{\mathrm{Tx}},a_{t-1}^{\mathrm{Tx}}) & \parbox[t]{.1\textwidth}{if $\ell\geq{t}>1$}\\
(0,0,\dots,0,0) & \parbox[t]{.1\textwidth}{otherwise}
\end{cases}
\label{scheduler history}
\end{equation}

The schedule decision at each time depends on both the state $o_{t}^{\mathrm{Tx}}$ and the history $h_t^{\mathrm{Tx}}$.

The scheduler adopts a Deep Q-Network (DQN) [31] to generate actions since DQNs are commonly used for solving decision-making problems with discrete actions. Different from the actor-critic NNs, a DQN only approximates the Q-values at an input state with different actions.
Let $Q_{\phi}(\cdot)$ denote the DQN with parameter set $\phi$. Then, $Q_{\phi}(h_t^{\mathrm{Tx}}, o_t^{\mathrm{Tx}}, a_t^{\mathrm{Tx}})$ is the Q-value given the state input $(h_t^{\mathrm{Tx}}, o_t^{\mathrm{Tx}})$ and the action $a_t^{\mathrm{Tx}}$.
The scheduling action is given by

\begin{equation}\label{scheduler}
a_t^{\mathrm{Tx}}=\mathop{\arg\max}_{a \in \{0,1\}} Q_\phi(h_t^{\mathrm{Tx}}, o_t^{\mathrm{Tx}}, a),
\end{equation}

where the action leading to the highest Q-value is chosen.
Due to the historical input, we adopt the same RNN structure in Fig. <ref> for the DQN.
We assume that the controller sends the one-bit scheduling signal $a_t^{\mathrm{Tx}}$ to the sensor via a perfect channel due to the negligible transmission overhead.

§.§ Joint Training of Controller and Scheduler

By taking into account the sensor transmission energy consumption $\bar{e}$, the per-step overall reward for training the controller and scheduler is the control reward deducted by the energy consumption as

\begin{equation}\label{total performance}
r_t^{\mathrm{total}}=\bar{r}_t - a^{\mathrm{Tx}}_t \bar{e},
\end{equation}

where the control reward $\bar{r}_t$ was defined in (<ref>).
As shown in Fig. <ref>, the overall reward can be calculated at the end of each time slot.

However, when training both the DRL-based controller and scheduler with the reward (<ref>), convergence to desired policies is often difficult to achieve.
This is mainly for two reasons:

1) At the beginning of the training, since the control reward $\bar{r}_t$ can be very low due to the lack of training of the controller, energy consumption is the dominant part of the overall reward. Thus, the scheduler tends not to schedule any sensor transmission. Note that any policy exploration, e.g., schedule of more sensor transmissions, will immediately lead to much lower overall rewards, especially when $\bar{e}$ is large.
Therefore, the controller can never be properly trained, and hence the control reward cannot be increased much, and the transmission energy consumption is always the dominant term. This inevitably leads to poorly trained control and scheduling policies.

2) When training the scheduler with the reward (<ref>), since the scheduler's action has no direct (short-term) impact on the control reward term, it would always try to reduce the number of transmissions to achieve a higher overall reward. This wrong indention makes the joint controller and scheduler training inefficient, leading to poor performance. Furthermore, when the energy consumption $\bar{e}$ is large, the scheduler's stochastic action results in large variations of the overall reward, making the training process difficult to converge.

To solve the first issue, we propose to have a pre-training phase of the controller before the joint training, where the sensor is always scheduled for transmission.
After the pre-training, the average control reward can be improved and comparable to the transmission energy consumption.

Training of the controller.
The algorithm is identical to the low-mobility scenario in Section <ref>, where the only differences are to use the overall reward $r_t^{\mathrm{total}}$, instead of the control reward, and to operate with the scheduler's training at the same time.

To solve the second issue, we should eliminate the impact of the instantaneous transmission energy cost on the scheduler's reward for the training purpose. Now, we propose to take the Q-value generated by the controller's DQN as the scheduler's reward, i.e.,

\begin{equation}\label{Q reward}
q_t = Q_{\varphi_1}(h_t^{\mathrm{C}},o_t^{\mathrm{C}},a_t^{\mathrm{C}}).
\end{equation}

This choice is motivated by the following two observations: 1) the controller's Q-value represents the expected long-term overall reward under the current control and scheduling policies, and a higher Q-value indicates a better schedule policy; and 2) the schedule action does not have a dominant instantaneous impact on the Q-value, which will avoid persistently reducing sensor transmissions.

Training of the transmission scheduler.
Each transition for scheduler training is denoted as

\begin{equation}\label{scheduler transition}
\mathcal{T}_i^{\mathrm{Tx}}\triangleq(h_{(i)}^{\mathrm{Tx}},o_{(i)}^{\mathrm{Tx}},a_{(i)}^{\mathrm{Tx}},q_{(i)},o_{(i')}^{\mathrm{Tx}})\in{\mathcal{D}_{\mathrm{Tx}}}
\end{equation}

where $\mathcal{D}_{\mathrm{Tx}}$ is the scheduler's reply buffer. Note that each scheduling action during training is selected based on the $\epsilon$-greedy policy.
We adopt the same importance sampling method as presented in Section <ref>. Given the importance-sampling weight $\omega_{(i)}^{\mathrm{Tx}}$, the loss function for optimizing the DQN is

\begin{equation}\label{eq:sch_loss}
J_\phi \triangleq \frac{1}{N} \sum_{i=1}^{N} \omega_{(i)}^{\mathrm{Tx}}\left(\mathsf{TD}_i\right)^2
\end{equation}

where the TD error is defined as

\begin{eqnarray}
\mathsf{TD}_i \triangleq y_{(i)}^{\mathrm{Tx}} - Q_\phi(h_{(i)}^{\mathrm{Tx}}, o_{(i)}^{\mathrm{Tx}}, a_{(i)}^{\mathrm{Tx}}),
\end{eqnarray}

\begin{equation}
y_{(i)}^{\mathrm{Tx}} = q_{(i)} + \gamma \max_{a}Q_{\phi}(h_{(i')}^{\mathrm{Tx}}, o_{(i')}^{\mathrm{Tx}}, a).
\end{equation}

Then, we have the gradient for DQN update as

\begin{equation}\label{actor update_DQN}
\nabla J_\phi =- \frac{2}N\sum_{\{\mathcal{T}_i^{\mathrm{C}}\}_{i=1}^N} \omega_{(i)}^{\mathrm{Tx}} \mathsf{TD}_i\nabla_\phi Q_{\phi}(h_{(i)}^{\mathrm{Tx}}, o_{(i)}^{\mathrm{Tx}}, a_{(i)}^{\mathrm{Tx}}).
\end{equation}

The details of the estimation-control-scheduler co-design in high-mobility WNCS are given in Algorithm <ref>.

Deep learning-based estimation-control-scheduler co-design for the high-mobility WNCS.

[1]

Initialize estimator network $\mathcal{E}_{\mu}$, scheduler network $Q_{\phi}$, actor network $\pi_{\theta}$, critic networks $Q_{\varphi_1},Q_{\varphi_2}$ with random parameters $\mu,\phi,\theta,\varphi_1,\varphi_2$
Initialize target networks $\phi'\gets{\phi}, \theta'\gets{\theta}, \varphi_1'\gets{\varphi_1}, \varphi_2'\gets{\varphi_2}$
Initialize experience replay buffers $\mathcal{D}_{\mathrm{E}}$, $\mathcal{D}_{\mathrm{Tx}}$ and $\mathcal{D}_{\mathrm{C}}$ with sizes $N_{\mathrm{E}}, N_{\mathrm{Tx}}$ and $N_{\mathrm{C}}$

Predict observation with estimator $\hat{o}_t=\mathcal{E}_\mu(h_t^{\mathrm{E}})$ to form scheduler input $o_{t}^{\mathrm{Tx}}\triangleq[\hat{o}_t, b_t^{\mathrm{U}}, n_{t-1}^{\mathrm{AoI}}]$
With probability $\epsilon_\mathrm{greedy}$ select a random action $a_t^{\mathrm{Tx}}\in{\{0,1\}}$ otherwise select action $a_t^{\mathrm{Tx}}$ according to (<ref>)

Execute control signal $u_t =\rho^{\mathrm{D}}_{t} a_t^{\mathrm{C}}$ according to Algorithm <ref> and generate Q-value $q_t = Q_{\varphi_1}(h_t^{\mathrm{C}},o_t^{\mathrm{C}},u_t)$

Store transitions $\mathcal{T}_{i=t}^{\mathrm{C}}$ in $\mathcal{D}_{\mathrm{C}}$ and $\mathcal{T}_{j=t}^{\mathrm{E}}$ in $\mathcal{D}_{\mathrm{E}}$ according to Algorithm <ref>
Store transition $\mathcal{T}_{l=t}^{\mathrm{Tx}}$ labeled with ranking value $V_l^{\mathrm{rank}} = -I_{l=t}^{\mathrm{AoI}}$ according to (<ref>) as the first one in $\mathcal{D}_{\mathrm{Tx}}$

Sort $\mathcal{D}_{\mathrm{C}}$ according to Algorithm <ref>
$t$ mod $N_{\mathrm{Tx}}$
Sort $\mathcal{D}_{\mathrm{Tx}}$ according to (<ref>)

Update estimator $\mathcal{E}_\mu$ according to Algorithm <ref>

Update actor $\pi_\theta$, twin critics $Q_{\varphi_1}, Q_{\varphi_2}$ and target networks according to Algorithm <ref>

Sample a mini-batch of $N$ transitions $\{\mathcal{T}_n^{\mathrm{Tx}}\}_{n=1}^N$ from $\mathcal{D}_{\mathrm{Tx}}$ according to (<ref>)
Calculate the TD-errors $\{\mathsf{TD}_{1,n}\}_{n=1}^N$ of sampled transitions and update their ranking values in $\mathcal{D}_{\mathrm{Tx}}$ according to (<ref>)
Update scheduler $Q_\phi$ by ${\phi} \gets {\phi} - \alpha_\phi{\nabla J_{\phi}}$ according to (<ref>) with learning rate $\alpha_\phi$
$t$ mod $n_\mathrm{target}'$
Update target network $\phi'\gets{\phi}$

§ NUMERICAL EXPERIMENTS

In this section, we design simulations to evaluate the performance of the proposed co-design algorithms for the low-mobility and high-mobility WNCSs in Sections <ref> and <ref>, and compare them with some benchmark policies.

§.§ Experiment Setups

The sensor measurement noise is assumed to be Gaussian, i.e., $v_t\sim{\mathcal{N}(0,\sigma^2)}$. We set the history length as $\ell=3$. The communication parameters for the two scenarios are as below.

§.§.§ Low-mobility WNCS

We consider $6$ scenarios with different static packet error probabilities of the uplink channel and the downlink channel as well as different measurement noise powers, as shown in Table <ref>.

§.§.§ High-mobility WNCS

We model the uplink and downlink fading channels as two-state Markov chains, that $b_t^{\mathrm{U}}\triangleq\{w_1^{\mathrm{U}}, w_2^{\mathrm{U}}\}$ and $b_t^{\mathrm{D}}\triangleq\{w_1^{\mathrm{D}}, w_2^{\mathrm{D}}\}$, where both the states $w_1^{\mathrm{U}}$ and $w_1^{\mathrm{D}}$ have a packet error probability of $5\%$, and $w_2^{\mathrm{U}}$ and $w_2^{\mathrm{D}}$ have $10\%$.
We consider two channel state transition probability matrices $M_1$ and $M_2$ as

\begin{equation}
\begin{bmatrix}
0.7 & 0.3\\
0.3 & 0.7
\end{bmatrix}
\end{equation}

\begin{equation}
\begin{bmatrix}
0.3 & 0.7\\
0.7 & 0.3
\end{bmatrix},
\end{equation}

respectively. It is clear that the fading channel $M_1$ has a longer average channel state holding time, while $M_2$ leads to a more frequent change of channel states.
For the communication cost, we consider two scenarios with low communication cost $\bar{e}=5$ and high communication cost $\bar{e}=10$ respectively.

The plant system with unknown dynamics can be modeled by one of the MuJoCo tasks, well-known benchmarks for RL algorithms, in the OpenAI Gym[https://gym.openai.com/envs/#mujoco] open-source simulation environment. We consider $3$ MuJoCo tasks: HalfCheetah-v2 (HalfCheetah), Hopper-v2 (Hopper) and InvertedDoublePendulum-v2 (InvDoublePen) to evaluate the performance of the proposed algorithms. A brief description and the state and control input dimensions of these MuJoCo tasks are given in Table <ref>, and the tasks with different states are illustrated in Fig. <ref>. The control performance of MuJoCo tasks is evaluated by the average sum of the control reward over a whole episode. A brief explanation of the control reward of the MuJoCo tasks is illustrated in Table <ref>. We note that the MuJoCo environment does not involve plant disturbance. Thus, we assume $\nu_t=0$ in our simulation.

Description of MuJoCo tasks

Plant     Description

State     Ctrl Input

HalfCheetah     Make a 2D cheetah robot run     17     6

Hopper     Make a 2D 1-legged robot hop     11     3

InvDoublePen     Balance 2-joint pole on a cart     11     1

Description of control reward in MuJoCo

Controlled plant     Reward formulation

HalfCheetah     FR - CC

Hopper     FR + AR - CC

InvDoublePen     AR - FR

* Forward reward (FR) depends on the displacement and velocity; Control cost (CC) depends on the magnitude of the control signal; Alive reward (AR) depends on the current state of the controlled plant https://mujoco.org/.

Three MuJoCo tasks at different states: from top to bottom, are HalfCheetah, Hopper and InvDoublePen, respectively

Comparison of control performance in the low-mobility scenario.

2cControlled plant     3cWNCS setup
3cControl performance

Name     Scenario label     Uplink channel dropout rate     Downlink channel dropout rate     Measurement noise $\sigma$     Hybrid AoI     Hybrid Uniform     MF Uniform

HalfCheetah      1     10%     10%     0.01     ${6110 }\pm{368}$     ${5306 }\pm{263}$     ${4338 }\pm{145}$

2     10%     5%      0.01     ${6909 }\pm{167}$     ${6263 }\pm{234}$     ${5377 }\pm{253}$

3     5%      10%     0.01     ${7072 }\pm{203}$     ${6627 }\pm{224}$     ${6129 }\pm{206}$

4     10%     0%      0.01     ${8408 }\pm{397}$     ${7511 }\pm{262}$     ${6450 }\pm{373}$

5     0%      10%     0.01     ${8092 }\pm{306}$     ${7555 }\pm{290}$     ${7510 }\pm{292}$

6     10%     5%      0.05     ${6034 }\pm{325}$     ${5393 }\pm{201}$     ${4724 }\pm{331}$

Hopper           1     10%     10%     0.01     ${2031 }\pm{177}$     ${1774 }\pm{223}$     ${1284}\pm{207}$

2     10%     5%      0.01     ${3102 }\pm{187}$     ${2794 }\pm{246}$     ${1825}\pm{354}$

3     5%      10%     0.01     ${2426 }\pm{267}$     ${2118 }\pm{240}$     ${1529 }\pm{359}$

4     10%     0%      0.01     ${3302 }\pm{179}$     ${3015 }\pm{178}$     ${2651 }\pm{297}$

5     0%      10%     0.01     ${2684 }\pm{261}$     ${2517 }\pm{269}$     ${2468 }\pm{273}$

6     10%     5%      0.05     ${2594 }\pm{301}$     ${2144 }\pm{279}$     ${1524 }\pm{199}$

InvDoublePen     1     10%     10%     0.01     ${1208 }\pm{146}$     ${1144 }\pm{191}$     ${949 }\pm{285}$

2     10%     5%      0.01     ${6647 }\pm{636}$     ${5576 }\pm{766}$     ${3832 }\pm{953}$

3     5%      10%     0.01     ${2203 }\pm{321}$     ${1770 }\pm{215}$     ${1324 }\pm{203}$

4     10%     0%      0.01     ${8844 }\pm{778}$     ${8019 }\pm{839}$     ${6294 }\pm{965}$

5     0%      10%     0.01     ${2555 }\pm{565}$     ${2032 }\pm{417}$     ${1990 }\pm{419}$

6     10%     5%      0.05     ${2506 }\pm{967}$     ${1864 }\pm{439}$     ${828 }\pm{133}$

Comparison of overall performance in the high-mobility scenario.

2cControlled plant     3cWNCS setup
3cOverall performance

Name     Scenario label     $M^{\mathrm{U}},M^{\mathrm{D}}$     Communication cost     Measurement noise $\sigma$     Scheduler-Q-Value     Scheduler-Reward     No-Scheduler

HalfCheetah      7     $M_1,M_1$     5     0.01     ${2622}\pm{434}$     ${1528}\pm{795}$     ${2332}\pm{79}$

8     $M_2,M_2$     5      0.01     ${2374}\pm{393}$     ${1192}\pm{762}$     ${2133}\pm{90}$

9     $M_1,M_1$      10     0.01     ${-1316}\pm{624}$     ${-1705}\pm{926}$     ${-2793}\pm{84}$

10     $M_2,M_2$     10      0.01     ${-1539}\pm{710}$     ${-2219}\pm{1367}$     ${-3126}\pm{87}$

The NN parameters of the estimator, controller, and scheduler are given in Table <ref>. The hyper-parameters for NN training are provided in Table <ref>.

NN parameters.

NN Layers:     2|<EMAIL_ADDRESS>1
2<EMAIL_ADDRESS>2
4<EMAIL_ADDRESS>3

NN type:     2<EMAIL_ADDRESS>2<EMAIL_ADDRESS>1
2<EMAIL_ADDRESS>2

Parameters:     DIM     AF     DIM     AF     DIM     AF     DIM     AF     DIM     AF

Estimator     $[n_{\mathrm{his}}^{\mathrm{E}},128]$     ReLU     $[128,n_{\mathrm{out}}^{\mathrm{E}}]$     Linear     $\backslash$     $\backslash$     $\backslash$     $\backslash$     $[128,128]$     Linear

DQN     $[n_{\mathrm{cur}}^{\mathrm{Tx}},128]$     ReLU     $[n_{\mathrm{his}}^{\mathrm{Tx}},128]$     ReLU     $[256,128]$     ReLU     $[128,2]$     Linear     $[128,128]$     Linear

Actor     $[n_{\mathrm{cur}}^{\mathrm{C}},128]$     ReLU     $[n_{\mathrm{his}}^{\mathrm{C}},128]$     ReLU     $[256,128]$     ReLU     $[128,n_{\mathrm{out}}^{\mathrm{C}}]$     Tanh     $[128,128]$     Linear

Critic     $[n_{\mathrm{cur}}^{\mathrm{C}}+n_{\mathrm{out}}^{\mathrm{C}},128]$     ReLU     $[n_{\mathrm{his}}^{\mathrm{C}},128]$     ReLU     $[256,128]$     ReLU     $[128,1]$     Linear     $[128,128]$     Linear

The parameters row presents the input-output dimension (DIM) and the activation function (AF) of each NN layer. The $n_{\mathrm{cur}}^{\mathrm{C}}, n_{\mathrm{cur}}^{\mathrm{Tx}}$ denote the dimensions of $o_t^\mathrm{C}, o_t^\mathrm{Tx}$ and $n_{\mathrm{his}}^{\mathrm{E}}, n_{\mathrm{his}}^{\mathrm{C}}, n_{\mathrm{his}}^{\mathrm{Tx}}$ denote the dimensions of $h_t^\mathrm{E}, h_t^\mathrm{C}, h_t^\mathrm{Tx}$ respectively, and $n_{\mathrm{out}}^{\mathrm{E}}, n_{\mathrm{out}}^{\mathrm{C}}$ are the dimensions of $o_t, a_t^{\mathrm{C}}$ respectively.
The parameters $n_{\mathrm{cur}}^{\mathrm{C}}, n_{\mathrm{cur}}^{\mathrm{Tx}}, n_{\mathrm{his}}^{\mathrm{E}}, n_{\mathrm{his}}^{\mathrm{C}}, n_{\mathrm{his}}^{\mathrm{Tx}}, n_{\mathrm{out}}^{\mathrm{E}}, n_{\mathrm{out}}^{\mathrm{C}}$ in the MuJoCo tasks are given as follows: HalfCheetah: $20, 19, 69, 78, 60, 17, 6$; Hopper: $14, 13, 42, 51, 42, 11, 3$; InvDoublePen: $14, 13, 36, 45, 42, 11, 1$.

Hyper-parameters for NN training

Hyper-parameter     Value

Size of replay buffers $N_{\mathrm{E}},N_{\mathrm{C}},N_{\mathrm{Tx}}$     $10^{5}$

Batch size $M, N$     $100$

History length $\ell$     $3$

Discount factor $\gamma$     $0.99$

Scheduler $\epsilon$-greedy probability $\epsilon_\mathrm{greedy}$     0.1

Actor exploration noise $\sigma_{\mathrm{expl}}$     0.1

Target update delay $n_\mathrm{target}$     $2$

Target update delay $n_\mathrm{target}'$     $100$

Target update rate $\tau$     $0.005$

Optimizer type     Adam

Estimator learning rate $\alpha_\mu$     $10^{-3}$

Scheduler learning rate $\alpha_\phi$     $3\times{10^{-4}}$

Actor learning rate $\alpha_\theta$     $3\times{10^{-4}}$

Critic learning rate $\alpha_\varphi$     $3\times{10^{-4}}$

Sampling probability prioritization $\alpha$     $1$

§.§ Performance Evaluation and Comparison

§.§.§ Joint vs. separative estimation-control methods of the low-mobility scenario.

We consider scenarios 2 and 3 of the InvDoublePen task in Table <ref> to compare the control performance between the joint and separative estimation-control methods.[The other scenarios show the same trend as the selected ones and thus are omitted for brevity.] For a separative estimation-control method, the estimator will first be trained by using an independent and identically distributed (i.i.d.) random control inputs, after which the trained estimator will be adopted to assist the intelligent controller training according to Section <ref>. We evaluate three setups of the separately training method.

We plot the learning curves of the joint and separative training methods in Fig. <ref>. We see that the control performance of the joint method during the training improves stably and is significantly higher than the separative ones. For example, the control performance doubled when completing the training in Scenario 2. This is because during the separative training of the estimator, the i.i.d. random control inputs are not able to explore the state-action space of the plant effectively. Thus, the DL-based estimator can only learn part of the plant dynamics and provide inaccurate estimations when training the controller. The proposed joint method explores the plant dynamics more efficiently for estimator training, which in turn enhances the controller's optimization.

Comparison of the learning curves of the joint and separative estimation-control methods. Hybrid AoI (in blue) denotes the proposed joint algorithm and the remaining three are the separative ones  (see Table <ref> for explanation).

§.§.§ Performance evaluation of the co-design algorithm in the low-mobility scenario

in Section <ref>, we propose a novel DRL framework for controller training with the hybrid MB-MF experience replay buffer and the AoI-based importance sampling method. To verify the effectiveness of the proposed algorithm, we compare it (Hybrid AoI) with two baseline algorithms, i.e., the one (Hybrid Uniform) with the hybrid MB-MF replay buffer and the uniform data sampling, and the one (MF Uniform) with a purely MF replay buffer (no plant state estimation) and the uniform sampling, as described in Table <ref>.

Benchmark Policies (low-mobility scenario).

Method     Description

Hybrid AoI (Sep-55)     lHybrid MB-MF controller with AoI-based
importance sampling, where the estimator and the
controller are trained separately with
$5 \times 10^5$ time steps each.

Hybrid AoI (Sep-46)     lHybrid MB-MF controller with AoI-based
importance sampling, where the estimator and the
controller are trained separately with
$4 \times 10^5$ and $6 \times 10^5$ time steps, respectively.

Hybrid AoI (Sep-64)     lHybrid MB-MF controller with AoI-based
importance sampling, where the estimator and the
controller are trained separately with
$6 \times 10^5$ and $4 \times 10^5$ time steps, respectively.

Hybrid AoI (proposed)     lHybrid MB-MF controller with AoI-based
importance sampling, where the estimator and the
controller are jointly trained.

Hybrid Uniform     lHybrid MB-MF controller with uniform random
importance sampling, where the estimator and the
controller are jointly trained.

MF Uniform     lMF controller with uniform random experience
replay (i.e., the controller receives zero-valued
observations when packet dropout occurs).

Performance evaluation of the proposed algorithm over three MuJoCo tasks:
Figs. <ref> and <ref> show the learning performance of the proposed algorithm and the baseline ones of three control tasks over six different WNCS settings (see Table <ref> for explanations).

We see that the InvDoublePen suffers most from high packet error probability and measurement noise while the HalfCheetah takes the minimum impact, e.g., when comparing scenario 1 (less reliable communication) with scenario 2 (more reliable communication). This is because the InvDoublePen is the most difficult control task due to the fact that the cart (i.e., the yellow part in the last row of Fig. <ref>) of the InvDoublePen is the only thing one can control for balancing the double-joint inverted pendulum. The task is impossible in the presence of relatively high packet error probability and measurement noise. On the other hand, for the HalfCheetah task, as shown in Table <ref>, the control reward does not consist of an alive reward, which means the HalfCheetah is more robust than the others and thus becomes less sensitive to packet dropouts and measurement noises.

Comparison of learning curves in the low-mobility scenario - Part 1: different channel conditions.

Comparison of learning curves in the low-mobility scenario - Part 2: different measurement noises.

Comparison between hybrid MB-MF and MF methods.
From Figs. <ref> and <ref>, it is clear that the proposed algorithm has the best learning performance.
In particular, when comparing the two baseline algorithms, the hybrid MB-BF replay buffer-based approach outperforms the conventional MF one in all scenarios. In scenario 2 of the Hopper task, a $50\%$ control performance improvement has been achieved.

Besides the learning curves, we have included the testing results of the control performance in Table <ref>. Comparing the proposed Hybrid AoI with the MF Uniform baseline[We assume that the control reward of the MF Uniform method is always known despite the presence of packet dropout. This is an ideal case for the MF Uniform method.], significant control performance improvements can be achieved. For example, the control performance has been improved by $70\%, 73\% \text{ and } 202\%$ in Hopper scenario 6, and InvDoublePen scenarios 2 and 6, respectively.
Even for the worst cases, i.e., HalfCheetah scenario 3 and 6, and InvDoublePen scenario 1, the improvement of $15\%, 28\% \text{ and } 27\%$ are obtained.

Comparison between the AoI-based importance sampling and the uniform sampling methods.
In Table <ref>, we compare the performance between the Hybrid AoI and the Hybrid Uniform algorithms over all scenarios. We see that improvements of $7\%-24\%$ can be obtained under scenarios with $5\%$ of uplink packet error rate, which further increases to $10\%-34\%$ under the $10\%$ uplink packet error rate, except for InvDoublePen scenario 1 with high packet dropout rates.

§.§.§ Performance evaluation of the co-design algorithm in the high-mobility scenario

In Section <ref>, we have proposed the joint estimator-controller-scheduler training algorithm with the consideration of the fading channel states and the communication costs. In particular, we propose to use the controller's Q-value for scheduler training.

For performance comparison, we consider two baselines in Table <ref>: “No-Scheduler", where the sensor transmits in each time slot, and “Scheduler-Reward", which use the conventional reward function for scheduler training.
We consider four WNCS settings with Markov fading channels in Table <ref>.
The comparisons of the learning curves and the corresponding testing results are presented in Fig. <ref> and Table <ref>, respectively.

Benchmark Policies (high-mobility scenario).

Method     Description

Scheduler-Q-Value (proposed)     lScheduler trained by Q-value (<ref>)

Scheduler-Reward     lScheduler train by overall control reward (<ref>)

No-Scheduler     lNo transmission scheduler

Comparison of learning curves in the high-mobility scenario.

Performance evaluation over fading channels.
Considering slow and fast fading channel conditions $M_1,M_1$ and $M_2,M_2$, i.e., the left column and the right column in Fig. <ref>, the overall performance in the slow fading case is $9\%$ to $28\%$ higher than the fast fading one, with the low and high communication costs (rows 1 and 2 of in Fig. <ref>). This is because it is harder for both the controller and the transmission scheduler to learn adaptive policies with frequently changing channel states.

Comparison with No-Scheduler.
In Table <ref>, comparing with the baseline No-Scheduler, the proposed Scheduler-Q-value algorithm increases the overall performance by about $11\%$ under communication cost $\bar{e}=5$, which further rises to $52\%$ with $\bar{e}=10$. It shows the effectiveness of the transmission scheduler in saving communication costs while guaranteeing control performance.

Q-value vs. immediate reward.
In Table <ref>,
comparing the proposed Scheduler-Q-value method with the baseline Scheduler-Reward when training the scheduler, it can be observed that the proposed algorithm achieves a $72\%$-$99\%$ performance improvement over the baseline with the communication cost $\bar{e}=5$, and a $23\%$-$31\%$ improvement under communication cost $\bar{e}=10$. Therefore, using the long-term reward for scheduler training is much more effective than the short-term reward-based one.
Furthermore, it can be observed that the Scheduler-Q-value method leads to a smaller standard deviation than the baseline, indicating better training stability.

§ CONCLUSIONS

We have proposed a novel DL-based WNCS system with the awareness of both the AoI and the channel states.
We have then developed a novel DRL algorithm for joint controller and scheduler optimization utilizing both model-free and model-based data. In particular, data accuracy has been taken into account for enhancing learning efficiency.
We also develop novel schemes to stabilize the joint training of the controller and the scheduler. Our experiment results have demonstrated significant performance gain compared to the benchmarks.
For future works, we will consider multi-loop WNCSs over uplink and downlink wireless channels and investigate the co-design problems therein.

[1]
P. Park, S. C. Ergen, C. Fischione, C. Lu, and K. H. Johansson, “Wireless
network design for control systems: A survey,” IEEE Commun. Surveys
Tuts., vol. 20, no. 2, pp. 978–1013, Dec. 2017.

[2]
M. Kountouris and N. Pappas, “Semantics-empowered communication for networked
intelligent systems,” IEEE Commun. Mag., vol. 59, no. 6, pp. 96–102,
Jun. 2021.

[3]
E. Uysal, O. Kaya, A. Ephremides, J. Gross, M. Codreanu, P. Popovski, M. Assad,
G. Liva, A. Munari, B. Soret et al., “Semantic communications in
networked systems: A data significance perspective,” IEEE Netw.,

[4]
P. Popovski, Č. Stefanović, J. J. Nielsen, E. De Carvalho,
M. Angjelichinoski, K. F. Trillingsgaard, and A.-S. Bana, “Wireless access
in ultra-reliable low-latency communication (URLLC),” IEEE Trans.
Commun., vol. 67, no. 8, pp. 5783–5801, May 2019.

[5]
L. Schenato, B. Sinopoli, M. Franceschetti, K. Poolla, and S. S. Sastry,
“Foundations of control and estimation over lossy networks,” Proc.
IEEE, vol. 95, no. 1, pp. 163–187, Jan. 2007.

[6]
K. Gatsis, M. Pajic, A. Ribeiro, and G. J. Pappas, “Opportunistic
control over shared wireless channels,” IEEE Trans. Autom. Control,
vol. 60, no. 12, pp. 3140–3155, Dec. Mar. 2015.

[7]
M. Eisen, M. M. Rashid, K. Gatsis, D. Cavalcanti, N. Himayat, and
A. Ribeiro, “Control aware radio resource allocation in low latency
wireless control systems,” IEEE Internet Things J., vol. 6, no. 5,
pp. 7878–7890, Apr. 2019.

[8]
E. G. W. Peters, D. E. Quevedo, and M. Fu, “Controller and scheduler
codesign for feedback control over IEEE 802.15.4 networks,” IEEE
Trans. Control Syst. Technol., vol. 24, no. 6, pp. 2016–2030, 2016.

[9]
K. Huang, W. Liu, Y. Li, A. Savkin, and B. Vucetic, “Wireless feedback control
with variable packet length for industrial IoT,” IEEE Wireless
Commun. Lett., vol. 9, no. 9, pp. 1586–1590, May 2020.

[10]
W. Liu, P. Popovski, Y. Li, and B. Vucetic, “Wireless networked control
systems with coding-free data transmission for industrial IoT,” IEEE
Internet Things J., vol. 7, no. 3, pp. 1788–1801, Dec. 2019.

[11]
W. Liu, G. Nair, Y. Li, D. Nesic, B. Vucetic, and H. V. Poor, “On the latency,
rate, and reliability tradeoff in wireless networked control systems for
IIoT,” IEEE Internet Things J., vol. 8, no. 2, pp. 723–733, Jul.

[12]
L. Zhao, W. Zhang, J. Hu, A. Abate, and C. J. Tomlin, “On the optimal
solutions of the infinite-horizon linear sensor scheduling problem,”
IEEE Trans. Autom. Control, vol. 59, no. 10, pp. 2825–2830, Mar.

[13]
D. Han, J. Wu, H. Zhang, and L. Shi, “Optimal sensor scheduling for multiple
linear dynamical systems,” Automatica, vol. 75, pp. 260–270, Jan.

[14]
A. S. Leong, S. Dey, and D. E. Quevedo, “Sensor scheduling in variance based
event triggered estimation with packet drops,” IEEE Trans. Autom.
Control, vol. 62, no. 4, pp. 1880–1895, Apr. 2017.

[15]
S. Wu, X. Ren, S. Dey, and L. Shi, “Optimal scheduling of multiple sensors
over shared channels with packet transmission constraint,”
Automatica, vol. 96, pp. 22 – 31, 2018.

[16]
A. S. Leong, A. Ramaswamy, D. E. Quevedo, H. Karl, and L. Shi, “Deep
reinforcement learning for wireless sensor scheduling in cyber–physical
systems,” Automatica, vol. 113, p. 108759, 2020.

[17]
W. Liu, K. Huang, D. E. Quevedo, B. Vucetic, and Y. Li, “Deep reinforcement
learning for wireless scheduling in distributed networked control,”
submitted to Automatica, 2021. [Online]. Available:
[18]
D. Baumann, J.-J. Zhu, G. Martius, and S. Trimpe, “Deep reinforcement learning
for event-triggered control,” in Proc. IEEE CDC, 2018, pp. 943–950.

[19]
V. Lima, M. Eisen, K. Gatsis, and A. Ribeiro, “Model-free design of control
systems over wireless fading channels,” Signal Processing, vol. 197,
p. 108540, 2022.

[20]
R. D. Yates, Y. Sun, D. R. Brown, S. K. Kaul, E. Modiano, and S. Ulukus, “Age
of information: An introduction and survey,” IEEE J. Sel. Areas
Commun., vol. 39, no. 5, pp. 1183–1210, Mar. 2021.

[21]
P. Sadeghi, R. A. Kennedy, P. B. Rapajic, and R. Shams, “Finite-state Markov
modeling of fading channels-a survey of principles and applications,”
IEEE Signal Process. Mag., vol. 25, no. 5, pp. 57–80, Aug. 2008.

[22]
A. Goldsmith, Wireless communications.1em plus 0.5em minus
0.4emCambridge university press, 2005.

[23]
Y. Jiang, J. Fan, T. Chai, F. L. Lewis, and J. Li, “Tracking control for
linear discrete-time networked control systems with unknown dynamics and
dropout,” IEEE Trans. Neural Netw. Learn. Syst., vol. 29, no. 10, pp.
4607–4620, Oct. 2017.

[24]
L. Meng, R. Gorbet, and D. Kulić, “Memory-based deep reinforcement
learning for POMDPs,” in Proc. IEEE IROS, Dec. 2021, pp.

[25]
Y. Tian and L. Pan, “Predicting short-term traffic flow by long short-term
memory recurrent neural network,” in Proc. IEEE SmartCity, Dec. 2015,
pp. 153–158.

[26]
J. Sung, J. K. Salisbury, and A. Saxena, “Learning to represent haptic
feedback for partially-observable tasks,” in Proc. IEEE ICRA, Jul.
2017, pp. 2802–2809.

[27]
L. Salmela, N. Tsipinakis, A. Foi, C. Billet, J. M. Dudley, and G. Genty,
“Predicting ultrafast nonlinear dynamics in fibre optics with a recurrent
neural network,” Nature Machine Intelligence, vol. 3, no. 4, pp.
344–354, Feb. 2021.

[28]
S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error
in actor-critic methods,” in Proc. ICML, Jul. 2018, pp. 1587–1596.

[29]
T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa,
D. Silver, and D. Wierstra, “Continuous control with deep reinforcement
learning,” arXiv preprint, Sep. 2015. [Online]. Available:
[30]
T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience
replay,” ICLR, 2016.

[31]
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare,
A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al.,
“Human-level control through deep reinforcement learning,” Nature,
vol. 518, no. 7540, pp. 529–533, Feb. 2015.