policies are constrained to be similar, e.g. $\left\Vert \mtbfw_{i}-\mtbfw_{j}\right\Vert $ and $\left\Vert \theta_{i}-\theta_{j}\right\Vert $ are small if $i\leftrightarrow j$ [@fyzhu_2014_JSTSP_RRLbS; @fyzhu_2015_PhDthesis]. With the network cohesion $\mtbfC=\left(c_{ij}\right)_{N\times N}$, the objective function for the critic updating is formed as follows [ $$\begin{aligned}
\mtbfW=\mhbfH & =\arg\min_{\mtbfH}\sum_{n=1}^{N}\sum_{\mtcalU_{i}\in\mtcalD_{n}}\left\Vert \mtbfx{}_{i}^{\intercal}\mtbfh_{n}-\left(r_{i}+\gamma\mtbfy{}_{i}^{\intercal}\mtbfw_{n}\right)\right\Vert _{2}^{2}\label{eq:CriticUpdat_Graph_theory}\\
\mtsubTo & \sum_{i,j=1}^{N}c_{ij}d\left(\mtbfh_{i},\mtbfh_{j}\right)\leq\delta_{1}\ \text{and}\ \sum_{i,j=1}^{N}c_{ij}d\left(\mtbfw_{i},\mtbfw_{j}\right)\leq\delta_{2},\nonumber \end{aligned}$$ ]{}where $\mtbfH\!=\left[\mtbfh_{1},\cdots,\mtbfh_{N}\right]\in\mtbbR^{u\times N}$ and $\mtbfW=\left[\mtbfw_{1},\cdots,\mtbfw_{N}\right]\in\mtbbR^{u\times N}$ are designed matrices that consist of all the $N$ users’ variables (each column summarizes the unknown varibile of one user); $d\left(\mtbfh_{i},\mtbfh_{j}\right)$ is a distance measure between two vectors; usually we set $d\left(\cdot,\cdot\right)$ as the Euler distance. With the matrix notations in Section \[sub:CriticUpdating\_LSTDQ\], we turn  into the following two-level nested optimization problems $$\begin{aligned}
\mhbfH\!=\arg\min_{\mtbfH} & \Bigg(\sum_{n=1}^{N}\left\Vert \mtbfX_{n}^{\intercal}\mtbfh_{n}-\left(\mtbfr_{n}+\gamma\mtbfY_{n}^{\intercal}\mtbfw_{n}\right)\right\Vert _{2}^{2}+\label{eq:criticUpdating_projection}\\
 & \quad\mu_{1}\sum_{i,j=1}^{N}c_{ij}\left\Vert \mtbfh_{i}-\mtbfh_{j}\right\Vert _{2}^{2}+\zeta_{1}\sum_{n=1}^{N}\left\Vert \mtbfh_{n}\right\Vert _{2}^{2}\Bigg),\nonumber \end{aligned}$$ $$\begin{aligned}
\mhbfW\!=\arg\min_{\mtbfW} & \Bigg(\sum_{n=1}^{N}\left\Vert \Phi_{n}\mtbfw_{n}-\Phi_{n}\mhbfh_{n}\right\Vert _{2}^{2}+\label{eq:criticUpdating_fixedPoint}\\
 & \quad\mu_{2}\!\sum_{i,j=1}^{N}c_{ij}\left\Vert \mtbfw_{i}-\mtbfw_{j}\right\Vert _{2}^{2}+\zeta_{2}\sum_{n=1}^{N}\left\Vert \mtbfw_{n}\right\Vert _{2}^{2}\Bigg),\nonumber \end{aligned}$$ where $\Phi_{n}$ is a designed matrix to facilitate the optimization of . The 1st level  projects the Bellman image onto a linear space (we refer  as the projection step); the 2nd level  deals with the fixed point problem (i.e. the fixed-point step) [@Mohammad_2011_RARL_RegularizedLSTD_L1L2].

The objective for the actor updating is defined as follows $$\left\{ \widehat{\theta}_{1},\cdots,\widehat{\theta}_{n},\cdots,\widehat{\theta}_{N}\right\} =\arg\max_{\left\{ \theta_{n}\right\} _{n=1}^{N}}\widehat{J}\left(\theta_{1},\cdots,\theta_{N}\right),\label{eq:objective_actorUpdate}$$ where $\Theta=\left[\theta_{1},\cdots,\theta_{N}\right]$, $Q\left(s_{i},a;\widehat{\mathbf{w}}_{\theta_{n}}\right)=\mathbf{x}\left(s_{i},a\right)^{T}\widehat{\mathbf{w}}_{\theta_{n}}$ is the estimated value 