for the $n$-th policy $\pi_{\theta_{n}}$ and $$\begin{aligned}
\widehat{J}\left(\Theta\right)\!= & \sum_{n=1}^{N}\left(\frac{1}{\left|\mtcalD_{n}\right|}\sum_{\mtcalU_{i}\in\mtcalD_{n}}\sum_{a\in\mtcalA}Q\left(s_{i},a;\mathbf{\mhbfw}_{\theta_{n}}\right)\pi_{\theta_{n}}\!\left(a|s_{i}\right)\right)\nonumber \\
 & \ -\frac{\mu_{3}}{2}\sum_{i,j=1}^{N}c_{ij}\left\Vert \theta_{i}-\theta_{j}\right\Vert _{2}^{2}-\frac{\zeta_{3}}{2}\sum_{n=1}^{N}\left\Vert \theta_{n}\right\Vert _{2}^{2}.\label{eq:actorUpdating_GraphRL}\end{aligned}$$ Although we are able to obtain a closed-form solution for the critic updating , to reduce the computational costs, we substitute the solution in value for $\left\{ \mhbfw_{n}\right\} _{n=1}^{N}$ rather than the closed-form expression of $\left\{ \mhbfw_{\theta_{n}}\right\} _{n=1}^{N}$ into the actor updating. The actor updating algorithm performs the maximization of  over $\Theta$, which is computed via the Sequential Quadratic Programming (SQP) algorithm. We use the implementation of SQP with finite-difference approximation to the gradient in the <span style="font-variant:small-caps;">fmincon</span> function of Matlab.

In the objectives , and , $\mu_{1}$, $\mu_{2}$ and $\mu_{3}$ are the tuning parmaters to control the strength of the network cohesion constraints. It is an advantage of our methods over the network based bandit [@Bianchi_2013_NIPS_GangOfBandit]. When $\mu_{1},\mu_{2},\mu_{3}\rightarrow\infty$, the connected users are enforced to have identical values and policies. When $\mu_{1},\mu_{2},\mu_{3}=0$, there is no network cohesion constraint. In such case, our method is equivalent to the separate online RL method. Compared with the Separate-RL, the model complexity of our methods is reduced since their parameter domain is constrained via the network cohesion regularization. Such case ensures our methods to work well 