when the sample size is small. However, the optimization of our method is much more complex than that of the separate-RL. The updating rules of all the users are independent with each other in the separate-RL; while in our method, the optimization of all the users is all coupled together. In the following section, two actor-critic RL algorithms are proposed to deal the objectives  and .

**Input**: $T,T_{0},\mu_{\left\{ 1,2,3\right\} },\zeta_{\left\{ 1,2,3\right\} },nAlg$ (i.e. the algorithm index).

Initialize the states $\left(s_{t,n}\right)_{p\times N}$, where $t=0$, and the policy parameters $\Theta=\left[\theta_{1},\cdots,\theta_{N}\right]\in\mtbbR^{m\times N}$ for $N$ users.

At time point $t,$ observe context $s_{t,n}$ for the $n$-th user.

Draw an action $a_{t,n}$ according to the policy $\pi_{\widehat{\theta}_{n}}(a|s_{t,n}).$

Observe an immediate reward $r_{t,n}$.

Construct the network cohesion via ,$\ $.

Data preparation and feature construction  for the critic update and the actor update.

Critic update to learn $\mhbfW_{t}$ (value parameter) via .

Critic update to learn $\mhbfW_{t}$ (value parameter) via .

Actor update to learn $\widehat{\Theta}_{t}$ (policy paramter) via .

**Output**: the policy for $N$ users, i.e. $\pi_{\widehat{\theta}_{n}}\left(a\mid s\right)$, for $\left\{ n\right\} _{n=1}^{N}$.

Algorithm\#1 for the Critic update
==================================

Updating Rules for the Projection Step  \[sub:UpdatingRule4ProjectionStep\]
---------------------------------------------------------------------------

We first discuss how to minimize the objective for 