feature for the $Q$-value approximation, $t$ is the current time point in the online RL learning procedure (i.e. the current trajectory length), $\mtbfy\left(s_{i}';\theta_{n}\right)=\sum_{a^{'}\in\mtcalA}\mathbf{x}\left(s_{i}',a'\right)\pi_{\theta_{n}}\left(a'\mid s_{i}'\right)$, and $\pi_{\theta_{n}}\left(a\mid s\right)$ is the policy for the $n$-th user. Let $\mtbfR=\left[\mtbfr_{1},\cdots,\mtbfr_{N}\right]\in\mtbbR^{t\times N}$ store the reward of all $N$ users at all the $t$ time points. To prevent the overfitting when $t$ is small at the beginning of online RL learning, the $\ell_{2}$ norm based constraint is considered in the objective as follows $$\mtbfw_{n}=\mhbfh_{n}=\arg\min_{\mtbfh_{n}}\left\Vert \mtbfX_{n}^{\intercal}\mtbfh_{n}-\left(\mtbfr_{n}+\gamma\mtbfY_{n}^{\intercal}\mtbfw_{n}\right)\right\Vert _{2}^{2}+\zeta_{c}\left\Vert \mtbfh_{n}\right\Vert _{2}^{2}$$ $\mtfor\ n\in\left\{ 1,\cdots,N\right\} $. The LSTD$Q$ provides a closed-form solution $$\mhbfw_{\theta_{n}}=\left[\mtbfX_{n}\left(\mtbfX_{n}-\gamma\mtbfY_{n}\right)^{\intercal}+\zeta\mtbfI\right]^{-1}\mtbfX_{n}\mtbfr_{n},\label{eq:LSTDQ_4_wn}$$ for $\left\{ n\right\} _{n=1}^{N}$, where $\mhbfw_{\theta_{n}}$ is a function of the policy parameter $\theta_{n}$.

The actor updating for policy improvement \[sub:ActorUpdating\_fmincon\]
------------------------------------------------------------------------

In mHealth, the reference distribution of states $d_{\text{ref}}\left(s\right)$ is unknown and hard to estimate due to the lack of samples. We set $d_{\text{ref}}\left(s\right)$ as the empirical distribution of states. Accordingly, the observations in the trajectory, i.e. $\mtcalD_{n}$, are used to form the objective for the actor updating $\widehat{\theta}_{n}={\displaystyle \arg\max_{\theta_{n}}\widehat{J}\left(\theta_{n}\right)}$, where $$\widehat{J}\left(\theta_{n}\right)=\frac{1}{\left|\mtcalD_{n}\right|}\sum_{s_{i}\in\mtcalD_{n}}\sum_{a\in\mtcalA}Q\left(s_{i},a;\mhbfw_{\theta_{n}}\right)\pi_{\theta_{n}}\!\left(a|s_{i}\right)-\frac{\zeta_{a}}{2}\left\Vert \theta_{n}\right\Vert _{2}^{2}\label{eq:Actor_objective_trajectory}$$ $\mtfor\ n\in\left\{ 1,\cdots,N\right\} $. Here $\left\Vert \theta_{n}\right\Vert _{2}^{2}$ is the constraint to make  a well-posed problem and $\zeta_{a}$ is the tuning parameter that controls the strength of the smooth penalization [@huitian_2014_NIPS_ActCriticBandit4JITAI]. We 