state and action pairs in the MDP; $D$ describes the distributions over the state and action pairs.

Since the state transition $\mtbfP$ is unknown and $\mebfX$ is too large to form in mHealth, we can not directly solve . Instead, we have to make use of the trajectories collected from $N$ users, i.e. $\mtcalD=\left\{ \mtcalD_{n}\right\} _{n=1}^{N}$, where $\mtcalD_{n}=\left\{ \mtcalU_{i}=\left(s_{i},a_{i,}r_{i},s_{i}'\right)\mid i=0,\cdots,t\right\} $ summarizes all the $t+1$ tuples for the $n$-th user and $\mtcalU_{i}$ is the $i$-th tuple in $\mtcalD_{n}$.

Current online contextual bandit (i.e. a special RL with $\gamma=0$) methods for mHealth assume that all users are completely heterogeneous. They share no information and run a separate algorithm for each user [@huitian_2014_NIPS_ActCriticBandit4JITAI]. Following this idea, we extend [@huitian_2014_NIPS_ActCriticBandit4JITAI] to the separate RL setting. The objective for the $n$-th user is defined as $$\begin{aligned}
\mtbfw_{n} & =\mhbfh_{n}=\arg\min_{\mtbfh_{n}}\sum_{\mtcalU_{i}\in\mtcalD_{n}}\left\Vert \mtbfx{}_{i}^{\intercal}\mtbfh_{n}-\left(r_{i}+\gamma\mtbfy{}_{i}^{\intercal}\mtbfw_{n}\right)\right\Vert _{2}^{2}\end{aligned}$$ $\mtfor\ n\in\left\{ 1,\cdots,N\right\} $, where $\mtbfx_{i}=\mtbfx\left(s_{i},a_{i}\right)$ is the value feature at time $i$ and $\mtbfy{}_{i}=\sum_{a'\in\mtcalA}\mathbf{x}\left(s_{i}',a'\right)\pi_{\theta}\left(a'\mid s_{i}'\right)$ is the value feature at the next time point. For the sake of easy derivation, we define the following matrices to store the actual observations $$\begin{aligned}
\mtbfX_{n} & =\left[\mathbf{x}\left(s_{1},a_{1}\right),\mathbf{x}\left(s_{2},a_{2}\right)\cdots,\mathbf{x}\left(s_{t},a_{t}\right)\right]\in\mathbb{R}^{u\times t}\nonumber \\
\mtbfY_{n} & =\left[\mtbfy\left(s_{1}';\theta_{n}\right),\cdots,\mtbfy\left(s_{t}';\theta_{n}\right)\right]\in\mathbb{R}^{u\times t}\label{eq:featureConstruction_4_value}\\
\mtbfr_{n} & =\left[r_{1},r_{2},\cdots,r_{t}\right]^{\intercal}\in\mathbb{R}^{t},\nonumber \end{aligned}$$ where $u$ is the length of the sample 