and states, i.e. $\phi\left(s,a\right)=\left[as^{\intercal},a\right]^{\intercal}\in\mtbbR^{m}$.

The HeartSteps Dataset
----------------------

To verify the performance of our method, we use a dataset from a mobile health study, called HeartSteps [@Walter_2015_Significance_RandomTrialForFitbitGeneration], to approximate the generative model. This is a 42-day mHealth intervention that aims to increase the users’ steps they take each day by providing positive treatments (i.e. interventions), which are adapted to users’ ongoing status, such as suggesting users to take a walk after long sitting [@Walter_2015_Significance_RandomTrialForFitbitGeneration], or to do some exercises after work.

A trajectory of $T$ tuples $\mtcalD=\left\{ \left(s_{i},a_{i,}r_{i}\right)\mid i=1,\cdots,T\right\} $ are generated for each user [@SusanMurphy_2016_CORR_BatchOffPolicyAvgRwd; @huitian_2014_NIPS_ActCriticBandit4JITAI]. The initial state is drawn from the Gaussian distribution $S{}_{0}\sim\mtcalN_{p}\left\lbrace 0,\Sigma\right\rbrace $, where $\Sigma$ is a $p\times p$ covariance matrix with pre-defined elements. The action $a_{t}$ for $0\leq t\leq T_{0}$ is drawn from the random policy, with a probability of $0.5$ to provide interventions, i.e. $\mu\left(1\mid s\right)=0.5$ for all states $s$. Such process is called drawing warm start trajectory (WST) via the micro-randomized trials [@PengLiao_2015_Proposal_offPolicyRL; @Walter_2015_Significance_RandomTrialForFitbitGeneration], and $T_{0}$ is the length of the WST. When $t\geq T_{0}$, we start the actor-critic updating, and the action is drawn from the learned policy, i.e. $a_{t}\sim\pi_{\widehat{\theta}_{t}}\left(\cdot\mid s_{t}\right)$. When $t\geq1$, the state and immediate reward are generated 