corresponding immediate reward for such transition where $\mtcalR\!:\mtcalS\times\mtcalA\times\mtcalS\mapsto\mtbbR$. For simplicity, the expected immediate reward $\mtcalR\left(s,a\right)=\mtexpect_{s'\sim\mtcalP}\left[\mtcalR\left(s,a,s'\right)\right]$ is assumed to be bounded over the state and action spaces. $\gamma\in[0,1)$ is the discount factor that reduces the influence of future rewards. To allow for the matrix operators, the state space $\mtcalS$ and action space $\mtcalA$ are assumed to be finite, though very large in mHealth.

The policy of an MDP is to choose actions for any state $s\in\mtcalS$ in the system [@Grondman_2012_IEEEts_surveyOfActorCriticRL; @Sutton_2012_Book_ReinforcementLearning]. There are two types of policies: (1) the deterministic policy $\pi:\mtcalS\mapsto\mtcalA$ selects an action directly for the state, and (2) the stochastic policy $\pi:s\in\mtcalS\mapsto\pi\left(\cdot\mid s\right)\in\mtcalP\left(\mtcalA\right)$ chooses the action for any state $s$ by providing $s$ with a probability distribution over all the possible actions [@Geist_2013_TNNLS_RL_valueFunctionApproximation]. In mHealth, the stochastic policy is preferred due to two reasons: (a) current evidence shows that some randomness in the action is likely to draw users’ interest, thus helpful to reduce the intervention burden/habituation [@Epstein_2009_AJCN_varietyInfluencesHab; @huitian_2014_NIPS_ActCriticBandit4JITAI; @SusanMurphy_2016_CORR_BatchOffPolicyAvgRwd]; (b) though some deterministic policy is theoretically optimal for the MDP, however, we do not know where it is for the large state space on the one hand and the MDP is a simplification for the complex behavioral 