with $\left|\mtcalS\right|\left|\mtcalA\right|$ elements; $\mtbfP\in\mtbbR^{\left|\mtcalS\right|\left|\mtcalA\right|\times\left|\mtcalS\right|}$ is the stochastic state transition matrix, in which $P\left(\left(s,a\right),s'\right)=\mtcalP\left(s,a,s'\right)$; $\Pi_{\pi}\in\mtbbR^{\left|\mtcalS\right|\times\left|\mtcalS\right|\left|\mtcalA\right|}$ is the stochastic policy matrix, where $\Pi_{\pi}\left(s,\left(s,a\right)\right)=\pi\left(a\mid s\right)$ [@Michail_2003_JMLR_LSPI_LSTDQ]. Once both the reward and the state transition models are given [@AndrewNg_2009_ICML_RLsparity], it is easy to obtain the analytical solution as $\mtbfq^{\pi}=\left(\mtbfI-\gamma\mtbfP\Pi_{\pi}\right)^{-1}\mtbfr$.

However, there are two factors making it impossible to have the analytical solution for the $Q$-value estimation: (a) in mHealth, both reward $\mtbfr$ and state transition $\mtcalP$ (i.e. $\mtbfP$) models are unknown. (b) the state space in mHealth is usually very large or even infinite, which makes it impossible to directly learn the $Q$-value due to lack of observations for sharper learning and too high storage requirements, i.e. $O\left(\left|\mtcalS\right|\left|\mtcalA\right|\right)$ to only store the $Q$-value table. We resolve these problems via the parameterized function approximation, which assumes that $Q^{\pi}$ is in a low dimensional space: $Q^{\pi}\approx Q_{\mtbfw}=\mtbfw^{\intercal}\mtbfx\left(s,a\right)$, where $\mtbfw\in\mtbbR^{u}$ is the unknown variable and $\mtbfx\left(s,a\right)$ is a feature processing step that combines information in the state and action. We then learn the value $Q_{\mtbfw}$ from observations via a supervised learning paramdigm, which, however, is much more challenging than the general supervised learning since the $Q$-value is not directly observed [@Geist_2013_TNNLS_RL_valueFunctionApproximation]. As a direct solution, the Monte 