Carlo (MC) method draws very deep trajectories to obtain the observation of actual $Q$ value. Although MC can provide an unbiased estimation of $Q_{\mtbfw}$, it is not suitable for mHealth since MC can’t learn from the incomplete trajectory [@Sutton_2012_Book_ReinforcementLearning]. Such case requires massive sampling from users, which, however, is very labor-intensive and expensive in time. As a central idea of RL [@Sutton_2012_Book_ReinforcementLearning], the temporal-difference (TD) learning is able to make use of the Bellman equation  and to learn the value from the incomplete trajectories. The learned result of TD has the property of low variance.

The critic updating: Least-Squares TD for $Q$-value (LSTD$Q$) Estimation {#sub:CriticUpdating_LSTDQ}
------------------------------------------------------------------------

In mHealth, though the data for all users is abundant, the data for each user is limited in size. We employ the least-square TD for the Q-value (LSTD$Q$) estimation, due to its advantage of efficient use of samples over the pure temporal-difference algorithms [@Michail_2003_JMLR_LSPI_LSTDQ; @Sakuma_2008_ICML_PrivacyPreservingRL]. The goal of LSTD$Q$ is to learn a $Q_{\mtbfw}$ to approximately satisfy the Bellman equation , by minimizing the TD error [@AndrewNg_2009_ICML_RLsparity] as $$\mtbfw=\mhbfh=\min_{\mtbfh\in\mtbbR^{K}}\left\Vert \mebfX^{\intercal}\mtbfh-\left(\mebfr+\gamma\mtbfP\Pi_{\pi}\mebfX^{\intercal}\mtbfw\right)\right\Vert _{D}^{2},\label{eq:LSTDQ_distribution}$$ where $\mtbfw=\mhbfh$ is a fixed point problem and $\mhbfh$ is a function of $\mtbfw$; $\mebfX$ is a designed matrix consisting of all the 