collect and analyze raw data (weather, location, social activity, stress, etc.). Based on that, the aim is to provide effective intervention that helps users to change to or adapt to healthy behaviors, such as reducing the alcohol abuse [@Gustafson_2014_JAMA_drinking; @Witkiewitz_2014_JAB_drinkingSmoking] and promoting physical activities [@Abby_2013_PlosONE_mobileIntervention]. The traditional adaptive treatment has restrictions on the time, location and frequency—patients have to visit the doctor’s office for treatments. Compared with them, mHealth is more affordable, portable and much more flexible in the sense that smart devices allow for the real-time collection and analysis of data as well as in-time delivery of interventions. Thus, mHealth technologies are widely used in lots of health-related tasks, such as physical activity [@Abby_2013_PlosONE_mobileIntervention], eating disorders [@Bauer_2012_JCCP_eatingDisorder], alcohol use [@Gustafson_2014_JAMA_drinking; @Witkiewitz_2014_JAB_drinkingSmoking], mental illness[@Depp_2010_JNMD_mentalIllness; @Ben_2013_APMHMHSRZZZZ_mentalIllness], obesity/weight management [@Patrick_2009_JMIR_weightManagement].

Formally, the mHealth intervention is modeled as a sequential decision making (SDM) problem. It aims to learn the optimal policy to determine when, where and how to deliver the intervention [@huitian_2014_NIPS_ActCriticBandit4JITAI; @PengLiao_2015_Proposal_offPolicyRL; @SusanMurphy_2016_CORR_BatchOffPolicyAvgRwd] to best serve users. This is a new research topic that lacks of methodological guidance. In 2014, Lei [@huitian_2014_NIPS_ActCriticBandit4JITAI] made a first attempt to formulate the mHealth intervention as an online actor-critic contextual bandit problem. Lei’s method served a 