good starting point for the mHealth study. However, this method did not consider the important delayed effect in the SDM—the current action may influence not only the immediate reward but also the next states and, through that, all the subsequent rewards [@Sutton_2012_Book_ReinforcementLearning; @YihongLi_2010_WWW_contextualBandit4newsArticleRecommend]. Dr. Murphy [@SusanMurphy_2016_CORR_BatchOffPolicyAvgRwd] proposed an average reward based RL to consider the delayed effect in the mHealth. However, those two methods rely on some ideal assumptions. They either assume that all the users are completely homogenous or completely heterogeneous. We find the truth lying between those extremes: a user might be similar with some, but not all, users. Their methods are easy to bring in too much bias or too much variance. Besides, [@SusanMurphy_2016_CORR_BatchOffPolicyAvgRwd] is in the batch learning setting, which is different from this paper’s focuses.

Recently, Dr. Cesa-Bianchi [@Bianchi_2013_NIPS_GangOfBandit] proposed a contextual bandit algorithm that considers the network information. It is for the recommendation system, which is very different from the mHealth task. Besides, there are three drawbacks making the method in [@Bianchi_2013_NIPS_GangOfBandit] impractical for the mHealth: (1) Cesa-Bianchi’s method focues on the bandit algorithm. It doesn’t consider the important delayed effect in mHealth. (2) They assume the network information is given beforehand from the 