overfitting when the sample size is very small, the conventional LSTD$Q$ usually employs the $\ell_{2}$ constraint on the variable $\mtbfH$ in the projection step. They do not put the $\ell_{2}$ constraint on the fixed-point variable $\mtbfW$ [@Mohammad_2011_RARL_RegularizedLSTD_L1L2; @AndrewNg_2009_ICML_RLsparity]. Following this idea, we have a simpler objective function for the critic update as $$\begin{aligned}
\mtbfw_{n} & =\mhbfh_{n}=\arg\min_{\mtbfh_{n}}\sum_{\mtcalU_{i}\in\mtcalD_{n}}\left\Vert \mtbfx{}_{i}^{\intercal}\mtbfh_{n}-\left(r_{i}+\gamma\mtbfy{}_{i}^{\intercal}\mtbfw_{n}\right)\right\Vert _{2}^{2},\\
 & \mtfor\ n\in\left\{ 1,\cdots,N\right\} \ \text{and}\ \mtsubTo\ \sum_{i,j=1}^{N}c_{ij}d\left(\mtbfh_{i},\mtbfh_{j}\right)\leq\delta_{1}.\nonumber \end{aligned}$$ According to the derivation in Section \[sub:UpdatingRule4ProjectionStep\] that considers the Frobenius norm based smooth constraint, the updating rule for the projection step is . In the fixed-point step, the objective is simply $\mtbfw_{n}=\mhbfh_{n}$ (i.e. a fixed-point problem), which leads to $$\mtvec\left(\mtbfW\right)=\mtvec\left(\mhbfH\right).$$ Thus, we have the closed-form solution for $\mtvec\left(\mtbfW\right)$ as follows $$\mtvec\left(\mhbfW\right)=\left[\mtbfF_{1}-\gamma\mtbfF_{3}+\mtbfL_{\otimes}\left(\mu_{1},\zeta_{1}\right)\right]^{-1}\mtbfF_{2}\mtvec\left(\mtbfR\right).\label{eq:CriticUpdatingRule_=00003D0000232_GraphRL}$$ It is simpler than the 1st updating rule for the critic update .

Experiment Results {#sec:Evaluation}
==================

We verify the proposed methods on the HeartSteps dataset. It has two choices for an action, i.e. $\left\{ 0,1\right\} ,$ where $a=1$ means sending the positive intervention, while $a=0$ indicates no intervention [@SusanMurphy_2016_CORR_BatchOffPolicyAvgRwd]. Specifically, the stochastic policy is assumed to be in the form $\pi_{\theta}\left(a\mid s\right)\!=\!\frac{\exp\left[-\theta^{\intercal}\phi\left(s,a\right)\right]}{\sum_{a'}\exp\left[-\theta^{\intercal}\phi\left(s,a'\right)\right]}$, where $\theta\in\mtbbR^{m}$ is the unknown parameter and $\phi\left(\cdot,\cdot\right)$ is a feature process that combines the information in actions 