policy $\pi^{*}$ that maximizes the $Q$-value for all the state-action pairs via interactions with the dynamic system [@Geist_2013_TNNLS_RL_valueFunctionApproximation]. The objective is ${\displaystyle \theta^{*}=\arg\max_{\theta}\widehat{J}\left(\theta\right)},$ where $$\widehat{J}\left(\theta\right)=\sum_{s\in\mtcalS}d_{\text{ref}}\left(s\right)\sum_{a\in\mtcalA}\pi_{\theta}\left(a\mid s\right)Q^{\pi_{\theta}}\left(s,a\right)\label{eq:actor-objective_thoery}$$ and $d_{\text{ref}}\left(s\right)$ is the reference distribution of states (e.g. the distribution of initial states); $Q^{\pi_{\theta}}$ is the value for the policy $\pi_{\theta}$. According to , we have to learn the $Q^{\pi_{\theta}}$ for all the state-action pairs to determine the objective  and, after then, to improve the policy. Thus in this paper, we employ the actor-critic algorithm. It is an alternating updating algorithm between two steps untill convergence. At each iteration, the critic updating estimates the $Q$-value function (i.e. policy evaluation, cf. Section \[sub:BellmanEquation\_MC\_TD\] and \[sub:CriticUpdating\_LSTDQ\]) for the lastest policy; the actor updating (i.e. policy improvement, cf. Section \[sub:ActorUpdating\_fmincon\]) learns a better policy based on the newly estimated $Q$-value. Moreover, the actor-critic algorithm has great properties of quick convergence with low variance and learning continuous policies [@Grondman_2012_IEEEts_surveyOfActorCriticRL].

Bellman Equation and Q-value Estimation {#sub:BellmanEquation_MC_TD}
---------------------------------------

It is well known that due to the Markovian property, the $Q$-value satisfies the linear Bellman equation [@Abe_2010_SIGKDD_optimizingDebt] for any policy $\pi$: $$\begin{aligned}
Q^{\pi}\left(s,a\right)=\  & \mtcalR\left(s,a\right)+\gamma\sum_{s^{'}\in\mtcalS}\mtcalP\left(s,a,s'\right)\sum_{a^{'}\in\mtcalA}\pi\left(a'\mid s'\right)Q^{\pi}\left(s',a'\right).\end{aligned}$$ It has the matrix form as $$\mtbfq^{\pi}=\mtbfr+\gamma\mtbfP\Pi_{\pi}\mtbfq^{\pi},\label{eq:Q_linearBellmanEquation_detail}$$ where $\mtbfq^{\pi}$ and $\mtbfr$ are vectors both 