gradient based approaches. We formally state and prove the corresponding three time-scale algorithm.

State Distribution in Policy Gradient Methods
=============================================

Policy gradient theorem  [@sutton] for the starting state formulation are given for an initial state distribution $\mathbf \alpha$, where the exact solution for the discounted objective is given by $J{\theta} = \mathbf \alpha^{T} v_{\theta} = \mathbf \alpha^{T} (\mathbf I - \gamma P_{\theta})^{-1} r_{\theta}$. In [@sutton], this is often known as the *discounted weighting of states* defined by $d_{\alpha, \gamma, \pi}^{T} = \mathbf \alpha^{T} (\mathbf I - \gamma P_{\theta})^{-1}$, where in the average reward case this reaches a stationary distribution implying that the process is independent of the initial states. However, the discounted weighting of states is not a distribution, or a stationary distribution in itself, since the rows of the matrix $(\mathbf I - \gamma P_{\theta})^{-1}$ do not sum to 1. The normalized version of this is therefore often considered, commonly known as the *discounted future state distribution*  [@Kakade2003OnTS] or the discounted state distribution  [@Thomas14]. Detailed analysis of the significance of the state distribution in policy gradient methods is further given in \[Bacon, 2018\]. $$\label{eq:gamma_discounted_future_state}
    \Bar{d}_{\alpha, \gamma, \pi} = (1 - \gamma) d_{\alpha, \gamma, \pi} = ( 1 - \gamma) 