\mathbf\alpha^{T} (\mathbf I - \gamma \mathbf P_{\pi})^{-1} = ( 1 - \gamma) \mathbf \alpha^{T} \sum_{t=0}^{\infty} \gamma^{t} P_{\pi} (s_t = s)$$ Given an infinite horizon MDP, and a stationary policy $\pi(a,s)$, equation is the $\gamma$ discounted future state distribution, i.e., the normalized version for the discounted weighting of states. We can draw samples from this distribution, by simulating $\pi$ and accepting each state as the sample with a probability $(1 - \gamma)$. With the discounted future state distribution, the equivalent policy gradient objective can therefore be given by $J({\theta}) = \Bar{ \mathbf  d}_{\alpha, \gamma, \theta}^{T} \mathbf r_{\theta}$. In practice, we want to express the policy gradient theorem with an expectation that we can estimate by sampling, but since the discounted weighting of states $d_{\alpha, \gamma, \pi}$ is not a distribution over states, we often use the normalized counterpart of the discounted weighting of states $\Bar{d}_{\alpha, \gamma, \pi}$ and correct the policy gradient with a factor of $\frac{1}{(1 - \gamma)}$. $$\nabla_{\theta} J(\theta) = \frac{1}{(1 - \gamma)} \expectation{_{\Bar{d}_{\alpha, \gamma, \theta}, a \sim \pi_{\theta} } } [ \nabla_{\theta} \log \pi_{\theta}(a,s) Q_{\pi_{\theta}}(s,a) ]$$ However, since the policy gradient objective is defined with respect to an initial distribution over states, the resulting policy is not *optimal* 