of the *discounted* state distribution ${\mathbb{H}}(d_{\alpha, \gamma, \pi})$ : 

The discounted state distribution $d_{\alpha, \gamma, \pi_\theta}$ can be computed as: $    \setlength{\abovedisplayskip}{3pt}
    d_{\alpha, \gamma, \pi_\theta}(s) = (1-\gamma) \mathbf \alpha^{T} \sum_{t=0}^{\infty}\gamma^t\prob(S_t = s), \quad \forall s \in \mathcal{S}$

We note that this is a theoretical construct and we cannot sample from this distribution, since it would require sampling each state with a probability $(1-\gamma)$ such that the accepted state is then distributed according to $d_{\alpha, \gamma, \pi_{\theta}}$. However, we can modify the state distribution $p(s)$ to a weighted distribution $\tilde p(s)$ as follows. We estimate $p(s)$ from samples as: $\setlength{\abovedisplayskip}{3pt}
p(s) = \frac{1}{T}\sum_{t=0}^{T}\IND({S_t = s}),
$ where the weight of each sample is $1/T$. To estimate $\tilde p(s)$, we use an importance sampling weighting of $(1-\gamma)\gamma^t$ to yield: $$\label{eq:tildep}
\setlength{\abovedisplayskip}{3pt}
\tilde p(s) \stackrel{(a)}{=} \frac{(1-\gamma)}{T}\sum_{t=0}^{T}\gamma^t\IND({S_t = s}) \stackrel{(b)}{=} (1-\gamma)\sum_{t=0}^T(\gamma^t\prob(S_t = s \mid S_0)) \stackrel{(c)}{\approx} d_{\gamma, \pi_\theta}(s),$$ where $(a)$ follows from the importance sampling approach, $(b)$ follows from the fact that $\frac{\IND({S_t = s})}{T} = \prob(S_t = s \mid S_o)$ and $(c)$ follows from above where the approximation is due to the finite truncation of the infinite horizon trajectory. Note that due to this finite truncation, our estimate of $d_{\gamma, \pi_\theta}$ will be 