\in \mathcal{S}}d_{1, \pi_\theta}(s) \log(d_{1, \pi_\theta}(s)) \stackrel{(b)}{\approx} - \frac{1}{T} \sum_{t=0}^{T} \log d_{1, \pi_\theta}(S_t) \stackrel{(c)}{=} - \frac{1}{T} \sum_{t=0}^{T} \log p(S_t),$$ where $T$ is a finite number of time-steps after which an infinite horizon episode can be truncated due to discounting. In deriving , $(a)$ follows from the definition of entropy, $(b)$ follows by assuming ergodicity, which allows us to replace an expectation over the state space with an expectation over time under all policies. The approximation here is due to the finite truncation of the infinite horizon to $T$. Step $(c)$ follows from the density estimation procedure.

**Estimating the entropy of discounted future state distribution:** In practice, we use a neural density estimator for estimating the discounted state distribution, based on the states induced by the policy $\pi_{\theta}$. The training samples for the density estimator is obtained by rolling out trajectories under the policy $\pi_{\theta}$. We train a variational inference based density estimator (similar to a variational auto-encoder) to maximize variational lower bound $\log p(s)$, where for the discounted case, we denote this as $\log \tilde p(s)$, as given in   and . We therefore obtain an approximation to the entropy of discounted future state distribution which can be used in the modified 