future state distribution induced by those policies as an auxiliary regularized objective. This distribution takes into account when various states are visited in addition to which states are visited. The main contribution of our work is to provide a practically feasible way to estimate the discounted future state distribution with a density estimator. Furthermore, we show that regularizing policy gradients with the entropy of this distribution can improve exploration. To the best of our knowledge, there are no previous works that provide a practical realization for estimating and regularizing with the entropy of the discounted state distribution. It is worthwhile to note that the estimation of the discounted/stationary state distribution is not readily achievable in practice. This is because the stationary distribution requires an estimate based on rollouts, as in value function estimates, under a given policy $\pi$. In contrast, the discounted state distribution requires estimation of discounted occupancy measures for the various states. Since the discounted occupancy measure is purely a theoretical construct, it is not possible to sample from this distribution using rollouts. In order to use this as an entropy regularizer, we also need the discounted or stationary distributions to be explicitly dependent on the policy parameters, 