algorithm, where learning a policy depends on both a value function and a state distribution estimation.

Appendix : Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods
======================================================================================================

Contributions : 
----------------

The key contributions of our work are as follows

-   We provide a practically feasible algorithm based on neural state density estimation, for entropy regularization with the *discounted* future state distributions in policy gradient based methods. This can be adapted for both episodic and infinite horizon environments, by switching between stationary and discounted state distributions with a $(1-\gamma) \gamma^{t}$ weighted importance sampling correction

-   For regularization with the state distribution, we require the estimated state distribution to be directly inlfuenced by the policy parameters $\theta$. We achieve this by learning the density estimator directly as a function of policy parameters $\theta$, denoted by $d_{\pi_{\theta}}$, such that the policy gradient update can be regularized with $\nabla_{\theta} {\mathbb{H}}(d_{\pi_{\theta}})$

-   Our approach requires estimating the state distribution, in addition to *any* existing actor-critic or policy optimization method, for state distribution entropy regularization. This leads to a three time scale algorithm based on our approach. We provide convergence for a *three-time-scale* algorithm and show that under sufficiently mild conditions, this approach 