where exploration plays a key role. In all the experiments, we use a state entropy regularization coefficient of $\lambda=0.1$ for our approach, and $\lambda=0.0$ for the baseline DDPG. Experiment results are averaged over $10$ random seeds [@DBLP:conf/aaai/0002IBPPM18][]{data-label="fig:ddpg_control"}](DDPG_Final_Walker2d-v1.pdf "fig:"){width="\linewidth"}

Summary and Discussion
======================

In this work, we provided a practically feasible algorithm for entropy regularization with the state distributions in policy optimization. We present a practically feasible algorithm, based on estimating the discounted future state distribution, for both episodic and infinite horizon environments. The key to our approach relies on using a density estimator for the state distribution $d_{\pi_{\theta}}$, which is a direct function of the policy parameters $\theta$ itself, such that we can regularize policy optimization to induce policies that can maximize state space coverage. We demonstrate the usefulness of this approach on a wide range of tasks, starting from simple toy tasks to sparse reward gridworld domains, and eventually extending our results to a range of continuous control suites. We re-emphasize that our approach gives a practically convenient handle to deal with the *discounted* state distribution, that are difficult to work with in practice. In addition, we provided a proof of convergence of our method as a three time-scale 