can converge to the optimal solution.

-   We demonstrate the usefulness of entropy regularization with the discounted state distribution on a wide range of deep reinforcement learning tasks, and discuss the usefulness of this approach compared to entropy regularization with stochastic policies, as commonly used in practice.

*Stationary* Distributions and *Discounted* Future State Distributions:
-----------------------------------------------------------------------

For environments where the stationary distribution exists, for all policies, it is more natural to use the entropy of the *stationary* state distribution as the regularizer. This is because we are interested in visiting all states in the state space, and not necessarily concerned with at what point in time the state is visited. However, there may be environments where the stationary distribution does not exist or environments in which the stationary distribution collapses to a unit measure on a single state. Episodic environments are examples of the latter where the support of the stationary state distribution only includes the terminal state. In such environments, it is more natural to use the *discounted* state distribution (normalized discounted occupancy measure).

Preliminaries
-------------

We consider the standard RL framework, where an agent acts in an environment that can be modelled as a Markov decision process (MDP). Formally 