method can be applied on top of *any* existing RL algorithm. In all our experiments, we use $\gamma$-*StateEnt* (or *Discounted StateEnt*) for denoting entropy regularization with the discounted future state distribution, and *StateEnt* for denoting the unnormalized counterpart of the discounted weighting of states.\
**Entropy regularization in Exact Policy Gradients with ${\mathbb{H}}(d_{\pi})$:** We first verify that entropy regularization with *exact* discounted future state distribution ${\mathbb{H}}(d_{\pi_{\theta}})$ can lead to benefits in policy optimization when used as a regularizer. We demonstrate this on three toy domains, varying the amount $\lambda$ of state distribution regularization, in the case where we can compute *exact policy gradient* given by $J(\pi) = (I - \gamma P_{\pi})^{-1} R$. In all these examples, the optimal solution can be found with value iteration algorithm.

[0.3]{} ![State distribution entropy regularized exact policy gradient can lead to a better converged solution on a simple two state MDP (taken from [@vf_polytope] (Figure (a)) and a discrete GridWorld domain (Figure (b)). The regularized objective has a faster convergence rate compared to the unregularized objective (with $\lambda=0.0$). Interestingly, in toy MDPs where there exists state aliasing, as in Figure (c) for MDP taken from the counterexample domain [@imani2018off], we find that state distribution entropy 