taken from the counterexample domain [@imani2018off], we find that state distribution entropy does not lead to significant improvements. This is an interesting result justifying that state space exploration may not necessarily be needed in all MDPs, especially when states are aliased[]{data-label="fig:twostateMDP_comp"}](Counterexample.pdf "fig:"){width="\linewidth"} \[fig:exact\_pg3\]

**Toy Domains:** Having verified our hypothesis in figure \[fig:twostateMDP\_comp\], we now present our approach based on separately learning a density estimator for the state distribution, on tabular domains with actor-critic algorithms. We compare our results for both the *discounted* state distributions and the unnormalized counterpart, with a baseline actor-critic (with $\lambda=0.0$ for the regularizer). Figure \[fig:toy\_actor\_critic\] summarizes our results.

[0.27]{} ![We show benefits of state distribution entropy regularization on toy domains, especially hard exploration tabular tasks such as FrozenLake. In all the tasks, we find that regularization with entropy of *discounted* state distribution performs significantly better compared to baseline and regularization with the unnormalized state distribution. In all tasks, we use $\lambda=0.1$ for our methods.[]{data-label="fig:toy_actor_critic"}](FrozenLake-v0.pdf "fig:"){width="\linewidth"} \[fig:frozelake\_comp\]

[0.27]{} ![We show benefits of state distribution entropy regularization on toy domains, especially hard exploration tabular tasks such as FrozenLake. In all the tasks, we find that regularization with entropy of *discounted* state distribution performs significantly better compared to baseline 