and regularization with the unnormalized state distribution. In all tasks, we use $\lambda=0.1$ for our methods.[]{data-label="fig:toy_actor_critic"}](GridWorld-v0.pdf "fig:"){width="\linewidth"} \[fig:gridworld\_comp\]

[0.27]{} ![We show benefits of state distribution entropy regularization on toy domains, especially hard exploration tabular tasks such as FrozenLake. In all the tasks, we find that regularization with entropy of *discounted* state distribution performs significantly better compared to baseline and regularization with the unnormalized state distribution. In all tasks, we use $\lambda=0.1$ for our methods.[]{data-label="fig:toy_actor_critic"}](WindyGridWorld-v0.pdf "fig:"){width="\linewidth"} \[fig:windygrid\_comp\]

**Complex Sparse Reward GridWorld Tasks:** We demonstrate the usefulness of our approach, with entropy of stationary (denoted StateEnt) and discounted (denoted $\gamma$ StateEnt) state distributions, on sparse reward complex gridworld domains. These are hard exploration tasks, where the agent needs to pass through slits and walls to reach the goal state (placed at the top right corner of the grid). We use REINFORCE [@reinforce] as the baseline algorithm, and for all comparisons use standard policy entropy regularization (denoted PolicyEnt for baseline).

[0.23]{} ![State space coverage on complex sparse reward double-slit-double-wall gridworld domains. Figure shows that regularization with the *discounted* state distribution indeed has more useful effects in terms of exploration and state space coverage compared to regularization with policies. We also find that state 