space coverage is more with entropy of *discounted* state distribution compared to *stationary* state distribution. All state visitation heat maps are shown after only $1000$ time steps of initial training phase.[]{data-label="fig:grid_reinforce"}](grid_discount_statent.png "fig:"){width="\linewidth"}

[0.23]{} ![State space coverage on complex sparse reward double-slit-double-wall gridworld domains. Figure shows that regularization with the *discounted* state distribution indeed has more useful effects in terms of exploration and state space coverage compared to regularization with policies. We also find that state space coverage is more with entropy of *discounted* state distribution compared to *stationary* state distribution. All state visitation heat maps are shown after only $1000$ time steps of initial training phase.[]{data-label="fig:grid_reinforce"}](grid_stateent.png "fig:"){width="\linewidth"}

[0.23]{} ![State space coverage on complex sparse reward double-slit-double-wall gridworld domains. Figure shows that regularization with the *discounted* state distribution indeed has more useful effects in terms of exploration and state space coverage compared to regularization with policies. We also find that state space coverage is more with entropy of *discounted* state distribution compared to *stationary* state distribution. All state visitation heat maps are shown after only $1000$ time steps of initial training phase.[]{data-label="fig:grid_reinforce"}](grid_baseline.png "fig:"){width="\linewidth"}

**Continuous Control Tasks:** We extend our proposed regularized policy gradient objective on standard continuous control Mujoco domains [@conf/iros/TodorovET12]. 