does not lead to significant improvements. This is an interesting result justifying that state space exploration may not necessarily be needed in all MDPs, especially when states are aliased[]{data-label="fig:twostateMDP_comp"}](mdp_fig2d_lr_0_01.pdf "fig:"){width="\linewidth"} \[fig:exact\_pg\]

[0.3]{} ![State distribution entropy regularized exact policy gradient can lead to a better converged solution on a simple two state MDP (taken from [@vf_polytope] (Figure (a)) and a discrete GridWorld domain (Figure (b)). The regularized objective has a faster convergence rate compared to the unregularized objective (with $\lambda=0.0$). Interestingly, in toy MDPs where there exists state aliasing, as in Figure (c) for MDP taken from the counterexample domain [@imani2018off], we find that state distribution entropy does not lead to significant improvements. This is an interesting result justifying that state space exploration may not necessarily be needed in all MDPs, especially when states are aliased[]{data-label="fig:twostateMDP_comp"}](GridworldPG.pdf "fig:"){width="\linewidth"} \[fig:exact\_pg2\]

[0.3]{} ![State distribution entropy regularized exact policy gradient can lead to a better converged solution on a simple two state MDP (taken from [@vf_polytope] (Figure (a)) and a discrete GridWorld domain (Figure (b)). The regularized objective has a faster convergence rate compared to the unregularized objective (with $\lambda=0.0$). Interestingly, in toy MDPs where there exists state aliasing, as in Figure (c) for MDP 