over the entire state space, ie, not *uniformly optimal*, but are rather optimal for one distribution over the initial states but may not be optimal for a different starting state distribution. This often leads to the large sample complexity of policy gradient methods [@Kakade2003OnTS] where a large number of samples may be required for obtaining good policies. The lack of exploration in policy gradient methods may often lead to large sample complexity to obtain accurate estimates of the gradient direction. It is often assumed that the restart, or starting state distribution in policy gradient method provides sufficient exploration. In this work, we tackle the exploration problem in policy gradient methods by explicitly using the entropy of the discounted future state distribution. We show that even for the starting state formulation of policy gradients, we can construct the normalized discounted future state distribution, where instead of sampling from this distribution (which is hard in practice, since sampling requires discounting with $(1-\gamma)$, we instead regularize policy optimization with the entropy ${\mathbb{H}}((1 - \gamma) \mathbf d_{\alpha, \gamma, \theta})$

Entropy Regularization with Discounted Future State Distribution
================================================================

The key idea behind our approach is to use regularization with the entropy of the state distribution 