policy optimization problems [@understanding_entropy]. Deep reinforcement learning algorithms have had enormous success with entropy regularized policies, commonly known as maximum entropy RL framework [@Ziebart]. These approaches ensure exploration in the action space, which indirectly contributes to exploration in the state space, but do not explicitly address the issue of state space exploration. This leads us to the question : *how do we regularize policies to obtain maximal coverage in the state space?*

One of the metrics to measure coverage in state space is the entropy of the *discounted* future state distribution, as proposed in [@kakade_entropy]. In their work, they prove that using the entropy of discounted future state distribution as a reward function, we can achieve improved coverage of the state space. Drawing inspiration from this idea, and to provide a practically feasible construct, we first propose an approach to estimate the discounted future state distribution. We then provide an approach for efficient exploration in policy gradient methods, to reduce sample complexity, by regularizing policy optimization based on the entropy of the discounted future state distribution. The implication of this is that the policy gradient algorithm yields policies that improve state space coverage by maximizing the entropy of the discounted 