state space can be based on the entropy of the induced state distribution. The proposed approach can be considered as a three time-scale algorithm and under some mild technical conditions, we prove its convergence to a locally optimal policy. Experimentally, we demonstrate usefulness of regularization with the discounted future state distribution in terms of increased state space coverage and faster learning on a range of complex tasks.'
author:
- |
    Riashat Islam\
    McGill University, Mila\
    School of Computer Science\
    `riashat.islam@mail.mcgill.ca`\
    Raihan Seraj\
    McGill University, CIM\
    Electrical and Computer Engineering\
    `raihan.seraj@mail.mcgill.ca`\
    Pierre-Luc Bacon\
    Stanford University\
    Computer Science\
    `plbacon@cs.stanford.edu`\
    Doina Precup\
    McGill University, Mila\
    School of Computer Science\
    Google Deepmind\
    `dprecup@cs.mcgill.ca`
bibliography:
- 'references.bib'
title: Entropy Regularization with Discounted Future State Distribution in Policy Gradient Methods
---

Introduction
============

Exploration in policy optimization methods is often tied to exploring in the policy parameter space. This is primarily achieved by adding noise to the gradient when following stochastic gradient ascent. More explicit forms of exploration within the state and action space include policy entropy regularization. This promotes stochasticity in policies, thereby preventing premature convergence to deterministic policies [@mhiha2c; @eq_pgq]. Such regularization schemes play the role of smoothing out the optimization landscape in non-convex 