surely to a local maximum of $J(\theta)$ under relatively mild technical conditions.

Maximum entropy based objectives are often used in policy gradient methods to help with exploration by avoiding premature convergence of stochastic policies to deterministic policies. This is often termed as entropy regularization which augments the rewards with an entropy weighted term given by $r_t = r_{t} + \lambda H( \pi(.|s_t )$. One of the common approaches is to use an entropy regularized objective, given by $    \tilde{J}(\theta) = \expectation_{\pi_{\theta}} \Bigl[\sum_{t=0}^{\infty} \gamma^{t} r(S_t, A_t) + \lambda {\mathbb{H}}(\pi_{\theta}(\cdot \mid S_t) \Bigm | S_0 \Bigr]$, where $\lambda \in \mathbb{R}$ is a hyperparameter and ${\mathbb{H}}(\pi_{\theta}(\cdot \mid S_t)$ is the entropy of the policy function (for stochastic policies).

Related Work
------------

Existing works in the literature achieve exploration by introducing entropy regularization during policy optimization [@williams1991function; @mnih2016asynchronous]. Entropy regularization is commonly used in deep RL tasks [@sac; @Ziebart] where preventing policies from quickly collapsing to a deterministic value is the key step for ensuring sufficient exploration. Information theoretic regularizers are also proposed in the literature that help policies to extract useful structures and priors from the tasks. This is often achieved with goal conditioned policies [@infobot] or taking information asymmetry under consideration [@galashov2019information]. 