A Unified Bellman Optimality Principle Combining Reward Maximization and Empowerment
Felix Leibfried Sergio Pascual-Diaz Jordi Grau-Moya
cs.LG cs.AI stat.ML


Global hypoellipticity for a class of
 periodic Cauchy operators
    Fernando de Ávila Silva
    December 30, 2023
================================================================




Empowerment is an information-theoretic method that can be used to intrinsically motivate learning agents. It attempts to maximize an agent's control over the environment by encouraging visiting states with a large number of reachable next states. Empowered learning has been shown to lead to complex behaviors, without requiring an explicit reward signal. In this paper, we investigate the use of empowerment in the presence of an extrinsic reward signal. We hypothesize that empowerment can guide reinforcement learning (RL) agents to find good early behavioral solutions by encouraging highly empowered states.
We propose a unified Bellman optimality principle for empowered reward maximization. Our empowered reward maximization approach generalizes both Bellman’s optimality principle as well as recent information-theoretical extensions to it. We prove uniqueness of the empowered values and show convergence to the optimal solution. We then apply this idea to develop off-policy actor-critic RL algorithms which we validate in high-dimensional continuous robotics domains (MuJoCo). Our methods demonstrate improved initial and competitive final performance compared to model-free state-of-the-art techniques. 




§ INTRODUCTION



In reinforcement learning <cit.> (RL), agents identify policies to collect as much reward as possible in a given environment. Recently, leveraging parametric function approximators has led to tremendous success in applying RL to high-dimensional domains such as Atari games <cit.> or robotics <cit.>. In such domains, inspired by the policy gradient theorem <cit.>, actor-critic approaches <cit.> attain state-of-the-art results by learning both a parametric policy and a value function. 

Empowerment is an information-theoretic framework where agents maximize the mutual information between an action sequence and the state that is obtained after executing this action sequence from some given initial state <cit.>. It turns out that the mutual information is highest for such initial states where the number of reachable next states is largest. Policies that aim for high empowerment can lead to complex behavior, e.g. balancing a pole in the absence of any explicit reward signal <cit.>. 

Despite progress on learning empowerment values with function approximators <cit.>, there has been little attempt in the combination with reward maximization, let alone in utilizing empowerment for RL in the high-dimensional domains it has become applicable just recently. We therefore propose a unified principle for reward maximization and empowerment, and demonstrate that empowered signals can boost RL in large-scale domains such as robotics. In short, our contributions are:


  * a generalized Bellman optimality principle for joint reward maximization and empowerment,

  * a proof for unique values and convergence to the optimal solution for our novel principle,

  * empowered actor-critic methods boosting RL in MuJoCo compared to model-free baselines.




§ BACKGROUND





 §.§ Reinforcement Learning



In the discrete RL setting, an agent, being in state s∈𝒮, executes an action a∈𝒜 according to a behavioral policy π_behave(a|s) that is a conditional probability distribution π_behave: 𝒮×𝒜→ [0,1]. The environment, in response, transitions to a successor state s^'∈𝒮 according to a (probabilistic) state-transition function 𝒫(s^'|s,a), where 𝒫: 𝒮×𝒜×𝒮→ [0,1]. Furthermore, the environment generates a reward signal r=ℛ(s,a) according to a reward function ℛ: 𝒮×𝒜→ℝ. The agent's aim is to maximize its expected future cumulative reward with respect to the behavioral policy max_π_behave𝔼_π_behave,𝒫[ ∑_t=0^∞γ^t r_t ], with t being a time index and γ∈ (0,1) a discount factor. Optimal expected future cumulative reward values for a given state s obey then the following recursion:

    V^⋆(s) = max_a( ℛ(s,a) + γ𝔼_𝒫(s^'|s,a)[ V^⋆(s^') ] )=: max_a Q^⋆(s,a) ,

referred to as Bellman's optimality principle <cit.>, where V^⋆ and Q^⋆ are the optimal value functions.



 §.§ Empowerment



Empowerment is an information-theoretic method where an agent executes a sequence of k actions a⃗∈𝒜^k when in state s∈𝒮 according to a policy π_empower(a⃗|s) which is a conditional probability distribution π_empower: 𝒮×𝒜^k → [0,1]. This is slightly more general than in the RL setting where only a single action is taken upon observing a certain state. The agent's aim is to identify an optimal policy π_empower that maximizes the mutual information I [ A⃗, S^'| s] between the action sequence a⃗ and the state s^' to which the environment transitions after executing a⃗ in s, formulated as:

    E^⋆(s) = max_π_empower I [ A⃗, S^'| s] = max_π_empower𝔼_π_empower(a⃗|s) 𝒫^(k)(s^'|s,a⃗)[ logp(a⃗|s^',s)/π_empower(a⃗|s)].

Here, E^⋆(s) refers to the optimal empowerment value and 𝒫^(k)(s^'|s,a⃗) to the probability of transitioning to s^' after executing the sequence a⃗ in state s, where 𝒫^(k):𝒮×𝒜^k ×𝒮→ [0,1]. Importantly, p(a⃗|s^',s) = 𝒫^(k)(s^'|s, a⃗) π_empower(a⃗|s)/∑_a⃗𝒫^(k)(s^'|s, a⃗) π_empower(a⃗|s) is the inverse dynamics model of π_empower. The implicit dependency of p on the optimization argument π_empower renders the problem non-trivial.

From an information-theoretic perspective, optimizing for empowerment is equivalent to maximizing the capacity <cit.> of an information channel 𝒫^(k)(s^'|s,a⃗) with input a⃗ and output s^' w.r.t. the input distribution π_empower(a⃗|s), as outlined in the following <cit.>.
Define the functional I_f(π_empower,𝒫^(k),q) := 𝔼_π_empower(a⃗|s) 𝒫^(k)(s^'|s,a⃗)[ logq(a⃗|s^',s)/π_empower(a⃗|s)], where q is a conditional probability q: 𝒮×𝒮×𝒜^k → [0,1]. Then the mutual information is recovered as a special case of I_f with I [ A⃗, S^'| s] = max_q I_f(π_empower,𝒫^(k),q) for a given π_empower. The maximum argument

    q^⋆ (a⃗|s^',s) = 𝒫^(k)(s^'|s,a⃗) π_empower(a⃗|s)/∑_a⃗𝒫^(k)(s^'|s,a⃗) π_empower(a⃗|s)
 
is the true Bayesian posterior p(a⃗|s^',s)—see <cit.> Lemma 10.8.1 for details.
Similarly, maximizing I_f(π_empower ,𝒫^(k),q) with respect to π_empower for a given q leads to:

    π_empower^⋆(a⃗|s) =  exp(𝔼_𝒫^(k)(s^'|s,a⃗)[ log q(a⃗|s^',s)])/∑_a⃗exp(𝔼_𝒫^(k)(s^'|s,a⃗)[ log q(a⃗|s^',s)] ) .

As explained e.g. in <cit.> page 335 similar to <cit.>. The above yields the subsequent proposition.

Maximum Channel Capacity.
 
Iterating through Equations (<ref>) and (<ref>) by computing q given π_empower and vice versa in an alternating fashion converges to an optimal pair (q^⋆, π^⋆_empower) that maximizes the mutual information max_π_empower I [ A⃗, S^'| s] = I_f(π^⋆_empower,𝒫^(k),q^⋆). The convergence rate is 𝒪(1/N), where N is the number of iterations, for any initial π^ini_empower with support in 𝒜^k  ∀s—see <cit.> Chapter 10.8 and <cit.>. This is known as Blahut-Arimoto algorithm <cit.>.


Remark. Empowerment is similar to curiosity concepts of predictive information that focus on the mutual information between the current and the subsequent state <cit.>.



§ MOTIVATION: COMBINING REWARD MAXIMIZATION WITH EMPOWERMENT



The Blahut-Arimoto algorithm presented in the previous section solves empowerment for low-dimensional discrete settings but does not readily scale to high-dimensional or continuous state-action spaces. While there has been progress on learning empowerment values with parametric function approximators <cit.>, how to combine it with reward maximization or RL remains open.
In principle, there are two possibilities for utilizing empowerment. The first is to directly use the policy π^⋆_empower obtained in the course of learning empowerment values E^⋆(s). The second is to train a behavioral policy to take an action in each state such that the expected empowerment value of the next state is highest (requiring E^⋆-values as a prerequisite). Note that the two possibilities are conceptually different. The latter seeks states with a large number of reachable next states <cit.>. The first, on the other hand, aims for high mutual information between actions and the subsequent state, which is not necessarily the same as seeking highly empowered states <cit.>.

We hypothesize empowered signals to be beneficial for RL, especially in high-dimensional environments and at the beginning of the training process when the initial policy is poor. In this work, we therefore combine reward maximization with empowerment inspired by the two behavioral possibilities outlined in the previous paragraph. Hence, we focus on the cumulative RL setting rather than the non-cumulative setting that is typical for empowerment. We furthermore use one-step empowerment as a reference, i.e. k=1, because cumulative one-step empowerment learning leads to high values in such states where the number of possibly reachable next states is high, and preserves hence the original empowerment intuition without requiring a multi-step policy—see Section <ref>.
The first idea is to train a policy that trades off reward maximization and learning cumulative empowerment:

    max_π_behave𝔼_π_behave,𝒫[ ∑_t=0^∞γ^t ( αℛ(s_t,a_t) + βlogp(a_t|s_t+1,s_t)/π_behave(a_t|s_t))  ] ,

where α≥ 0 and β≥ 0 are scaling factors, and p indicates the inverse dynamics model of π_behave in line with Equation (<ref>). Note that p depends on the optimization argument π_behave, similar to ordinary empowerment, leading to a non-trivial Markov decision problem (MDP).

The second idea is to learn cumulative empowerment values a priori by solving Equation (<ref>) with α=0 and β=1. The outcome of this is a policy π^⋆_empower (and its inverse dynamics model p) that can be used to construct an intrinsic reward signal which is then added to the external reward:

    max_π_behave𝔼_π_behave,𝒫[ ∑_t=0^∞γ^t ( αℛ(s_t,a_t) + β𝔼_π^⋆_empower(a|s_t) 𝒫(s^'|s_t,a)[ logp(a|s^',s_t)/π^⋆_empower(a|s_t)] )  ] .

Importantly, Equation (<ref>) poses an ordinary MDP since the reward signal is merely extended by another stationary state-dependent signal.

Both proposed ideas require to solve the novel MDP as specified in Equation (<ref>). In Section <ref>, we therefore prove the existence of unique values and convergence of the corresponding value iteration scheme (including a grid world example). We also show how our formulation generalizes existing formulations from the literature. In Section <ref>, we carry our ideas over to high-dimensional continuous state-action spaces by devising off-policy actor-critic-style algorithms inspired by the proposed MDP formulation. We evaluate our novel actor-critic-style algorithms in MuJoCo demonstrating better initial and competitive final performance compared to model-free state-of-the-art baselines. 



§ JOINT REWARD MAXIMIZATION AND EMPOWERMENT LEARNING IN MDPS



We state our main theoretical result in advance, proven in the remainder of this section (an intuition follows): the solution to the MDP from Equation (<ref>) implies unique optimal values V^⋆ obeying the Bellman recursion

    V^⋆(s)     = max_π_behave𝔼_π_behave,𝒫[ ∑_t=0^∞γ^t ( αℛ(s_t,a_t) + βlogp(a_t|s_t+1,s_t)/π_behave(a_t|s_t)) | s_0 = s] 
        = max_π_behave,q𝔼_π_behave(a|s)[ αℛ(s,a) +  𝔼_𝒫(s^'|s,a)[ βlogq(a|s^',s)/π_behave(a|s) + γ V^⋆(s^') ] ] 
        = βlog∑_aexp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s) + γ/β V^⋆(s^')  ] ),

where 

    q^⋆(a | s^', s) = 𝒫(s^'|s, a) π^⋆_behave(a|s)/∑_a𝒫(s^'|s, a) π^⋆_behave(a|s)  = p(a|s^',s)

is the inverse dynamics model of the optimal behavioral policy π^⋆_behave that assumes the form:

    π_behave^⋆(a|s) =  exp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a|s^',s) + γ/β V^⋆(s^') ])/∑_aexp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a|s^',s)  + γ/β V^⋆(s^') ]) ,

where the denominator is just exp ( (1/β) V^⋆(s) ). While the remainder of this section explains how Equations (<ref>) to (<ref>) are derived in detail, it can be insightful to understand at a high level what makes our formulation non-trivial. The difficulty is that the inverse dynamics model p = q^⋆ depends on the optimal policy π^⋆_behavioral and vice versa leading to a non-standard optimal value identification problem. Proving the existence of V^⋆-values and how to compute them poses therefore our main theoretical contribution, and implies the existence of at least one (q^⋆,π^⋆_behave)-pair that satisfies the recursive relationship of Equations (<ref>) and (<ref>). This proof is given in Section <ref> and leads naturally to a value iteration scheme to compute optimal values in practice. The convergence of this scheme is proven in Section <ref> and we also demonstrate value learning in a grid world example—see Section <ref>. In Section <ref>, we elucidate how our formulation generalizes and relates to existing MDP formulations.



 §.§ Existence of Unique Optimal Values



Following the second line from Equation (<ref>), let's define the Bellman operator B_⋆: ℝ^|𝒮|→ℝ^|𝒮| as

    B_⋆ V(s) := max_π_behave,q𝔼_π_behave(a|s)[ αℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ βlogq(a|s^',s)/π_behave(a|s) + γ V(s^') ] ] .
 

Existence of Unique Optimal Values.

Assuming a bounded reward function ℛ, the optimal value vector V^⋆ as given in Equation (<ref>) exists and is a unique fixed point V^⋆ = B_⋆ V^⋆ of the Bellman operator B_⋆ from Equation (<ref>).


Proof. The proof of Theorem <ref> comprises three steps. First, we prove for a given (q,π_behave)-pair the existence of unique values V^(q,π_behave)  which obey the following recursion

    V^(q,π_behave)(s) = 𝔼_π_behave(a|s)[ αℛ(s,a) +  𝔼_𝒫(s^'|s,a)[ βlogq(a|s^',s)/π_behave(a|s) + γ V^(q,π_behave)(s^') ] ] .

This result is obtained through Proposition <ref> following <cit.> where we show that the value vector V^(q,π_behave) is a unique fixed point of the operator B_q,π_behave: ℝ^|𝒮|→ℝ^|𝒮| given by

    B_q,π_behaveV(s) := 𝔼_π_behave(a|s)[ αℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ βlogq(a|s^',s)/π_behave(a|s) + γ V(s^') ] ] .

Second, we prove in Proposition <ref> that solving the right hand side of Equation (<ref>) for the pair (q,π_behave) can be achieved with a Blahut-Arimoto-style
algorithm in line with <cit.>. Third, we complete the proof in Proposition <ref> based on Proposition <ref> and <ref> by showing that V^⋆ = max_π_behave,qV^(q,π_behave), where the vector-valued max-operator is well-defined because both π_behave and q are conditioned on s. The proof completion follows again <cit.>. □

Existence of Unique Values for a Given (q,π_behave)-Pair.

Assuming a bounded reward function ℛ, the value vector V^(q,π_behave) as given in Equation (<ref>) exists and is a unique fixed point V^(q,π_behave) = B_q,π_behave V^(q,π_behave) of the Bellman operator B_q,π_behave from Equation (<ref>).


As opposed to the Bellman operator B_⋆, the operator B_q,π_behave does not include a max-operation that incurs a non-trivial recursive relationship between optimal arguments. The proof for existence of unique values follows hence standard methodology <cit.> and is given in Appendix <ref>.

Blahut-Arimoto for One Value Iteration Step.

Assuming that ℛ is bounded, the maximization problem max_π_behave, q from Equation (<ref>) in the Bellman operator B_⋆ can be solved for (q,π_behave) by iterating through the following two equations in an alternating fashion:
    
    q^(m) (a|s^',s) = 𝒫(s^'|s,a) π^(m)_behave(a|s)/∑_a𝒫(s^'|s,a) π^(m)_behave(a|s) ,

    
    π^(m+1)_behave(a|s) =  exp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^(m)(a|s^',s) + γ/β V(s^') ])/∑_aexp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^(m)(a|s^',s)  + γ/β V(s^') ]),

where m is the iteration index. The convergence rate is 𝒪(1/M) for arbitrary initial π_behave^(0) with support in 𝒜 ∀s. M is the total number of iterations. The complexity for a single s is 𝒪(M |𝒮| |𝒜|).


Proof Outline. The problem in Proposition <ref> is mathematically similar to the maximum channel capacity problem <cit.> from Proposition <ref> and proving convergence follows similar steps that we outline here—details can be found in Appendix <ref>. First, we prove that optimizing the right-hand side of Equation (<ref>) w.r.t. q for a given π_behave results in Equation (<ref>) according to <cit.> Lemma 10.8.1. Second, we prove that optimizing w.r.t. π_behave for a given q results in Equation (<ref>) following standard techniques from variational calculus and Lagrange multipliers. Third, we prove convergence to a global maximum when iterating alternately through Equations (<ref>) and (<ref>) following <cit.>.

Completing the Proof of Theorem <ref>.

The optimal value vector is given by V^⋆ = max_π_behave,qV^(q,π_behave) and is a unique fixed point V^⋆ = B_⋆ V^⋆ of the Bellman operator B_⋆.


Completing the proof of Theorem <ref> requires two ingredients: the existence of unique V^(q,π_behave)-values for any (q,π_behave)-pair as proven in Proposition <ref>, and the fact that the optimal Bellman operator can be expressed as B_⋆ = max_π_behave, q B_q,π_behave where max_π_behave, q is the max-operator from Proposition <ref>. The proof follows then standard methodology <cit.>, see Appendix <ref>.



 §.§ Value Iteration and Convergence to Optimal Values



In the previous section, we have proven the existence of unique optimal values V^⋆ that are a fixed point of the Bellman operator B_⋆. This section devises a value iteration scheme based on the operator B_⋆ and proves its convergence. We commence by a corollary to express B_⋆ more concisely.

Optimal Bellman Operator.

The operator B_⋆ from Equation (<ref>) can be written as

    B_⋆ V(s)  = βlog∑_aexp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^converged(a | s^', s) + γ/β V(s^')  ] ),

where q^converged(a | s^', s) is the result of the converged Blahut-Arimoto scheme from Proposition <ref>. 


This result is obtained by plugging the converged solution π^converged_behave from Equation (<ref>) into Equation (<ref>) and leads naturally to a two-level value iteration algorithm that proceeds as follows: the outer loop updates the values V by applying Equation (<ref>) repeatedly; the inner loop applies the Blahut-Arimoto algorithm from Proposition <ref> to identify q^converged required for the outer value update.

Convergence to Optimal Values.

Assuming bounded ℛ and let ϵ∈ℝ be a positive number such that ϵ < η/1-γ where η = αmax_s,a|ℛ(s,a)|  + βlog|𝒜|. If the value iteration scheme with initial values of V(s) = 0  ∀s is run for i ≥⌈log_γϵ (1-γ)/η⌉ iterations, then ‖ V^⋆ - B_⋆^(i)V ‖_∞≤ϵ, where the notation B_⋆^(i)V means to apply B_⋆ to V i-times consecutively.


Proof. Via a sequence of inequalities, one can show that the following holds true: ‖ V^⋆ - B^(i)_⋆ V ‖_∞≤γ‖ V^⋆ - B^(i-1)_⋆ V ‖_∞≤γ^i ‖ V^⋆ - V ‖_∞≤γ^i 1/1-γη—see Appendix <ref> for a more detailed derivation. This implies that if ϵ≥γ^i 1/1-γη then i ≥⌈log_γϵ (1-γ)/η⌉ presupposing ϵ < η/1-γ. □

Conclusion. Together, Theorems <ref> and <ref> prove that our proposed value iteration scheme convergences to optimal values V^⋆ in combination with a corresponding optimal pair (q^⋆,π^⋆_behave) as described at the beginning of this section in the third line of Equation (<ref>) and in Equations (<ref>) and (<ref>) respectively. The overal complexity is 𝒪(i M |𝒮|^2 |𝒜|) where i and M refer to outer and inner iterations.

Remark. Our value iteration is required for both objectives from Section <ref> to combine reward maximization with empowerment. Equation (<ref>) motivated our scheme in the first place, whereas Equation (<ref>) requires cumulative empowerment values without reward maximization (α = 0, β = 1).




 §.§ Practical Verification in a Grid World Example



In order to practically verify our value iteration scheme from the previous section, we conduct experiments on a grid world example. The outcome is shown in Figure <ref> demonstrating how different configurations for α and β, that steer cumulative reward maximization versus empowerment learning, affect optimal values V^⋆. Importantly, the experiments show that our proposal to learn cumulative one-step empowerment values recovers the original intuition of empowerment in the sense that high values are assigned to states where many other states can be reached and low values to states where the number of reachable next states is low, but without the necessity to maintain a multi-step policy.

 



 §.§ Generalization of and Relation to Existing MDP formulations



Our Bellman operator B_⋆ from Equation (<ref>) relates to prior work as follows (see also Appendix <ref>).

    
  * Ordinary value iteration <cit.> is recovered as a special case for α=1 and β=0.
    
  * Cumulative one-step empowerment is recovered as a special case for α=0 and β=1, with non-cumulative one-step empowerment <cit.> as a further special case of the latter (γ→ 0).
    
  * When setting q(a|s^',s)=q(a|s), using a distribution that is not conditioned on s^' and omitting maximizing w.r.t. q, one recovers as a special case the soft Bellman operator presented e.g. in <cit.>. Note that this soft Bellman operator also occurred in numerous other work on MDP formulations and RL <cit.>. 
    
  * As a special case of the previous, when q(a|s^',s)=𝒰(𝒜) is the uniform distribution in action space, one recovers cumulative entropy regularization <cit.> that inspired algorithms such as soft Q-learning <cit.> and soft actor-critic <cit.>.
    
  * When dropping the conditioning on s^' and s by setting q(a|s^',s)=q(a) but without omitting maximization w.r.t. q, one recovers a formulation similar to <cit.> based on mutual-information regularization <cit.> that spurred RL algorithms such as <cit.>.
    
  * When replacing q(a|s^',s) with q(a|s_',a_'), where s_' and a_' refers to the state-action pair of the previous time step, one recovers a formulation similar to <cit.> based on the information-theoretic principle of directed information <cit.>.




§ SCALING TO HIGH-DIMENSIONAL ENVIRONMENTS



In the previous section, we presented a novel Bellman operator in combination with a value iteration scheme to combine reward maximization and empowerment. In this section, by leveraging parametric function approximators, we validate our ideas in high-dimensional state-action spaces and when there is no prior knowledge of the state-transition function. In Section <ref>, we devise novel actor-critic algorithms for RL based on our MDP formulation since they are naturally capable of handling both continuous state and action spaces. In Section <ref>, we practically confirm that empowerment can boost RL in the high-dimensional robotics simulator domain of MuJoCo using deep neural networks.



 §.§ Empowered Off-Policy Actor-Critic Methods with Parametric Function Approximators



Contemporary off-policy actor-critic approaches for RL <cit.> follow the policy gradient theorem <cit.> and learn two parametric function approximators: one for the behavioral policy π_ϕ(a|s) with parameters ϕ, and one for the state-action value function Q_θ(s,a) of the parametric policy π_ϕ with parameters θ. The policy learning objective usually assumes the form: max_ϕ𝔼_s∼𝒟[ 𝔼_π_ϕ(a|s)[ Q_θ(s,a) ] ], where 𝒟 refers to a replay buffer <cit.> that stores collected state transitions from the environment. Following <cit.>, Q-values are learned most efficiently by introducing another function approximator V_ψ for state values of π_ϕ with parameters ψ using the objective:

    min_θ𝔼_s,a, r, s^'∼𝒟[ ( Q_θ(s,a) - ( α r + γ V_ψ(s^') ) )^2 ],

where (s,a, r, s^') refers to an environment interaction sampled from the replay buffer (r stands for the observed reward signal). We multiply r by the scaling factor α from our formulation because Equation (<ref>) can be directly used for the parametric methods we propose. Learning policy parameters ϕ and value parameters ψ requires however novel objectives with two additional approximators: one for the inverse dynamics model p_χ(a|s^',s) of π_ϕ, and one for the transition function 𝒫_ξ(s^'|s,a) (with parameters χ and ξ respectively). While the necessity for p_χ is clear, e.g. from inspecting Equation (<ref>), the necessity for 𝒫_ξ will fall into place shortly as we move forward.

In order to preserve a clear view, let's define the quantity f(s,a) := 𝔼_𝒫_ξ(s^'|s,a)[ log p_χ(a|s^',s) ] - logπ_ϕ(a|s), which is short-hand notation for the empowerment-induced addition to the reward signal—compare to Equation (<ref>). We then commence with the objective for value function learning:

    min_ψ𝔼_s∼𝒟[ ( V_ψ(s) - 𝔼_π_ϕ (a|s)[  Q_θ(s,a) + β f(s,a) ] )^2 ],

which is similar to the standard value objective but with the added term β f(s,a) as a result of joint cumulative empowerment learning. At this point, the necessity for a transition model 𝒫_ξ becomes apparent. In the above equation, new actions a need to be sampled from the policy π_ϕ for a given s. However, the inverse dynamics model (inside f) depends on the subsequent state s^' as well, requiring therefore a prediction for the next state. Note also that (s,a, r, s^')-tuples from the replay buffer as in Equation (<ref>) can't be used here, because the expectation over a is w.r.t. to the current policy whereas tuples from the replay buffer come from a mixture of policies at an earlier stage of training.

Extending the ordinary actor-critic policy objective with the empowerment-induced term f yields: 

    max_ϕ𝔼_s∼𝒟[ 𝔼_π_ϕ(a|s)[ Q_θ(s,a) + β f(s,a) ] ] .


The remaining parameters to be optimized are χ and ξ from the inverse dynamics model p_χ and the transition model 𝒫_ξ. Both problems are supervised learning problems that can be addressed by log-likelihood maximization using samples from the replay buffer, leading to max_χ𝔼_s∼𝒟[ 𝔼_π_ϕ(a|s)𝒫_ξ(s^'|s,a)[ log p_χ(a|s^',s) ] ] and max_ξ𝔼_s,a,s^'∼𝒟[ log𝒫_ξ(s^'|s,a) ].

Coming back to our motivation from Section <ref>, we propose two novel empowerment-inspired actor-critic approaches based on the optimization objectives specified in this section. The first combines cumulative reward maximization and empowerment learning following Equation (<ref>) which we refer to as empowered actor-critic. The second learns cumulative empowerment values to construct intrinsic rewards following Equation (<ref>) which we refer to as actor-critic with intrinsic empowerment.

Empowered Actor-Critic (EAC). In line with standard off-policy actor-critic methods <cit.>, EAC interacts with the environment iteratively storing transition tuples (s,a,r,s^') in a replay buffer. After each interaction, a training batch {(s,a,r,s^')^(b)}_b=1^B ∼𝒟 of size B is sampled from the buffer to perform a single gradient update on the objectives from Equation (<ref>) to (<ref>) as well as the log likelihood objectives for the inverse dynamics and transition model—see Appendix <ref> for pseudocode.

 Actor-Critic with Intrinsic Empowerment (ACIE). By setting α=0 and β=1, EAC can train an agent merely focusing on cumulative empowerment learning. Since EAC is off-policy, it can learn with samples obtained from executing any policy in the real environment, e.g. the actor of any other reward-maximizing actor-critic algorithm. We can then extend external rewards r_t at time t of this actor-critic algorithm with intrinsic rewards 𝔼_π_ϕ(a|s_t) 𝒫_ξ(s^'|s_t,a)[ logp_χ(a|s^',s_t)/π_ϕ(a|s_t)] according to Equation (<ref>), where (ϕ,ξ,χ) are the result of concurrent raw empowerment learning with EAC. This idea is similar to the preliminary work of <cit.> using non-cumulative empowerment as intrinsic motivation for deep value-based RL with discrete actions in the Atari game Montezuma's Revenge.



 §.§ Experiments with Deep Function Approximators in MuJoCo



We validate EAC and ACIE in the robotics simulator MuJoCo <cit.> with deep neural nets under the same setup for each experiment following <cit.>—see Appendix <ref> for details. While EAC is a standalone algorithm, ACIE can be combined with any RL algorithm (we use the model-free state of the art SAC <cit.>). We compare against DDPG <cit.> and PPO <cit.> from RLlib <cit.> as well as SAC on the MuJoCo v2-environments (ten seeds per run <cit.>). 

The results in Figure <ref> confirm that both EAC and ACIE can attain better initial performance compared to model-free baselines. While this holds true for both approaches on the pendulum benchmarks (balancing and swing up), our empowered methods can also boost RL in demanding environments like Hopper, Ant and Humanoid (the latter two being amongst the most difficult MuJoCo tasks). EAC significantly improves initial learning in Ant, whereas ACIE boosts SAC in Hopper and Humanoid. While EAC outperforms PPO and DDPG in almost all tasks, it is not consistently better then SAC. Similarly, the added intrinsic reward from ACIE to SAC does not always help. This is not unexpected as it cannot be in general ruled out that reward functions assign high (low) rewards to lowly (highly) empowered states, in which case the two learning signals may become partially conflicting.



For the sake of completeness, we report Figure <ref> which is similar to Figure <ref> but shows episodic rewards and not maximum episodic rewards obtained so far <cit.>. Also, limits of y-axes are preserved for the pendulum tasks. Note that our SAC baseline is comparable with the SAC from <cit.> on Hopper-v2, Walker2d-v2, Ant-v2 and Humanoid-v2 after 5· 10^5 steps (the SAC from <cit.> uses the earlier v1-versions of Mujoco and is hence not an optimal reference). However, there is a discrepancy on HalfCheetah-v2. This was earlier noted by others who tried to reproduce SAC results in HalfCheetah-v2 but failed to obtain episodic rewards as high as in <cit.>, leading to a GitHub issue <https://github.com/rail-berkeley/softlearning/issues/75>. The final conclusion of this issue was that differences in performance are caused by different seed settings and are therefore of statistical nature (comparing all algorithms under the same seed settings is hence valid).




 


§ CONCLUSION



This paper provides a theoretical contribution via a unified formulation for reward maximization and empowerment that generalizes Bellman's optimality principle and recent information-theoretic extensions to it. We proved the existence of and convergence to unique optimal values, and practically validated our ideas by devising novel parametric actor-critic algorithms inspired by our formulation. These were evaluated on the high-dimensional MuJoCo benchmark demonstrating that empowerment can boost RL in challenging robotics tasks (e.g. Ant and Humanoid). 

The most promising line of future research is to investigate scheduling schemes that dynamically trade off rewards vs. empowerment with the prospect of obtaining better asymptotic performance. Empowerment could also be particularly useful in a multi-task setting where task transfer could benefit from initially empowered agents.




  §.§.§ Acknowledgments

We thank Haitham Bou-Ammar for pointing us in the direction of empowerment.


abbrv





§ THEORETICAL ANALYSIS



This section provides more details regarding the theoretical analysis of the main paper to prove the existence of unique optimal values as well as convergence of the value iteration scheme.



 §.§ Proof of Proposition <ref> from the Main Paper



Proof. Following <cit.>, let's start by defining P_π_behave:𝒮×𝒮→ [0, 1] and g_q,π_behave: 𝒮→ℝ:

    P_π_behave (s,s^') := 𝔼_π_behave(a|s)[ 𝒫(s^'|s,a) ],


    g_q,π_behave(s):= 𝔼_π_behave(a|s)[ αℛ(s,a) + β𝔼_𝒫(s^'|s,a)[ logq(a|s^',s)/π_behave(a|s)] ] .

We can then express the Bellman operator B_q,π_behave in vectorized form yielding B_q,π_behaveV = g_q,π_behave + γ P_π_behave V. Defining B_q,π_behave^(i) as short-hand notation for applying B_q,π_behave to a value vector V i-times consecutively (i=0 leaves V unaffected), we arrive at:

    V^(q,π_behave) := lim_i →∞ B_q,π_behave^(i) V = lim_i →∞∑_t=0^i-1γ^t P_π_behave^t g_q,π_behave + γ^i P_π_behave^i V_→ 0,

where P_π_behave^t denotes the t-times multiplication of P_π_behave with itself (P_π_behave^0 is the identity matrix). This means that the convergence of B_q,π_behave does not depend on the initial value vector V, therefore:

    B_q,π_behave V^(q,π_behave)    = g_q,π_behave + γ  P_π_behavelim_i →∞∑_t=0^i-1γ^t P_π_behave^t g_q,π_behave
        = γ^0 P_π_behave^0 g_q,π_behave + lim_i →∞∑_t=1^i γ^t P_π_behave^t g_q,π_behave
        = lim_i →∞∑_t=0^i-1γ^t P_π_behave^t g_q,π_behave + γ^i P_π_behave^i g_q,π_behave_→ 0 = V^(q,π_behave),

proving that V^(q,π_behave) is a fixed point of B_q,π_behave. The uniqueness proof follows next. Assume there was another fixed point V^' of B_q,π_behave, then lim_i →∞ B_q,π_behave^(i) V^' = V^(q,π_behave) because the convergence behavior of B_q,π_behave does not depend on the initial V^', hence V^' =  V^(q,π_behave). □



 §.§ Proof of Proposition <ref> from the Main Paper



Proof. Proving Proposition <ref> from the main paper is similar to the maximum channel capacity problem from information theory <cit.>. The proof follows hence similar steps as the one for Proposition <ref> from the background section on empowerment in the main paper, in the following accomplished via Lemma <ref>, <ref> and <ref>. □

Inverse Dynamics.

Maximizing the right-hand side of the Bellman operator B_⋆ V(s) = max_π_behave,q B_q,π_behaveV(s) w.r.t. to q for a given π_behave yields:

    argmax_qB_q,π_behaveV(s) = 𝒫(s^'|s,a) π_behave(a|s)/∑_a𝒫(s^'|s,a) π_behave(a|s) .



Proof. It holds that argmax_qB_q,π_behaveV(s) = argmax_q𝔼_π_behave(a|s) 𝒫(s^'|s,a)[ logq(a|s^',s)/π_behave(a|s)] because neither ℛ nor V depends on q. It then follows that

    𝔼_π_behave(a|s) 𝒫(s^'|s,a)[ logq(a|s^',s)/π_behave(a|s)] ∀ q≤ I(A,S^'|s) = 𝔼_π_behave(a|s) 𝒫(s^'|s,a)[ logp(a|s^',s)/π_behave(a|s)],

where p is the true Bayesian posterior—see <cit.> Lemma 10.8.1. □

Optimal Policy.

Maximizing the right-hand side of the Bellman operator B_⋆ V(s) = max_π_behave,q B_q,π_behaveV(s) w.r.t. to π_behave for a given q yields:

    argmax_π_behaveB_q,π_behaveV(s) = exp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q(a|s^',s) + γ/β V(s^') ])/∑_aexp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q(a|s^',s)  + γ/β V(s^') ]) .



Proof. Maximizing B_q,π_behaveV(s) w.r.t. π_behave subject to the constraint ∑_aπ_behave(a|s)=1 yields the Lagrangian:

    L(π_behave,λ) = B_q,π_behaveV(s) - λ( ( ∑_aπ_behave(a|s) ) - 1 ),

where λ is a Lagrange multiplier. The derivatives of the Lagrangian w.r.t. π_behave(ã|s), where ã refers to a specific action, and λ are given by:

    ∂ L(π_behave,λ)/∂π_behave(ã|s) =  αℛ(s,ã) +  𝔼_𝒫(s^'|s,ã)[ βlogq(ã|s^',s)/π_behave(ã|s) + γ V(s^') ] - β - λ ,


    ∂ L(π_behave,λ)/∂λ =  - ( ( ∑_aπ_behave(a|s) ) - 1 ) .

Equating the first derivative with 0 and resolving w.r.t. π_behave(ã|s), one arrives at:

    π_behave(ã|s) = exp( α/βℛ(s,ã) +  𝔼_𝒫(s^'|s,ã)[  log q(ã|s^',s) + γ/β V(s^') ] -β + λ/β).

Plugging this result into the second derivative and equating with 0 yields:

    exp( - β + λ/β) = ( ∑_aexp( α/βℛ(s,a) +  𝔼_𝒫(s^'|s,a)[ log q(a|s^',s) + γ/β V(s^') ]) )^-1.

Plugging the latter back into the result for π_behave(ã|s) completes the proof. □

Blahut-Arimoto.

Assuming bounded ℛ, iterating through Equations (<ref>) and (<ref>) from the main paper converges to argmax_π_behave,qB_q,π_behaveV(s) at a rate of 𝒪(1/M) for arbitrary initial π_behave^(0) having support in 𝒜 ∀s, with M being the total number of iterations.


Proof. Evaluating the operator B_q,π_behaveV(s) at the pair (q^(m),π_behave^(m+1)), we obtain:

    B_q^(m),π_behave^(m+1)V(s) =  βlog∑_aexp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^(m)(a|s^',s) + γ/β V(s^')  ] ).


Due to Lemma <ref>, we know that max_π_behave,qB_q,π_behaveV(s) is upper bounded:

    max_π_behave,qB_q,π_behaveV(s) ≤
       𝔼_π_behave^`⋆`(a|s)[ αℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ βlog q^(m)(a|s^',s) +γ V(s^') ] - βlogπ^(m)_behave(a|s) ] = 
       𝔼_π_behave^`⋆`(a|s)[ βlog( exp( α/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^(m)(a|s^',s) + γ/β V(s^') ] ) ) - βlogπ^(m)_behave(a|s) ],

where the notation ^`⋆` indicates optimality of a single value iteration step, as opposed to the notation (q^⋆,π_behave^⋆) from the main paper that refers to optimality after the entire value iteration scheme has converged—see Lemma <ref>.

By using the definition of π_behave^(m+1)(a|s) from Equation (<ref>), the upper two equations enable us to derive the following upper bound:

    max_π_behave,qB_q,π_behaveV(s) - B_q^(m),π_behave^(m+1)V(s) ≤β𝔼_π_behave^`⋆`(a|s)[ logπ^(m+1)_behave(a|s)/π^(m)_behave(a|s)] .


From there it follows that for M steps of the Blahut-Arimoto scheme

    1/M∑_m=0^M-1( max_π_behave,qB_q,π_behaveV(s) - B_q^(m),π_behave^(m+1)V(s) ) ≤1/Mβ𝔼_π_behave^`⋆`(a|s)[ logπ^(M)_behave(a|s)/π^(0)_behave(a|s)] ≤
       1/Mβ𝔼_π_behave^`⋆`(a|s)[ log1/π^(0)_behave(a|s)] ≤1/Mβmax_a[ log1/π^(0)_behave(a|s)] .


However, since the upper term is lower-bounded by 0 and since B_q^(0),π_behave^(0)V(s) ≤ B_q^(0),π_behave^(1)V(s) ≤ B_q^(1),π_behave^(1)V(s) ≤ ... because of the alternating optimization procedure, this implies convergence at a rate of 𝒪(1/M). □

Upper Value Bound for One Value Iteration Step.

Let's introduce the following notation (q^`⋆`,π_behave^`⋆`) := argmax_π_behave,qB_q,π_behaveV(s) where the symbol ^`⋆` indicates optimality of a single value iteration step, as opposed to the notation (q^⋆,π_behave^⋆) from the main paper that refers to optimality after the entire value iteration scheme has converged. Let's define κ^(m)(s,a) := αℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ βlog q^(m)(a|s^',s) +γ V(s^') ]. It then holds that max_π_behave,qB_q,π_behaveV(s) ≤𝔼_π_behave^`⋆`(a|s)[ κ^(m)(s,a) - βlogπ^(m)_behave(a|s) ].


Proof. Let's first note that (q^`⋆`,π_behave^`⋆`) exists because B_q,π_behaveV is bounded. B_q,π_behaveV is bounded because it is a sum of three weighted terms that are bounded—see Equation (<ref>) of the main paper: 

    
  * 𝔼_π_behave(a|s)[ℛ(s,a) ] is bounded because the reward is bounded by assumption,
    
  * 𝔼_π_behave(a|s) 𝒫(s^'|s,a)[ logq(a|s^',s)/π_behave(a|s)] is a lower bound to the mutual information I( A, S^'| s) (which is bounded) according to <cit.> Lemma 10.8.1,
    
  * and V(s^') is bounded when the value iteration schemes (both using B_⋆ and B_q,π_behave) are initialized, and remains bounded in each value iteration step because B_q,π_behaveV(s) is bounded due to the previous two points and initial bounded V(s).

It then holds that

    max_π_behave,q  B_q,π_behaveV(s)  = B_q^`⋆`,π^`⋆`_behaveV(s) 
        = 𝔼_π_behave^`⋆`(a|s)[ αℛ(s,a) + γ𝔼_𝒫(s^'|s,a) [ V(s^') ]  + β𝔼_𝒫(s^'|s,a) [ log𝒫(s^'|s,a)/∑_a𝒫(s^'|s,a) π_behave^`⋆`(a|s)] ] 
       ≤𝔼_π_behave^`⋆`(a|s)[ αℛ(s,a) + γ𝔼_𝒫(s^'|s,a) [ V(s^') ] + β𝔼_𝒫(s^'|s,a) [ log𝒫(s^'|s,a)/∑_a𝒫(s^'|s,a) π_behave^(m)(a|s)] ],

where the equality is obtained by plugging in q^`⋆` using Equation (<ref>), and where the inequality leverages one more time <cit.> Lemma 10.8.1.

At the same time, we can plug Equation (<ref>) from the main paper into κ^(m)(s,a), yielding:

    κ^(m)(s,a) =  αℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ βlog𝒫(s^'|s,a)  π_behave^(m)(a|s)/∑_a𝒫(s^'|s,a) π_behave^(m)(a|s) +γ V(s^') ] .


Rearranging the upper equation results in:

    β𝔼_𝒫(s^'|s,a)[ log𝒫(s^'|s,a) /∑_a𝒫(s^'|s,a) π_behave^(m)(a|s)] = 
       κ^(m)(s,a) -βlogπ_behave^(m)(a|s) - αℛ(s,a) - γ𝔼_𝒫(s^'|s,a) [ V(s^') ] .


Plugging the latter result into the earlier derived inequality completes the proof. □





 §.§ Proof of Proposition <ref> from the Main Paper



Proof. The mechanics of the proof are in line with <cit.>. Let's denote (q^⋆,π_behave^⋆) = argmax_π_behave, q V^(q,π_behave) and V^⋆ = V^(q^⋆,π^⋆_behave). It then holds that

    V^⋆ = B_q^⋆,π_behave^⋆ V^⋆≤max_π_behave,q B_q,π_behave V^⋆ =: B_q^',π_behave^' V^⋆≤ V^(q^',π^'_behave) ,

where the last inequality is because of the consistency of values as proven in Lemma <ref>. But by definition it holds that V^⋆ = max_π_behave,qV^(q,π_behave)≥ V^(q^',π_behave^'). This implies that V^⋆ = V^(q^',π_behave^'). The latter means that V^⋆ = max_π_behave,q B_q, π_behave V^⋆ = B_⋆ V^⋆ which proves that V^⋆ is a fixed point of the operator B_⋆.

The uniqueness of values proof comes next. Assume there was another fixed point of the operator B_⋆ denoted as V^' = V^(q^', π^'_behave), then

    V^⋆ = B_q^⋆, π^⋆_behave V^⋆ = max_π_behave,q B_q,π_behave V^⋆≥ B_q^', π^'_behave V^⋆≥ V^(q^', π^'_behave) = V^' ,

where the last inequality is again because of Lemma <ref>. One can show similarly that V^'≥ V^⋆, which does hence imply that V^' = V^⋆. □

Value Consistency for the Evaluation Operator.

If V ≤ B_q,π_behaveV then B^(i)_q,π_behave V ≤ V^(q,π_behave) ∀ i ∈ℕ, and similarly if V ≥ B_q,π_behaveV then B^(i)_q,π_behave V ≥ V^(q,π_behave) ∀ i ∈ℕ.


Proof. The proof follows via induction. The base case is V (≥)≤ B_q,π_behaveV. The inductive step is as follows. If B_q,π_behave^(i-1) V (≥)≤B_q,π_behave^(i) V then

    B_q,π_behave^(i+1) V = g_q,π_behave + γ P_π_behave B_q,π_behave^(i) V (≤)≥ g_q,π_behave + γ P_π_behave B_q,π_behave^(i-1) V = B_q,π_behave^(i) V,

which completes the induction with help of the concise notation from Appendix <ref>. □



 §.§ Proof Details of Theorem <ref> from the Main Paper



This section is to shed more light on the proof of Theorem <ref> from the main paper to show that B_⋆ is a contraction map via the subsequent proposition.

Contraction Map.

Assuming bounded ℛ and let η∈ℝ^+ be a positive constant η = αmax_s,a|ℛ(s,a)|  + βlog|𝒜|. Then ‖ V^⋆ - B_⋆^(i)V ‖_∞≤γ^i 1/1 - γη with initial V(s) = 0  ∀s.


Proof. The proposition is proven by the following sequence of inequalities:

    ‖ V^⋆ - B_⋆^(i)V ‖_∞ =: | V^⋆(s^⋆) - B_⋆^(i)V(s^⋆) | = 
       | max_π_behave,q B_q,π_behave V^⋆(s^⋆) - max_π_behave,q B_q,π_behave B_⋆^(i-1)V(s^⋆) |  ≤
       max_π_behave,q| B_q,π_behave V^⋆(s^⋆) - B_q,π_behave B_⋆^(i-1)V(s^⋆) | = 
       max_π_behave| γ𝔼_π_behave(a|s) 𝒫(s^'|s,a)[ V^⋆(s^') ] - γ𝔼_π_behave(a|s) 𝒫(s^'|s,a)[ B_⋆^(i-1)V(s^') ] | ≤
       γ‖ V^⋆ - B_⋆^(i-1)V ‖_∞recursion≤γ^i ‖ V^⋆ - V ‖_∞V  is 0=γ^i ‖ V^⋆‖_∞≤γ^i 1/1-γη,

where η is a positive constant to upper-bound V^⋆-values, see Corollary <ref>. □


Upper Value Bound for Optimal Values.

Optimal values are upper-bounded according to | V^⋆(s) | ≤1/1-γ( αmax_s,a|ℛ(s,a)|  + βlog|𝒜| )  ∀s.


This follows straightforwardly from worst-case assumptions and properties of the geometric series and the mutual information. The empowerment-induced addition to the reward signal is upper-bounded by a mutual information term, which is upper-bounded by the worst-case entropy in action space.

Remark. A contraction proof for B_⋆  with any two initial value vectors V^' and V follows similar steps as outlined in Proposition <ref> by replacing V^⋆ accordingly.



 §.§ Limit Cases of Equation (<ref>)


In the following, we consider limit cases of Equation (<ref>).



  §.§.§ Value Iteration Recovered



Here, we consider α = 1 and β→ 0. While one can easily recover value iteration as a special case by inspecting Equation (<ref>) from the main paper simply by setting α=1 and β=0, it can be insightful how to obtain Bellman's classical optimality principle as a limit case from Equation (<ref>):

    lim_β→ 0 V^⋆(s)  = 
       lim_β→ 0βlog∑_aexp( 1/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s) + γ/β V^⋆(s^')  ] ) L'Hospital if max_a( ℛ(s,a) + γ𝔼_𝒫(s^'|s,a)[ V^⋆(s^')  ] ) > 0=
       lim_β→ 0∑_aexp( 1/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s) + γ/β V^⋆(s^')  ] ) ( -1/β^2)(  ℛ(s,a) + γ𝔼_𝒫(s^'|s,a)[V^⋆ (s^') ] )/( -1/β^2)∑_aexp( 1/βℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s) + γ/β V^⋆(s^')  ] ) = 
       max_a( ℛ(s,a) + γ𝔼_𝒫(s^'|s,a)[V^⋆ (s^') ] ) .

The above is true if ( ℛ(s,a) + γ𝔼_𝒫(s^'|s,a)[ V^⋆(s^')  ] ) > 0 for at least one action a given the state s, because numerator and denominator are then dominated by the maximum sum element. If max_a( ℛ(s,a) + γ𝔼_𝒫(s^'|s,a)[ V^⋆(s^')  ] ) ≤ 0 given s, then one needs to focus on the second line of the above expression because L'Hospital does not apply anymore. In this case, the maximum element will dominate the sum dwarfing the non-maximum elements. As a consequence log and exp cancel each other and β cancels with (1 / β). β hence only multiplies with the intrinsic motivation term induced by empowerment. The latter is going to therefore vanish since β→ 0, resulting in the same expression as in the last line above. 



  §.§.§ Cumulative One-Step Empowerment Recovered



Here we consider α→ 0 and β =1. In line with the previous section, recovering cumulative one-step empowerment can be easily obtained from Equation (<ref>) by setting α=0 and β=1. The limit case of Equation (<ref>) is trivially given by:

    lim_α→ 0 V^⋆(s) = 
       lim_α→ 0log∑_aexp( αℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s) + γ V^⋆(s^')  ] ) = 
       log∑_aexp(𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s) + γ V^⋆(s^')  ] ) .




  §.§.§ Non-Cumulative One-Step Empowerment Recovered


In addition to α→ 0 and β =1 from the former section, we consider here γ→ 0 in the following:

    lim_α→ 0, γ→ 0 V^⋆(s) = 
       lim_α→ 0, γ→ 0log∑_aexp( αℛ(s,a) + 𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s) + γ V^⋆(s^')  ] ) = 
       log∑_aexp(𝔼_𝒫(s^'|s,a)[ log q^⋆(a | s^', s)  ] ) .

The latter can be also obtained by running one-step empowerment (k=1) according to the Blahut-Arimoto scheme from the main paper's background section in Proposition <ref> until convergence, and subsequently plugging the converged solution π_empower^⋆ from Equation (<ref>) into Equation (<ref>).



§ PSEUDOCODE FOR THE EMPOWERED ACTOR-CRITIC (EAC)



Let's restate the optimization objectives from Section <ref> as functions of the optimization parameters and a batch ℬ = { (s,a,r,s^')^(b)}_b=1^B sampled from the replay buffer, where B is the batch size:

    J_Q(θ, ℬ) = 1/B∑_b=1^B ( Q_θ(s^(b),a^(b)) - ( α r^(b) + γ V_ψ(s^'^(b)) ) )^2 ,


    J_V(ψ, ℬ) = 1/B∑_b=1^B  ( V_ψ(s^(b)) - 𝔼_π_ϕ(a|s^(b))[  Q_θ(s^(b),a) + β f(s^(b),a) ] )^2 ,


    J_π(ϕ, ℬ) = -1/B∑_b=1^B  𝔼_π_ϕ(a|s^(b))[ Q_θ(s^(b),a) + β f(s^(b),a) ]  ,


    J_p(χ, ℬ) = -1/B∑_b=1^B  𝔼_π_ϕ(a|s^(b))𝒫_ξ(s^'| s^(b),a)[ log p_χ(a|s^',s^(b)) ] ,


    J_𝒫(ξ, ℬ) = -1/B∑_b=1^B  log𝒫_ξ(s^'^(b)|s^(b),a^(b))  .


Denoting the corresponding learning rates as δ_θ,δ_ψ,δ_ϕ,δ_χ and δ_ξ, we can phrase pseudocode for the empowered actor-critic conveniently.


Note that practically when updating the Q-value parameters θ, we recommend replacing the value target V_ψ with an exponentially averaged value target V_ψ̅ instead where ψ̅← (1-τ) ψ̅ + τψ with horizon parameter τ —see <cit.>.

Note also that our second proposed method, actor-critic with intrinsic empowerment (ACIE), can use the same algorithm for learning parametric function approximators by setting α=0 and β=1. Since Algorithm <ref> is an off-policy method that uses a replay buffer, it can be combined with any other actor-critic algorithm whose actor is collecting samples from the environment. An ACIE-agent can hence be trained concurrently and used to generate intrinsic rewards according to Equation (<ref>) from the main paper. These intrinsic rewards are then added to the extrinsic rewards of the agent that collects samples from the environment to encourage visiting states with high cumulative empowerment. 



§ EXPERIMENTS



The following subsections provide a detailed description of the setups that we used for the grid world and MuJoCo experiments.



 §.§ Grid World



In the grid world setting from the main paper (Section <ref>), the agent has to reach a goal in the lower left of a 16 × 16 grid, which is rewarded with +2. The agent can execute nine actions in each grid cell: left, right, up, down, as well as diagonally or stay in place. The transition function is deterministic.
The discount factor γ was set to γ=0.95 in the experiments. The stopping criterion for the value iteration scheme was when the infinity norm of two consecutive value vectors dropped below ϵ_outer=5 · 10^-4. The stopping criterion for the inner Blahut-Arimoto scheme for each value iteration step was when the maximum absolute difference between the probability values in consecutive q and π_behave dropped below the threshold ϵ_inner=5 · 10^-4.

Below is another grid world example similar to the one from the main paper, where the agent has to reach a goal in the upper right of a 16 × 16 grid. Reaching the goal is rewarded with +1 and terminates the episode whereas every step is penalized with -1. The transition function is probabilistic. Whenever the agent takes a step, the agent ends up at the intended next grid cell with only a 20%-chance. There is either a 30%-chance of a horizontal perturbation by one step, or a 30%-chance of a vertical perturbation by one step, or a 20%-chance of a diagonal perturbation by one step.
The discount factor γ was set to γ=0.6 (leading to more myopic policies).

 



 §.§ MuJoCo



For all our MuJoCo experiments, we followed standard literature regarding hyperparameter settings <cit.>. We used Adam <cit.> as optimizer for all parametric functions with a learning rate δ=3 · 10^-4. The discount factor γ was set to γ = 0.99, the replay buffer size was 5 · 10^5 and the batch size for training was 256. All neural networks were implemented in PyTorch. The critic and policy networks had two hidden layers whereas the transition and inverse dynamics model networks had three hidden layers. The number of units per hidden layer was 256 using ReLU activations. In line with <cit.>, we used an exponentially averaged V-value target for updating Q-value parameters with a horizon parameter τ=0.01—explained at the end of Appendix <ref>. Our specific trade-off parameters α and β were set to α=10 and β=0.1 respectively (both for EAC and ACIE experiments) as determined through initial experiments on InvertedDoublePendulum-v2 and HalfCheetah-v2. ACIE-generated intrinsic rewards were furthermore clipped to not exceed an absolute value of 20.

Both policy and inverse dynamics model assume that actions are distributed according to a multivariate Gaussian with diagonal covariance. They receive as input the (concatenated) vectors of s and (s,s^') respectively. They output the mean and the log standard deviation vectors from which real-valued actions can be sampled. The real-valued actions are subsequently squashed through a sigmoid function because MuJoCo has bounded action spaces. We used tanh <cit.> scaled by the environment-specific bounds. The transition network assumes that states are distributed according to a multivariate isotropic Gaussian with a given standard deviation of 10^-5. It receives as input the concatenated vectors of (s,a) and outputs the mean of s^'. The value networks merely output a single real number for cumulative reward prediction given the input. The input to the Q-value network are the concatenated vectors of (s,a) whereas the input to the V-value network is s.

Following <cit.>, we used a twin Q-critic rather than a single Q-critic. This means that two Q-critic networks Q_θ_1(·,·^') and Q_θ_2(·,·^') are trained. When updating the V-critic and the policy, Q_θ(·,·^') is replaced with min{Q_θ_1(·,·^'),Q_θ_2(·,·^')} to prevent value overestimation. To train the policy parameters, we applied the reparameterization trick on the actions <cit.>—see <cit.> Appendix C. We also found it helpful to bound the log standard deviation of the policy and inverse dynamics networks according to <cit.> Appendix A.1 to make our implementation more stable.

We compare against an SAC baseline with hyperparameters chosen according to the original paper <cit.>, except using a reward scale of 10 to ensure comparability with our methods EAC and ACIE. We furthermore compare against the DDPG and PPO baselines from RLlib <cit.> using hyperparameters settings following <cit.> and <cit.>, but with the same neural network architectures as used in EAC, ACIE and SAC to ensure a fair comparison.

Note that in neither Figure <ref> nor Figure <ref> from the main paper do we report results from DDPG on Ant because the RLlib baseline implementation of that algorithm was not able to learn with our experimental protocol in that specific environment. In initial trials, we observed that DDPG in Ant leads to a rapid drop in performance to large negative values after the very first few episodes and never recovers from there within the next 5· 10^5 environment steps. This performance pattern is in line with the experiments conducted in previous literature and can be seen by carefully inspecting Figure 1(d) from the SAC-paper <cit.>.

