Autoregressive Policies for Continuous Control Deep Reinforcement Learning
Dmytro Korenkevych A. Rupam Mahmood Gautham Vasan James Bergstra


Privacy Protection in Street-View Panoramas using Depth 
 and Multi-View Imagery
    Ries Uittenbogaard^1,  ^*Clint Sebastian^2,  Julien Vijverberg^3, 

Bas Boom^3,  Dariu M. Gavrila^1,  Peter H.N. de With^2 
 

^1Intelligent Vehicles Group, TU Delft, ^2VCA Group, TU Eindhoven, ^3Cyclomedia B.V

c.sebastian@tue.nl 

^*corresponding author

    
====================================================================================================================================================================================================================================================================




Reinforcement learning algorithms rely on exploration to discover new behaviors, which is typically achieved by following a stochastic policy. In continuous control tasks, policies with a Gaussian distribution have been widely adopted. Gaussian exploration however does not result in smooth trajectories that generally correspond to safe and rewarding behaviors in practical tasks. In addition, Gaussian policies do not result in an effective exploration of an environment and become increasingly inefficient as the action rate increases. This contributes to a low sample efficiency often observed in learning continuous control tasks. We introduce a family of stationary autoregressive (AR) stochastic processes to facilitate exploration in continuous control domains. We show that proposed processes possess two desirable features: subsequent process observations are temporally coherent with continuously adjustable degree of coherence, and the process stationary distribution is standard normal. We derive an autoregressive policy (ARP) that implements such processes maintaining the standard agent-environment interface. We show how ARPs can be easily used with the existing off-the-shelf learning algorithms. Empirically we demonstrate that using ARPs results in improved exploration and sample efficiency in both simulated and real world domains, and, furthermore, provides smooth exploration trajectories that enable safe operation of robotic hardware. 




§ INTRODUCTION

Reinforcement Learning (RL) is a promising approach to solving complex real world tasks with physical robots, supported by recent successes <cit.>. Exploration is an integral part of RL responsible for discovery of new behaviors. It is typically achieved by executing a stochastic behavior policy <cit.>. In continuous control domain, for instance, policies with a parametrized Gaussian distribution have been commonly used <cit.>. The samples from such policies are temporally coherent only through the distribution mean. In most environments this coherence is not sufficient to provide consistent and effective exploration. In early stages of learning in particular, with randomly initialized policy parameters, exploration essentially relies on a white noise process around zero mean. In environments where actions represent low-level motion control, e.g. velocity or torque, such exploration rarely produces a consistent motion that could lead to discovery of rewarding behaviors. This contributes to low sample efficiency of learning algorithms <cit.>. For real world robotic applications a short reaction time is often desirable, however, as action rate increases, a white noise exploration model becomes even less viable, effectively locking the robot in place <cit.>. In addition, temporally incoherent exploration results in a jerky movement on physical robots leading to hardware damage and safety concerns <cit.>. 

 In this work we observe that the parameters of policy distribution typically exhibit high temporal coherence, particularly at higher action rates. Specifically, in the case of Gaussian policy distribution, the action can be represented as a sum of deterministic parametrized mean and a scaled white noise component. The mean part typically changes smoothly between subsequent states. It is the white noise part that results in inconsistent exploration behavior. We propose to replace the white noise component with a stationary autoregressive Gaussian process that has stationary standard normal distribution, while exhibiting temporal coherence between subsequent observations. We derive a general form of these processes of an arbitrary order and show how the degree of temporal coherence can be continuously adjusted with a scalar parameter. We demonstrate an advantage of higher orders processes compared to the first order ones used in prior work. Further, we propose an agent's policy structure that directly implements the autoregessive computation in such processes. Temporal action smoothing mechanism therefore is not hidden from the agent but is made explicit through its policy function. In order to achieve this, we require a fixed length history of past states and actions. However, the set of resulting history-dependent policies contains the set of Markov deterministic policies, and those contain the optimal policies in many tasks <cit.>. We find that, in practical applications, the search for such optimal policies can be more efficient and safe in a space of history-dependent stochastic policies with special structure, compared to conventional search in a space of Markov stochastic policies. 

Empirically we show that proposed autoregressive policies can be used with off-the-shelf learning algorithms and result in superior exploration and learning in sparse reward tasks compared to conventional Gaussian policies, while achieving similar or slightly better performance in tasks with dense reward. We also show that the drop in learning performance due to increasing action rate can be greatly mitigated by increasing the degree of temporal coherence in the underlying autoregressive process. In the real world robotic experiments we demonstrate that the autoregressive policies result in a  smoother and safer movement. [See accompanying video at <https://youtu.be/NCpyXBNqNmw>]



§ RELATED WORK


The problem of exploration in Reinforcement Learning has been studied extensively. One approach has been to modify the environment by changing its reward function to make it easier for an agent to obtain any rewards or to encourage the agent to visit new environment states. This approach includes work on reward shaping <cit.> and auxiliary reward components such as curiosity <cit.>. Note that regardless of chosen reward function temporally consistent behavior would still be beneficial in most tasks as it would discover rewarding behaviors more efficiently. A randomly initialized agent is unaware of the reward function and for example will not exhibit curiosity until after some amount of learning, which already requires visiting new states and discovering rewarding behaviors in the first place. 



A second approach, particularly common in practical robotic applications, has been to directly enforce temporal coherence between subsequent motion commands. In the most simple case a low-pass filter is applied, e.g. the agent actions are exponentially averaged over the fixed or infinite length window <cit.>. A similar alternative is to employ a derivative control where agent's actions represent higher order derivatives of the control signal <cit.>. Both of these approaches correspond to acting in a modified MDP with different state and action spaces and result in a less direct connection between agent's action and its consequence in the environment, which can make the learning problem harder. They also make the process less observable unless the agent has access to the history of past actions used in smoothing and to the structure of a smoothing mechanism itself, which is typically not the case. 
As in the case with modified reward function, the optimal policies in the new MDP generally may not correspond to the optimal policies in the original MDP. 

A third approach has been to learn parameters of predefined parametrized controllers, such as motor primitives, instead of learning control directly in the actuation space <cit.>. This approach is attractive, as it allows to ensure safe robot behaviour and often results in an easier learning problem <cit.>. However, it requires expert knowledge to define appropriate class of controllers and limits possible policies to those, representable within the selected class. In complex tasks <cit.> it may be non-trivial to design a sufficiently rich primitives set.

Several studies have considered applying exploration noise to policy distribution parameters such as network weights and hidden units activations. Plappert et al. plappert2017parameter applied Gaussian exploration noise to policy parameters at the beginning of each episode, demonstrating a more coherent and efficient exploration behavior compared to only adding Gaussian noise to the action itself. Fortunato et al. fortunato2017noisy similarly applied independent Gaussian noise to policy parameters, where the scale of the noise was also learned via gradient descent. Both of these works demonstrated improved learning performance compared to baseline Gaussian action space exploration, in particular in tasks with sparse rewards. Our approach is fully complimentary to auxiliary rewards and parametric noise ideas, as both still rely on exploration noise in the action space in addition to other noise sources and can benefit from consistent and temporally smooth exploration trajectories provided by our method. 



In the context of continuous control deep RL our work is most closely related to the use of temporally coherent Gaussian noise during exploration. Wawrzynski wawrzynski2015control used moving average process for exploration where temporal smoothness of exploration trajectories was determined by the integer size of an averaging window. They showed that learning with such process results in a similar final performance as with standard Gaussian exploration, while providing smoother behavior suitable for physical hardware applications. Hoof et al. van2017generalized proposed a stationary first order AR exploration process in parameters space.
Lillicrap et al. lillicrap2015continuous and Tallec et al. tallec2019making used Ornstein–Uhlenbeck (OU) process for exploration in off-policy learning. The latter work showed that adjusting process parameters according to the time step duration helps to maintain exploration performance at higher action rates. It can be shown that in a discrete time form OU process is a first order Gaussian AR process, which makes it a particular case of our model. AR processes derived in this work generalize the processes used in these studies, providing a wider space of possible exploration trajectories. In addition, the current work proposes policy structure that directly implements autoregressive computation, in contrast to the above studies, where the agent was unaware of the noise structure. Due to this explicit policy formulation, autoregressive exploration can be used in both, on-policy and off-policy learning.

Autoregressive architectures have been proposed in the context of high-dimensional discrete or discretized continuous action spaces <cit.> with regression defined over action components. The objective of such architectures was to reduce dimensionality of the action space. In contrast, we draw on autoregressive stochastic processes literature, and define regression over time steps directly in a multidimensional continuous action space with the objective of enforcing temporally coherent behaviour. 



§ BACKGROUND




 §.§ Reinforcement Learning

Reinforcement Learning (RL) framework <cit.> describes an agent interacting with an environment at discrete  time steps. At each step t the agent receives the environment state s_t ∈𝒮 and a scalar reward signal r_t∈ R. The agent selects an action a_t ∈𝒜 according to a policy defined by a probability distribution 
π(a|s)  P{ a_t=a | s_t=s }. At the next time step t + 1 in part due to the agent's action, the environment transitions to a new state s_t+1 and produces a new reward r_t+1 according to a transition probability distribution  p(s^', r|s, a) { s_t+1 = s^', r_t+1=r | s_t = s, a_t = a }.
The objective of the agent is to find a policy that maximizes the expected return defined as the future  accumulated rewards G_t ∑_k=t^∞γ^k-t r_k+1, where γ∈[0,1] is a discount factor.
In practice, the agent observes the environment's state partially through a real-valued observation vector o_t.



 §.§ Autoregressive processes


An autoregressive process of order p ∈ℕ (AR-p) is defined as


    X_t    = ∑_k = 1^p ϕ_k X_t-k + Z_t,

where ϕ_k ∈ R, k = 1, …, p are real coefficients, and Z_t is a white noise with zero mean and finite variance,  Z_t ∼ WN(0, σ_Z^2), σ_Z^2 < ∞.

An autoregressive process {X_t} is called weakly stationary, if its mean function μ_X(t) = 𝔼[X_t] is independent of t and its covariance function γ_X(t + h, t) = cov(X_t + h, X_t) is independent of t for each h. In the future we will use the term stationary implying this definition. The process (<ref>)  is stationary if the roots G_i, i = 1, …, p (possibly complex) of its characteristic polynomial 


    P(z)  z^p - ∑_i = 1^pϕ_i z^p-i
 lie within a unit circle, e.g. |G_i| < 1, i = 1, …, p (see e.g. <cit.>). 

An autocovariance function γ_k is defined as γ_k = cov(X_t, X_t - k), k = 0, ± 1, ± 2, …. From definition, γ_0 = var(X_t) = σ_X^2. For a stationary AR-p process a linear system of Yule-Walker equations holds:



    [ γ_1; γ_2; γ_3;   ⋮; γ_p ]  = 
    [   γ_0   γ_1     … γ_p-1;   γ_1   γ_0     … γ_p-2;   γ_2   γ_1     … γ_p-3;     ⋮     ⋮     ⋱     ⋮; γ_p-1 γ_p-2     …   γ_0;       ][ ϕ_1; ϕ_2; ϕ_3;   ⋮; ϕ_p ]
    and   
       γ_0 = ∑_i = 1^p ϕ_i γ_i + σ_Z^2.


The system (<ref>) has a unique solution with respect to the variables {γ_k}, k = 0, …, p.


§ STATIONARY AUTOREGRESSIVE GAUSSIAN PROCESSSES


In this section we derive a family of stationary AR-p Gaussian processes for any p ∈ℕ, such that X_t ∼𝒩(0, 1) ∀ t, meaning X_t has a marginal standard normal distribution at each t. We also show how the degree of temporal smoothness of trajectories formed by subsequent observations of such processes can be continuously tuned with a scalar parameter.



For any p ∈ℕ and for any α_k ∈ [0, 1),
 k = 1, …, p consider a set of coefficients

    {ϕ̃_k}_k=0^p    = (-1)^k + 1∑_1 
    ≤ i_1 < i_2 … < i_k ≤ pα_i_1α_i_2…α_i_k.

The Yule-Walker system (<ref>) with coefficients {ϕ̃_k} has a unique solution with respect to (γ̃_0, γ̃_1, …, γ̃_p,σ̃_Z^2), such that γ̃_0 = 1 and σ̃_Z^2 > 0.  
Furthermore, the autoregressive process

    X_t    = ∑_k = 1^p ϕ̃_k X_t-k + Z_t
    
    Z_t    ∼𝒩(0, σ̃_Z^2),

is a stationary Gaussian process with zero mean and unit variance, meaning X_t ∼𝒩(0, 1)  ∀ t.



The proof follows from the observation that {ϕ̃_k} are coefficients of a polynomial P(z) = (z - α_1)(z - α_2)…(z - α_p) with roots {α_k} that all lie within a unit circle. Since P(z) is a characteristic polynomial of a process (<ref>), the process is stationary. The existence of a unique solution to the system (<ref>) with γ̃_0 = 1, σ̃_Z^2 > 0 follows from the observation that for any σ̃_Z^2 > 0 the system (<ref>) has a unique solution with respect to (γ̃_0, γ̃_1, …, γ̃_p), while it is homogeneous with respect to (γ̃_0, γ̃_1, …, γ̃_p,σ̃_Z^2). The result then follows from observing that γ̃_0 > 0, and scaling the solution (γ̃_0, γ̃_1, …, γ̃_p,σ̃_Z^2) by 1/γ̃_0. A complete proof can be found in Appendix <ref>.



For any p ∈ℕ and for any α∈ [0, 1) let {X_t} be the autoregressive process:

    X_t    = ∑_k = 1^p ϕ̃_k X_t-k + Z_t
    ϕ̃_k    = (-1)^k + 1p kα^k
     
    Z_t    ∼𝒩(0, σ̃_Z^2),

where p k = p!/k!(p - k)! is a binomial coefficient, σ̃_Z^2 is a solution to the system (<ref>) with {ϕ_k = ϕ̃_k}_k=1^p and γ_0=1. Then X_t ∼𝒩(0, 1)  ∀ t.


Set α_k = α ∀ k in (<ref>) and apply Proposition <ref>.

An example of a third order process AR-3 defined by (<ref>) is given in Appendix <ref>.

Proposition <ref> allows to formulate stationary autoregressive processes of an arbitrary order p for arbitrary values α_k ∈ [0, 1), k = 1, …, p, such that the marginal distributions of realizations of these processes are standard normal at each time step. This gives us great flexibility and power in defining properties of these processes, such as the degree of temporal coherence between process realizations at various time lags. If we were to use these processes as a source of exploration behavior in RL algorithms, this flexibility would translate into a flexibility in defining the shape and smoothness of exploration trajectories. Note, that the process (<ref>) trivially generalizes to a vector form by defining Z_t a multivariate white noise with a diagonal covariance.
 

Autoregressive processes in the general form (<ref>) possess a number of interesting properties that can be utilized in reinforcement learning. However, for the purposes of the discussion in the following sections, from now on we will consider a simpler subfamily of processes, defined by (<ref>). Notice, that α = 0 results in ϕ̃_k = 0 ∀ k, and X_t becomes a white Gaussian noise. On the other hand, α→ 1 results in σ_Z^2 → 0, and X_t becomes a constant function. Therefore, tuning a single scalar parameter α from 0 to 1 continuously adjusts temporal smoothness of X_t ranging from white noise to a constant function. 
Figure <ref> shows realizations of such processes at different values of p and α. 

The realizations are initialized from the same set of 3 random seeds for each p, α pair. 



§ AUTOREGRESSIVE POLICIES

In continuous control RL a policy is often defined as a parametrized diagonal Gaussian distribution: 

    p_θ(a_t | s_t) = 𝒩(μ_θ(s_t), σ_θ^2(s_t)· I),

where s_t is a state at time t, μ_θ(s_t) and σ_θ^2(s_t) are vectors parametrized by deep neural networks.
The actions, sampled from such distribution, can be represented as a_t = μ_θ(s_t) + σ_θ(s_t)ε_t, where ε_t ∼𝒩(0, I) is a white Gaussian noise.
We propose to replace ε_t with observations of an AR-p process {X_t} defined by (<ref>) for some p ∈ N and α∈ [0, 1):

    a_t = μ_θ(s_t) + σ_θ(s_t)X_t.

Both ε_t and X_t follow marginal standard normal distribution at each step t, therefore such substitution does not change the network output to noise ratio in sampled actions, however for α > 0 the sequence {X_t} possesses temporal coherence and can provide a more consistent exploration behavior. We would like to build an agent that implements stochastic policy with samples, defined by (<ref>).
From definition (<ref>) of the process {X_t}, (<ref>) can be expanded as

    a_t   =μ_θ(s_t) + σ_θ(s_t)∑_k = 1^p ϕ̃_k X_t-k + σ_θ(s_t)σ̃_Z ε_t,
    ε_t    ∼𝒩(0, I).

From (<ref>) also, X_t = (a_t - μ_θ(s_t))/σ_θ(s_t) ∀ t,
hence (<ref>) can be rewritten as

    a_t    =μ_θ(s_t) + σ_θ(s_t)∑_k = 1^p ϕ̃_k a_t-k - μ_θ(s_t-k)/σ_θ(s_t-k) +
       + σ_θ(s_t)σ̃_Z ε_t,
    ε_t    ∼𝒩(0, I).

Denote f_θ, t = ∑_k = 1^p ϕ̃_k a_t-k - μ_θ(s_t-k)/σ_θ(s_t-k) the auto-regressive "history" term in (<ref>). Note that f_θ, t is a function of past p states and actions, f_θ, t = f({s_t-k, a_t-k}_k=1^p, θ). Then a_t follows the distribution:

    a_t    ∼𝒩(μ_θ(s_t) + σ_θ(s_t) f_θ, t, σ_θ^2(s_t)σ̃^2_Z · I),
    ϕ̃_k   , σ̃^2_Z  are defined by (<ref>).


In order to implement such action distribution, we need to define a history-dependent policy π(a_t|s_t, h^p_t), where h^p_t = (s_t-p, a_t-p, …, s_t-1, a_t-1) is a history of past p states and actions. In general, history-dependent policies do not induce Markov stochastic processes, even if the environment transition probabilities are Markovian <cit.>. However when the dependence is only on a history of a fixed size, such policy induces a Markov stochastic process in an extended state space, where states are defined as pairs (h^p_t, s_t). In order to be able to lean on existing theoretical results, such as Policy Gradient Theorem <cit.>, and to use existing learning algorithms, we will talk about learning policies in this extended MDP. 

More formally, let M = (S, A, P(·|a, s), r(s, a)) be a given MDP with S and A denoting state and action sets, and P and r denoting transition probability and reward functions respectively. Let p be an arbitrary integer number. We define a modified MDP M̃^p = (S̃, Ã, P̃(· |a, s̃), r̃(s̃, a)) with the elements S̃ = {S × A}^p × S, where {C}^p denotes Cartesian product of set C with itself p times, Ã = A, and P̃ and r̃ defined as follows:




    ∀s̃, s̃^'   ∈S̃:
    s̃   = (s_1, a_1, …, s_p, a_p, s_p+1), a_k ∈ A, s_k ∈ S ∀ k
    s̃^'   = (s^'_1, a^'_1, …,  s^'_p, a^'_p, s^'_p+1), a^'_k ∈ A, s^'_k ∈ S ∀ k
    P̃(s̃^' |a, s̃)    =  P(s^'_p+1|a, s_p+1),    if [t]
           s^'_k    = s_k+1, k ≤ p
    
           a^'_k    = a_k+1, k < p
    
           a^'_p    = a
           
       
    
    0    otherwise
    r̃(s̃, a)    = r(s_p+1, a)


In other words, transitions in a modified MDP M̃^p correspond to transitions in the original MDP M with states in M̃^p containing also the history of past p states and actions in M. 
The interaction between the agent and the environment, induced by M̃^p, occurs in the following way. At each time t the agent is presented with the current state s̃_t = (s_t-p, a_t-p, …, s_t-1, a_t-1, s_t). Based on this state and its policy, it chooses an action a_t from the set A and sends it to the environment. Internally, the environment propagates the action a_t to the original MDP M, currently in state s_t, which responds with a reward value r_t+1 and transitions to a new state s_t+1. At this moment, the MDP M̃^p transitions to a new state s̃_t+1 = (s_t-p +1, a_t-p+1, …, s_t, a_t, s_t+1) and presents it to the agent together with the reward r_t+1. Let s_0 be an element of the set of the initial states of M. A corresponding initial state of M̃^p is defined as s̃_0 = (s_0, a_0, …, s_0, a_0_p  repetitions, s_0), where a_0 is any element of an action set A, for example zero vector in the case of continuous space. The particular choice of a_0 is immaterial, since it does not affect future transitions and rewards (more details in Appendix <ref>).  


We define an autoregressive policy (ARP) over M̃^p as:

    ∀s̃_t    = (s_t-p, a_t-p, …, s_t-1, a_t-1, s_t): 
    π_θ(a_t|s̃_t)    = 𝒩(μ_θ(s_t) + σ_θ(s_t) f_θ(s̃_t), σ^2_θ(s_t)σ̃^2_Z I),
    
    f_θ(s̃_t)    = ∑_k = 1^p ϕ̃_k a_t-k - μ_θ(s_t-k)/σ_θ(s_t-k),
    ϕ̃_k, σ̃^2_Z     are defined by (<ref>),

where μ_θ(·) and σ_θ(·) are parametrized function approximations, such as deep neural networks. For notation brevity, we omitted dependence of policy π_θ on {ϕ̃_k} and σ̃_Z, since these values are constant once the autoregressive model {X_t} is selected. In this parametrization, μ_θ(s_t - k), k = 0, …, p should be thought of as the same parametrized function μ_θ(·) applied to different parts of the state vector s̃_t, therefore each occurence of μ_θ(·) in (<ref>) contributes to the gradient w.r.t. parameters θ. Similarly, each occurence of σ_θ(·) contributes to the gradient w.r.t. θ. Note, that including history of states and actions does not affect the dimensionality of the input to the function approximations, as both μ_θ(·) and σ_θ(·) accept only states from the original space as inputs.   
 

The history-dependent policy (<ref>) results in the desired action distribution (<ref>) in the original MDP M, at the same time with respect to M̃^p it is just a particular case of a Gaussian policy (<ref>). Formally, we will perform learning in M̃^p, where π_θ is Markov, and therefore all the related theoretical results apply, and any off-the-shelf learning algorithm, applicable to policies of type (<ref>), can be used. In particular, the value function in e.g. actor-critic architectures is learned with usual methods. Empirically we found that conditioning value function only on a current state s_t from the original MDP instead of an entire vector s̃_t gives more stable learning performance. It also helps to maintain the critic network size invariant to the AR process order p.


By design, for each sample path (s̃_0, a_0, s̃_1, a_1, …) in M̃^p there is a corresponding sample path (s_0, a_0, s_1, a_1, …) in M with identical rewards. Therefore, improving the policy and the obtained rewards in M̃^p results in identical improvement of a corresponding history-dependent policy in M. Notice also, that if σ_θ(s_t) → 0 in (<ref>), then π_θ reduces to a Markov deterministic policy a_t = μ_θ(s_t) in M. Therefore, the optimal policy in the set of ARPs defined by (<ref>) is  at least as good, as the best deterministic policy in the set of policies  a_t = μ_θ(s_t). This is in contrast with action averaging approaches, where temporal smoothing is typically imposed on the entire action vector and not just on the exploration component, limiting the space of possible deterministic policies. 

It is important to point out that for any history-dependent policy there exists an equivalent Markov stochastic policy with identical expected returns <cit.>. For the policy (<ref>), for example, it can be constructed as π_θ^M(a | s) = ∑_h^p ∈ H^pπ_θ(a | s, h^p)p(h^p|s, π_θ) ∀ (a, s), where H^p is a set of all histories of size p. However, π_θ^M(a | s) is a non-trivial function of a state s, unknown to us at the beginning of learning. It is certainly not given by a random initialization of (<ref>), while a random initialization of (<ref>) already provides consistent and smooth behavior. π^M(a | s) also cannot be derived analytically from (<ref>), since computing p(h^p|s, π_θ) requires knowledge of environment transition probabilities, which we cannot expect to have for each given task. From these considerations, the particular form of policy parametrization defined by (<ref>) can also be thought of as an additional structure, enforced upon the general class of Markov policies, such as policies defined by (<ref>), restricting possible behaviors to temporally coherent ones. 


Although autoregressive term f_θ(s̃_t) in (<ref>) is formally a part of the distribution mean, numerically it corresponds to a stationary zero mean random process F_t = ∑_k=1^p ϕ̃_k X_t-k, where {X_t} is an underlying AR process defined by (<ref>). Therefore, f_θ(s̃_t) can be thought of as a part of an action exploration component around the "true" mean, given by μ_θ(s_t). It is this part that ensures a consistent and smooth exploration, as will be demonstrated in the next section. 


In principle, one could define π_θ in (<ref>) using arbitrary values of coefficients {ϕ̃_k} and σ̃_Z^2. The role of particular values of {ϕ̃_k} computed according to (<ref>) is to make sure, that the underlying AR process {X_t} is stationary and the autoregressive part f_θ(s̃_t) does not explode. The role of σ̃_Z^2 computed by solving (<ref>) with coefficients {ϕ_k = ϕ̃_k} and γ_0=1 is to make sure, that the variance of the underlying process {X_t} is 1. The total variance around μ_θ(s_t) is then conveniently defined by an agent controlled σ_θ(s_t).


Since linear system (<ref>) with coefficients (<ref>) and γ_0 = 1 has a unique solution according to the Proposition <ref>, its matrix has a full rank, and therefore the system is well-determined and can be solved numerically to an arbitrary precision. In practice we solve it with numpy.linalg.solve function.



§ EXPERIMENTS


We compared conventional Gaussian policy with ARPs on a set of tasks with both, sparse and dense reward functions, in simulation and the real world. In the following learning experiments we used the Open AI Baselines PPO algorithm implementation <cit.>. The results with Baselines TRPO <cit.> are provided in Appendix <ref>. For each experiment we used identical algorithm hyper-parameters and neural network structures to parametrize μ_θ, σ_θ and the value networks for both Gaussian and ARP policies. We used the same set of random seeds to initialize neural networks and the same set of random seeds to initialize environments that involve uncertainty. Detailed parameters for each task are included in Appendix <ref>.  We did not perform a hyper-parameter search to optimize for ARP performance, as our primary objective is to demonstrate the advantage of temporally coherent exploration even in the setting, tuned for a standard Gaussian policy.  The video of agent behaviors can be found at <https://youtu.be/NCpyXBNqNmw>. The code to reproduce experiments is available at <https://github.com/kindredresearch/arp>.



 §.§ The order of an autoregressive process

 
From Figure <ref> one can notice that the temporal smoothness of realizations of AR processes (<ref>) empirically increases with both, parameter α and order p. 
Why do we need higher order processes if we can simply increase α to achieve a higher degree of temporal coherence? To answer this question it is helpful to consider an autocorrelation function (ARF) ρ_τ = cov(X_t, X_t+τ)/var(X_t) = γ_τ/γ_0 of these processes.
White Gaussian noise by definition has autocorrelation function equal to zero at any τ other than 0. An autoregressive process with non-zero coefficients generally has non-zero values of autocorrelation function at all τ.

One of the reasons we are interested in autoregressive processes for exploration is that they provide smooth trajectories that do not result in jerky movement and do not damage physical robot hardware. 
Intuitively, the smoothness of the process realization is defined by a correlation between subsequent observations corr(X_t, X_t+1) = ρ_1, which for a given p increases with increasing α. However, given the same value ρ_1, processes of different orders p behave differently. Figure <ref>a shows ARFs for different processes defined by (<ref>)  and their corresponding values of α  with the same value of ρ_1 = 0.99, while Figure <ref>b shows realizations of these processes.
ARF values at higher orders p decrease much faster with increasing time lag τ compared to the 1st order process, where correlation between past and future observations lingers over long periods of time.
As shown on Figure <ref>b, the 1st order AR process produces nearly a constant function, while the 5th order process exhibits a much more diverse exploratory behavior. Given the same value of correlation between subsequent realizations, higher order autoregressive processes exhibit lower correlation between observations distant in time, resulting in trajectories with better exploration potential. In robotics applications where smoothness of the trajectory can be critical, higher order autoregressive processes may be a preferable choice. 
Empirically we found that the 3-rd order  processes provide sufficiently smooth trajectories while exhibiting a good exploratory behavior, and used p=3 in all our subsequent learning experiments varying only the smoothing parameter α. 


 §.§ Toy environment with sparse reward

  
To demonstrate the advantage of temporally consistent exploration, in particular at high action rates, we designed a toy Square environment with a 2D continuous state space bounded by a 10x10 square arena. The agent controls a dot through a continuous direct velocity control. The agent is initialized in the middle of the arena at the start of each episode and receives a -1 reward at each time step scaled by time step duration. The target is generated at a random location on a circle of diameter 5 centered at the middle of the arena to make episodes homogenous in difficulty. The episode is over when the agent approaches the target to within a distance of 0.5. The action space is bounded within a two-dimensional [-1, 1]^2 interval. The observation vector contains the agent's position, velocity, and the vector difference between agent position and the target position.

To compare exploration efficiency we ran random ARP (p=3) and Gaussian agents with μ_θ(·), σ_θ initialized to 0⃗ and 1⃗ respectively for 10 million simulated seconds at different action rates. Figure <ref>a shows average time to reach the target as a function of an action rate.
The results show that the optimal degree of temporal coherence depends on the environment properties, such as action rate. At low control frequency a white Gaussian exploration is more effective than ARPs with high α, as in the latter the agent quickly reaches the boundary of the state space and gets stuck there. The efficiency of Gaussian exploration drops dramatically with the increase of action rate. However it is possible to recover the same exploration performance in ARP by increasing accordingly the α parameter. This effect is visualized on Figure <ref>b which shows five 10 seconds long exploration trajectories at 10Hz and 100Hz control for Gaussian and ARP policies. Although ran for the same amount of simulated time, Gaussian exploration at 100Hz covers substantially smaller area of state space compared to 10Hz control, while increasing α from 0.8 to 0.95 (the values were chosen empirically) results in ARP trajectories covering similar space at both action rates. 
Note that the issue with Gaussian policy can not be fixed by simply increasing the variance, as most actions will just be clipped at the [-1, 1]^2 boundary, resulting in a similarly poor exploration. Figure <ref>b top right plot shows exploration trajectories for σ(θ) = 1⃗ (blue) and σ(θ) = 1⃗0⃗ (gray). To the contrary of the common intuition, in bounded action spaces Gaussian exploration with high variance does not produce a diverse state-action visitation.  





The advantage of ARPs in exploration translates into an advantage in learning. 
Figure <ref> shows learning curves (averaged over 5 random seeds) on Square environment at different action rates ran for 50,000 seconds of total simulated time with episodes limited to 1000 simulated seconds.
Not only ARPs exhibit better learning, but the initial random behaviour gives much higher returns compared to initial Gaussian agent behaviour. At higher action rates ARPs with higher α produce better results.

In the formulation of the AR-1 process used in Lillicrap et al. lillicrap2015continuous and in Tallec et al. tallec2019making, parameter α corresponds to 1 - κ dt, where dt is a time step duration. Hence, in that formulation α naturally approaches 1 as dt approaches zero. 
Note, that in order to achieve the best performance on each given task, parameter κ still needs to be tuned, just as parameter α needs to be tuned in our formulation. The optimal values of these parameters depend not only on action rate, but also on environment properties, such as a size of a state space relative to the typical size of an agent step.



 §.§ Mujoco experiments

Figure <ref> shows the learning results on standard OpenAI Gym Mujoco environments <cit.>. These environments have dense rewards, so consistent exploration is less crucial here compared to tasks with sparse rewards. Nevertheless, we found that ARPs perform similarly or slightly better, than a standard Gaussian policy. On a Swimmer-v2 environment ARP resulted in a much better performance compared to Gaussian policy, possibly because in this environment smooth trajectories are highly rewarded. 

 



 §.§ Physical robot experiments

On a UR5 robotic arm we were able to obtain results similar to those in the toy environment. We designed a sparse reward version of a UR5 Reacher 2D task introduced in <cit.>. In a modified task at each time step the agent receives a -1 reward scaled by a time step duration. The episode is over when the agent reaches the target within a distance of 0.05. In order to provide sufficient time for exploration in a sparse reward setting we doubled the episode time duration to 8 seconds. Figure <ref> shows the learning curves for 25Hz and 125Hz control. Each curve is an average across 4 random seeds. The Gaussian policy fails to learn in a 125Hz control setting within a 5 hours time limit, while ARP was able to find an effective policy in 50% of the runs, and the effectiveness at higher α increased at a higher action rate. 




§ CONCLUSIONS

We introduced autoregressive Gaussian policies (ARPs) for temporally coherent exploration in continuous control deep reinforcement learning. The policy form is grounded in the theory of stationary autoregressive stochastic processes. We derived a family of stationary Gaussian autoregressive stochastic processes for an arbitrary order p with continuously adjustable degree of temporal coherence between subsequent observations. We derived an agent policy that implements these processes with a standard agent-environment interface. Empirically we showed that ARPs result in a superior exploration and learning in sparse reward tasks and perform on par or better compared to standard Gaussian policies in dense reward tasks. On physical hardware, ARPs result in smooth trajectories that are safer to execute compared to the trajectories provided by conventional Gaussian exploration. 


named
[3]#3




§ PROOF OF PROPOSITION <REF>




For any p ∈ℕ and for any α_k ∈ [0, 1), k = 1, …, p the autoregressive process

    X_t    = ∑_k = 1^p ϕ̃_k X_t-k + Z_t
    ϕ̃_k    = (-1)^k + 1∑_1 
    ≤ i_1, i_2, …, i_k ≤ pα_i_1α_i_2…α_i_k, k = 1, …, p
    
    Z_t    ∼WN(0, σ_Z^2), σ_Z^2 < ∞,
 
is stationary.
 

Consider a polynomial

    P(z) = (z - α_1)(z - α_2)…(z - α_p).

It can be written in a form

    P(z)    = z^p - ∑_k = 1^pϕ̃_k z^p-k,
    ϕ̃_k    = (-1)^k + 1∑_1 
    ≤ i_1, i_2, …, i_k ≤ pα_i_1α_i_2…α_i_k, k = 1, …, p
 
consequently P(z) is a characteristic polynomial of the process (<ref>). By design, P(z) has roots α_k, k = 1, …, p which all lie within a unit circle, therefore the process (<ref>) is stationary.




Let {X_t} be an autoregressive process defined in (<ref>).
If its white noise component Z_t is Gaussian, i.e. Z_t ∼𝒩(0, σ_Z^2), then X_t are identically distributed normal random variables with zero mean and finite variance var(X_t) = σ_X ^2 < ∞ ∀ t.


According to lemma <ref> the process (<ref>) is stationary, meaning X_t are identically distributed random variables with finite variance var(X_t) = σ_X ^2 < ∞ ∀ t. If Z_t is Gaussian, then X_t are identically distributed normal variables. Let us denote μ_X the mean of this distribution. Taking expectation of both sides of (<ref>) gives



    𝔼[X_t]    = ∑_k = 1^p ϕ̃_k 𝔼[X_t-k] + 𝔼[Z_t]  ⇒
    μ_X    = ∑_k = 1^p ϕ̃_k μ_X  ⇒
    μ_X    (1 - ∑_k = 1^p ϕ̃_k) = 0.

(1 - ∑_k = 1^p ϕ̃_k)  0 since polynomial (<ref>) does not have a root z = 1, therefore μ_X = 0.


We established that under Gaussian white noise Z_t the process (<ref>) represents a series of identically distributed normal variables with zero mean and finite variance. From the linear form of the process it is clear that the variance σ_X^2 linearly depends on the variance σ_Z^2 of a white noise component. Scaling σ_Z^2 by a factor β > 0 results in scaling σ_X^2 by the same factor β. Therefore, it should be possible to pick σ_Z such that the variance σ_X^2 would have any desired positive value, in particular, value of 1. The following lemma formalizes this observation.



For any p ∈ℕ let {ϕ_k, k = 0, …, p} be a set of coefficients corresponding to a stationary AR-p process {X_t} defined in (<ref>). Then the linear system 


    [ γ_1; γ_2; γ_3;   ⋮; γ_p ]  = 
    [     1   γ_1   γ_2     … γ_p-1;   γ_1     1   γ_1     … γ_p-2;   γ_2   γ_1     1     … γ_p-3;     ⋮     ⋮     ⋮     ⋱     ⋮; γ_p-1 γ_p-2 γ_p-3     …     1;       ][ ϕ_1; ϕ_2; ϕ_3;   ⋮; ϕ_p ]
    and   
        1 = ∑_i = 1^p ϕ_i γ_i + σ_Z^2,

has a unique solution (γ̃_1, …, γ̃_p,σ̃_Z^2), where σ̃_Z^2 > 0. Furthermore, the autoregressive process


    X̃_t    = ∑_k = 1^p ϕ_k X̃_t-k + Z̃_t,
    var(Z̃_t)   =σ̃_Z^2

is stationary with variance σ̃_X^2 = var(X̃_t) = 1. 



For any σ̃_Z^2 > 0 stationarity of {X̃_t} follows from stationarity of {X_t}, since both processes share the same coefficients {ϕ_k}, and therefore, the same characteristic polynomial. 

Since {X_t} is stationary, the corresponding system of Yule-Walker equations (<ref>) has a unique solution with respect to (γ_0, γ_1, …, γ_p) <cit.>. Notice, however, that the system (<ref>) is homogenous with respect to the variables (γ_0, γ_1, …, γ_p, σ_Z^2), meaning that if (γ_0^', γ_1^', …, γ_p^', σ_Z^' 2) is a solution, then (βγ_0^', βγ_1^', …, βγ_p^', βσ_Z^'2) is also a solution ∀β∈ℛ.
Therefore, since γ_0 = var(X_t) > 0, exists a unique solution (γ̃_0, γ̃_1, …, γ̃_p, σ̃_̃Z̃^2) such that γ̃_0 = 1. We can find it by substituting γ_0 = 1 into a linear system (<ref>), resulting in (<ref>). This solution corresponds to a stationary process {X̃_t} with var(X̃_t) = γ̃_0 = 1  ∀ t. 




Now the proof of proposition <ref> is straightforward. By lemma <ref> coefficients {ϕ̃_k} correspond to a stationary autoregressive process, therefore by lemma <ref> the system (<ref>) has a unique solution σ̃_Z^2 > 0 and the process (<ref>) is a stationary process with unit variance. Since Z_t is normal, by lemma (<ref>) the process (<ref>) is Gaussian with zero mean. Therefore, X_t ∼𝒩(0, 1)  ∀ t.






§ EXAMPLE OF AN AR-3 GAUSSIAN PROCESS


A third order process AR-3 defined by (<ref>) has a form:

    X_t    = (α_1 + α_2 + α_3)X_t-1  - (α_1 α_2 + α_2α_3 + α_1α_3)X_t-2 + α_1 α_2 α_3 X_t-3 + σ_Z 𝒩(0, 1),



where σ_Z^2 is a solution of a system:



    [ γ_1; γ_2; γ_3;     ]  = 
    [   1 γ_1 γ_2; γ_1   1 γ_1; γ_2 γ_1   1;     ][ ϕ_1; ϕ_2; ϕ_3;     ],
       ϕ_1 = α_1 + α_2 + α_3
       ϕ_2 =  - (α_1 α_2 + α_2α_3 + α_1α_3)
       ϕ_3 = α_1 α_2 α_3
    and   
        1 = ∑_i = 1^3 ϕ_i γ_i + σ_Z^2,



resulting in

    σ_Z^2 = (1 - α_1^2)(1 - α_2^2)(1 - α_3^2)(1 - α_1 α_2)(1 - α_2α_3)(1 - α_1α_3)/(1 + α_1 α_2 + α_2α_3 + α_1 α_3 - α_1  α_2  α_3  (α_1α_2α_3 + α_1 + α_2 + α_3)).


For any α_1, α_2, α_3 ∈ [0, 1) this process is stationary with X_t ∼𝒩(0, 1) ∀ t.

If α_1 = α_2 = α_3 = α, the process reduces to 


    X_t    = 3 α X_t-1  - 3 α^2 X_t-2 + α^3 X_t-3 + σ_Z 𝒩(0, 1),
    σ_Z^2    = (1 - α^2)^6/1 + 3 α^2 - 3 α^4 - α^6.





§ LEARNING RESULTS IN SIMULATION WITH ARPS AND OPENAI BASELINES TRPO


We ran a set of experiments with ARPs (p = 3) and OpenAI Baselines TRPO algorithm. Figures <ref> and <ref> show learning curves in Mujoco and Square environments respectively. The hyper-parameters are specified in the Appendix <ref>. TRPO delivered a similar performance to PPO in Mujoco tasks (with a more stable performance on a Reacher-v2 task), however PPO produced better results on a sparse reward Square environment.



§ ALGORITHM PARAMETERS


In our experiments we used the default Open AI Baselines parameters specified in <https://github.com/openai/baselines/blob/master/baselines> with the exception of γ and λ parameters, for which we used a larger value of 0.995. For experiments in Square environment at 10Hz action rate we used 4 times larger batch and optimization batch sizes to account for longer episodes. For Square and UR5 Reacher 2D environments at higher action rates we used the same parameter values as for basic versions (10Hz and 25Hz respectively), but scaled the batch and the optimization batch sizes accordingly to make sure that the data within a batch at all action rates corresponds to the same amount of simulated time (e.g. for UR5 Reacher 2D at 125Hz we used 5 times bigger batch and optimization batch compared to those in UR5 Reacher 2D at 25Hz). In all experiments we used fully connected networks with the same hidden sizes  to parametrize policy and value networks. For each experiment identical parameters and network architectures were used for standard Gaussian policy and ARP. The table below shows the parameters values for basic versions of the environments: 

PPO:



 
 Hyper-parameter     Square at 10Hz     Mujoco and UR5 Reacher 2D at 25Hz 
 
 batch size     8192      2048 

 Gray
 step-size     4 × 10^-3     4 × 10^-3 
 
 opt. batch size     256     64 

  Gray
 opt. epochs     10     10 
 
  γ     0.995     0.995 

  Gray
λ     0.995     0.995 

 clip. ε     0.2     0.2 

 Gray
 hidden layers     2     2 

 hidden sizes     64     64 

 




TRPO:


 
 Hyper-parameter     Square at 10Hz     Mujoco
 
 batch size     8192      1024

 Gray
 max-kl     0.01     0.01
 
 cg-iters     10     10

  Gray
 vf-iters     5     5
 
 vf-step-size     10^-3     10^-3

 Gray
  γ     0.995     0.995

 
λ     0.995     0.995
 
 Gray
 hidden layers     2     2

 hidden sizes     64     64

 





§ IMPLEMENTATION DETAILS


The equation (<ref>) in the main text, replicated for convenience below, defines a stationary autoregressive policy distribution in M̃^p under given parameters θ:

    π_θ(a_t|s̃_t)    = 𝒩(μ_θ(s_t) + σ_θ(s_t) f_θ(s̃_t), σ^2_θ(s_t)σ̃^2_Z I),
    
    f_θ(s̃_t)    = ∑_k = 1^p ϕ̃_k a_t-k - μ_θ(s_t-k)/σ_θ(s_t-k),
    ϕ̃_k, σ̃^2_Z     are defined by (<ref>).

The two aspects that need to be considered separately for a practical application are the initialization at the start of a roll-out and the learning updates. 

At t < p, we do not include in the model the terms ϕ̃_k a_t-k - μ_θ(s_t-k)/σ_θ(s_t-k) where t - k < 0. This corresponds to having X_t-k = 0, t-k < 0 in the underlying AR process. Initializing an AR process with zero values results in a well behaved time-series that quickly equilibrates to the stationary distribution without large spikes in values. In contrast, initializing from arbitrary values often results in temporal spikes of large values before the process equilibrates to its stationary behaviour. This restriction can be expressed in the following policy formulation:   

    π_θ(a_t|s̃_t)    = 𝒩(μ_θ(s_t) + σ_θ(s_t) f_θ(s̃_t), σ^2_θ(s_t)σ̃^2_Z I),
    
    f_θ(s̃_t)    = ∑_k = 1^min(p, t)ϕ̃_k a_t-k - μ_θ(s_t-k)/σ_θ(s_t-k),
    ϕ̃_k, σ̃^2_Z     are defined by (<ref>).

Formally, dependence on t makes the policy non-stationary, however the term t plays a role only within the first p steps in each sample path, and, as long as the exact action a_0 is not encountered again in the sample path, which is the case for continuous action spaces,  the policy could be equivalently formulated by having the agent count number of occurences of a_0 in each state s̃_t and based on that decide on the number of terms in f_θ(s̃_t). Such formulation would make the policy stationary, however it would not allow a similarly compact representation and would unnecessarily burden the formulation. Notice, that proposed formulation also means that a_0 will never be part of policy computation, and therefore will not affect future transitions.

As we pointed out in the main text, if the learning updates are performed within an episode, the change in the parameters θ can temporally distort stationarity of the underlying AR process, since the terms ϕ̃_k a_t-k - μ(s_t-k, θ_new)/σ(s_t-k, θ_new) change their values compared to ϕ̃_k a_t-k - μ(s_t-k, θ_old)/σ(s_t-k, θ_old). This may cause temporal spikes in values of f_θ(s̃_t) until the process equilibrates again. To avoid this, corresponding previously computed values of terms ϕ̃_k a_t-k - μ(s_t-k, θ_old)/σ(s_t-k, θ_old) where t - k < t_update should be used in computing f_θ for the next p steps after an update.



