∎

11institutetext:  1College of Mechanical and Electrical Engineering, Harbin
Engineering University, Harbin 150001, China: {ihuhuhu, 993740540, pangyiming,
<EMAIL_ADDRESS>22institutetext:
2College of Shipbuilding Engineering, Harbin Engineering University, Harbin
150001, China<EMAIL_ADDRESS>
# Realistic Actor-Critic: A Framework for Balance Between Value Overestimation
and Underestimation ††thanks: An early preprint is available on
arXiv(https://arxiv.org/abs/2110.09712) with CC BY: Creative Commons
Attribution License. This work has been submitted to the Springer Nature for
possible publication. Copyright may be transferred without notice, after which
this version may no longer be accessible. This work was supported in part by
the National Natural Science Foundation of China under Grant 51779059, in part
by the National Natural Science Foundation of Heilongjiang Province (Grant
No.YQ2020E028), and National Key Research and Development Project (Grant
No.2018YFF01012900). All correspondences should be sent to G. Wang with email:
<EMAIL_ADDRESS>
Sicen Li1 Qinyun Tang1 Yiming Pang1 Xinmeng Ma1 and Gang Wang*2

(Received: date / Accepted: date)

###### Abstract

This paper proposes a reinforcement learning framework to enhance the
exploration-exploitation trade-off by learning a range of policies concerning
various confidence bounds. The underestimated values provide stable updates
but suffer from inefficient exploration behaviors. On the other hand,
overestimated values can help the agent escape local optima, but it might
cause over-exploration on low-value areas and function approximation errors
accumulation. Algorithms have been proposed to mitigate the above
contradiction. However, we lack an understanding of how the value bias impact
performance and a method for efficient exploration while keeping value away
from catastrophic overestimation bias accumulation. In this paper, we 1)
highlight that both under- and overestimation bias can improve learning
efficiency, and it is a particular form of the exploration-exploitation
dilemma; 2) propose a unified framework called Realistic Actor-Critic(RAC),
which employs Universal Value Function Approximators (UVFA) to simultaneously
learn policies with different value confidence-bond with the same neural
network, each with a different under-overestimation trade-off. This allows us
to perform directed exploration without over-exploration using the upper
bounds while still avoiding overestimation using the lower bounds. Through
carefully designed experiments, We empirically verify that RAC achieves 10x
sample efficiency and 25% performance improvement compared to Soft Actor-
Critic on the most challenging Humanoid environment. All the source codes are
available at https://github.com/ihuhuhu/RAC.

###### Keywords:

Reinforcement learning(RL)estimation biasexploration-exploitation dilemma
realistic actor-critic(RAC)

††journal: International Journal of Machine Learning and Cybernetics

##  1 Introduction

Sample efficiency is one of the main challenges that prevent reinforcement
learning(RL) from applying to real-world systems where obtaining samples is
expensive Levine et al (2018); Dulac-Arnold et al (2020); Sutton and Barto
(2018). This paper aims at mitigating this problem by a better balance between
exploration and exploitation.

Undesirable overestimation bias and accumulation of function approximation
errors in temporal difference methods may lead to sub-optimal policy updates
and divergent behaviors Thrun and Schwartz (1993); Pendrith et al (1997);
Fujimoto et al (2018). Most model-free off-policy RL methods learn approximate
lower confidence bound of Q-function Lan et al (2020); Fujimoto et al (2018);
Lee et al (2021); Chen et al (2021); Kuznetsov et al (2020) to avoid
overestimation by introducing underestimation bias.

However, if the lower bound has a spurious maximum, it will discourage policy
to explore potential higher uncertain regions, resulting in stochastic local-
maximum and causing pessimistic underexploration Ciosek et al (2019).
Moreover, directionally uninformed Ciosek et al (2019) policies, such as
Gaussian policies, are unable to avoid fully explored actions, which are
wasteful.

Optimistic exploration methods Brafman and Tennenholtz (2002); Pathak et al
(2019); Kim et al (2019) learn upper confidence bounds of the Q-function from
an epistemic uncertainty estimate. These methods are directionally informed
and encourage policy to execute overestimated actions to help agents escape
local optimum. However, such upper confidence bound might cause an agent to
over-explore low-value regions. In addition, it increases the risk of value
overestimation since transitions with high uncertainty may have higher
function approximation errors to make the value overestimated.

To avoid the above problems, one has to adjust hyperparameters carefully and
control the bias to keep the value at a balance point between lower and higher
bounds: supporting stable learning while providing good exploration behaviors.
We highlight that this balance is a particular form of the exploration-
exploitation dilemma Sutton and Barto (2018). Unfortunately, most prior works
have studied the overestimation and pessimistic underexploration in isolation
and have ignored the under-overestimation trade-off aspect.

We formulate Realistic Actor-Critic (RAC), whose main idea is to learn
together values and policies with different trade-offs between underestimation
and overestimation in the same network. Policies guided by lower bounds
control overestimation bias to provide consistency and stable convergence.
Each policy guided by different upper bounds provides a unique exploration
strategy to generate overestimated actions, so that the policy family can
directionally explore overestimated state-action pairs uniformly and avoid
over-exploration. All transitions are stored in a shared replay buffer, and
all policies benefit from them to escape spurious maximum.

Such a family of policies is jointly parameterized with the Universal Value
Function Approximators (UVFA) Schaul et al (2015). The learning process can be
considered as a set of auxiliary tasks Badia et al (2020); Lyle et al (2021)
that help build shared state representations and sills.

However, learning such policies with diverse behaviors in a single network is
challenging since policies vary widely in behavior. We introduce punished
Bellman backup, which calculate uncertainty as punishment to correct value
estimations. Punished Bellman backup provide fine-granular estimation control
to make value approximation shift smoothly between upper and lower boundsy,
allowing for more efficient training. An ensemble of critics is learned to
produce well-calibrated uncertainty estimations (i.e., standard deviation) on
unseen samples Lee et al (2021); Pathak et al (2019); Amos et al (2018). We
show empirically that RAC controls the std and the mean of value estimate bias
to close to zero for most of the training. Benefit from well bias control,
critics are trained with a high Update-To-Data (UTD) ratio Chen et al (2021)
to improve sample efficiency significantly.

Empirically, we implement RAC with SAC Haarnoja et al (2018) and TD3 Fujimoto
et al (2018) in continuous control benchmarks (OpenAI Gym Brockman et al
(2016), MuJoCo Todorov et al (2012)). Results demonstrate that RAC
significantly improves the performance and sample efficiency of SAC and TD3.
RAC outperforms the current state-of-the-art algorithms (MBPO Janner et al
(2019), REDQ Chen et al (2021) and TQC Kuznetsov et al (2020)), achieving
state-of-the-art sample efficiency on the Humanoid benchmark. We perform
ablations and isolate the effect of the main components of RAC on performance.
Moreover, we perform hyperparameter ablations and demonstrate that RAC is
stable in practice.

This paper makes the following contributions:

  * (i)

using a conditional architecture with shared weights to learn a family of
policies with respect to various confidence bounds that maintain uniformly
exploration and stable updates;

  * (ii)

defining a variant of soft Bellman backup to provide fine granular bias
control to smooth value approximations that can make the learning process more
efficiently; and

  * (iii)

experimental evidence that the performance and sample efficiency of the
proposed method are better than state-of-the-art methods on continuous control
tasks.

The paper is organized as follows. Section 2 describes related works and their
results. Section 3 describes the problem setting and preliminaries of RL.
Section 4 explains the under-overestimation trade-off. Section 5 introduces
the punished Bellman backup and RAC algorithm. Section 6 presents experimental
results that show the sample efficacy and final performance of RAC. Finally,
Section 7 presents our conclusions.

##  2 Related works

Underestimation and overestimation of Q-function. The maximization update rule
in Q-learning has been shown to suffer from overestimation bias which will
seriously hinder learning Thrun and Schwartz (1993).

Minimizing the value ensemble is a standard method to deal with overestimation
bias. Clipped double Q-learning (CDQ) Fujimoto et al (2018) takes the minimum
value between a pair of critics to limit overestimation. SAC Haarnoja et al
(2018) then combined CDQ with entropy maximization to get impressive
performance in continuous control tasks. Maxmin Q-learning Lan et al (2020)
mitigated the overestimation bias by using a minimization over multiple
action-value estimates. But minimizing a Q-function set is unable to filter
out abnormally small values, which causes undesired pessimistic
underexploration problem Ciosek et al (2019). Using minimization to control
overestimation is coarse and wasteful as it ignores all estimates except the
minimal one Kuznetsov et al (2020).

REDQ Chen et al (2021) proposed in-target minimization, which used a
minimization across a random subset of Q functions from the ensemble to
alleviate the above problems. REDQ Chen et al (2021) showed their method
reduces the std of the Q-function bias to close to zero for most of the
training. Truncated Quantile Critics (TQC) Kuznetsov et al (2020) truncates
the right tail of the distributional value ensemble by dropping several of the
topmost atoms to control overestimation. Weighted bellman backup Lee et al
(2021) and uncertainty weighted actor-critic Wu et al (2021) prevents error
propagation Kumar et al (2020) in Q-learning by reweighing sample transitions
based on uncertainty estimations from the ensembles Lee et al (2021) or Monte
Carlo dropout Wu et al (2021); Srivastava et al (2014). Unlike prior works,
our work does not reweight sample transitions but directly adds uncertainty
estimations to punish the target value.

The effect of underestimation bias on learning efficiency is environment-
dependent Lan et al (2020). Therefore, it may be hard to choose the suitable
parameters to balance under- and overestimation for entirely different
environments. Our work proposed to solve this problem by learning about an
optimistic and pessimistic policy family

Ensemble methods. In deep learning, ensemble methods often used to solve the
two key issues, uncertainty estimations Wen et al (2020); Abdar et al (2021)
and out-of-distribution robustness Havasi et al (2020); Dusenberry et al
(2020); Wenzel et al (2020). In reinforcement learning, using ensemble to
enhance value function estimation was widely studied, such as, averaging a
Q-ensemble Anschel et al (2017); Peer et al (2021), bootstrapped actor-critic
architecture Kalweit and Boedecker (2017); Zheng12 et al (2018), calculate
uncertainty to reweight sample transitions Lee et al (2021), minimization over
ensemble estimates Lan et al (2020); Chen et al (2021) and update the actor
with a value ensemble Kuznetsov et al (2020); Chen et al (2021).

A high-level policy can be distilled from a policy ensemble Chen and Peng
(2019); Badia et al (2020) by density-based selection Saphal et al (2020),
selection through elimination Saphal et al (2020), choosing action that max
all Q-functions Lee et al (2021); Parker-Holder et al (2020); Jung et al
(2020), Thompson Sampling Parker-Holder et al (2020) and sliding-window UCBs
Badia et al (2020). Leveraging uncertainty estimations of the ensemble, Osband
et al (2016); Kalweit and Boedecker (2017); Zheng12 et al (2018) simulated
training different policies with a multi-head architecture independently to
generate diverse exploratory behaviors.

Ensemble methods were also used to learn joint state presentation to improve
sample efficiency. There were two main methods: multi-heads Osband et al
(2016); Kalweit and Boedecker (2017); Zheng12 et al (2018); Goyal et al (2019)
and UVFA Schaul et al (2015); Badia et al (2020, 2020). This paper uses
uncertainty estimation to reduce value overestimation bias, a simple max
operator to get the best policy, and learning joint state presentation with
UVFA.

Optimistic exploration. Pessimistic initialisation Rashid et al (2020) and
learning policy that maximizes a lower confidence bound value could suffer a
pessimistic underexploration problem Ciosek et al (2019). Optimistic
exploration is a promising solution to ease the above problem by applying the
principle of optimism in the face of uncertainty Brafman and Tennenholtz
(2002). Disagreement Pathak et al (2019) and EMI Kim et al (2019) considered
uncertainty as intrinsic motivation to encourage agents to explore the high
uncertainty areas of the environment. Uncertainty punishment proposed in this
paper can also be a particular intrinsic motivation. Different with Pathak et
al (2019); Kim et al (2019) which usually choose the weighting $\geq 0$ to
encourage exploration, punished Bellman backup using the weighting $\leq 0$ to
control value bias.

SUNRISE Lee et al (2021) proposed an optimistic exploration that chooses the
action that maximizes an upper confidence bound Chen et al (2017) of
Q-functions. OAC Ciosek et al (2019) proposed an off-policy exploration policy
that is adjusted to a linear fit of upper bounds to the critic with the
maximum Kullback-Leibler(KL) divergence constraining between the exploration
policies and the target policy. Most importantly, our work provides a unified
framework for the under-overestimation trade-off.

##  3 Problem setting and preliminaries

In this section, we describe the notations and introduce the concept of
maximum entropy RL.

###  3.1 Notation

We consider the standard reinforcement learning notation, with states
$\mathbf{s}$, actions $\mathbf{a}$, reward $r(\mathbf{s},\mathbf{a})$, and
dynamics $p(\mathbf{s^{\prime}}\mid\mathbf{s},\mathbf{a})$. The discounted
return $R_{t}={\textstyle\sum_{k=0}^{\infty}}\gamma^{k}r_{k}$ is the total
accumulated rewards from timestep $t$, $\gamma\in[0,1]$ is a discount factor
determining the priority of short-term rewards. The objective is to find the
optimal policy $\pi_{\phi}(\mathbf{s}\mid\mathbf{a})$ with parameters $\phi$,
which maximizes the expected return
$J(\phi)=\mathbb{E}_{p_{\pi}}\left[R_{t}\right]$.

###  3.2 Maximum Entropy RL

The maximum entropy objective Ziebart (2010) encourages the robustness to
noise and exploration by maximizing a weighted objective of the reward and the
policy entropy:

$\displaystyle\pi^{*}=\arg\max_{\pi}\sum_{t}\mathbb{E}_{\mathbf{s}\sim
p,\mathbf{a}\sim\pi}\left[r(\mathbf{s},\mathbf{a})+\alpha\mathcal{H}\left(\pi\left(\cdot\mid\mathbf{s}\right)\right)\right],$
(1)

where $\alpha$ is the temperature parameter used to determine the relative
importance of entropy and reward. Soft Actor-Critic(SAC) Haarnoja et al (2018)
seeks to optimize the maximum entropy objective by alternating between a soft
policy evaluation and a soft policy improvement. A parameterized soft
Q-function $Q_{\theta}\left(\mathbf{s},\mathbf{a}\right)$ , known as the
critic in actor-critic methods, is trained by minimizing the soft Bellman
backup:

$\displaystyle\mathcal{L}_{\tt
critic}(\theta)=\mathbb{E}_{\tau\sim\mathcal{B}}[\left(Q_{\theta}(\mathbf{s},\mathbf{a})-y\right)^{2}],$
(2) $\displaystyle
y=r-\gamma\mathbb{E}_{\mathbf{a^{\prime}}\sim\pi_{\phi}}\big{[}Q_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})-\alpha\log\pi_{\phi}(\mathbf{a^{\prime}}\mid\mathbf{s^{\prime}})\big{]},$
(3)

where $\tau=(\mathbf{s},\mathbf{a},r,\mathbf{s^{\prime}})$ is a transition,
$\mathcal{B}$ is a replay buffer, $\bar{\theta}$ are the delayed parameters
which is updated by exponential moving average
$\bar{\theta}\leftarrow\rho\theta+(1-\rho)\bar{\theta}$, $\rho$ is the target
smoothing coefficient, $y$ is the target value. The target value
$Q_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})$ is obtained by
using two networks
$Q_{\bar{\theta}}^{1}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})$ and
$Q_{\bar{\theta}}^{2}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})$ with minimum
operator:

$\displaystyle
Q_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})=min(Q_{\bar{\theta}}^{1}(\mathbf{s^{\prime}},\mathbf{a^{\prime}}),Q_{\bar{\theta}}^{2}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})).$
(4)

The parameterized policy $\pi_{\phi}$ , known as the actor, is updated by
minimizing the following object:

$\displaystyle\mathcal{L}_{\tt
actor}(\phi)=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},\mathbf{a}\sim\pi_{\phi}}\left[\alpha\log\left(\pi_{\phi}\left(\mathbf{a}\mid\mathbf{s}\right)\right)-Q_{\theta}\left(\mathbf{a},\mathbf{s}\right)\right].$
(5)

SAC uses an automate entropy adjusting mechanism to update $\alpha$ with
following objective:

$\displaystyle\mathcal{L}_{\tt
temp}(\alpha)=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},\mathbf{a}\sim\pi_{\phi}}\left[-\alpha\log\pi_{\phi}\left(\mathbf{a}\mid\mathbf{s}\right)-\alpha\overline{\mathcal{H}}\right],$
(6)

where $\overline{\mathcal{H}}$ is the target entropy.

##  4 Understanding under-overestimation trade-off

This section briefly discusses the estimation bias issue and empirically shows
that a better under-overestimation trade-off may improve learning performance.

###  4.1 Under-overestimation trade-off

Figure 1: The balance between value underestimation and overestimation in
actor-critic methods. The state $s$ is fixed. The graph shows $Q_{\pi}$(in
black), which is unknown to the algorithm, estimated lower bound
$\hat{Q}_{LB}$ (in blue), higher bound $\hat{Q}_{UB}$ (in red), two policies
$\pi_{LB}$ (in blue) and $\pi_{past}$(in black) at different time-steps of the
algorithm, exploration policies $\pi_{UB}$(in red) for optimistic exploration.

Under-overestimation trade-off is a special form of the exploration-
exploitation dilemma. This is illustrated in Figure 1. At first, the agent
starts with a policy $\pi_{past}$, trained with lower bound
$\hat{Q}_{LB}(\mathbf{s},\mathbf{a})$, becoming $\pi_{LB}$. We divide the
current action space into four regions:

  1. 1)

High uncertainty, low-value. Highly stochastic regions also have low values;
overestimation bias might cause an agent to over-explore a low-value area.

  2. 2)

High uncertainty, excessive errors. This region has high uncertainty but is
full of unseen transitions that can have excessive-high approximation errors,
which may cause catastrophic overestimation and need fewer samples.

  3. 3)

High uncertainty, controllable errors. This region has high uncertainty and is
closer to the $\pi_{LB}$, with controllable approximation errors, needs more
samples.

  4. 4)

Full explored. Since $\pi_{past}$ is gradually updated to $\pi_{LB}$, the area
is fully explored and needs less samples.

(a) Visit frequency of the optimal state(%)

(b) Visit frequency of low value states

(c) Q bias

Figure 2: Results of the simple MDP. In-target minimization target from REDQ
is used as $\hat{Q}_{LB}$. $\hat{Q}_{b}=mean(\hat{Q}^{\prime}_{LB})+\beta
std(\hat{Q}^{\prime}_{LB})$ can perform optimistic exploration. $\beta$ is a
key parameter to control value bias. If $\beta=0$, $\hat{Q}_{b}$ is equal to
$\hat{Q}_{LB}$. As $\beta$ increases, $\hat{Q}_{b}$ gradually approaching
$\hat{Q}_{UB}$. The horizontal axis indicates the number of time steps. Visit
frequency of the optimal state, shown in 2a, is the ratio of the frequency of
visiting the optimal state among all termination states. The higher the value,
the lower the probability that the agent is stuck in a local optimum. Visit
frequency of low value states, shown in 2b, is the ratio of the visit
frequency of low value state 2-8 and the optimal state 9. The lower the value,
the fewer steps the agent wastes in low-value states. This value has been
subtracted by seven, as the minimum step size to reach the optimum state is
seven. Q bias, shown in 2c, measures the difference between the estimated Q
values and true Q values. All results are estimated by the Monte Carlo method
and averaged over eight seeds.

To prevent catastrophic overestimation bias accumulation, SAC Haarnoja et al
(2018), TD3 Fujimoto et al (2018) and REDQ Chen et al (2021) introduce
underestimation bias to learn lower confidence bounds of Q-functions, similar
to equation 4. However, directionally uninformed policies, such as gaussian
policies, will sample actions located in region 4 with half probability. If
the lower bound has a spurious maximum and policies are directionally
uninformed Ciosek et al (2019), lower bound policy $\pi_{LB}$ may be stuck at
the junction of regions 3 and 4. This is wasteful and inefficient, causing
pessimistic underexploration.

$\pi_{UB}$, which is used in optimistic exploration methods Brafman and
Tennenholtz (2002); Pathak et al (2019); Kim et al (2019), can encourage
agents to execute overestimated actions and explore potential high-value
regions with high uncertainty. But regions with high and overestimated
actions, such as region 2, may have excessive function approximation errors.
Alternatively, if highly uncertain regions also have low values (like region
1), overestimation bias might cause an agent to over-explore a low-value
region.

Ideally, the exploration policies are located in region 3 to provide better
exploration behaviors and keep stable updates. There are two ways to achieve
this: 1) enforcing a KL constraint between $\pi_{UB}$ and $\pi_{LB}$ (like OAC
Ciosek et al (2019)), 2) balancing $\hat{Q}$ between $\hat{Q}_{LB}$ and
$\hat{Q}_{UB}$, and we call it an under-overestimation trade-off.

However, in practical applications, $Q_{\pi}$ is unknown, and it isn’t easy to
tune to ideal conditions through constant hyperparameters.

Figure 3: A simple episodic MDP Lan et al (2020), adapted from Figure 6.5 in
Sutton and Barto (2018). This MDP has two terminal states: state 9 and state
0. Every episode starts from state 1, which has two actions: Left and Right.
The MDP is deterministic. Once the agent takes into any states, the MDP will
reward back: $r=0.1$ for terminal states 0, $r=1$ for terminal states $9$ and
a reward $r\sim U(-1,1)$ for non-terminal states 1-8. State 9 is the optimal
state, state 0 is a local optimum, states 1-8 are high uncertainty and low-
value states.

###  4.2 A simple MDP

We show this effect in a simple Markov decision process(MDP), shown in Figure
3. Any state’s optimal policy is the left action. If the agent wants to go to
state 9, it must go through states 1-8 with high uncertainty and low values.

In the experiment, we used a discount factor $\gamma=0.9$; a replay buffer
with size $5000$; a boltzmann policy with $temperature=0.1$; tabular action-
values with uniform noisy respect to a Uniform distribution $U(-0.1,0.1)$,
initialized with a Uniform distribution $U(-5,5)$; and a learning rate of
$0.01$ for all algorithms.

The results in Figure 2 verify our hypotheses in Section 4.1. All algorithms
converge, but each has a different convergence speed. $\hat{Q}_{LB}$
underestimates too much and converges to a suboptimal policy in the early
learning stage, causing slow convergence. For $\beta=0.5$ and $1.0$,
optimistic exploration drives the agent to escape local optimum and learn
faster. However, $\hat{Q}$ overestimates too much for $\beta=2.0$,
significantly impairing the convergence speed of the policy. In addition, no
matter what parameter $\beta$ takes, the agent still over-explore low-value
states at different time steps(see figure 2b).

RAC avoids over-exploration in low-value states and is the fastest to converge
to the optimal policy. Furthermore, RAC maintains the Q bias close to zero
without catastrophic overestimation throughout the learning process,
indicating that RAC keeps an outstanding balance between underestimation and
overestimation.

##  5 Realistic Actor-Critic

We present Realistic Actor-Critic (RAC), which can be used in conjunction with
most modern off-policy actor-critic RL algorithms in principle, such as SAC
Haarnoja et al (2018) and TD3 Fujimoto et al (2018). We describe only the SAC
version of RAC (RAC-SAC) in the main body for the exposition. The TD3 version
of RAC (RAC-TD3) follows the same principles and is fully described in
Appendix B.

###  5.1 Punished Bellman backup

Punished Bellman backup is a variant of soft Bellman backup(2). The idea is to
maintain an ensemble of $N$ soft Q-functions
$Q_{\theta_{i}}(\mathbf{s},\mathbf{a})$, where $\theta_{i}$ denote the
parameters of the $i-th$ soft Q-function, which are initialized randomly and
independently for inducing an initial diversity in the models Osband et al
(2016), but updated with the same target.

Given a transition $\tau_{t}$, punished Bellman backup considers following
punished target $y$:

$\displaystyle
y=r_{t}+\gamma\mathbb{E}_{\mathbf{a^{\prime}}\sim\pi_{\phi}}[\bar{Q}_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})-\beta\hat{s}({Q}_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}}))-$
(7)
$\displaystyle\qquad\alpha\log\pi_{\phi}\left(\mathbf{a^{\prime}}\mid\mathbf{s^{\prime}}\right)],$

where $\bar{Q}_{\bar{\theta}}(\mathbf{s},\mathbf{a})$ is the sample mean of
the target Q-functions, $\hat{s}({Q}_{\bar{\theta}}(\mathbf{s},\mathbf{a}))$
is the sample standard deviation of target Q-functions with bessel’s
correction Warwick and Lininger (1975). Punished Bellman backup uses
$\hat{s}({Q}_{\bar{\theta}}(\mathbf{s},\mathbf{a}))$ as uncertainty estimation
to punish value estimation. $\beta\geq 0$ is the weighting of the punishment.
Note that we do not propagate gradient through the uncertainty
$\hat{s}({Q}_{\bar{\theta}}(\mathbf{s},\mathbf{a}))$.

We write $Q_{\mathbf{s}\mathbf{a}}^{i}$ instead of
$Q_{\theta_{i}}(\mathbf{s},\mathbf{a})$,
$Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{i}$ instead of
$Q_{\theta_{i}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}})$ for compactness.
Assuming each Q-function has random approximation error
$e_{\mathbf{s}\mathbf{a}}^{i}$ Thrun and Schwartz (1993); Lan et al (2020);
Chen et al (2021) which is a random variable belonging to some distribution

$\displaystyle
Q_{\mathbf{s}\mathbf{a}}^{i}=Q_{\mathbf{s}\mathbf{a}}^{*}+e_{\mathbf{s}\mathbf{a}}^{i},$
(8)

where $Q_{\mathbf{s}\mathbf{a}}^{*}$ is the ground truth of Q-functions. $M$
is the number of actions applicable at state $\mathbf{s^{\prime}}$. Define the
estimation bias $Z_{MN}$ for a transition $\tau_{t}$ to be

$\displaystyle Z_{MN}\stackrel{{\scriptstyle\text{def}}}{{=}}$
$\displaystyle\left[r+\gamma\max_{\mathbf{a}^{\prime}}(Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{mean}-\beta
Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{std})\right]-$ (9)
$\displaystyle\left(r+\gamma\max_{\mathbf{a}^{\prime}}Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{*}\right)$
$\displaystyle=$
$\displaystyle\gamma\left[\max_{\mathbf{a}^{\prime}}(Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{mean}-\beta
Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{std})-\max_{\mathbf{a}^{\prime}}Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{*}\right],$

where

$\displaystyle Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{mean}$
$\displaystyle\approx\frac{1}{N}\sum_{i=1}^{N}Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{i}=\frac{1}{N}\sum_{i=1}^{N}(Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{*}+e_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{i})$
(10)
$\displaystyle=Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{*}+\frac{1}{N}\sum_{i=1}^{N}e_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{i}=Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{*}+\bar{e}_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}},$

$\displaystyle Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{std}$
$\displaystyle\approx\sqrt{\frac{1}{N-1}\sum_{i=1}^{N}\left(Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{i}-Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{mean}\right)^{2}}$
(11)
$\displaystyle=\sqrt{\frac{1}{N-1}\sum_{i=1}^{N}\left(Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{*}+e_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{i}-Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{*}+\bar{e}_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}\right)^{2}}$
$\displaystyle=\sqrt{\frac{1}{N-1}\sum_{i=1}^{N}\left(e_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{i}-\bar{e}_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}\right)^{2}}=\hat{s}(e_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}).$

Then

$\displaystyle
Z_{MN}\approx\gamma\left[\max_{\mathbf{a}^{\prime}}(Q_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}^{*}+\bar{e}_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}-\beta\hat{s}(e_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}))-\max_{\mathbf{a}^{\prime}}Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{*}\right].$
(12)

If one could choose
$\beta=\frac{\bar{e}_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}}}{\hat{s}(e_{\mathbf{s^{\prime}}\mathbf{a^{\prime}}})}$,
$Q_{\mathbf{s}\mathbf{a}}^{i}$ will be resumed to
$Q_{\mathbf{s}\mathbf{a}}^{*}$, then $Z_{MN}$ can be reduced to near 0.
However, it’s hard to adjust a suitable constant $\beta$ for various state-
action pairs actually. We develop vanilla RAC which uses a constant $\beta$
appendix B.3 to research this problem.

Algorithm 1 RAC: SAC version

1:Initialize actor network $\phi$

2:Initialize $N$ critic networks $\theta_{i},i=1,\ldots,N$

3:Initialize temperature network $\psi$

4:Initialize empty replay buffer $\mathcal{B}$

5:Initialize target network $\bar{\theta_{i}}\longleftarrow\theta_{i}$, for
$i=1,2,\ldots,N$

6:Initialize uniform distribution $U_{1}$ and $U_{2}$

7:for each iteration do

8: // optimistic exploration

9: execute an action $a\sim\pi_{\phi}\left(\cdot\mid s,\beta\right),\beta\sim
U_{2}$.

10: Observe reward $r_{t}$, new state $s^{\prime}$

11: Store transition tuple
$\mathcal{B}\leftarrow\mathcal{B}\cup\left\\{\left(s,a,r_{t},s^{\prime}\right)\right\\}$

12: for $G$ updates do

13: // update critics via punished bellman backup

14: Sample random minibatch:

15: $\\{\tau_{j}\\}_{j=1}^{B}\sim\mathcal{B}$, $\\{\beta_{m}\\}_{m=1}^{B}\sim
U_{1}$

16: Compute the Q target (15)

17: for $i=1,\ldots,N$ do

18: Update $\theta_{i}$ by minimize $\mathcal{L}^{\tt RAC}_{\tt critic}$ (14)

19: Update target networks:

20: $\bar{\theta_{i}}\leftarrow\rho\bar{\theta_{i}}+(1-\rho)\theta_{i}$

21: // update actors and temperatures according to $U_{1}$

22: Update $\phi$ by minimize $\mathcal{L}^{\tt RAC-SAC}_{\tt actor}$ (16)

23: Update $\psi$ by minimize $\mathcal{L}^{\tt RAC}_{\tt temp}$ (13)

Shifting smoothly between higher and lower bounds. For $\beta$ = 0, the update
is simple average Q-learning which causes overestimation bias Chen et al
(2021). As $\beta$ increasing, increasingly penalties
$Q_{\mathbf{s}^{\prime}\mathbf{a}^{\prime}}^{std}$ decrease $E[Z_{MN}]$
gradually, and encourage Q-functions transit smoothly from higher-bounds to
lower-bounds.

###  5.2 Realistic Actor-Critic agent

We demonstrate how to use punished Bellman backup to incorporate various
bounds of value approximations into a full agent that maintains diverse
policies, each with a different under-overestimation trade-off. The pseudocode
for RAC-SAC is shown in Algorithm 1.

RAC use UVFA Schaul et al (2015) to extend the critic and actor as
$Q_{\theta_{i}}(\mathbf{s},\mathbf{a},\beta)$ and
$\pi_{\phi}\left(\cdot\mid\mathbf{s^{\prime}},\beta\right)$, $U_{1}$ is a
uniform traning distribution $\mathcal{U}[0,a]$, $a$ a is a positive real
number, $\beta\sim U_{1}$ that generates various bounds of value
approximations.

An independent temperature network $\alpha_{\psi}$ parameterized by $\psi$ is
used to accurately adjust the temperature with respect to $\beta$, which can
improve the performance of RAC. Then the objective (6) becomes:

$\displaystyle\mathcal{L}^{\tt RAC}_{\tt temp}(\psi)$
$\displaystyle=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},\mathbf{a}\sim\pi_{\phi},\beta\sim
U_{1}}[-\alpha_{\psi}(\beta)\log\pi_{\phi}\left(\mathbf{a}\mid\mathbf{s},\beta\right)-$
(13) $\displaystyle\quad\alpha_{\psi}(\beta)\overline{\mathcal{H}}].$

The extended Q-ensemble use punished Bellman backup to simultaneously
approximate a soft Q-function family:

$\displaystyle\mathcal{L}^{\tt RAC}_{\tt
critic}(\theta_{i})=\mathbb{E}_{\tau\sim\mathcal{B},\beta\sim
U_{1}}[\left(Q_{\theta_{i}}(\mathbf{s},\mathbf{a},\beta)-y\right)^{2}],$ (14)
$\displaystyle
y=r+\gamma\mathbb{E}_{\mathbf{a^{\prime}}\sim\pi_{\phi}}[\bar{Q}_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}},\beta)-\beta\hat{s}({Q}_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}},\beta))$
$\displaystyle\quad-\alpha_{\psi}(\beta)\log\pi_{\phi}(\mathbf{a^{\prime}}\mid\mathbf{s^{\prime}},\beta)],$
(15)

where $\bar{Q}_{\bar{\theta}}(\mathbf{s},\mathbf{a},\beta)$ is the sample mean
of target Q-functions,
$\hat{s}({Q}_{\bar{\theta}}(\mathbf{s},\mathbf{a},\beta))$ is the corrected
sample standard deviation of target Q-functions.

The extended policy $\pi_{\phi}$ is updated by minimizing the following
object:

$\displaystyle\mathcal{L}^{\tt RAC-SAC}_{\tt actor}(\phi)$
$\displaystyle=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},\beta\sim
U_{1}}[\mathbb{E}_{\mathbf{a}\sim\pi_{\phi}}[\alpha_{\psi}(\beta)\log\left(\pi_{\phi}\left(\mathbf{a}\mid\mathbf{s},\beta\right)\right)$
$\displaystyle\quad-\bar{Q}_{\theta}\left(\mathbf{a},\mathbf{s},\beta\right)]],$
(16)

where $\bar{Q}_{\theta}\left(\mathbf{a},\mathbf{s},\beta\right)$ is the sample
mean of Q-functions.

A larger UTD ratio $G$ improves sample utilization. We find that a smaller
replay buffer capacity slightly improves the sample efficiency of RAC in
Section 6.5.

Note that we find that applying different samples, which are generated by
binary masks from the Bernoulli distribution Lee et al (2021); Osband et al
(2016), to train each Q-function won’t improve RAC performance in our
experiments; therefore, RAC does not apply this method.

Figure 4: Visualization of RAC. The serial numbers in the figure correspond to
Section 4.1 and Figure 1. For better illustration, $\hat{Q}$ is discretized.
In fact, $\hat{Q}_{n}$ learned by RAC is infinite and changes continuously.
$Q^{\star}(\mathbf{s},\mathbf{a})$ is the optimal Q-funtion that is unknown.
Q-functions distributed between $\hat{Q}_{UB}$ and $\hat{Q}_{LB}$ and their
policies distributed between $\pi_{UB}$ and $\pi_{LB}$. $\pi_{UB}$, $\pi_{1}$
and $\pi_{2}$ are used as exploration policies.

RAC circumvents direct adjustment of $\beta$. RAC leaners with a distribution
of $\beta$ instead of a constant $\beta$. one could evaluate the policy family
to find the best $\beta$. We employ a discrete number $H$ of values
$\left\\{\beta_{i}\right\\}_{i=1}^{H}$ (see details in Appendix A.1) to
implement a distributed evaluation for computational efficiency, and apply the
max operator to get best $\beta$.

Optimistic exploration. When interacting with the environment, we propose to
sample $\beta$ uniformly from a uniform explore distribution
$U_{2}=\mathcal{U}[0,b]$, where $b<a$ is a positive real number, to get
optimistic exploratory behaviors to avoid pessimistic underexploration Ciosek
et al (2019).

###  5.3 How RAC solves the under-overestimation trade-off

Similar to the idea of NGU Badia et al (2020), RAC decouples exploration and
exploitation policies. RAC uses UVFA to simultaneously learn policies with the
same neural network, each with different trade-offs between underestimation
and overestimation. Using UVFA to learn different degrees of confidence bounds
allows us to learn a powerful representation and set of skills that can be
quickly transferred to the expected policy. With punished Bellman backup, RAC
has a larger number of policies and values that change smoothly, allowing for
more efficient training.

This is illustrated in Figure 4. Q-functions that are close to $\hat{Q}_{LB}$
(like $\hat{Q}_{n}$) control overestimation bias to provide consistency and
stable convergence. Exploration policies (such as $\pi_{UB}$, $\pi_{1}$ and
$\pi_{2}$) are far from the spurious maximum of $\hat{Q}_{LB}$ and
$\hat{Q}_{n}$, overestimated actions sampled from them located in regions 1,
2, and 3 lead to a quick correction to the critic estimate. All transitions
are stored in a shared replay buffer, and all policies benefit from them to
escape spurious maximums. Since exploration policies are not symmetric to the
mean of $\pi_{LB}$ and $\pi_{n}$, RAC also avoids directional uninformedness.

Although RAC cannot always keep the exploration policies located in region 3,
the policy family avoids all behaviors concentrated in regions 1 or 2.
Exploration behaviors uniformly distribute in regions 1, 2, and 3, preventing
over-exploration in any area.

Moreover, such policies could be quite different from a behavior standpoint
and generate varied action sequences to visit unseen state-action pairs
following the principle of optimism in the face of uncertainty Ciosek et al
(2019); Lee et al (2021); Chen et al (2017).

Table 1: Performance on OpenAI Gym. The maximum value for each task is bolded. ± corresponds to a single standard deviation over eight runs. The best results are indicated in bold. Results of SAC, TD3, and TQC are obtained at $6\times 10^{6}$ timesteps for Humanoid and HalfCheetah, $3\times 10^{6}$ timesteps for other environments. Results of RAC, REDQ, and TQC20 are obtained at $1\times 10^{6}$ timesteps for Humanoid and HalfCheetah and $3\times 10^{5}$ timesteps for other environments. Results of MBPO are obtained at $3\times 10^{5}$ timesteps for Ant, Humanoid, and Walker2d, $4\times 10^{5}$ for HalfCheetah, $1.25\times 10^{5}$ for Hopper. | RAC-SAC | RAC-TD3 | REDQ | MBPO | TQC20 | TD3 | SAC | TQC
---|---|---|---|---|---|---|---|---
Humanoid |  | 11107$\pm$475
---
|  9321$\pm$1126
---
|  5504$\pm$120
---
|  5162$\pm$350
---
|  7053$\pm$857
---
|  7014$\pm$643
---
|  7681$\pm$1118
---
|  10731$\pm$1296
---
Ant |  |  6283$\pm$549
---
|  6470$\pm$165
---
|  5475$\pm$890
---
|  5281$\pm$699
---
|  4722$\pm$567
---
| 6796$\pm$277
---
|  6433$\pm$332
---
|  6402$\pm$1371
---
Walker |  | 5860$\pm$440
---
|  5114$\pm$489
---
|  5034$\pm$711
---
|  4864$\pm$488
---
|  5109$\pm$696
---
|  4419$\pm$1682
---
|  5249$\pm$554
---
|  5821$\pm$457
---
Hopper |  |  3421$\pm$483
---
|  3495$\pm$672
---
| 3563$\pm$94
---
|  3280$\pm$455
---
|  3208$\pm$538
---
|  3433$\pm$321
---
|  2815$\pm$585
---
|  3011$\pm$866
---
HalfCheetah |  |  15717$\pm$1063
---
|  15083$\pm$1113
---
|  10802$\pm$1179
---
|  13477$\pm$443
---
|  12123$\pm$2600
---
|  14462$\pm$1982
---
|  16330$\pm$323
---
| 17245$\pm$293
---
Swimmer |  | 143$\pm$6.8
---
|  71$\pm$83
---
|  98$\pm$31
---
| -
---
|  143$\pm$9.6
---
|  53$\pm$8.8
---
|  51$\pm$4.2
---
|  65$\pm$5.8
---
Table 2: Sample efficiency comparison. Sample efficiency Chen et al (2021); Dorner (2021) is measured by the ratio of the number of samples collected when RAC and some algorithms reach the specified performance. The last four rows show how many times RAC is more sample efficient than other algorithms in achieving that performance. | RAC-SAC | REDQ | MBPO | TQC | TQC20 | REDQ/RAC-SAC | MBPO/RAC-SAC | TQC/RAC-SAC | TQC20/RAC-SAC
---|---|---|---|---|---|---|---|---|---
Humanoid at $2000$ |  | 63K
---
| 109K
---
| 154K
---
| 145K
---
| 147K
---
| 1.73
---
| 2.44
---
| 2.30
---
| 2.33
---
Humanoid at $5000$ |  | 134K
---
| 250K
---
| 295K
---
| 445K
---
| 258K
---
| 1.87
---
| 2.20
---
| 3.32
---
| 1.93
---
Humanoid at $10000$ |  | 552K
---
| -
---
| -
---
| 3260K
---
| -
---
| -
---
| -
---
| 5.91
---
| -
---
Ant at $1000$ |  | 21K
---
| 28K
---
| 62K
---
| 185K
---
| 42K
---
| 1.33
---
| 2.95
---
| 8.81
---
| 2.00
---
Ant at $3000$ |  | 56K
---
| 56K
---
| 152K
---
| 940K
---
| 79K
---
| 1.00
---
| 2.71
---
| 16.79
---
| 1.41
---
Ant at $6000$ |  | 248K
---
| -
---
| -
---
| 3055K
---
| -
---
| -
---
| -
---
| 12.31
---
| -
---
Walker at $1000$ |  | 27K
---
| 42K
---
| 54K
---
| 110K
---
| 50K
---
| 1.56
---
| 2.00
---
| 4.07
---
| 1.85
---
Walker at $3000$ |  | 53K
---
| 79K
---
| 86K
---
| 270K
---
| 89K
---
| 1.49
---
| 1.62
---
| 10.75
---
| 1.68
---
Walker at $5000$ |  | 147K
---
| 272K
---
| -
---
| 960K
---
| 270K
---
| 1.85
---
| -
---
| 6.53
---
| 1.84
---

(a) Humanoid

(b) Ant

(c) HalfCheetah

(d) Hopper

(e) Walker2d

(f) Swimmer

Figure 5: The learning curves on six Mujoco environments. The horizontal axis
indicates the number of time steps. The vertical axis shows the average
undiscounted return. The shaded areas denote one standard deviation over eight
runs.

##  6 Experiments

We designed our experiments to answer the following questions:

  * •

Can Realistic Actor-Critic outperform state-of-the-art algorithms in
continuous control tasks?

  * •

Can Realistic Actor-Critic better balance between value overestimation and
underestimation?

  * •

What is the contribution of each technique in Realistic Actor-Critic?

###  6.1 Setups

We implement RAC with SAC and TD3 as RAC-SAC( ) and RAC-TD3( )(see Appendix
B).

The baseline algorithms are REDQ Chen et al (2021)( ), MBPO Janner et al
(2019)( ), SAC Haarnoja et al (2018)( ), TD3 Fujimoto et al (2018)( ) and TQC
Kuznetsov et al (2020)( ). All hyper-parameters we used for evaluation are the
same as those in the original papers. For
MBPO111https://github.com/JannerM/mbpo,
REDQ222https://github.com/watchernyu/REDQ, TD3333https://github.com/sfujim/TD3
and TQC444https://github.com/SamsungLabs/tqc_pytorch, we use the authors’
code. For SAC, we implement it following Haarnoja et al (2018), and the
results we obtained are similar to previously reported results. TQC20( ) is a
variant of TQC with UTD $G=20$ for a fair comparison.

We compare baselines on six challenging continuous control tasks (Walker2d,
HalfCheetah, Hopper, Swimmer, Ant, and Humanoid) from MuJoCo environments
Todorov et al (2012) in the OpenAI gym benchmark Brockman et al (2016).

The time steps for training instances on Walker2d, Hopper, and Ant are
$3\times 10^{5}$, and $1\times 10^{6}$ for Humanoid and HalfCheetah. All
algorithms explore with a stochastic policy but use a deterministic policy for
evaluation similar to those in SAC. We report the mean and standard deviation
across eight seeds.

For all algorithms, We use a fully connected network with two hidden layers,
and 256 units per layer, with Rectified Linear Unit in each layer Glorot et al
(2011), for both actor and critic. All the parameters are updated by the Adam
optimizer Kingma and Ba (2014) with a fixed learning rate. All algorithms
adopt almost the same NN architecture and hyperparameter.

For all experiments, our learning curves show the total undiscounted return.

Using the Monte Carlo method, we estimate the mean and std of normalized
Q-function bias Chen et al (2021) as main analysis indicators to analyze the
value approximation quality (described in Appendix A). The average bias lets
us know whether $Q_{\theta}$ is overestimated or underestimated, while std
measures whether $Q_{\theta}$ is overfitting.

Sample efficiency (SE) Chen et al (2021); Dorner (2021) is measured by the
ratio of the number of samples collected when RAC and some algorithm reach the
specified performance. Hopper is not in the comparison object as the
performance of algorithms is almost indistinguishable.

(a) Mean of normalized Q bias, Ant (b) Std of normalized Q bias, Ant (c) Mean
of normalized Q bias, Humanoid (d) Std of normalized Q bias, Humanoid (e) Mean
of normalized Q bias, Walker2d (f) Std of normalized Q bias, Walker2d

Figure 6: Estimated mean and std of normalized Q bias of RAC-SAC, REDQ, TQC,
and TQC20 for Ant and Humanoid with Monte Carlo method.

(a) Performance respect to different $\beta$, Ant (b) Performance respect to
different $\beta$, Humanoid (c) Performance respect to different $\beta$,
Walker2d (d) Performance respect to different $\beta$, Hopper

Figure 7: Performance of various value confidence bounds with respect to
different $\beta$. We visualize different $\beta$ belong to traning
distribution $U_{1}=\mathcal{U}[0,a]$ during training processes.

###  6.2 Comparative evaluation

OpenAI Gym. Figure 5 and Table 1 shows learning curves and performance
comparison. RAC consistently improves the performance of SAC and TD3 across
all environments and performs better than other algorithms. In particular, RAC
learns significantly faster for Humanoid and has better asymptotic performance
for Ant, Walker2d, and HalfCheetah. RAC yields a much smaller variance than
SAC and TQC, indicating that the optimistic exploration helps the agents
escape from bad local optima.

Sample efficiency comparison. Table 2 shows the sample efficiency comparison
with baselines. Compared with TQC, RAC-SAC reaches 3000 and 6000 for Ant with
16.79x and 12.31x sample efficiency. RAC-SAC performs 1.5x better than REDQ
halfway through training and 1.8x better at the end of training in Walker and
Huamnmoid. They show that a better under overestimation trade-off can achieve
better sample-efficiency performance than the MuJoCo environments’ state-of-
the-art algorithms.

Value approximation analysis. Figure 6 presents the results for Ant, Humanoid,
and Walker2d.

In Ant and Walker2d, TQC20 has a high normalized mean of bias, indicating that
TQC20 prevents catastrophic overestimation fails accumulation. TQC20 also has
a high normalized std of bias, indicating that the bias is highly non-uniform,
which can be detrimental. Since distributional RL is prone to overfitting with
few samples, it may not be appropriate to use a high UTD ratio for TQC. In
Humanoid, which has a high-dimensional state, overfitting still exists but has
been alleviated.

Relative to TQC and TQC20, REDQ and RAC-SAC have a very low normalized std of
bias for most of the training, indicating the bias across different state-
action pairs is about the same. But the Q estimation of REDQ is too
conservative in Humanoid, and the large negative bias makes REDQ trapped in a
bad locally optimal policy, suffering from pessimistic underexploration. For
Ant and Walker2d, although this poor exploration does not harm the performance
of the policy, it still slows down convergence speed compared to RAC.

Relative to REDQ, RAC-SAC keeps the Q bias nearly zero without overestimation
accumulation; this benign overestimation bias significantly improves
performance. RAC-SAC strikes a good balance between overestimation bias (good
performance without being trapped in a bad local optima) and underestimation
bias (slight overestimation bias and consistently small std of bias).

(a) relay buffer capacity, Ant

(b) right side of $U_{1}$ ($a$), Ant

(c) right side of $U_{2}$ ($b$), Ant

(d) Update-To-Data (UTD) ratio, Ant

(e) Ensemble size, Ant

(f) replay buffer capacity, Humanoid

(g) right side of $U_{1}$ ($a$), Humanoid

(h) right side of $U_{2}$ ($b$), Humanoid

(i) Update-To-Data (UTD) ratio, Humanoid

(j) Ensemble size, Humanoid

Figure 8: Hyperparameter ablations of RAC.

(a) Humanoid

(b) Ant

(c) Walker2d

(d) Hopper

Figure 9: Performance of RAC and its variants. The in-target minimization
version of RAC is shown in Appendix B.4. RAC without policy family is named
Vanilla RAC (details in Appendix B.3. In this case, OAC Ciosek et al (2019) is
used as optimistic exploration method.)

###  6.3 Why Humanoid is hard for most baselines?

Figure 7 visualizes the performance respect to various value confidence
bounds. Humanoid is extremely sensitive to value bias. The huge state-action
space of Humanoid leads to a large approximation error of the value function
with small samples. The approximate lower bound inevitably has the spurious
maxima, while a small overestimated bias can seriously destabilize updates. It
is hard to choose appropriate confidence bound for Humanoid by tuning the
hyperparameters, resulting in a difficult under-overestimation trade-off.

Algorithms(like REDQ) that rely on constant hyperparameters to control the
value bias have to conservatively introduce a large underestimation error
(Figure 6) to stabilize updates, leading the policy to fall into pessimistic
underexploration. In contrast, other algorithms (such as TQC20) plagued by
overestimation and overfitting require more samples.

Compared to Humanoid, the state-action space of other environments(see
HalfCheetah and Swimmer in Appendix C) is much smaller. The approximate
Q-functions can easily fit the true Q values accurately, significantly
reducing the possibility of the spurious maxima. Therefore, optimistic
exploration may not be a required component for these environments. So we can
see that they are not very sensitive to various value confidence bound from
figure 7. An underestimated value is enough to guide the policy to learn
stably.

###  6.4 Variants of RAC

We evaluate the performance contributions of ingredients of RAC( )(punished
Bellman backup, policy family, optimistic exploration, independent temperature
network, and learning rate warm-up) on a subset of four environments (see
Figure 9).

Punished Bellman backup( ). When using the in-target minimization instead of
punished Bellman backup, RAC is stable, but the performance is significantly
worse in Humanoid. Punished Bellman backup provides more finer-grained bias
control than in-target minimization, reducing the difficulty of learning
representations. Compared with other environments, Humanoid has stronger
requirements for state representation learning Chen et al (2021). Thus
punished Bellman backup far outperforms in-target minimization in Humanoid and
almost the same in other environments.

Policy family( ). The policy family is paramount to performance. This is
consistent with Section 5.3’s conjecture. Even with OAC, agent can only
converge to a local optimum without the policy family in Humanoid, indicating
a single optimistic exploration method can not solve the pessimistic
underexploration well. In addition, the convergence speed of the policy has
decreased in Walker2d and Ant.

Optimistic exploration( ). Experimental results support the point in Section
6.3. Optimistic exploration can help the agent escape from local optima in
Humanoid. But in simple environments (like Ant, Walker2d and Hopper),
optimistic exploration has little impact on performance.

Independent temperature network( ). Except for Walker2d, the independent
temperature network has little effect on RAC performance. The learned
temperatures are shown in Appendix C. In practice, we find that the
independent temperature network can control the entropy of the policy more
quickly and stably.

Learning rate warm-up( ). A high UTD ratio can easily lead to an excessive
accumulation of overestimation errors in the early stage of learning. The
learning rate warm-up can alleviate this problem and stabilize the learning
process. Without the learning rate warm-up, RAC learns slower at the beginning
of the training process.

###  6.5 Hyperparameter ablations

RAC introduces some hyperparameters, 1) replay buffer capacity, 2) right side
of exploitation distribution $U_{1}$ ($a$), 3) right side of exploration
distribution $U_{2}$ ($b$), 4) Update-To-Data (UTD) ratio G in Algorithm 1, 5)
Ensemble size. Table 3 show that hyperparameters sweeps range, while Figure 8
show the numerical results.

Table 3: Ablation study Hyperparameters Sweeps Hyperparameters | Range
---|---
replay buffer capacity | $\\{5\times 10^{4},10^{5},2\times 10^{5},3\times 10^{5},10^{6}\\}$
right side of $U_{1}$ ($a$) | {0.2, 0.3, 0.4, 0.6, 0.8}
right side of $U_{2}$ ($b$) | {0.4, 0.6, 0.8, 1.0, 1.2}
Update-To-Data (UTD) ratio G | {1, 5, 10, 20}
Ensemble size | {3, 5, 8, 10}

Replay buffer capacity (Figure 8a and 8f). RAC can benefit from a smaller
capacity but will be hurt when the capacity is excessively small.

right side of $U_{1}$ ($a$) (Figure 8b and 8g). $a$ is a key hyperparameter of
RAC. Because $a$ controls the underestimation bias of RAC, which determines
the lower bound of Q-functions. The learning process becomes stable with $a$
increasing. However, if $a$ is too large, it will reduce the learning
opportunity of optimistic policies, thereby reducing the learning efficiency.

right side of $U_{2}$ ($b$) (Figure 8c and 8h). Exploration policies become
more conservative with $b$ increasing, and the performance of RAC gradually
declines. The increasing standard deviation means that more and more agents
fall into local optimal policies. But if $b$ is too small, policies may over-
explore the overestimated state, resulting in a decrease in learning
efficiency.

The ensemble size (8e and 8j) and the UTD ratio (Figure 8d, 8i). RAC appears
to benefit greatly from The ensemble size and the UTD ratio. When the ensemble
size and the UTD ratio are increased, we generally get a more stable average
bias, a lower std of bias, and stronger performance.

##  7 Conclusions

This paper presents Realistic Actor-Critic(RAC) to improve the sample
efficiency of existing off-policy actor-critic RL algorithms by effectively
balancing value under- and overestimation. The proposed agent achieves high
scores and sample efficiency in four challenging mujoco benchmarks.

This work has three main contributions: (i) a framework for involving policies
with different value confidence bonds. The exploratory nature of optimistic
policies and the stability of pessimistic policies are considered. The method
produces a way to auto-find the best value estimation degree in an unknown
environment; (ii) a method for learning Q-functions, punished Bellman backup,
provides fine-granular estimation bias control to make value approximation
smoothly shift between upper bounds and lower bounds; (iii) Experiments show
advantageous properties of RAC: low-value approximation error and brilliant
sample efficiency. Furthermore, continuous control benchmarks suggest that RAC
consistently improves performances and sample efficiency of existing off-
policy RL algorithms, such as SAC and TD3.

Our results suggest that directly incorporating uncertainty to value functions
and learning a powerful policy family can provide a promising avenue for
improved sample efficiency and performance. Further exploration of ensemble
methods, including high-level policies or more rich policy classes, is an
exciting avenue for future work.

##  8 Conflict of interest

The authors declare that there is no conflict of interest.

##  Appendix A Experiments Setups and methodology

###  A.1 Evaluation method

For all training instances, the policies are evaluated every $R_{eval}=10^{3}$
time steps. The agent fixes its policy at each evaluation phase and
deterministically interacts with the same environment, separate from obtaining
$10$ episodic rewards. The mean and standard deviation of these $10$ episodic
rewards is the performance metrics of the agent at the evaluation phase.

In the case of RAC, we employ a discrete number $H$ of values
$\left\\{\beta_{i}\right\\}_{i=1}^{H}$ to get $H$ policies:

$\displaystyle\beta_{i}=b/H\cdot i,i=1,\ldots,H.$ (17)

Each $H$ policy fixes its policy at the evaluation phase and deterministically
interacts with the environment with the fixed policy to obtain $10$ episodic
rewards. First, the $10$ episodic rewards are averaged for each policy, and
then the maximum of the $10$-episode-average rewards of the $H$ policies is
taken as the performance at that evaluation phase.

We performed this operation for $8$ different random seeds used in the
computational packages(NumPy Van Der Walt et al (2011), PyTorch Paszke et al
(2019)) and environments(OpenAI gym Brockman et al (2016)). The mean and
standard deviation of the learning curve are obtained from these $8$
simulations.

###  A.2 The normalized value bias estimation

Given a state-action pair, the normalized value bias is defined as:

$\displaystyle\left(\bar{Q}_{\theta}(\mathbf{s},\mathbf{a})-Q^{\pi}(\mathbf{s},\mathbf{a})\right)/\left|E_{\bar{\mathbf{s}},\bar{\mathbf{a}}\sim\pi}\left[Q^{\pi}(\bar{\mathbf{s}},\bar{\mathbf{a}})\right]\right|,$
(18)

where

  * •

$Q^{\pi}(\mathbf{s},\mathbf{a})$ be the action-value function for policy $\pi$
using the standard infinite-horizon discounted Monte Carlo return definition.

  * •

$\bar{Q}_{\theta}(\mathbf{s},\mathbf{a})$ the estimated Q-value, defined as
the mean of $Q_{\theta_{i}}(\mathbf{s},\mathbf{a}),i=1,\ldots,N$.

For RAC, the normalized value bias is defined as:

$\displaystyle\left(\bar{Q}_{\theta}(\mathbf{s},\mathbf{a},\beta^{*})-Q^{\pi^{*}}(\mathbf{s},\mathbf{a})\right)/\left|E_{\bar{s},\bar{a}\sim\pi^{*}}\left[Q^{\pi^{*}}(\bar{\mathbf{s}},\bar{\mathbf{a}})\right]\right|,$
(19)

where

  * •

$\pi^{*}$ is the best-performing policy in the evaluation among $H$ policies
A.1.

  * •

$Q^{\pi^{*}}(\mathbf{s},\mathbf{a})$ be the action-value function for policy
$\pi^{*}$ using the standard infinite-horizon discounted Monte Carlo return
definition.

  * •

$\bar{Q}_{\theta}(\mathbf{s},\mathbf{a},\beta^{*})$ the estimated Q-value
using $Q_{\theta}$ of $\beta^{*}$ which corresponds to the policy $\pi^{*}$,
defined as the mean of
$Q_{\theta_{i}}(\mathbf{s},\mathbf{a},\beta^{*}),i=1,\ldots,N$.

To get various target state-action pairs, we first execute the policy in the
environment to obtain 100 state-action pairs and then sample the target state-
action pair without repetition. Starting from the target state-action pair,
run the Monte Carlo processes until the max step limit is reached. Table 4
lists common parameters of the normalized value bias estimation.

Table 4: Parameters of the normalized value bias estimation Parameter | Value
---|---
| number of Monte Carlo process | 20
| number of target state-action pairs | 20
| Max step limit | 1500

##  Appendix B Hyperparameters and implementation details

We implement all RAC algorithms with Pytorch Paszke et al (2019) and use
Ray[tune] Liaw et al (2018) to build and run distributed applications. For all
the algorithms and variants, we first obtain $5000$ data points by randomly
sampling actions from the action space without making any parameter updates.
Then, to stabilize the early learning of critics, a linear learning rate warm-
up strategy is applied to critics in the start stage of training for RAC and
its variants:

$\displaystyle l=l_{init}(1-p)+p\cdot
l_{target},p=\operatorname{clip}(\frac{t-t_{start}}{t_{target}-t_{start}},0,1),$
(20)

where $l$ is current learning rate, $l_{init}$ is the initial value of
learning rate, $l_{target}$ is the target value of learning rate, $t_{start}$
is the time steps to start adjusting the learning rate, $t_{target}$ is the
time steps to arrive at $l_{target}$.

For all RAC algorithms and variants, We parameterize both the actor and
critics with feed-forward neural networks with $256$ and $256$ hidden nodes,
respectively, with rectified linear units (ReLU) Nair and Hinton (2010)
between each layer. $\beta$ is log-scaled before input into actors and
critics. In order to prevent $\beta$ sample from being zero, a small value
$\varepsilon=10^{-7}$ is added to the left side of $U_{1}$ and $U_{2}$ to be
$U_{1}[\varepsilon,a]$ and $U_{2}[\varepsilon,b]$. Weights of all networks are
initialized with Kaiming Uniform Initialization He et al (2015), and biases
are zero-initialized. We normalize actions to a range of $[-1,1]$ for all
environments.

###  B.1 RAC-SAC algorithm

Here, the policy is modeled as a Gaussian with mean and covariance given by
neural networks to handle continuous action spaces. The way RAC optimizes the
policy makes use of the reparameterization trick Kingma and Welling (2013);
Haarnoja et al (2018), in which a sample is drawn by computing a deterministic
function of the state, policy parameters, and independent noise:

$\displaystyle\tilde{\mathbf{a}}_{\phi}(\mathbf{s},\beta,\xi)=\tanh\left(\mu_{\phi}(\mathbf{s},\beta)+\sigma_{\phi}(\mathbf{s},\beta)\odot\xi\right),\quad\xi\sim\mathcal{N}(0,I).$
(21)

The actor network outputs the Gaussian’s means and log-scaled covariance, and
the log-scaled covariance is clipped in a range of $[-10,2]$ to avoid extreme
values. Then, the actions are bounded to a finite interval by applying an
invertible squashing function ($tanh$) to the Gaussian samples, and the
Squashed Gaussian Trick Haarnoja et al (2018) calculates the log-likelihood of
actions.

The temperature is parameterized by a one-layer feedforward neural network
$T_{\psi}$ of $64$ with rectified linear units (ReLU). To prevent temperature
be negative, we parameterize temperature as:

$\displaystyle\alpha_{\psi}(\beta)=e^{T_{\psi}(\operatorname{log}(\beta))+\xi},$
(22)

where $\xi$ is constant controlling the initial temperature,
$\operatorname{log}(\beta)$ is log-scaled $\beta$,
$T_{\psi}(\operatorname{log}(\beta))$ is the output of the neural network.

###  B.2 RAC-TD3 algorithm

We implement RAC-TD3 referring to https://github.com/sfujim/TD3. A final
$tanh$ unit following the output of the actor. Unlike TD3, we did not use a
target network for the actor and delayed policy updates. For each update of
critics, a small amount of random noise is added to the policy and averaged
over mini-batches:

$\displaystyle
y=r+\gamma\left[\bar{Q}_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}},\beta)-\beta\hat{s}({Q}_{\bar{\theta}}(\mathbf{s^{\prime}},\mathbf{a^{\prime}},\beta))\right],$
(23)
$\displaystyle\mathbf{a^{\prime}}=\operatorname{clip}(\pi_{\phi}\left(\cdot\mid\mathbf{s^{\prime}},\beta\right)+\epsilon,-1,1),$
(24)
$\displaystyle\epsilon\sim\operatorname{clip}(\mathcal{N}(0,\sigma),-c,c).$
(25)

The extended policy $\pi_{\phi}$ is updated by minimizing the following
object:

$\displaystyle\mathcal{L}^{\tt RAC-TD3}_{\tt
actor}(\phi)=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},\beta\sim
U_{1}}\left[-\bar{Q}_{\theta}\left(\mathbf{a},\mathbf{s},\beta\right)\right],\mathbf{a}=\pi_{\phi}\left(\cdot\mid\mathbf{s},\beta\right).$
(26)

The pseudocode for RAC-TD3 is shown in Algorithm 2.

Algorithm 2 RAC: TD3 version

1:Initialize actor network $\phi$

2:Initialize $N$ critic networks $\theta_{i},i=1,\ldots,N$

3:Initialize empty replay buffer $\mathcal{B}$

4:Initialize target network $\bar{\theta_{i}}\longleftarrow\theta_{i}$, for
$i=1,2,\ldots,N$

5:Initialize uniform distribution $\mathcal{U}_{1}$ and $U_{2}$

6:for each iteration do

7: execute an action:

8:
$\mathbf{a}=\pi_{\phi}\left(\cdot\mid\mathbf{s},\beta\right)+\epsilon,\epsilon\sim\mathcal{N}(0,\sigma),\beta\sim
U_{2}$.

9: Observe reward $r$, new state $\mathbf{s^{\prime}}$

10: Store transition tuple
$\mathcal{B}\leftarrow\mathcal{B}\cup\left\\{\left(\mathbf{s},\mathbf{a},r,\mathbf{s^{\prime}}\right)\right\\}$

11: for $G$ updates do

12: Sample random minibatch:

13: $\\{\tau_{j}\\}_{j=1}^{B}\sim\mathcal{B}$, $\\{\beta_{m}\\}_{m=1}^{B}\sim
U_{1}$

14: Compute the Q target (23)

15: for $i=1,...,N$ do

16: Update $\theta_{i}$ by minimize $\mathcal{L}^{\tt RAC}_{\tt critic}$ (14)

17: Update target networks:

18: $\bar{\theta_{i}}\leftarrow\rho\bar{\theta_{i}}+(1-\rho)\theta_{i}$

19: Update $\phi$ by minimize $\mathcal{L}^{\tt RAC-TD3}_{\tt actor}$ (26)

(a) Humanoid (b) Ant (c) Hopper (d) Walker2d

Figure 10: Performance of Vanilla RAC with different $\beta$.

(a) Humanoid (b) Ant (c) Hopper (d) Walker2d

Figure 11: Mean of normalized Q bias for Vanilla RAC with different $\beta$.

(a) Humanoid (b) Ant (c) Hopper (d) Walker2d

Figure 12: Std of normalized Q bias for Vanilla RAC with different $\beta$.

###  B.3 Vanilla RAC algorithm

UVFA is not needed for vanilla RAC as $\beta$ is a constant. The actor is
updated by minimizing the following object:

$\displaystyle\mathcal{L}^{\tt{vanillaRAC}}_{\tt
actor}(\phi)=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},\mathbf{a}\sim\pi_{\phi}}\left[\alpha\log\left(\pi_{\phi}\left(\mathbf{a}\mid\mathbf{s}\right)\right)-\bar{Q}_{\theta}\left(\mathbf{a},\mathbf{s}\right)\right].$
(27)

The pseudocode for Vanilla RAC is shown in Algorithm 3. Figture 10, 11, and 12
shows performance and Q bias of Vanilla RAC with different $\beta$.

Algorithm 3 Vanilla RAC

1:Initialize actor network $\phi$

2:Initialize $N$ critic networks $\theta_{i},i=1,\ldots,N$

3:Initialize empty replay buffer $\mathcal{B}$

4:Initialize target network $\bar{\theta_{i}}\longleftarrow\theta_{i}$, for
$i=1,2,\ldots,N$

5:for each iteration do

6: execute an action $\mathbf{a}\sim\pi_{\phi}\left(\cdot\mid s\right)$.

7: Observe reward $r$, new state $\mathbf{s^{\prime}}$

8: Store transition tuple
$\mathcal{B}\leftarrow\mathcal{B}\cup\left\\{\left(\mathbf{s},\mathbf{a},r,\mathbf{s^{\prime}}\right)\right\\}$

9: for $G$ updates do

10: Sample random minibatch $\\{\tau_{j}\\}_{j=1}^{B}\sim\mathcal{B}$

11: Compute the Q target (7)

12: for $i=1,\ldots,N$ do

13: Update $\theta_{i}$ by minimize $\mathcal{L}_{\tt critic}$ (2)

14: Update target networks:

15: $\bar{\theta_{i}}\leftarrow\rho\bar{\theta_{i}}+(1-\rho)\theta_{i}$

16: Update $\phi$ by minimize $\mathcal{L}^{\tt vanillaRAC}_{\tt actor}$ (27)

17: Update $\alpha$ by minimize $\mathcal{L}_{\tt temp}$ (6)

###  B.4 RAC with in-target minimization

We implement RAC with in-target minimization referring to authors’s code
https://github.com/watchernyu/REDQ. The critics and actor are extended as
$Q_{\theta_{i}}(\mathbf{s},\mathbf{a},k)$ and
$\pi_{\phi}\left(\cdot\mid\mathbf{s^{\prime}},k\right)$, $U_{1}$ is a uniform
traning distribution $\mathcal{U}[1,a]$, $a>1$, $k\sim U_{1}$ that determine
the size of the random subset $\mathcal{M}$. When $k$ is not an integer, the
size of $\mathcal{M}$ will be sample beweetn $\operatorname{floor}(k)$ and
$\operatorname{floor}(k+1)$ according to the Bernoulli distribution
$\mathcal{B}(p)$ with parameter $p=k-\operatorname{floor}(k)$, where
$\operatorname{floor}$ is a round-towards-zero operator.

An independent temperature network $\alpha_{\psi}$ parameterized by $\psi$ is
updated with the following object:

$\displaystyle\mathcal{L}^{\tt RAC}_{\tt temp}(\psi)$
$\displaystyle=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},\mathbf{a}\sim\pi_{\phi},k\sim
U_{1}}[-\alpha_{\psi}(k)\log\pi_{\phi}\left(\mathbf{a}\mid\mathbf{s},k\right)$
$\displaystyle\qquad-\alpha_{\psi}(k)\overline{\mathcal{H}}].$ (28)

In-target minimization is used to calculate the target $y$:

$\displaystyle y$
$\displaystyle=r+\gamma\mathbb{E}_{\mathbf{a^{\prime}}\sim\pi_{\phi}}[\min_{i\in\mathcal{M}}Q_{\bar{\theta_{i}}}\left(\mathbf{s^{\prime}},\mathbf{a^{\prime}},k\right)$
(29)
$\displaystyle\qquad-\alpha_{\psi}(k)\log\pi_{\phi}\left(\mathbf{a^{\prime}}\mid\mathbf{s^{\prime}},k\right)],$

Then each $Q_{\theta_{i}}(\mathbf{s},\mathbf{a},k)$ is updated with the same
target:

$\displaystyle\mathcal{L}^{\tt RAC}_{\tt
critic}(\theta_{i})=\mathbb{E}_{\tau_{t}\sim\mathcal{B},k\sim
U_{1}}[\left(Q_{\theta_{i}}(\mathbf{s},\mathbf{a},k)-y\right)^{2}].$ (30)

The extended policy $\pi_{\phi}$ is updated by minimizing the following
object:

$\displaystyle\mathcal{L}^{\tt RAC}_{\tt actor}(\phi)$
$\displaystyle=\mathbb{E}_{\mathbf{s}\sim\mathcal{B},k\sim
U_{1}}[\mathbb{E}_{\mathbf{a}\sim\pi_{\phi}}[\alpha_{\psi}(k)\log\left(\pi_{\phi}\left(\mathbf{a}\mid\mathbf{s},k\right)\right)$
$\displaystyle\qquad-\bar{Q}_{\theta}\left(\mathbf{a},\mathbf{s},k\right)]].$
(31)

When interacting with the environment, obtaining exploration behaviors by
sample $k$ from exploration distribution $U_{2}=\mathcal{U}[1,b]$, $a>b>1$.

The pseudocode for RAC with in-target minimization is shown in Algorithm 4.

Algorithm 4 RAC with in-target minimization

1:Initialize actor network $\phi$

2:Initialize $N$ critic networks $\theta_{i},i=1,\ldots,N$

3:Initialize temperature network $\psi$

4:Initialize empty replay buffer $\mathcal{B}$

5:Initialize target network $\bar{\theta_{i}}\longleftarrow\theta_{i}$, for
$i=1,2,\ldots,N$

6:Initialize uniform distribution $U_{1}$ and $U_{2}$

7:for each iteration do

8: execute an action
$\mathbf{a}\sim\pi_{\phi}\left(\cdot\mid\mathbf{s},k\right),k\sim U_{2}$.

9: Observe reward $r$, new state $\mathbf{s^{\prime}}$

10: Store transition tuple
$\mathcal{B}\leftarrow\mathcal{B}\cup\left\\{\left(\mathbf{s},\mathbf{a},r,\mathbf{s^{\prime}}\right)\right\\}$

11: for $G$ updates do

12: Sample random minibatch:

13: // update critics via in-target minimization

14: $\\{\tau_{j}\\}_{j=1}^{B}\sim\mathcal{B}$, $\\{k_{j}\\}_{j=1}\sim U_{1}$

15: Sample a set $\mathcal{M}$ of $k$ distinct indices from
$\\{1,2,\ldots,N\\}$

16: Compute the Q target (29)

17: for $i=1,\ldots,N$ do

18: Update $\theta_{i}$ by minimize $\mathcal{L}^{\tt RAC}_{\tt critic}$ (30)

19: Update target networks:

20: $\bar{\theta_{i}}\leftarrow\rho\bar{\theta_{i}}+(1-\rho)\theta_{i}$

21: Update $\phi$ by minimize $\mathcal{L}^{\tt RAC-SAC}_{\tt actor}$ (31)

22: Update $\psi$ by minimize $\mathcal{L}^{\tt RAC}_{\tt temp}$ (28)

Figure 13: Visualisations of learned temperatures for RAC-SAC with different $\beta$. Table 5: Environment dependent hyperparameters Hyperparameters | Humanoid | Walker | Ant | Hopper | HalfCheetah | Swimmer
---|---|---|---|---|---|---
RAC-SAC |  |  |  |  |  |
| replay buffer capacity | $3\times 10^{5}$ | $10^{5}$ | $2\times 10^{5}$ | $1\times 10^{6}$ | $2.5\times 10^{5}$ | $2\times 10^{5}$
| exploitation distribution $U_{1}$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.8]$
| exploration distribution $U_{1}$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.1]$ | $\mathcal{U}[10^{-7},0.3]$
RAC-TD3 |  |  |  |  |  |
| replay buffer capacity | $3\times 10^{5}$ | $10^{5}$ | $2\times 10^{5}$ | $1\times 10^{6}$ | $2.5\times 10^{5}$ | $2\times 10^{5}$
| exploitation distribution $U_{1}$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.8]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.8]$
| exploration distribution $U_{1}$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.3]$ | $\mathcal{U}[10^{-7},0.1]$ | $\mathcal{U}[10^{-7},0.3]$
Vanilla RAC |  |  |  |  |  |
| replay buffer capacity | $10^{6}$ | $10^{6}$ | $10^{6}$ | $10^{6}$ | $10^{6}$ | $10^{6}$
| uncertainty punishment ($\beta$) | 0.2 | 0.3 | 0.2 | 0.2 | 0.1 | 0.4
RAC-in-target |  |  |  |  |  |
| replay buffer capacity | $3\times 10^{5}$ | $10^{5}$ | $2\times 10^{5}$ | $1\times 10^{6}$ | $1\times 10^{6}$ | $1\times 10^{6}$
Table 6: Specific Hyperparameters Hyperparameters | Value
---|---
RAC-SAC |
| initial temperature coefficient ($\xi$) | $-5$
RAC-TD3 |
| exploration noisy | $\mathcal{N}(0,0.1)$
| policy noisy ($\sigma$) | $0.2$
| policy noisy clip ($c$) | $0.5$
Vanilla RAC |
| initial temperature | $\exp(-3)$
| uncertainty punishment ($\beta$) | 0.3
RAC with in-target minimization |
| initial temperature coefficient ($\xi$) | $-5$
| exploitation distribution $U_{1}$ | $\mathcal{U}[1,1.5]$
| exploration distribution $U_{2}$ | $\mathcal{U}[1,2.0]$
Table 7: Shared hyperparameters Hyperparameters | Value
---|---
| optimizer |  Adam Kingma and Ba (2014)
| actor learning rate | $3\times 10^{-4}$
| temperature learning rate | $3\times 10^{-4}$
|  initial critic learning rate ($l_{init}$) | $3\times 10^{-5}$
|  target critic learning rate ($l_{target}$) | $3\times 10^{-4}$
|  time steps to start learning rate adjusting ($t_{start}$) | 5000
|  time steps to reach target learning rate ($t_{target}$) | $10^{4}$
|  number of hidden layers (for $\phi$ and $\theta_{i}$) | 2
|  number of hidden units per layer (for $\phi$ and $\theta_{i}$) | 256
|  number of hidden layers (for $T_{\psi}$) | 1
|  number of hidden units per layer (for $T_{\psi}$) | 64
|  discount ($\gamma$) | 0.99
| nonlinearity | ReLU
| evaluation frequence | $10^{3}$
| minibatch size | 256
|  target smoothing coefficient ($\rho$) | 0.005
|  Update-To-Data (UTD) ratio ($G$) | 20
|  ensemble size ($N$) | 10
| number of evaluation episodes | 10
| initial random time steps | 5000
| frequence of delayed policy updates | 1
| log-scaled covariance clip range | $[-10,2]$
|  number of discrete policies for evaluation ($H$) | 12

(a) Performance respect to different $\beta$, HalfCheetah (b) Performance
respect to different $\beta$, Swimmer

Figure 14: Performance of various value confidence bounds for different
$\beta$ for RAC-SAC. RAC leaners with a distribution of $\beta$ instead of a
constant $\beta$. We visualize different $\beta$ belong to traning
distribution $U_{1}=\mathcal{U}[0,a]$ during training processes.

###  B.5 Hyperparameter setting

Table 7 and 6 lists the hyperparameters for RAC and variants used in
experiments.

##  Appendix C Visualisations

learned temperatures. The Figure 13 shows the visualization of learned
temperatures concerning different $\beta$ during training. The figure
demonstrates that learned temperatures are quite different. It is challenging
to consider the temperature of different $\beta$ with a single temperature.

Performance of various value confidence bounds. The Figure 14 shows the
performance of various value confidence bounds for different $\beta$ for RAC-
SAC on Half-Cheetah.

## Acknowledgment

The authors are grateful to the Editor-in-Chief, the Associate Editor, and
anonymous reviewers for their valuable comments.

## References

  * Levine et al (2018) Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research 37(4-5):421–436
  * Dulac-Arnold et al (2020) Dulac-Arnold G, Levine N, Mankowitz DJ, Li J, Paduraru C, Gowal S, Hester T (2020) An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:200311881
  * Sutton and Barto (2018) Sutton RS, Barto AG (2018) Reinforcement learning: An introduction. MIT press
  * Thrun and Schwartz (1993) Thrun S, Schwartz A (1993) Issues in using function approximation for reinforcement learning. In: Proceedings of the Fourth Connectionist Models Summer School, Hillsdale, NJ, pp 255–263
  * Pendrith et al (1997) Pendrith MD, Ryan MR, et al (1997) Estimator variance in reinforcement learning: Theoretical problems and practical solutions. University of New South Wales, School of Computer Science and Engineering
  * Fujimoto et al (2018) Fujimoto S, Hoof H, Meger D (2018) Addressing function approximation error in actor-critic methods. In: International Conference on Machine Learning, PMLR, pp 1587–1596
  * Lan et al (2020) Lan Q, Pan Y, Fyshe A, White M (2020) Maxmin q-learning: Controlling the estimation bias of q-learning. arXiv preprint arXiv:200206487
  * Lee et al (2021) Lee K, Laskin M, Srinivas A, Abbeel P (2021) Sunrise: A simple unified framework for ensemble learning in deep reinforcement learning. In: International Conference on Machine Learning, PMLR, pp 6131–6141
  * Chen et al (2021) Chen X, Wang C, Zhou Z, Ross K (2021) Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:210105982
  * Kuznetsov et al (2020) Kuznetsov A, Shvechikov P, Grishin A, Vetrov D (2020) Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In: International Conference on Machine Learning, PMLR, pp 5556–5566
  * Ciosek et al (2019) Ciosek K, Vuong Q, Loftin R, Hofmann K (2019) Better exploration with optimistic actor critic. Advances in Neural Information Processing Systems 32
  * Brafman and Tennenholtz (2002) Brafman RI, Tennenholtz M (2002) R-max-a general polynomial time algorithm for near-optimal reinforcement learning. Journal of Machine Learning Research 3(Oct):213–231
  * Pathak et al (2019) Pathak D, Gandhi D, Gupta A (2019) Self-supervised exploration via disagreement. In: International Conference on Machine Learning, PMLR, pp 5062–5071
  * Kim et al (2019) Kim H, Kim J, Jeong Y, Levine S, Song HO (2019) Emi: Exploration with mutual information. In: International Conference on Machine Learning, PMLR, pp 3360–3369
  * Schaul et al (2015) Schaul T, Horgan D, Gregor K, Silver D (2015) Universal value function approximators. In: International conference on machine learning, PMLR, pp 1312–1320
  * Badia et al (2020) Badia AP, Sprechmann P, Vitvitskyi A, Guo D, Piot B, Kapturowski S, Tieleman O, Arjovsky M, Pritzel A, Bolt A, et al (2020) Never give up: Learning directed exploration strategies. arXiv preprint arXiv:200206038
  * Lyle et al (2021) Lyle C, Rowland M, Ostrovski G, Dabney W (2021) On the effect of auxiliary tasks on representation dynamics. In: International Conference on Artificial Intelligence and Statistics, PMLR, pp 1–9
  * Amos et al (2018) Amos B, Dinh L, Cabi S, Rothörl T, Colmenarejo SG, Muldal A, Erez T, Tassa Y, de Freitas N, Denil M (2018) Learning awareness models. arXiv preprint arXiv:180406318
  * Haarnoja et al (2018) Haarnoja T, Zhou A, Hartikainen K, Tucker G, Ha S, Tan J, Kumar V, Zhu H, Gupta A, Abbeel P, et al (2018) Soft actor-critic algorithms and applications. arXiv preprint arXiv:181205905
  * Brockman et al (2016) Brockman G, Cheung V, Pettersson L, Schneider J, Schulman J, Tang J, Zaremba W (2016) Openai gym. arXiv preprint arXiv:160601540
  * Todorov et al (2012) Todorov E, Erez T, Tassa Y (2012) Mujoco: A physics engine for model-based control. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, pp 5026–5033
  * Janner et al (2019) Janner M, Fu J, Zhang M, Levine S (2019) When to trust your model: Model-based policy optimization. Advances in Neural Information Processing Systems 32
  * Wu et al (2021) Wu Y, Zhai S, Srivastava N, Susskind J, Zhang J, Salakhutdinov R, Goh H (2021) Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:210508140
  * Kumar et al (2020) Kumar A, Gupta A, Levine S (2020) Discor: Corrective feedback in reinforcement learning via distribution correction. Advances in Neural Information Processing Systems 33:18,560–18,572
  * Srivastava et al (2014) Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1):1929–1958
  * Wen et al (2020) Wen Y, Tran D, Ba J (2020) Batchensemble: an alternative approach to efficient ensemble and lifelong learning. arXiv preprint arXiv:200206715
  * Abdar et al (2021) Abdar M, Pourpanah F, Hussain S, Rezazadegan D, Liu L, Ghavamzadeh M, Fieguth P, Cao X, Khosravi A, Acharya UR, et al (2021) A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information Fusion
  * Havasi et al (2020) Havasi M, Jenatton R, Fort S, Liu JZ, Snoek J, Lakshminarayanan B, Dai AM, Tran D (2020) Training independent subnetworks for robust prediction. arXiv preprint arXiv:201006610
  * Dusenberry et al (2020) Dusenberry M, Jerfel G, Wen Y, Ma Y, Snoek J, Heller K, Lakshminarayanan B, Tran D (2020) Efficient and scalable bayesian neural nets with rank-1 factors. In: International conference on machine learning, PMLR, pp 2782–2792
  * Wenzel et al (2020) Wenzel F, Snoek J, Tran D, Jenatton R (2020) Hyperparameter ensembles for robustness and uncertainty quantification. Advances in Neural Information Processing Systems 33:6514–6527
  * Anschel et al (2017) Anschel O, Baram N, Shimkin N (2017) Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In: International Conference on Machine Learning, PMLR, pp 176–185
  * Peer et al (2021) Peer O, Tessler C, Merlis N, Meir R (2021) Ensemble bootstrapping for q-learning. arXiv preprint arXiv:210300445
  * Kalweit and Boedecker (2017) Kalweit G, Boedecker J (2017) Uncertainty-driven imagination for continuous deep reinforcement learning. In: Conference on Robot Learning, PMLR, pp 195–206
  * Zheng12 et al (2018) Zheng12 Z, Yuan C, Lin12 Z, Cheng12 Y (2018) Self-adaptive double bootstrapped ddpg. International Joint Conference on Artificial Intelligence
  * Chen and Peng (2019) Chen G, Peng Y (2019) Off-policy actor-critic in an ensemble: Achieving maximum general entropy and effective environment exploration in deep reinforcement learning. arXiv preprint arXiv:190205551
  * Badia et al (2020) Badia AP, Piot B, Kapturowski S, Sprechmann P, Vitvitskyi A, Guo ZD, Blundell C (2020) Agent57: Outperforming the atari human benchmark. In: International Conference on Machine Learning, PMLR, pp 507–517
  * Saphal et al (2020) Saphal R, Ravindran B, Mudigere D, Avancha S, Kaul B (2020) Seerl: Sample efficient ensemble reinforcement learning. arXiv preprint arXiv:200105209
  * Parker-Holder et al (2020) Parker-Holder J, Pacchiano A, Choromanski KM, Roberts SJ (2020) Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems 33:18,050–18,062
  * Jung et al (2020) Jung W, Park G, Sung Y (2020) Population-guided parallel policy search for reinforcement learning. arXiv preprint arXiv:200102907
  * Osband et al (2016) Osband I, Blundell C, Pritzel A, Van Roy B (2016) Deep exploration via bootstrapped dqn. Advances in neural information processing systems 29
  * Goyal et al (2019) Goyal A, Sodhani S, Binas J, Peng XB, Levine S, Bengio Y (2019) Reinforcement learning with competitive ensembles of information-constrained primitives. arXiv preprint arXiv:190610667
  * Rashid et al (2020) Rashid T, Peng B, Böhmer W, Whiteson S (2020) Optimistic exploration even with a pessimistic initialisation. In: International Conference on Learning Representations (ICLR)
  * Chen et al (2017) Chen RY, Sidor S, Abbeel P, Schulman J (2017) Ucb exploration via q-ensembles. arXiv preprint arXiv:170601502
  * Ziebart (2010) Ziebart BD (2010) Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University
  * Warwick and Lininger (1975) Warwick DP, Lininger CA (1975) The sample survey: Theory and practice. McGraw-Hill
  * Dorner (2021) Dorner FE (2021) Measuring progress in deep reinforcement learning sample efficiency. arXiv preprint arXiv:210204881
  * Glorot et al (2011) Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics, JMLR Workshop and Conference Proceedings, pp 315–323
  * Kingma and Ba (2014) Kingma DP, Ba J (2014) Adam: A method for stochastic optimization. arXiv preprint arXiv:14126980
  * Van Der Walt et al (2011) Van Der Walt S, Colbert SC, Varoquaux G (2011) The numpy array: a structure for efficient numerical computation. Computing in science & engineering 13(2):22–30
  * Paszke et al (2019) Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, et al (2019) Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32
  * Liaw et al (2018) Liaw R, Liang E, Nishihara R, Moritz P, Gonzalez JE, Stoica I (2018) Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:180705118
  * Nair and Hinton (2010) Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Icml
  * He et al (2015) He K, Zhang X, Ren S, Sun J (2015) Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision, pp 1026–1034
  * Kingma and Welling (2013) Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:13126114