Philip J. Ball (ball@robots.ox.ac.uk), Laura Smith and Ilya Kostrikov
({smithlaura<EMAIL_ADDRESS>
# Efficient Online Reinforcement Learning with Offline Data

Philip J. Ball  Laura Smith  Ilya Kostrikov  Sergey Levine  UC Berkeley

###### Abstract

Sample efficiency and exploration remain major challenges in online
reinforcement learning (RL). A powerful approach that can be applied to
address these issues is the inclusion of offline data, such as prior
trajectories from a human expert or a sub-optimal exploration policy. Previous
methods have relied on extensive modifications and additional complexity to
ensure the effective use of this data. Instead, we ask: _can we simply apply
existing off-policy methods to leverage offline data when learning online?_ In
this work, we demonstrate that the answer is yes; however, a set of minimal
but important changes to existing off-policy RL algorithms are required to
achieve reliable performance. We extensively ablate these design choices,
demonstrating the key factors that most affect performance, and arrive at a
set of recommendations that practitioners can readily apply, whether their
data comprise a small number of expert demonstrations or large volumes of sub-
optimal trajectories. We see that correct application of these simple
recommendations can provide a $\mathbf{2.5\times}$ improvement over existing
approaches across a diverse set of competitive benchmarks, with no additional
computational overhead.

##  1 Introduction

Deep reinforcement learning (RL) has achieved success in a number of complex
domains, such as Atari [Mnih et al., 2015] and Go [Silver et al., 2016], as
well as real-world applications like Chip Design [Mirhoseini et al., 2021] and
Human Preference Alignment [Ouyang et al., 2022]. In many of these settings,
strong RL performance is predicated on having large amounts of online
interaction with an environment, which is usually made feasible through the
use of simulators. In real-world problems, however, we are often confronted
with scenarios where samples are expensive, and furthermore, rewards are
sparse, often exacerbated by high dimensional state and action spaces.

Figure 1: Our approach, RLPD, extends standard off-policy RL and achieves
reliable state-of-the-art online performance on a number of tasks using
offline data. Here we show the difficult D4RL AntMaze domain (10 seeds, 1 std.
shaded), averaged over all 6 tasks. We run RLPD for 300k steps due to early
convergence.

One promising way to resolve this issue is via the inclusion of data generated
by a previous policy or human expert when training deep RL algorithms (often
referred to as _offline data_ [Levine et al., 2020]), as evidenced
theoretically [Wagenmaker and Pacchiano, 2022, Song et al., 2023] and in real-
world examples by Cabi et al. [2019], Nair et al. [2020], Lu et al. [2021].
This can alleviate challenges due to sample efficiency and exploration by
providing the algorithm with an initial dataset to “kick-start" the learning
process, either in the form of high-quality expert demonstrations, or even
low-quality but high-coverage exploratory trajectories. This also provides us
an with avenue to leverage large pre-collected datasets in order to learn
useful policies.

Some prior work has focused on using this data through pre-training, while
other approaches introduce constraints when training online to handle issues
with distribution shift. However, each approach has its drawbacks, such as
additional training time and hyperparamters, or limited improvement beyond the
behavior policy respectively. Taking a step back, we note that _standard off-
policy algorithms_ should be able to take advantage of this offline data, and
furthermore issues with distribution shift should be alleviated in this
setting, as we can explore the environment online. Thus far however, such
methods have seen limited success in this problem setting. Therefore, in this
work, we ask the following question: _can we simply apply existing off-policy
methods to leverage offline data when learning online, without offline RL
pretraining or explicit imitation terms that privilege the prior offline
data?_

Through a set of thorough experiments on a collection of widely studied
benchmarks, we show that the answer to this question is yes. However, naïvely
applying existing online off-policy RL algorithms can result in comparatively
poor performance, as we see in Figure 1 comparing ‘SAC + Offline Data’ with
‘IQL + Finetuning’. Instead, a minimal set of key _design choices_ must be
taken into consideration to ensure their success. Concretely, we first
introduce a remarkably simple approach to sampling the offline data, which we
call “symmetric sampling", that performs well over a large variety of domains
with no hyperparameter tuning. Then, we see that in complex settings (e.g.,
sparse reward, low volume of offline data, high dimensionality, etc.), it is
vital that value functions are prevented from over-extrapolation. To this end,
we provide a novel perspective on how Layer Normalization Ba et al. [2016]
implicitly prevents catastrophic value over-extrapolation, thereby greatly
improving sample-efficiency and stability in many scenarios, while being a
minimal modification to existing approaches. Then, to improve the rate at
which the offline data are utilized, we incorporate and compare the latest
advances in sample-efficient model-free RL, and find that large ensembles are
remarkably effective across a variety of domains. Finally, we identify and
provide evidence that key design choices in recent RL literature are in fact
environment sensitive, showing the surprising result that environments which
share similar properties in fact require entirely different choices, and
recommend a _workflow for practitioners_ to accelerate their application of
our insights to new domains.

We demonstrate that our final approach, which we call RLPD (Reinforcement
Learning with Prior Data) outperforms previously reported results, as we see
in Figure 1, on many competitive domains, sometimes by $\mathbf{2.5\times}$.
Crucially, as our changes are minimal, we maintain the attractive properties
of online algorithms, such as ease of implementation and computational
efficiency. Furthermore, we see the generality of our approach, which achieves
strong performance across a number of diverse offline datasets, from those
containing limited expert demonstrations, through to data comprised of high-
volume sub-optimal trajectories.

We believe that our insights are valuable to the community and practitioners.
We show that online off-policy RL algorithms can be remarkably effective at
learning with offline data. However, we show their reliable performance is
predicated on several key design choices, namely the way the offline data are
sampled, a crucial way of normalizing the critic update, and using large
ensembles to improve sample efficiency. While the individual ingredients of
RLPD are refreshingly simple modifications on existing RL components, we show
that their combination delivers state-of-the-art performance on a number of
popular online RL with offline data benchmarks, exceeds the performance of
significantly more complex prior methods, and generalizes to a number of
different types of offline data, whether it be expert demonstrations or sub-
optimal trajectories. We have released RLPD here: github.com/ikostrikov/rlpd.

##  2 Related work

#### Offline RL pre-training.

We note connections to offline RL Ernst et al. [2005], Fujimoto et al. [2019],
Levine et al. [2020]; many prior works perform offline RL, followed by online
fine-tuning [Hester et al., 2018, Kalashnikov et al., 2018, Nair et al., 2020,
Lee et al., 2021, Kostrikov et al., 2022]. Notably, Lee et al. [2021] also
considers large ensembles and multiple gradient-step per timestep regimes when
learning online. However, our approach uses a significantly simpler sampling
mechanism with no hyperparameters and does not rely on costly offline pre-
training, which introduces yet _additional_ hyperparameters. We also emphasize
that our normalized update is _not an offline RL method_ —we do not perform
any offline pre-training but run online RL from scratch with offline data
included in a replay buffer.

#### Constraining to prior data.

An alternative to the offline RL pre-training paradigm is to explicitly
constrain the online agent updates such that it exhibits behavior that
resembles the offline data [Levine and Koltun, 2013, Fox et al., 2016, Hester
et al., 2018, Nair et al., 2018a, Rajeswaran et al., 2018, Rudner et al.,
2021]. Particularly relevant to our approach is work by Rajeswaran et al.
[2018], which augments a policy gradient update with a weighted update that
explicitly includes _demonstration_ data. In contrast, we use a sample-
efficient off-policy paradigm, and _do not_ perform any pre-training. Also
similar to our work is that by Nair et al. [2018a], who also use an off-policy
algorithm with a fixed offline replay buffer. However, we do not restrict the
policy using a behavior cloning term, and do not reset to demonstration
states. Moreover, we note that these approaches generally require the offline
data to be high quality (i.e., ‘learning from demonstration data’ [Asada and
Hanafusa, 1979, Schaal, 1996]), while our approach is, importantly, agnostic
to the quality of the data.

#### Unconstrained methods with prior data.

Prior work has also considered ways of incorporating offline data without any
constraints. Some methods focus on initializing a replay buffer with offline
data [Večerík et al., 2017, Hester et al., 2018], while other works have
utilized a balanced sampling strategy to handle online and offline data [Nair
et al., 2018b, Kalashnikov et al., 2018, Hansen et al., 2022]. Most recently,
Song et al. [2023] presented a theoretical analysis of such approaches,
showing that balanced sampling is important both in theory and practice. In
our experiments, we also show that balanced sampling helps online RL with
offline data; however, directly using this approach on a range of benchmark
tasks is insufficient, and other design decisions that we present are critical
in achieving good performance across all tasks.

##  3 Preliminaries

We consider problems that can be formulated as a Markov Decision Process (MDP)
[Bellman, 1957], described as a tuple
$(\mathcal{S},\mathcal{A},\gamma,p,r,d_{0})$ where $\mathcal{S}$ is the state
space, $\mathcal{A}$ is the action space and $\gamma\in(0,1)$ is the discount
factor. The dynamics are governed by a transition function
$p(s^{\prime}|s,a)$; there is a reward function $r(s,a)$ and initial state
distribution $d_{0}(s)$. The goal of RL is then to maximize the expected sum
of discounted rewards:
$\mathbb{E}_{\pi}\left[\sum_{t=1}^{\infty}\gamma^{t}r(s_{t},a_{t})\right].$

In this work, we focus on RL while having access to offline datasets
$\mathcal{D}$ Levine et al. [2020], a collection of $(s,a,r,s^{\prime})$
tuples generated from a particular MDP. A key property of offline datasets is
they usually do not provide complete state-action coverage, i.e.,
$\\{s,a\in\mathcal{D}\\}$ is a small subset of $\mathcal{S}\times\mathcal{A}$.
Due to this lack of on-policy coverage, methods using function approximation
may over-extrapolate values when learning on this data, leading to a
pronounced effect on learning performance [Fujimoto et al., 2019].

##  4 Online RL with Offline Data

As outlined in Section 3, we consider the standard RL setting with the
addition of a pre-collected dataset. In this work, we aim to design a general
approach that is agnostic to the _quality and quantity_ of this pre-collected
data. For instance, this data could take the form of a handful of human
demonstrations, or swathes of sub-optimal, exploratory data. Furthermore, we
wish to make recommendations that are agnostic to the nature of the problem
setting, such as whether the observations are state or pixel-based, or whether
the rewards are sparse or dense.

To this end, we present an approach based on off-policy model-free RL, without
pre-training or explicit constraints, which we call RLPD (Reinforcement
Learning with Prior Data). As discussed in Section 4.5, we base our algorithm
design on SAC [Haarnoja et al., 2018a, b], though in principle similar design
choices may improve other off-policy RL approaches. First, we propose a simple
mechanism for incorporating the prior data. Then, we identify a pathology that
exists when naïvely applying off-policy methods to this problem setting, and
propose a simple and minimally invasive solution. After, we improve the rate
the offline data are utilized by incorporating the latest approaches in
sample-efficient RL. Finally, we highlight common design choices in recent
deep RL that are in fact environment sensitive, and should be adjusted
accordingly by practitioners.

###  4.1 Design Choice 1: A Simple and Efficient Strategy to Incorporate
Offline Data

We start with a simple approach that incorporates prior data which adds no
computational overhead, yet is agnostic to the nature of the offline data. We
call this ‘symmetric sampling’, whereby for each batch we sample 50% of the
data from our replay buffer, and the _remaining 50% from the offline data
buffer_ , resembling the scheme used by Ross and Bagnell [2012]. As we will
see in later sections, this sampling strategy is surprisingly effective across
a variety of scenarios, and we extensively ablate various elements of this
scheme (see Section 5.1). However, applying this approach to canonical off-
policy methods, such as SAC [Haarnoja et al., 2018a], does not yield strong
performance, as we see in Figure 1, and further design choices must be taken
into consideration.

###  4.2 Design Choice 2: Layer Normalization Mitigates Catastrophic
Overestimation

Standard off-policy RL algorithms query the learned Q-function for out-of-
distribution (OOD) actions, which might not be defined during learning.
Consequently, there can be significant overestimation of actual values due to
the use of function approximation [Thrun and Schwartz, 1993]. In practice,
this phenomenon leads to training instabilities and possible _divergence_ when
the critic is trying the catch up with a constantly increasing value.

Figure 2: Using SAC with our symmetric sampling method can result in
instabilities due to diverging Q-values; with LayerNorm in the critic this
disappears, improving performance.

In particular, we find this to be the case when naïvely applying our symmetric
sampling approach for complex tasks (see Figure 2). Critic divergence is a
well-studied problem, particularly in the _offline_ regime, where the policy
cannot generate new experience. In our problem setting, however, we _can_
sample from the environment. Therefore, instead of creating a mechanism that
explicitly discourages OOD actions, which can be viewed as anti-exploration
[Rezaeifar et al., 2022], we instead need to simply ensure that the learned
functions do not extrapolate in an unconstrained manner. To this end, we show
that Layer Normalization (LayerNorm) [Ba et al., 2016] can bound the
extrapolation of networks but, crucially, _does not_ explicitly constrain the
policy to remain close to the offline data. This in turn does not discourage
the policy from exploring unknown and _potentially valuable_ regions of the
state-action space. In particular, we demonstrate that LayerNorm bounds the
values and empirically prevents catastrophic value extrapolation. Concretely,
consider a Q-function $Q$ parameterized by $\theta,w$, applying LayerNorm and
intermediate representation $\psi_{\theta}(\cdot,\cdot)$. For any $a$ and $s$
we can say111For simplicity, we consider LayerNorm without bias terms. This
does not change the analysis, as it is a constant.:

$\displaystyle\|Q_{\theta,w}(s,a)\|$
$\displaystyle=\|w^{T}\text{relu}(\psi_{\theta}(s,a))\|$
$\displaystyle\leq\|w\|\|\text{relu}(\psi_{\theta}(s,a))\|\leq\|w\|\|\psi(s,a)\|$
$\displaystyle\leq\|w\|$

Therefore, the Q-values of a Layer Normalized neural network are bounded by
the norm of the weight layer, including actions outside the dataset. Thus, the
effect of erroneous action extrapolation is greatly mitigated, as their
Q-values are unlikely to be significantly greater than those already seen in
the data. Indeed, referring back to Figure 2, we see introducing LayerNorm
into the critic greatly improves performance through mitigating critic
divergence.

To illustrate this, we generate a dataset with inputs $x$ distributed in a
circle with radius $0.5$ and labels $y=\|x\|$. We study how a standard two-
layer MLP with ReLU activations (common in deep RL) extrapolates outside of
the data distribution, and the effect of adding LayerNorm. In Figure 3, the
standard parameterization leads to unbounded extrapolation outside of the
support, while LayerNorm bounds the values, greatly reducing the effect of
uncontrolled extrapolation.

###  4.3 Design Choice 3: Sample Efficient RL

We now have an online approach leveraging offline data that also suppresses
extreme value extrapolation, whilst maintaining the freedom of an
unconstrained off-policy method. However, a benefit of offline and constrained
approaches is that they have an explicit mechanism to efficiently incorporate
prior data, such as through pre-training [Hester et al., 2018, Lee et al.,
2021], or an auxiliary supervision term [Nair et al., 2018a, Rudner et al.,
2021] respectively. In our case, the incorporation of prior data is implicit
through the use of online Bellman backups over offline transitions. Therefore,
it is imperative that these Bellman backups are performed as sample-
efficiently as possible.

Figure 3: We fit data (left) with a two-layer MLP without LayerNorm (center)
and with LayerNorm (right). LayerNorm bounds the values and prevents
catastrophic overestimation.

One way to achieve this is to increase the number of updates we perform per
environment step (also referred to as update-to-data (UTD) ratio), allowing
the offline data to become “backed-up" more quickly. However, as highlighted
in recent literature in online RL, this can create issues in the optimization
process and ironically _reduce sample efficiency_ , due to statistical over-
fitting [Li et al., 2022]. To ameliorate this, prior work has suggested a
number of regularization approaches, such as simple L2 normalization [Večerík
et al., 2017], Dropout [Srivastava et al., 2014, Hiraoka et al., 2022] and
random ensemble distillation [Chen et al., 2021]. In this work, we settle on
the latter approach of random ensemble distillation; we will demonstrate
through ablations that this performs strongest, particularly on sparse reward
tasks.

We also note that value over-fitting issues exist when performing TD-learning
from images [Cetin et al., 2022]. Therefore in these settings, we further
include random shift augmentations [Kostrikov et al., 2021, Yarats et al.,
2022].

###  4.4 Per-Environment Design Choices

Having highlighted the 3 key design choices for our approach that can be
applied generally to all environments and offline datasets, we now to turn our
attention to design choices that are commonly taken for granted, but can in
fact be environment-sensitive. It is well documented that deep RL algorithms
are sensitive to implementation details [Henderson et al., 2018, Engstrom et
al., 2020, Andrychowicz et al., 2021, Furuta et al., 2021]. As a result, many
works in deep RL require per-environment hyperparameter tuning. Given the huge
variety of tasks we consider in our experiments, we believe it is important to
contribute to this discourse, and highlight that certain design choices, which
are often simply inherited from previous implementations, should in fact be
_carefully reconsidered_ , and may explain why off-policy methods have not
been competitive thus far on the problems we consider. We therefore take a
view that, given the well-documented sensitivity of deep RL, it is important
to demonstrate a critical path of design choices to consider when assessing
new environments, and provide a _workflow_ to simplify this process for
practitioners.

#### Clipped Double Q-Learning (CDQ).

Value-based methods combined with function approximation and stochastic
optimization suffer from estimation uncertainty, which, in combination with
the maximization objective of Q-learning, leads to value overestimation van
Hasselt et al. [2016] (as also discussed in Section 4.2). In order to mitigate
this issue, Fujimoto et al. [2018] introduce Clipped Double Q-Learning (CDQ)
which involves taking a minimum of an ensemble of two Q-functions for
computing TD-backups. They define the targets for updating the critics as
follows:

$y=r(s,a)+\gamma\min_{i=1,2}Q_{\theta_{i}}(s^{\prime},a^{\prime})\mbox{ where
}a^{\prime}\sim\pi(\cdot|s^{\prime}).$

However, this corresponds to fitting target Q-values that are 1 std. below the
actual target values, and recent work [Moskovitz et al., 2021] suggests that
this design choice may not be universally useful as it can be too
conservative. Therefore, this is important to reconsider, especially outside
of the domains for which it was originally designed, such as sparse reward
tasks prevalent in our problem setting.

#### Maximum Entropy RL.

MaxEnt RL augments the traditional return objective with an entropy term:

$\max_{\pi}\mathbb{E}_{s\sim\rho^{\pi},a\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}(r_{t}+\alpha\mathcal{H}(\pi(a|s)))\right].$

This corresponds to maximizing the discounted reward and expected policy
entropy at each time step. The motivation for these approaches is centered
around robustness and exploration (i.e., maximize reward while behaving as
randomly as possible). Approaches relying on this objective are empirically
impressive [Haarnoja et al., 2018a, b, Chen et al., 2021, Hiraoka et al.,
2022]. We, therefore, believe this design choice is of interest in the context
of online fine-tuning, where rewards are often sparse and require exploration.

#### Architecture.

Network architecture can have a significant impact on deep RL performance, and
the same architecture that can be optimal in one environment can be sub-
optimal in another Furuta et al. [2021]. To simplify the search space, we
consider the impact of having 2 or 3 layers in the actor and critic, which
have been shown to affect performance, even on canonical tasks [Ball and
Roberts, 2021].

###  4.5 RLPD: Approach Overview

Here we present pseudo-code for our approach, highlighting in Green elements
that are important to our approach, and in Purple, environment-specific design
choices.

The key factors of RLPD reside in lines 1 and 13 of Algorithm 1 with adopting
LayerNorm, large ensembles, sample efficient learning and a symmetric sampling
approach to incorporate online and offline data. For environment specific
choices, we recommend the following as a starting point:

  * •

Line 3: Subset 2 critics

  * •

Line 16: Remove entropy

  * •

Line 1: Utilize a _deeper_ 3 layer MLP

As a pragmatic workflow, we recommend ablating these Purple design choices
first, and in the order stated above.

Algorithm 1 Online RL with Offline Data (RLPD)

1: Select LayerNorm, Large Ensemble Size $E$, Gradient Steps $G$, and
architecture.

2: Randomly initialize Critic $\theta_{i}$ (set targets
$\theta_{i}^{\prime}=\theta_{i}$) for $i=1,2,\dots,E$ and Actor $\phi$
parameters. Select discount $\gamma$, temperature $\alpha$ and critic EMA
weight $\rho$.

3: Determine number of Critic targets to subset $Z\in\\{1,2\\}$

4: Initialize empty replay buffer $\mathcal{R}$

5: Initialize buffer $\mathcal{D}$ with offline data

6: while True do

7: Receive initial observation state $s_{0}$

8: for t = 0, T do

9: Take action $a_{t}\sim\pi_{\phi}(\cdot|s_{t})$

10: Store transition $(s_{t},a_{t},r_{t},s_{t+1})$ in $\mathcal{R}$

11: for $g=1,G$ do

12: Sample minibatch $b_{R}$ of $\frac{N}{2}$ from $\mathcal{R}$

13: Sample minibatch $b_{D}$ of $\frac{N}{2}$ from $\mathcal{D}$

14: Combine $b_{R}$ and $b_{D}$ to form batch $b$ of size $N$

15: Sample set $\mathcal{Z}$ of $Z$ indices from $\\{1,2,\dots,E\\}$

16: With $b$, set
$y=r+\gamma\left(\min_{i\in\mathcal{Z}}Q_{\theta_{i}^{\prime}}(s^{\prime},\tilde{a}^{\prime})\right),\quad\tilde{a}^{\prime}\sim\pi_{\phi}(\cdot|s^{\prime})$

17: Add entropy term $y=y+\alpha\log\pi_{\phi}(\tilde{a}^{\prime}|s^{\prime})$

18: for $i=1,E$ do

19: Update $\theta_{i}$ minimizing loss:
$L=\frac{1}{N}\sum_{i}(y-Q_{\theta_{i}}(s,a))^{2}$

20: Update target networks
$\theta_{i}^{\prime}\leftarrow\rho\theta_{i}^{\prime}+(1-\rho)\theta_{i}$

21: With $b$, update $\phi$ maximizing objective:
$\frac{1}{E}\sum_{i=1}^{E}Q_{\theta_{i}}(s,\tilde{a})-\alpha\log\pi_{\phi}(\tilde{a}|s),\quad\tilde{a}\sim\pi_{\phi}(\cdot\mid
s)$

Figure 4: RLPD exceeds prior state-of-the-art performance on a number of
different popular benchmarks whilst being significantly simpler. Results are
aggregated over 21 different environments (10 Seeds, 1 std. shaded). In each
case, we compare to the prior best known work (IQL + Finetuning in Adroit and
AntMaze, Off2On in Locomotion), and SACfD, a canonical off-policy approach
using offline data.

##  5 Experiments

We design our experiments to not only demonstrate the importance of our design
choices, but also provide the insights that allow practitioners to quickly
adapt RLPD to their problems. As such, we aim to answer the following
questions:

  1. 1.

Is RLPD competitive with prior work despite using _no pretraining nor having
explicit constraints_?

  2. 2.

Does RLPD transfer to _pixel-based_ environments?

  3. 3.

Does LayerNorm _mitigate value divergence_?

  4. 4.

Does the proposed workflow around environment-specific design choices lead to
_reliable performance_?

For (1), we compare RLPD to those which have been designed to use offline data
to accelerate online learning. For (2), we consider an additional suite of
tasks to study RLPD’s applicability to vision-based domains. Then, for (3) we
perform analysis to demonstrate the importance of using LayerNorm. Lastly, for
(4) we demonstrate our proposed workflow on the most challenging tasks (see
Subsection 4.5).

#### How does RLPD compare?

We consider the following 21 tasks from established benchmarks:

  * •

Sparse Adroit [Nair et al., 2020]. These 3 dexterous manipulation tasks—pen-
spinning, door-opening, ball relocation—are challenging, sparse-reward tasks.
The offline data are multi-modal, with a small set of human demonstrations and
a large set of trajectories from a behavior-cloned policy trained on this
human data. We follow the _rigorous evaluation criteria_ of Kostrikov et al.
[2022], whereby performance is based on completion speed, rather than success
rate. IQL + Finetuning represents the strongest prior work [Kostrikov et al.,
2022].

  * •

D4RL AntMaze [Fu et al., 2020]. These 6 sparse reward tasks require an Ant
agent to learn to walk and navigate through a maze to reach a goal. The
offline data comprise _only sub-optimal_ trajectories that can in principle be
“stitched" together. Again, IQL + Fine-tuning represents the strongest prior
work [Kostrikov et al., 2022].

  * •

D4RL Locomotion [Fu et al., 2020]. Lastly, we have 12 dense reward, locomotion
tasks featuring offline data with varying levels of expertise. Off2On [Lee et
al., 2021] has state-of-the-art performance on this suite of tasks.

For evaluation, we first include a baseline studied in prior work [Večerík et
al., 2017, Nair et al., 2020] we refer to here as SACfD, which similarly to
RLPD is an off-policy approach that incorporates offline data during training.
SACfD however simply initializes the online replay buffer with the offline
data. We implement this baseline using SAC without the additional design
decisions disucssed in subsection 4.5. Then, since there is no _single_ prior
method that achieves the best performance across all groups of tasks, we
compare to the state-of-the-art method specific to each group (as listed
above). We refer to this comparison in our plots as Prior SoTA. For all
experiments in this work, we report the mean and standard deviation across _10
seeds_ and aggregate the results across tasks within the three groups listed.
For full detailed results broken down by task, see Appendix A, and for more
environment details, see Section B.1.

We see that RLPD performs very well, either matching or _significantly
exceeding_ the best known prior work on these challenging benchmarks(see
Figure 4). We reiterate that we present results for our method and SACfD
without doing any pre-training, unlike the Prior SoTA methods. So, while prior
work (shown in blue) may achieve strong initial performance, the online
improvement is more modest. On the other hand, our method reaches or surpasses
that performance in just on the order of 100k online samples. Notably, we
outperform the best reported performance on the Sparse Adroit ‘Door’ task by
$\mathbf{2.5\times}$. Moreover, to our knowledge, our method is the first to
effectively ‘solve’ the AntMaze tasks. Furthermore, we are able to do so in
_less than a third_ the time-step budget allocated to prior methods.

#### Does RLPD transfer to pixels?

Here we consider the medium and expert locomotion tasks V-D4RL, an offline
dataset Lu et al. [2022] with _only pixel observations_ 222V-D4RL normalized
return is episode return divided by 10.. V-D4RL is particularly challenging as
the behavior policies are state-based, which means the data is partially
observable in pixel-space. This is most obvious in ‘Humanoid Walk’, where body
parts are visually occluded, thus behavior cloning (BC) can struggle to
achieve strong performance. For evaluation, as our method seeks to accelerate
online learning with offline data, we focus on data-efficiency—we introduce a
challenge that we call “10%DMC", which involves training policies using only
10% of the total timesteps recommended by Yarats et al. [2022]. As we use the
medium and expert data, we include a BC baseline. To evaluate RLPD’s ability
to efficiently use offline data to boost online learning, we compare to a
baseline approach that does not use the offline data. To isolate the effect of
the utilization of offline data, we use the same architecture and policy
optimizer as our method and label this baseline as Online in our plots. Then,
to evaluate how this compares to the state-of-the-art sample-efficient RL
method from pixels, we compare to DrQ-v2 [Yarats et al., 2022].

Figure 5: Our approach generalizes to vision-based domains, providing
consistent improvements over existing approaches.

We see in Figure 5 that RLPD provides consistent improvements over purely
online approaches, and in many cases greatly improves over a BC baseline. We
see through the difference in RLPD (dashed black and purple lines) and the
Online baseline that RLPD _clearly_ effectively utilizes the offline data to
bootstrap learning. And this conclusion holds as we compare additionally to
the SoTA vision-based RL method (blue).

Figure 6: Increasing UTD with RLPD greatly improves sample efficiency from
pixels.

Lastly, we test increasing UTD to 10 on one of the tasks—Cheetah Run with
Expert offline data. Figure 6 shows a remarkable improvement in performance
when learning with the offline dataset. To our knowledge, this is the first
demonstration of a high UTD approach improving model-free pixel-based
continuous control.

###  5.1 RLPD Analysis and Ablation Study

Here, we address (3) and (4) by quantifying the effect of LayerNorm, and
demonstrating the reliability of our proposed workflow (see Subsection 4.5).

#### Does LayerNorm mitigate value divergence?

We now illustrate the importance of LayerNorm in mitigating catastrophic value
divergence, and focus on tasks where overestimation is particularly prone. In
Figure 7, we see LayerNorm is crucial for strong performance in the Adroit
domain; excluding LayerNorm results in significantly higher variance across
seeds and reduces mean performance.

To more clearly illustrate this effect, we construct a dataset of _only the
expert human demonstration data_ from the Adroit Sparse tasks (see "Expert
Adroit Sparse Tasks" in Figure 7). This subset comprises just 22 of the 500
trajectories in the original dataset and is much more narrowly distributed by
nature—representing a task with sparse rewards, limited demonstrations, and
narrow offline data coverage—likely to exacerbate value divergence. Here we
see a remarkable result: RLPD still _exceeds_ prior work, despite significant
restrictions in data. Moreover, removing LayerNorm now results in collapsed
performance, with no progress made on any task. We further observe
improvements in sample efficiency through the inclusion of LayerNorm in
AntMaze and Humanoid Walk through reducing excessive extrapolation. Additional
experiments and results are in Appendix A.

Figure 7: LayerNorm is crucial for strong performance, particularly when data
are limited or narrowly distributed.

#### Design choice workflow.

We now motivate our workflow in Section 4.5, demonstrating the importance of
these design choices. We focus on the hardest tasks, namely ‘Relocate’ in
Adroit Sparse, ‘Large Diverse’ in AntMaze, and ‘Humanoid Expert’ in V-D4RL, as
we found these to be most sensitive. We provide full results on all tasks in
Appendix A, noting that optimal decisions in the harder domains also produce
optimal results in the easier domains.

Figure 8: Our recommended starting design choices and workflow leads to strong
performance on all tasks.

We see in Figure 8 that with the recommended environment-specific design
choices provide strong performance; we see using entropy backups and smaller
networks always results in worse performance. In ‘AntMaze Large Diverse’
however we see that with CDQ, performance deteriorates. Following our
workflow, and ablating this by subsetting 1 critic is crucial to recovering
strong performance. The same applies to ‘Humanoid Walk Expert’, whereby we
surprisingly see that CDQ is detrimental to performance, despite its
popularity in recent implementations. We also show the surprising positive
effect of larger ensembles in pixel-based tasks, with the standard 2 member
critic ensemble performing worse than the 10 member ensemble we use by default
in RLPD.

Lastly, we conduct additional ablations to understand the importance of the
individual design choices that we propose to combine for our method, showing
that, though the individual design decisions are simple, they are important
for good performance.

#### Critic regularization.

We now examine the effects of critic regularization on performance. We compare
3 approaches: weight-decay333We found a weight decay value of 0.01 worked best
across a number of settings. [Večerík et al., 2017], DropOut [Hiraoka et al.,
2022] and ensembling [Chen et al., 2021]. We choose a subset of the prior
experiments to evaluate on from the AntMaze, Adroit, and Locomotion sections.

Figure 9: In general, critic ensembling provides the best performance. Dropout
performs worse in sparse reward tasks.

In Figure 9, ensembling is the strongest form of regularization. We see that
Dropout performs well in the Locomotion domain ‘walker2d-medium-v0’, as
affirmed by Hiraoka et al. [2022], but does not generalize to challenging
sparse reward environments. We also see that weight-decay regularization is
less performant in all domains.

#### Buffer initialization.

In this section, we compare symmetric sampling with that of one which relies
on _initializing_ a replay buffer with offline data.

Figure 10: Symmetric sampling improves sample efficiency and reduces variance
across seeds, and does not work by simply increasing the reward density in a
batch.
Figure 11: Initializing the buffer with large amounts of data limits
improvement.

First returning to the challenging Human Expert Demonstrations setting in
Adroit, in Figure 10 we show two contrasting examples that demonstrate
symmetric sampling effectively trades-off between replay and offline data. We
observe that in the Pen environment, symmetric sampling clearly improves
exploration by ensuring increasing reward density within mini-batches. In
contrast, we see that although the buffer initialization approach explores
just as well in the Door environment, symmetric sampling has the important
effect of improving stability and decreases variance due to relying less on
the higher-variance data generated by the online policy.

We now consider a situation whereby we have abundant sub-optimal offline data,
and see that again, our balanced sampling approach improves sample-efficiency.
In Figure 11, initializating the buffer with high volumes of medium quality
locomotion data gives initial improvement over online performance, but
struggles asymptotically, likely due to a lack of on-policy data sampling,
vital for online improvement. On the other hand, our symmetric sampling
approach improves sample efficiency and matches asymptotic performance while
reducing variance.

#### Sampling proportion sensitivity.

We assess the sensitivity of our sampling approach away away from the
“symmetric" ratio of 50% online/offline.

Figure 12: RLPD is not sensitive to replay proportion; 50% offers the best
compromise between variance, speed of convergence, and asymptotic performance.

As we see in Figure 12, RLPD is not very sensitive to sampling proportion.
While sampling 25% offline can marginally help asymptotic performance in
‘walker2d-medium-v0’, this comes at the expense of variance and sample
efficiency in sparse reward tasks. We also affirm with our 100% offline
results that RLPD is not an offline method, and key to its success is how it
controls for divergence _without restricting exploration or behavior
learning_.

##  6 Conclusion

In this work, we show that off-policy approaches can be adapted to leverage
offline data when training online. We see that with careful application of key
design choices, RLPD can attain remarkably strong performance, and demonstrate
this on a total of 30 different tasks. Concretely, we show that the unique
combination of symmetric sampling, LayerNorm as a value extrapolation
regularizer and sample efficient learning is key to its success, resulting in
our outperforming prior work by up to $\mathbf{2.5\times}$ on a large variety
of competitive benchmarks. Moreover, our recommendations have negligible
impact on computational efficiency compared to pure-online approaches, and are
simple, allowing practitioners to easily incorporate the insights of this work
into existing approaches. To further improve adoptability, we recommend and
demonstrate a workflow for practitioners that improves performance on a wide
variety of tasks, and thus demonstrate that certain canonical design choices
should be reconsidered when applying off-policy methods. Finally, to
facilitate future research, we have released the RLPD codebase here, which
features highly optimized off-policy algorithms for proprioceptive and pixel-
based tasks in a single codebase written in JAX [Bradbury et al., 2018].

## Acknowledgments

PJB is funded through the Willowgrove Foundation and the Les Woods Memorial
Fund. This research was partly supported by the DARPA RACER program, ARO
W911NF-21-1-0097, the Office of Naval Research under N00014-21-1-2838 and
N00014-19-12042. This research used the Savio computational cluster resource
provided by the Berkeley Research Computing program at the University of
California, Berkeley (supported by the UC Berkeley Chancellor, Vice Chancellor
for Research, and Chief Information Officer).

## References

  * Andrychowicz et al. [2021] Marcin Andrychowicz, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem.  What matters for on-policy deep actor-critic methods? a large-scale study.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=nIAxjsniDzg.
  * Asada and Hanafusa [1979] Haruhiko Asada and Hideo Hanafusa.  Playback control of force teachable robots.  _Transactions of the Society of Instrument and Control Engineers_ , 15(3):410–411, 1979.  10.9746/sicetr1965.15.410.
  * Ba et al. [2016] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization, 2016.  URL https://arxiv.org/abs/1607.06450.
  * Ball and Roberts [2021] Philip J. Ball and Stephen J. Roberts.  Offcon3: What is state of the art anyway?, 2021.  URL https://arxiv.org/abs/2101.11331.
  * Bellman [1957] Richard Bellman.  A markovian decision process.  _Indiana Univ. Math. J._ , 6:679–684, 1957.  ISSN 0022-2518.
  * Bradbury et al. [2018] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang.  JAX: composable transformations of Python+NumPy programs, 2018\.  URL http://github.com/google/jax.
  * Cabi et al. [2019] Serkan Cabi, Sergio Gomez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott E. Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden, Mel Vecerík, Oleg O. Sushkov, David Barker, Jonathan Scholz, Misha Denil, Nando de Freitas, and Ziyun Wang.  Scaling data-driven robotics with reward sketching and batch reinforcement learning.  _Robotics: Science and Systems XVI_ , 2019.
  * Cetin et al. [2022] Edoardo Cetin, Philip J Ball, Stephen Roberts, and Oya Celiktutan.  Stabilizing off-policy deep reinforcement learning from pixels.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, _Proceedings of the 39th International Conference on Machine Learning_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 2784–2810. PMLR, 17–23 Jul 2022.  URL https://proceedings.mlr.press/v162/cetin22a.html.
  * Chen et al. [2021] Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross.  Randomized ensembled double q-learning: Learning fast without a model.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=AY8zfZm0tDd.
  * Engstrom et al. [2020] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry.  Implementation matters in deep rl: A case study on ppo and trpo.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=r1etN1rtPB.
  * Ernst et al. [2005] Damien Ernst, Pierre Geurts, and Louis Wehenkel.  Tree-based batch mode reinforcement learning.  _Journal of Machine Learning Research_ , 6(18):503–556, 2005.  URL http://jmlr.org/papers/v6/ernst05a.html.
  * Fox et al. [2016] Roy Fox, Ari Pakman, and Naftali Tishby.  Taming the noise in reinforcement learning via soft updates.  In _32nd Conference on Uncertainty in Artificial Intelligence (UAI)_ , 2016.
  * Fu et al. [2020] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine.  D4rl: Datasets for deep data-driven reinforcement learning, 2020.
  * Fujimoto et al. [2018] Scott Fujimoto, Herke van Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  _ArXiv_ , abs/1802.09477, 2018.
  * Fujimoto et al. [2019] Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pages 2052–2062. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/fujimoto19a.html.
  * Furuta et al. [2021] Hiroki Furuta, Tadashi Kozuno, Tatsuya Matsushima, Yutaka Matsuo, and Shixiang Gu.  Co-adaptation of algorithmic and implementational innovations in inference-based deep reinforcement learning.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=vLyI__SoeAe.
  * Haarnoja et al. [2018a] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _International Conference on Machine Learning (ICML)_ , 2018a.
  * Haarnoja et al. [2018b] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine.  Soft actor-critic algorithms and applications, 2018b.  URL https://arxiv.org/abs/1812.05905.
  * Hansen et al. [2022] Nicklas Hansen, Yixin Lin, Hao Su, Xiaolong Wang, Vikash Kumar, and Aravind Rajeswaran.  Modem: Accelerating visual model-based reinforcement learning with demonstrations.  _arXiv preprint_ , 2022.
  * Henderson et al. [2018] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger.  Deep reinforcement learning that matters.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), Apr. 2018.  10.1609/aaai.v32i1.11694.  URL https://ojs.aaai.org/index.php/AAAI/article/view/11694.
  * Hester et al. [2018] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John Agapiou, Joel Leibo, and Audrunas Gruslys.  Deep q-learning from demonstrations.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), Apr. 2018.  10.1609/aaai.v32i1.11757.  URL https://ojs.aaai.org/index.php/AAAI/article/view/11757.
  * Hiraoka et al. [2022] Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka.  Dropout q-functions for doubly efficient reinforcement learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=xCVJMsPv3RT.
  * Kalashnikov et al. [2018] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine.  Scalable deep reinforcement learning for vision-based robotic manipulation.  In Aude Billard, Anca Dragan, Jan Peters, and Jun Morimoto, editors, _Proceedings of The 2nd Conference on Robot Learning_ , volume 87 of _Proceedings of Machine Learning Research_ , pages 651–673. PMLR, 29–31 Oct 2018.  URL https://proceedings.mlr.press/v87/kalashnikov18a.html.
  * Kostrikov et al. [2021] Ilya Kostrikov, Denis Yarats, and Rob Fergus.  Image augmentation is all you need: Regularizing deep reinforcement learning from pixels.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=GY6-6sTvGaf.
  * Kostrikov et al. [2022] Ilya Kostrikov, Ashvin Nair, and Sergey Levine.  Offline reinforcement learning with implicit q-learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=68n2s9ZJWF8.
  * Lee et al. [2021] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin.  Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.  In _5th Annual Conference on Robot Learning_ , 2021.  URL https://openreview.net/forum?id=AlJXhEI6J5W.
  * Levine and Koltun [2013] Sergey Levine and Vladlen Koltun.  Guided policy search.  In Sanjoy Dasgupta and David McAllester, editors, _Proceedings of the 30th International Conference on Machine Learning_ , volume 28 of _Proceedings of Machine Learning Research_ , pages 1–9, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.  URL https://proceedings.mlr.press/v28/levine13.html.
  * Levine et al. [2020] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.  URL https://arxiv.org/abs/2005.01643.
  * Li et al. [2022] Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine.  Efficient deep reinforcement learning requires regulating statistical overfitting.  In _Deep Reinforcement Learning Workshop NeurIPS 2022_ , 2022.  URL https://openreview.net/forum?id=Jwfa-oyQduy.
  * Lu et al. [2022] Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh.  Challenges and opportunities in offline reinforcement learning from visual observations.  In _Workshop on Learning from Diverse, Offline Data_ , 2022.  URL https://openreview.net/forum?id=bPOBIKaqLba.
  * Lu et al. [2021] Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, and Sergey Levine.  Aw-opt: Learning robotic skills with imitation andreinforcement at scale.  In _5th Annual Conference on Robot Learning_ , 2021.
  * Mirhoseini et al. [2021] Azalia Mirhoseini, Anna Goldie, Mustafa Yazgan, Joe Wenjie Jiang, Ebrahim M. Songhori, Shen Wang, Young-Joon Lee, Eric Johnson, Omkar Pathak, Azade Nazi, Jiwoo Pak, Andy Tong, Kavya Srinivasa, Will Hang, Emre Tuncer, Quoc V. Le, James Laudon, Richard Ho, Roger Carpenter, and Jeff Dean.  A graph placement methodology for fast chip design.  _Nature_ , 594 7862:207–212, 2021.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, February 2015.  ISSN 00280836.  URL http://dx.doi.org/10.1038/nature14236.
  * Moskovitz et al. [2021] Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, and Michael Jordan.  Tactical optimism and pessimism for deep reinforcement learning.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 12849–12863. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/6abcc8f24321d1eb8c95855eab78ee95-Paper.pdf.
  * Nair et al. [2018a] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel.  Overcoming exploration in reinforcement learning with demonstrations.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , page 6292–6299. IEEE Press, 2018a.  10.1109/ICRA.2018.8463162.  URL https://doi.org/10.1109/ICRA.2018.8463162.
  * Nair et al. [2018b] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel.  Overcoming exploration in reinforcement learning with demonstrations.  In _2018 IEEE international conference on robotics and automation (ICRA)_ , pages 6292–6299. IEEE, 2018b.
  * Nair et al. [2020] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine.  AWAC: Accelerating online reinforcement learning with offline datasets.  _ArXiv e-prints_ , June 2020.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=TG8KACxEON.
  * Rajeswaran et al. [2018] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine.  Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations.  In _Proceedings of Robotics: Science and Systems (RSS)_ , 2018.
  * Rezaeifar et al. [2022] Shideh Rezaeifar, Robert Dadashi, Nino Vieillard, Léonard Hussenot, Olivier Bachem, Olivier Pietquin, and Matthieu Geist.  Offline reinforcement learning as anti-exploration.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(7):8106–8114, Jun. 2022.  10.1609/aaai.v36i7.20783.  URL https://ojs.aaai.org/index.php/AAAI/article/view/20783.
  * Ross and Bagnell [2012] Stéphane Ross and J. Andrew Bagnell.  Agnostic system identification for model-based reinforcement learning.  In _Proceedings of the 29th International Coference on International Conference on Machine Learning_ , ICML’12, page 1905–1912, Madison, WI, USA, 2012. Omnipress.  ISBN 9781450312851.
  * Rudner et al. [2021] Tim G. J. Rudner, Cong Lu, Michael Osborne, Yarin Gal, and Yee Whye Teh.  On pathologies in KL-regularized reinforcement learning from expert demonstrations.  In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , 2021.  URL https://openreview.net/forum?id=sS8rRmgAatA.
  * Schaal [1996] Stefan Schaal.  Learning from demonstration.  In M.C. Mozer, M. Jordan, and T. Petsche, editors, _Advances in Neural Information Processing Systems_ , volume 9. MIT Press, 1996.  URL https://proceedings.neurips.cc/paper/1996/file/68d13cf26c4b4f4f932e3eff990093ba-Paper.pdf.
  * Silver et al. [2016] David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis.  Mastering the game of Go with deep neural networks and tree search.  _Nature_ , 529(7587):484–489, January 2016.  10.1038/nature16961.
  * Song et al. [2023] Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun.  Hybrid RL: Using both offline and online data can make RL efficient.  In _International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=yyBis80iUuU.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Thrun and Schwartz [1993] Sebastian Thrun and A. Schwartz.  Issues in using function approximation for reinforcement learning.  In _Proceedings of 4th Connectionist Models Summer School_. Erlbaum Associates, June 1993.
  * van Hasselt et al. [2016] Hado van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 30(1), Mar. 2016.  10.1609/aaai.v30i1.10295.  URL https://ojs.aaai.org/index.php/AAAI/article/view/10295.
  * Večerík et al. [2017] Matej Večerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller.  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _ArXiv e-prints_ , July 2017.
  * Wagenmaker and Pacchiano [2022] Andrew Wagenmaker and Aldo Pacchiano.  Leveraging offline data in online reinforcement learning.  _arXiv preprint arXiv:2211.04974_ , 2022.
  * Yarats et al. [2022] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.  Mastering visual continuous control: Improved data-augmented reinforcement learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=_SJ-_yyes8.

##  Appendix A Detailed Experiments

###  A.1 Sparse Adroit

#### Full Results

Here we present the full results for Sparse Adroit, including LayerNorm and
environment-specific design choice ablations.

Figure 13: Full Adroit Results.

We see that LayerNorm greatly reduces variance on the Pen and Door
environments, however slightly harms mean performance on Relocate. Taken
together however, LayerNorm is still a vital ingredient for reliable
performance on this domain.

###  A.2 AntMaze

#### Full Results

Here we present the full results for AntMaze, including LayerNorm and
environment-specific design choice ablations.

Figure 14: Full AntMaze results

We see that our recommended design choices are vital for strong performance on
the harder Large tasks, and furthermore see that using deeper 3-layer networks
seems to help stability across all tasks. As we see, the best design choices
on the Large environments are also optimal for the Umaze and Medium
environments.

#### Gradient Steps per Environment Step Ablations

We see LayerNorm is incredibly effective in a setting whereby we perform a
single gradient update per time-step in Figure 15. This may be required for
learning on real robots, where computational efficiency is important.

Figure 15: SAC with and without Layer Normalization.

In Figure 16, we see the impact of increasing the number of gradient steps per
time-step. As we see, both sample efficiency stability improves greatly.

Figure 16: RLPD v.s. SAC + LN + Offline Data.

#### Full IQL comparison

Here we compare to IQL + Finetuning. We show pre-training gradient steps on
IQL as negative indices in the x-axis. As we see in Figure 17, despite IQL
learning a strong initialization, it struggles to improve beyond this. In
comparison, RLPD is able to quickly match, then greatly exceed IQL.

Figure 17: RLPD v.s. IQL + Finetuning. We show offline pre-training steps for
IQL using negative indices.

###  A.3 D4RL Locomotion

#### Full Results

Here we present the full results for D4RL Locomotion, including LayerNorm and
baselines. We emphasize that these tasks are not necessarily a good use case
for online learning with offline datasets as the tasks can be solved
relatively quickly with purely online approaches, and there is no inherent
difficulty regarding exploration.

Figure 18: D4RL Ablations.

As we see, the impact of LayerNorm is not so clear cut in Figure 18, however
this is to be expected as online approaches already achieve strong results in
this domain. Notably, we see strong performance compared to baselines in the
medium-expert domains.

Figure 19: D4RL Ablation on Expert Data.

We also test on the narrow Expert dataset, not tested by [Lee et al., 2021].
In Figure 19 we see LayerNorm can marginally help both sample efficiency, and
asymptotic performance.

###  A.4 V-D4RL Locomotion

#### Full Results

Here we present the full results for V-D4RL Locomotion, including LayerNorm
and environment-specific design choice ablations.

Figure 20: VD4RL Ablations.

As we see, LayerNorm helps significantly in the Walker and Humanoid
environments. We also see the positive impact of our recommended design
choices in the complex Humanoid domain.

##  Appendix B Experimental Details

###  B.1 Further Environment Details

(a) The Sparse Adroit Domain. Pen, Door and Relocate tasks respectively.

(b) The AntMaze Domain. Umaze, Medium and Large tasks respectively.

(c) The V-D4RL Domain. Walker Walk, Cheetah Run and Humanoid Walk
respectively.

Figure 21: Visualizations of the environments we consider.

We provide further details about the key domains we evaluate on. In Figure 21
we provide visualizations of the environments.

#### Sparse Adroit

In these tasks, reward is a binary variable that indicates whether the task
has been completed successfully or not. In prior work [Nair et al., 2020,
Rudner et al., 2021], it is common to see success rate used as the metric,
which simply determines if the task has been completed in any time step.
However Kostrikov et al. [2022] use a more challenging metric that involves
speed of completion. Concretely, return is calculated as the percentage of the
total timesteps in which the task is considered solved (note that there are no
early terminations). For example, in the ‘Pen’ task, where the horizon is 100
timesteps, if a policy achieves a Normalized Score of 80, that means in 80
timesteps, the task is considered solved. This effectively means that the
policy was able to solve the task in 20 timesteps. At each evaluation, we
perform 100 trials.

#### D4RL: AntMaze

In these tasks, reward is a binary variable that indicates whether the agent
has reached the goal. Upon reaching the goal, the episode terminates. The
normalized return is therefore the proportion of evaluation trials that were
successful. We follow prior work, and perform 100 trials, and measure
Normalized Return as the percentage of successful trials.

###  B.2 Hyperparameters

Here we list the hyperparameters for RLPD in Table 1, and the environment-
specific hyperparameters in Table 2.

Table 1: RLPD hyperparameters. Parameter | Value
---|---
Online batch size | 128
Offline batch size | 128
Discount ($\gamma$) | 0.99
Optimizer | Adam
Learning rate | $3\times 10^{-4}$
Ensemble size ($E$) | 10
Critic EMA weight ($\rho$) | 0.005
Gradient Steps (State Based) ($G$ or UTD) | 20
Network Width | 256 Units
Initial Entropy Temperature ($\alpha$) | 1.0
Target Entropy | $-\dim(\mathcal{A})/2$
Pixel-Based Hyperparameters
Action repeat | 2
Observation size | [64, 64]
Image shift amount | 4
Table 2: Environment specific hyperparameters. Environment | CDQ | Entropy Backups | MLP Architecture
---|---|---|---
Locomotion | True | True | 2 Layer
AntMaze | False | False | 3 Layer
Adroit | True | False | 3 Layer
DMC (Pixels) | False | False | 2 Layer


# Measurement of hyperfine constants and the isotope shift of rubidium 5P1/2
excited-state using Saturated Absorption Spectroscopy

P.M. Rupasinghe, Fiona Wee, Thomas Bullock, Jiaxing Liu  Department of
Physics, State University of New York at Oswego, Oswego, NY 13126

###### Abstract

The Saturated Absorption Spectroscopy (SAS) was performed to measure the
hyperfine energy splittings of rubidium 5P1/2 excited state using a homemade
external-cavity diode laser (ECDL) operating at 795 nm. Any nonlinearities
associated with ECDL scans were removed by using a low-expansion confocal
Fabry-Perot cavity and hence created a linearized frequency axis for the
spectra collected in a fully automated fashion. We report our measurements for
the magnetic dipole coupling constants 120.79(29) and 407.75(50) for
${}^{85}Rb$ and ${}^{87}Rb$ respectively.

###### pacs:

32.10.Fn, 31.30.Gs, 27.80.1+w

††preprint: APS/123-QED

##  I Introduction

Precise measurements of hyperfine structure and isotope shifts of multi-
electron atomic systems provide valuable insight into the nuclear properties
as well as our understanding of the electronic wavefunctions near the nucleus.
Therefore, they play an important role in the development of new atomic
theories that may rely on various approximation techniques. For example, two
different theoretical methods estimate 85Rb magnetic-dipole coupling constants
for the 5P1/2 state as 69.8 MHz and 120.4 MHz while the later being close to
the experimental results[1].

Rubidium can be considered as one of well studied alkali systems with wide
range of applications in frequency stabilization techniques, laser cooling and
trapping, atomic clocks, development of atomic sensors, and quantum computing.
A natural sample of rubidium contains two isotopes ${}^{85}Rb$ and ${}^{87}Rb$
with 70$\%$ and 30$\%$ abundances respectively. This system is an attractive
candidate for experimental study using laser spectroscopy techniques primarily
due to convenience in accessibility of D1 (5S1/2$\rightarrow$5P1/2) and D2
(5S1/2$\rightarrow$5P3/2) excitation transitions using commonly available
laser diodes operating at 795 nm and 780 nm as well as the availability of
sufficient vapor pressure at or near the room temperature. For such an atomic
system the energy of the hyperfine level measured from the center-of-mass
energy can be written as,

$W(F)=\frac{1}{2}AK+B\frac{(3/2)K(K+1)-2I(I+1)J(J+1)}{2I(2I-1)2J(2J-1)}$ (1)

where $K=F(F+1)-I(I+1)-J(J+1)$, $A$ and $B$ are the magnetic-dipole and
electric-quadrupole hyperfine coupling constants, $I$ is the nuclear spin, $J$
is the total electronic angular momentum, and $F$ is the total angular
momentum of the atom. The second term only contributes when $J>\frac{1}{2}$.

Hyyperfine constants of rubidium 5$P_{1/2}$ excited state have been measured
previously with different techniques[2, 3, 4, 5]. Here, we present another
measurement of hyperfine coupling constants and the isotope shift of rubidium
5$P_{1/2}$ state using a distinct approach from the previous techniques.

##  II Experimental Details

###  II.1 Optical System and Experimental Layout

We excited ground state rubidium using 795 nm laser light according to the
scheme shown in Figure 1.

Figure 1: Hyperfine structure of relevant states for two naturally occurring
isotopes of rubidium (${}^{85}Rb$ ($~{}70\%$ abundance) $\&$ ${}^{87}Rb$ (
30$\%$ abundance)). $\Delta^{85}_{e}$, $\Delta^{87}_{e}$, and $TIS$ represent
hyperfine splitting of ${}^{85}Rb$, hyperfine splitting of ${}^{87}Rb$, and
the transition isotope shift of the 5P1/2 excited state. Similarly,
$\Delta^{85}_{g}$ and $\Delta^{87}_{g}$ represent hyperfine splittings for the
ground state. $\Delta\nu_{85}$ and $\Delta\nu_{87}$ represent the least
frequency gaps between excited state and the ground state in 85Rb and 87Rb
respectively.

The external-cavity diode laser (ECDL) used in this experiment is a homemade
system based on an existing design[6, 7] with a 10 mW laser diode. As shown in
the Figure 2 two low-power parallel beams with each having power of
approximately 25 $\mu$W transmitted through the Rb vapor cell held at 47 oC
are directed to a differential photodiode (Thorlabs PDB450A). While the laser
is scanning through the relevant transition each beam produces an identical
Doppler-broadened direct-absorption signal at individual photodiodes. This
corresponds to a nearly zero-signal at the detector output (difference
signal). However, when a relatively strong beam (pump-beam) of power
approximately 140 $\mu$W is present, a large number of atoms absorb light from
the strong beam. As a result, the direct-absorption signal corresponds to the
relevant probe-beam shows absorption dips widely known as (hole burning) at
certain frequencies.

Figure 2: Experimental setup

The hole burning can occur at hyperfine transition frequencies as well as at
exactly in the middle of each pair of hyperfine transition frequencies due to
the Doppler effect-caused transitions (cross-over resonances). These dip-
absorption features show up due to the fact that there are two velocity
classes of atoms exist in the cell such that the pump beam is in resonance
with one hyperfine transition, and the probe beam is in resonance with the
other[8].

When the pump (strong beam) is present the output signal (difference signal)
of the differential photodetector shows sharp peaks only correspond to
hyperfine as well as cross-over resonances as shown in Figures 5 and 6.

###  II.2 Data acquisition system

After tuning the laser to near transition frequency we scan the laser by
applying a triangular voltage signal such that we collect up-scans and down-
scans alternatively. During each scan period our labview program samples about
1000 data points per channel in order to create spectra using National
Instruments BNC2110 DAQ device. We record hundreds of spectra under various
experimental conditions in fully automated fashion with 20-30 scans in each
run. Specifically, for each scan we collect the laser scan voltage ramp (up
and down), corresponding Fabry-Perot signal, and the vapor cell signal (the
saturated absorption spectrum).

##  III Data analysis and results

###  III.1 Creating a linearized frequency axis

Data analysis procedure starts with Fabry-Perot spectrum. We fit Fabry-Pérot
signal to modified Airy function as a function of normalized data point$\\#$
($x$) as shown in equation 2.

$A(x)=\frac{b_{0}+b_{1}x}{1+Fsin^{2}\left((\pi/\Delta)f(x)\right)}$ (2)

where, $b_{0}$ and $b_{1}$ are constants, F is the coefficient of finesse,
$\Delta$ is the free-spectral range in MHz, and
$f(x)=a_{0}+a_{1}x+a_{2}x^{2}+...$ is a polynomial. This fit model allows us
not only transform data point numbers to a frequency axis but also remove any
nonlinearities associated with the laser scan due to the hysteresis of the PZT
device that used to change the angle of the diffraction grating of the ECDL.
Such a fit is shown in Figure 3, notice the slightly increasing separations
between peaks in the spectrum which is attributed to a slight nonlinearity of
the laser scan.

Figure 3: A sample fit of the Fabry-Perot spectra. Notice the slightly
increasing spacing between peaks. In this fit, a 4th order polynomial has been
used for f(x).

During the fit all the parameters are floated except the free-spectral range
which is kept fixed at its nominal value (500 MHz in our case). Initial
conditions for parameters $a_{0}$ and $a_{1}$ were set by mapping peak
positions $x_{i}$, where, $i$ is the peak number, to relative frequency values
separated by FSR and fitting $f(x_{i})$ vs $x_{i}$ to a linear function
$a_{0}+a_{1}x$. Higher order coefficients $a_{2}$, $a_{3}$,… were set to zero.
After the first fit all the initial conditions has been updated with new fit
parameters and refit for the improved results (Bootstrapping). The final fit
parameters $a_{0}$, $a_{1}$, … defines our new frequency axis $f(x)$.

Figure 4 shows Fabry-Perot signal plotted against new frequency axis $f(x)$
with a 4th order polynomial. Unlike the Figure 3, in Figure 4 equal spacing
between peaks indicates that the non-linearity associated with laser scans has
been removed. There were no statistically significant change of results was
observed by using higher order polynomials which further assures that the 4th
order polynomial used in this analysis was sufficient to completely remove the
nonlinearity. This procedure has been used to create a linearized frequency
axes for atomic spectra produced by ECDL scans in many occasions in recent
years[9, 10, 11].

Figure 4: Fabry-Perot spectra is plotted against the new frequency axis
$f(x)$. Notice the almost equal spacing between peaks.

Next we plot the saturated absorption data against the new frequency axis
$f(x)$ and fit them to a model function containing sum of Lorentzian line-
shapes. Samples of these fits are shown in figures 5, 6, and 8 for 85Rb, 87Rb,
and both isotope spectra respectively.

###  III.2 Single isotope spectra analysis

Figure 5: 85Rb single transition spectrum in frequency domain. An upper
transition spectrum corresponds to (F = 3 $\rightarrow$ F’ = 2, 3) is shown.
Figure 6: 87Rb single transition spectrum in frequency domain. An upper
transition spectrum corresponds to (F = 2 $\rightarrow$ F’ = 1, 2) is shown.

Hyperfine splitting values for 85Rb and 87Rb spectra shown in figures 5 and 6
are derived in three independent ways. $\nu_{3}$-$\nu_{1}$ (direct),
2($\nu_{2}$-$\nu_{1}$)(indirect), and 2($\nu_{3}$-$\nu_{2}$)(indirect), where,
$\nu_{1}$, $\nu_{2}$, and $\nu_{2}$ are first, second, and third peak
positions respectively. These direct and indirect values are a good systematic
error check for our frequency axis. For each spectrum, direct and indirect
methods yielded similar results reassuring that our frequency axis
linearization is done correctly.

Figure 7 shows a histogram of 85Rb hyperfine splitting values extracted in
this manner. Similar behavior is observed for 87Rb splittings as well as
isotope shift measurements.

Figure 7: Histogram of the 85Rb hyperfine splitting measurements extracted
from several hundred of scans. The red curve shows the fitted Gaussian
function.

###  III.3 Dual isotope spectra analysis

Figure 8: Fitted saturated absorption dual isotope spectrum in frequency
domain.

The dual isotope spectrum shown in figure 8 can be used to determine hyperfine
splittings for individual isotopes as well as the transition isotope shift
between them. Using this spectrum individual hyperfine splittings are
extracted using similar analysis done for the single isotope spectra.
According the level diagram described in the figure 1, transition isotope
shift can be determined by the following equation.

$IS=\left(\frac{5}{12}\Delta^{85}_{g}+\frac{7}{12}\Delta^{85}_{e}\right)-\left(\frac{3}{8}\Delta^{87}_{g}+\frac{5}{8}\Delta^{87}_{e}\right)+\left(\delta\nu^{85}-\delta\nu^{87}\right)$
(3)

Here, $\left(\delta\nu^{85}-\delta\nu^{87}\right)$ term can be identified as
the difference between 1st and 4th peak in the dual isotope spectrum shown in
the figure 8. Similar to hyperfine splitting values, here we can extract the
isotope shift values in several independent ways. For example, the term
$\left(\delta\nu^{85}-\delta\nu^{87}\right)$ can be determined using various
peak positions with the aid of $\Delta^{85}_{e}$ and $\Delta^{87}_{e}$. The
ground state splittings $\Delta^{85}_{g}$ and $\Delta^{87}_{g}$ are taken from
[12] and [5] respectively.

###  III.4 Systematic error search and calibration

In order to assess any potential systematic effects occurred during the course
of experiment we divided our data files into different categories and
determine the results of our measurements based on laser sweep direction (up
scan vs down scan), scan speed, excitation transition, and scan linearization
(direct vs indirect measurements). Such a analysis conducted for ${}^{85}Rb$
hyperfine splitting is shown in the figure 9. The plot suggests each subset of
data converge to a similar value within their statistical uncertainty.
Similarly, we found no evidence of any bias or trends associated with our
subset measurements.

Figure 9: Plot of values obtained for hyperfine splitting of ${}^{85}Rb$ based
on laser scan direction, scan speed, and excitation scheme (F=2 $\rightarrow$
F’= 2,3) vs (F=3 $\rightarrow$ F’= 2,3).

During the Fabry-Perot data fitting described in III(a), we used the nominal
value of 500 MHz as cavity’s FSR. However, this value does not guarantee the
actual FSR value present during the experiment. In order to ensure the
frequency calibration is done correctly, we measured the FSR few times during
the course of the experiment which spanned few weeks to a month. All of our
FSR measurements resulted similar results and found the final value of 500.45
MHz. Based on this value we corrected all of our final (uncalibrated)
measurements by multiplying a correction factor defines as $c_{cal}$ =
500.45/500.00 = 1.0009.

###  III.5 Results

Our final (calibrated) hyperfine splittings and isotope shift results are
summarized in Table 1.

Table 1: Hyperfine splittings and isotope shift measurements Hyperfine Splitting | $\Delta^{85}_{e}$ | $\Delta^{87}_{e}$ | $TIS$
---|---|---|---
Final result (MHz) | 362.37 | 815.49 | 77.56
Stat. error (MHz) | 0.39 | 0.75 | 0.51
Systematic error source: |  |  |
Scan direction (up vs down) | 0.67 | 0.61 | 0.85
Scan speed | 0.10 | 0.16 | 0.24
Excitation scheme | 0.36 | - | -
Scan linearization | 0.02 | 0.23 | 0.16
Combined error (MHz) | 0.86 | 1.00 | 1.03

The resulted hyperfine $A$ (magnetic-dipole) coupling constants are listed in
Table 2 along with previous measurements for comparison.

Table 2: Magnetic-dipole coupling constant measurements Source |  85Rb |  87Rb
---|---|---
This work (MHz) | 120.79(29) | 407.75(50)
Ref. [2] (MHz) | 120.640(20) | 406.147(15)
Ref. [3] (MHz) | 120.499(10) | 408.328(15)
Ref. [5] (MHz) | 120.72(25) | 406.2(8)

##  IV Concluding remarks

Through saturated absorption spectroscopy (SAS), the hyperfine splittings of
Rb 5P1/2 excited state have been measured using a homemade ECDL operating at
795 nm. Using these splittings hyperfine A coupling constants for both
isotopes have been extracted. Our 85Rb value shows an agreement with previous
measurements. However, our 87Rb value shows a slight disagreement with
previous measurements. It is also important to note that previous measurements
described in [2] and [3] slightly disagree with the values obtained for both
isotopes. We believe our measurements will add a significant contribution to
the existing pool of measurements in the literature.

###### Acknowledgements.

The authors would like to thank SUNY-Oswego Office of Research and Sponsored
Programs (ORSP) for supporting this work.

## References

  * Safronova  _et al._ [1999] M. S. Safronova, W. R. Johnson, and A. Derevianko, Phys. Rev. A 60, 6 (1999).
  * Barnergee  _et al._ [2004] A. Barnergee, D. das, and V. Natarajan, Europhys. Lett., 65, 172 (2004).
  * Barwood  _et al._ [1991] G. P. Barwood, P. Gill, and W. R. C. Rowley, Appl. Phys. B, 53, 142 (1991).
  * Beacham and Andrew [1971] J. R. Beacham and K. L. Andrew, Journal of Opt. Soc. of America 61, 2 (1971).
  * Arimondo  _et al._ [1977] E. Arimondo, M. Inguscio, and P. Violino, Reviews of Modern Physics, 49, 1 (1977).
  * Arnold  _et al._ [1998] A. S. Arnold, J. S. Wilson, and M. G. Boshier, Rev. Sci. Instrum. 69, 1236 (1998).
  * Hawthorn  _et al._ [2001] C. J. Hawthorn, K. P. Weber, and R. E. Scholtena, Rev. Sci. Instrum. 72, 12 (2001).
  * Preston [1996] D. W. Preston, Am. J. Physics, 64, 11 (1996).
  * Ranjit  _et al._ [2013] G. Ranjit, N. A. Schine, A. T. Lorenzo, A. E. Schneider, and P. K. Majumder, Phys. Rev. A 87, 032506 (2013).
  * Ranjit  _et al._ [2014] G. Ranjit, D. Kealhofer, G. Vukasin, and P. K. Majumder, Phys. Rev. A 89, 012511 (2014).
  * Augenbraun  _et al._ [2016] B. L. Augenbraun, A. Carter, P. M. Rupasinghe, and P. K. Majumder, Phys. Rev. A 94, 022515 (2016).
  * Bize  _et al._ [1999] S. Bize, Y. Sortais, M. S. Santos, C. Mandache, A. Clairon, and C. Salomon, Europhysics Letters 45, 558 (1999).