# Imaginary Hindsight Experience Replay: Curious Model-based Learning for
Sparse Reward Tasks

Robert McCarthy† and Stephen J. Redmond† †UCD School of Electronic and
Electrical Engineering, University College Dublin, Belfield, Dublin 4,
Ireland. e-mail<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Model-based reinforcement learning is a promising learning strategy for
practical robotic applications due to its improved data-efficiency versus
model-free counterparts. However, current state-of-the-art model-based methods
rely on shaped reward signals, which can be difficult to design and implement.
To remedy this, we propose a simple model-based method tailored for sparse-
reward multi-goal tasks that foregoes the need for complicated reward
engineering. This approach, termed Imaginary Hindsight Experience
Replay111Code available soon, minimises real-world interactions by
incorporating imaginary data into policy updates. To improve exploration in
the sparse-reward setting, the policy is trained with standard Hindsight
Experience Replay and endowed with curiosity-based intrinsic rewards. Upon
evaluation, this approach provides an order of magnitude increase in data-
efficiency on average versus the state-of-the-art model-free method in the
benchmark OpenAI Gym Fetch Robotics tasks.

##  I INTRODUCTION

Deep reinforcement learning (RL) has achieved remarkable successes in recent
years, for example learning to play Atari computer games [1] or defeating the
world champion in the game of Dota 2 [2]. However, these successes have
generally relied on the availability of vast amounts of training data and a
dense/well-shaped reward function to guide exploration, neither of which are
easily obtained in practical robotic scenarios. Thus, while RL has impressed
in virtual worlds, successes in the real world have been relatively limited.

The data hungry nature of RL is a fundamental barrier to its application in
robotics; unlike in video games, gathering data in the real world is
prohibitively time-consuming and expensive. However, model-based RL has shown
promise in improving sample efficiency and tackling this issue. Unlike
imitation learning [3] or simulation to real transfer (sim-to-real) [4], these
algorithms do not come with a hard requirement for expert domain knowledge or
significant engineering effort. Unlike model-free RL methods, model-based RL
learns to explicitly model the transition dynamics of the environment, using
this model to aid the learning of a control policy. Since the model can be
learned in an entirely supervised manner, the primary advantage of model-based
RL is a significant improvement in sample efficiency. In fact, recent advances
in the state-of-the-art have allowed model-based methods to match the
asymptotic performance of model-free methods, while requiring orders of
magnitude less data [5, 6, 7]. However, data-efficiency issues aside, these
state-of-the-art techniques generally rely on dense reward signals, and in
this respect are still somewhat impractical for robotics applications.

In robotic manipulation tasks, significant expertise can be required to design
and implement a dense reward that can successfully guide learning. For
example, Popov et al. [8] require a highly complex reward function to learn to
stack blocks, while Andrychowicz et al. [9] demonstrate that an over-
simplified dense reward function can be detrimental to learning performances.
Suitable shaped rewards are not only difficult to design, but also difficult
to implement in the real world; unlike in simulation, often computer vision
systems must be employed to track reward-relevant information, such as object
positions [10].

To bypass the challenge of hand-crafting dense reward signals, methods have
been developed which can learn from sparse rewards; rewards with little
information that are generally only received once the task has been
successfully completed. These sparse rewards are much easier to design and
implement, but are conversely much more difficult to learn from. Deep RL
methods such as curiosity-driven exploration [11, 12] and Hindsight Experience
Replay (HER) [9] have made significant progress in the sparse-reward setting,
however, few works have successfully integrated these techniques with model-
based methods to obtain the benefits of both in difficult robotic tasks.

In this paper, a novel model-based RL method specifically tailored for sparse-
reward tasks is proposed to jointly address the data-efficiency and reward
engineering issues inherent in the application of RL to robotics. The method
trains the policy with HER to ensure maximum efficacy in the multi-goal tasks
commonly found in robotics, and incorporates imaginary/model-generated data
into policy updates to reduce sample complexity. To ensure the learned policy
does not overfit to model inaccuracies, imagined data is generated by an
ensemble of models [13], and is regenerated each time the model is updated. To
improve environment exploration, the ensemble also provides the agent with
intrinsic rewards based on the disagreement between its predictions [14]. To
allow the agent to adapt its behaviors specifically for the real environment,
real and imaginary data are distinguished from each other when input to the
policy. We term this technique Imaginary Hindsight Experience Replay (I-HER).

I-HER is evaluated in the challenging OpenAI Gym Fetch Robotics tasks [15,
16], and converges using, on average, an order of magnitude less data than the
state-of-the-art model-free algorithm, matching its asymptotic performance in
all but one task. The diversity of the Fetch tasks is found to expose various
modes in which model-based RL can struggle, but an ablation study demonstrates
how I-HER’s components help to overcome these issues. Compared to state-of-
the-art model-free and model-based methods, I-HER offers increased feasibility
for robotic applications due to its combined data-efficiency and ability to
effectively tackle sparse-reward tasks. Moreover, I-HER’s empirical results
outperform those presented by other model-based methods also tailored for
multi-goal sparse-reward tasks.

##  II Related Work

###  II-A Model-based Reinforcement Learning

The data-efficiency of model-based RL methods make them promising for real-
world applications, and many variants have been proposed. Learned dynamics
models have been used for model predictive control (MPC), but these methods
have struggled to match the asymptotic performance of model-free methods [17],
and generally rely on a dense reward function to identify good trajectories
[10, 18, 19]. While MPC is purely model-based, often model-based and model-
free components are combined to yield both high data efficiency and high
asymptotic performance. In one such approach, a dynamics model can be used to
predict into the future to improve the value estimates of a model-free
algorithm [20, 21]. Another approach is to train a model-free policy with
cheap model-generated data; a concept first introduced by Dyna-Q [22].

Like any model-based method, Dyna-Q-type approaches are limited by
inaccuracies in the learned dynamics model [23, 24]. Kalweit and Boedecker
[25] prevent an inaccurate model from limiting asymptotic performance by
sampling less imaginary data as uncertainty in the Q-function decreases. Other
works employ an ensemble of dynamics models to prevent the policy from
overfitting to the inaccuracies of a single model [13, 26]. SLBO [27]
minimises overfitting to any single iteration of the model by updating the
model regularly. MBPO [5] optimises the length of model-generated rollouts to
avoid compounding model errors, while Ha and Schmidhuber [28] optimise
stochasticity in the imagined environment to prevent exploitation of a
deterministic model.

This paper extends these Dyna-Q approaches, accounting for model inaccuracies
by (i) employing an ensemble of models, (ii) introducing a step which
regenerates the data in the imaginary replay buffer each time the model is
updated so as to eradicate inaccuracies produced by older iterations of the
model (unlike in [25]), and (iii) introducing an explicit distinguishing of
real and imaginary transitions to allow the policy to account for differences
in their dynamics.

###  II-B Curiosity-Driven Exploration

An agent can be endowed with curiosity by encouraging it to visit novel
states, which can greatly improve exploration of the state space in the
absence of dense extrinsic rewards. In large continuous state spaces, the
prediction error of a learned dynamics model can act as a measure of novelty,
and thus be leveraged as an intrinsic reward signal to create a curious agent
[29, 30]. However, this approach can lead to the agent becoming ‘obsessed’
with inherently unpredictable elements of the environment [31]. To balance
this issue, Pathak et al. [11] make predictions in a feature space learned by
an inverse dynamics model, while Burda et al. [12] employ a randomly
initialised neural network to remove any stochasticity in the production of
prediction targets. Intrinsic rewards have also be generated based on the
disagreement between the next-state predictions of an ensemble of dynamics
models [14, 32]. Such disagreement is more pronounced in regions of the state-
space where training data is limited, thus acts as a good measure of novelty.
Moreover, when repeatedly exposed to stochastic aspects of the environment,
models will converge to predicting the mean and disagreement will diminish.

###  II-C Model-Based Learning with Sparse Rewards

Few proposed model-based methods have been well-equipped to handle difficult
sparse-reward tasks. Indeed, Yang et al. [33] find that MBPO and MVE [20]
perform poorly in goal-based sparse-reward tasks. Multi-DEX [34], a method
tailored for sparse rewards, uses a curious model-based policy search
algorithm which (as part of its multi-objective) seeks maximally novel state
trajectories. Orthogonally, exploration policies have been learned efficiently
in imagination via ensemble disagreement intrinsic rewards, aiding quick
adaptation to downstream tasks [35, 36]. However, these methods do not
leverage the learning efficiencies that HER can provide in multi-goal tasks.

MHER [33] increases the efficiency of HER by generating imagined future
achieved goals via a learned dynamics model, but this is only demonstrated in
relatively trivial reaching tasks. Taking inspiration from HER, PlanGAN [37]
trains an ensemble of generative models to generate trajectories that lead
from the current state towards a specified goal, and use these trajectories
for MPC. This improves efficiency versus HER, but entails a highly
computationally expensive online planning procedure.

Unlike these competing methods ([33, 37]), ours is demonstrated to be robust
across the full set of benchmark Fetch tasks, providing roughly an order of
magnitude increase in data-efficiency versus HER. This is achieved via a novel
combination of curiosity-driven exploration, model-based learning, and HER,
yielding an algorithm well suited to sparse-reward multi-goal tasks.

##  III Background

###  III-A Multi-goal Reinforcement Learning

Our reinforcement learning agent interacts with a Markov decision process
(MDP), defined by the tuple
$(\mathcal{S},\mathcal{A},\mathcal{G},p,r,\gamma,\rho_{0})$. $\mathcal{S}$,
$\mathcal{A}$, and $\mathcal{G}$ are the state, action and goal spaces,
respectively, and $\gamma\in(0,1)$ is the discount factor. The state
transition distribution is denoted as $p(s^{\prime}|s,a)$, the initial state
distribution as $\rho_{0}(s)$, and the reward function as $r(s,g)$. The goal
of the RL agent is to find the optimal policy $\pi^{*}$ that maximizes the
expected sum of discounted rewards in this MDP:
$\pi^{*}=\operatorname*{argmax}_{\pi}\mathbb{E}_{\pi}[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},g_{t})]$.

The reward function is assumed to be known but the dynamics
$p(s^{\prime}|s,a)$ are unknown. Model-based reinforcement learning constructs
a model of the dynamics, $\hat{f}_{\theta}(s^{\prime}|s,a)$, using data
collected from the MDP and supervised learning.

###  III-B Hindsight Experience Replay (HER)

HER can be used with any off-policy RL algorithm in multi-goal tasks, proving
highly effective in these settings [9]. In these tasks, transition tuples
collected in the MDP take the form of $(s_{t},a_{t},r_{t},s_{t+1},g)$, where
$g$ is the goal of an episode, and the reward function is generally binary:

$r(s,g)=\begin{cases}0&\text{if $g$ is satisfied by $s$}\\\
-1&\text{otherwise}\end{cases}$ (1)

Now, if the goal is never reached in an episode in this MDP, all rewards will
be negative and the collected transitions will provide minimal information for
policy optimization. To solve this, HER employs a simple trick: a proportion
of sampled transitions have their goal altered to $g’$, where $g’$ is a goal
achieved later in the episode. The rewards of these altered transitions are
then recalculated with respect to $g’$, leaving the altered transition tuples
as $(s_{t},a_{t},r_{t}^{\prime},s_{t+1},g^{\prime})$. Even if the original
episode was unsuccessful, the altered transitions will teach the agent how to
achieve $g^{\prime}$, thus accelerating its acquisition of skills.

##  IV Imaginary Hindsight Experience Replay (I-HER)

Excellent data-efficiency and the ability to learn from sparse rewards are
appealing qualities for an RL algorithm being deployed in a robotic scenario.
While HER has excelled in sparse-reward tasks where standard model-free RL
algorithms have tended to fail [9, 38], it is apparent that its data
efficiency could be improved through the use of a model-based component. In
light of this, Imaginary Hindsight Experience Replay (I-HER) is now presented
as a technique that incorporates imaginary/model-generated data into the HER
learning procedure to improve its data efficiency. We follow the original
paper by employing Deep Deterministic Policy Gradients (DDPG) [39] as the base
off-policy RL algorithm. See Algorithm 1 for the I-HER pseudo-code and refer
to the appendix for comprehensive hyperparameter details.

###  IV-A I-HER Overview

I-HER consists of a continuous cycle of: (i) collecting real experience and
updating the ensemble of dynamics models, then; (ii) generating imaginary
experience and updating the policy. Collected real and imaginary MDP
transitions are stored in their respective replay buffers,
$\mathcal{D}_{real}$ and $\mathcal{D}_{imag}$. The dynamics model is trained
on $\mathcal{D}_{real}$. The RL policy is trained on data sampled from both
$\mathcal{D}_{real}$ and $\mathcal{D}_{imag}$, combined into a single batch
for the mini-batch gradient descent step. The sampling ratio here is based on
the ratio of real to imaginary data collected so far:
$p(imag)=\frac{N(imag)}{N(imag)+N(real)}$, where $p(imag)$ is the proportion
of sampled transitions that are imaginary, and $N(imag)$ and $N(real)$ are the
respective number of imaginary and real transitions collected so far. This
unbiased approach aims to avoid any overfitting that may occur from
oversampling real experiences.

###  IV-B Generating Imaginary Experience

In I-HER, an ensemble of dynamics models
$\\{\hat{f}_{\theta_{1}},...,\hat{f}_{\theta_{k}}\\}$ is maintained. Following
[17, 13], each model is a feed-forward neural network trained on data from
$\mathcal{D}_{real}$ to minimize the $L_{2}$ one-step prediction loss:
$\|f(s_{t},a_{t};\theta)-(s_{t+1}-s_{t})\|_{2}^{2}$, with normalized inputs
and output targets. To generate imaginary experience, at each time step a
model is sampled uniformly from the ensemble to predict $s_{t+1}$. This serves
to minimise policy exploitation of any single models inaccuracies and thus
improve policy robustness in the real environment [13]. Diversity is
maintained in the ensemble by varying model weight initializations and
training input sequences [13, 26]. Imaginary rollouts are of equal length to
the real rollouts (50 steps in this case).

###  IV-C Regenerating Imaginary Experience

Off-policy RL algorithms learn from all previously collected experience
maintained in the replay buffer. This presents an issue when learning from
imaginary transitions: older imaginary transitions are likely more inaccurate
than those generated by the latest dynamics model (which has been trained with
the most data). Ideally, all imaginary experience should at least be
consistent with the latest dynamics model, and I-HER ensures this is the case.
Recognising that imaginary experience can be generated rapidly via batch
parallelization, $\mathcal{D}_{imag}$ is simply emptied each time the dynamic
model is updated and refilled to its previous level with data generated by the
up-to-date model222On an Intel Xeon Gold 6252 CPU, it takes less than 30
seconds to refill a full imaginary replay buffer with 1 million transitions..
To ensure the buffer is still filled with a diverse set of experiences (an
important factor for off-policy algorithms [40, 41]), and to maintain learning
stability, the new imaginary transitions are collected with the current and
many older versions of the policy (see appendix for more details).

###  IV-D Intrinsic Rewards

In I-HER, the agent is provided with a useful intrinsic reward $r^{i}$, added
to the extrinsic reward $r^{e}$ during DDPG updates in the same manner as
[30], to give a total reward of: $r_{t}=r_{t}^{e}+r_{t}^{i}$. Following [14],
this intrinsic reward is based on the variance between the ensemble
predictions of $s_{t+1}$. Importantly, these ensemble-generated intrinsic
rewards can be applied to imaginary data [36], and thus are applied to both
real and imaginary transitions during policy updates. To weight the intrinsic
rewards, and to ensure they do not overshadow the extrinsic rewards, they are
scaled and clipped [30]. Thus, the intrinsic reward is calculated as:
$r_{t}^{i}=\textrm{clip}(\nu\sigma_{t},0,\eta)$, where $\sigma_{t}$ is the
variance between ensemble predictions, $\nu$ is the scaling factor, and
$\eta>0$ is the maximum value $r_{t}$ can take.

This intrinsic reward signal serves two important purposes: (i) to strengthen
the dynamics model by encouraging the collection of data in areas where it
experiences epistemic uncertainty (uncertainty due to a lack of data), and
(ii) to improve general exploration of the state-space.

###  IV-E Distinguishing Real from Imaginary Experiences

When training in a imaginary environment, a model-based RL algorithm can
overfit to inaccuracies in the imaginary dynamics at the expense of its real
world performance. Now, to account for differences between a training
environment and a target environment, meta-learning is commonly employed. For
example, a recurrent policy can be learned whose memory allows it to discern
the current dynamics and alter its behaviour appropriately [4]. Alternatively,
a policy capable of quickly adapting to slight environmental differences in
one gradient step can be learned [26]. A simpler solution is used in I-HER,
albeit one that requires the use of some real data to train the policy; the
policy is told whether input observations are from the real or imaginary
environment by appending to them a binary variable; a 1 if the observation is
real, a 0 otherwise. This is done to allow the policy to detect and account
for noticeable differences between the real and imaginary environments, and
thus learn a set of behaviours fine-tuned for the real world.

Algorithm 1 Imaginary Hindsight Experience Replay (I-HER)

1:Initialize off-policy RL algorithm $\mathbb{A}$ with policy $\pi_{\phi}$

2:Initialize ensemble of models
$\\{\hat{f}_{\theta_{1}},..,\hat{f}_{\theta_{k}}\\}$

3:Initialize empty data sets $\mathcal{D}_{real}$, $\mathcal{D}_{imag}$,
$\mathcal{D}_{\pi}$

4:for E epochs do

5: Collect R real episodes with $\pi_{\phi}$ and add to $\mathcal{D}_{real}$

6: Train $\\{\hat{f}_{\theta_{1}},..,\hat{f}_{\theta_{k}}\\}$ on
$\mathcal{D}_{real}$

7: Clear $\mathcal{D}_{imag}$ and refill using
$\\{{\hat{f}_{\theta_{1}},..,\hat{f}_{\theta_{k}}}\\}$ and policies of
$\mathcal{D}_{\pi}$

8: for C cycles do

9: Collect I imaginary episodes from
$\\{{\hat{f}_{\theta_{1}},..,\hat{f}_{\theta_{k}}}\\}$ with $\pi_{\phi}$ and
add to $\mathcal{D}_{imag}$

10: for N batches do

11: Sample minibatch B from $\mathcal{D}_{real}$ and $\mathcal{D}_{imag}$

12: Apply HER to B

13: Compute intrinsic rewards of B

14: Update $\pi_{\phi}$ with $\mathbb{A}$ and B

15: Add copy of $\pi_{\phi}$ to $\mathcal{D}_{\pi}$

##  V Experiments

###  V-A Environments

I-HER is evaluated in the OpenAI Gym Fetch Robotics environments [16], which
consist of four diverse and challenging continuous control tasks: Reach, Push,
PickAndPlace, and Slide (see Fig. 1). Rewards in these multi-goal tasks are
sparse and binary (as in Eq. (1)). Observations include relevant state
information.

In these tasks, HER is known to outperform standard model-free algorithms,
which struggle due to the sparse reward function [9, 38]. Although there have
been several iterations on the original HER (e.g. [42, 43]), these have
generally provided incremental improvements in performance, and could be
integrated into the I-HER algorithm if desired. As such, we regard HER as the
state-of-the-art model-free method in the benchmark Fetch tasks for the
purposes of comparison to I-HER.

Figure 1: The OpenAI Gym Fetch Robotics tasks: Reach, Push, Slide, and
PickAndPlace [16].

###  V-B Benchmark Results

The performance of I-HER is now compared to HER333All presented experiments
were performed over 3 random seeds. Both methods were implemented with DDPG as
the base RL algorithm. To enable fair comparison, shared hyperparameters were
maintained at equal values, largely based on those employed by [16]. The
results of this comparative evaluation are shown in Fig. 5.

The model-based I-HER matches the asymptotic performance of the model-free HER
in all tasks except Slide, requiring on average an order of magnitude less
real data to learn. Roughly 30$\times$, 15$\times$, 7$\times$, and 4$\times$
less data are required in Reach, Push, PickAndPlace, and Slide, respectively
(notably, gains diminish as task difficulty increases). To solve the non-
trivial Push task, I-HER requires 40 minutes of experience versus the 10 hours
required by HER. Interestingly, I-HER achieves a slightly higher asymptotic
performance in the PickAndPlace task, perhaps due to the improved exploration
encouraged by its intrinsic rewards.

Turning now to the reported results of competing model-based methods. Of the
Fetch tasks, MHER [33] is only tested in the trivial Reach but does not match
our 30 times improvement in data-efficiency versus HER. PlanGAN [37] displays
similar improvements in PickAndPlace but do not match our 15 times improvement
in efficiency in Push.

Figure 2: I-HER’s performance is compared to HER, the state-of-the-art in the
Fetch tasks. The x-axis, in log-scale, denotes the number of time steps of
real environment data used. The y-axis denotes task success rate in the real
environment. These figures demonstrate that I-HER provides substantial
improvements in sample complexity versus HER.555Unlike in Figs 3 and 4, the
I-HER results in Fig 5 do not include within epoch evaluations, hence their
smoother appearance. Figure 3: Contributions of I-HER’s components. The full
I-HER method (red) is compared to: I-HER without the distinguishing of real
and imaginary experiences (blue), without regenerating imaginary data (green),
and without intrinsic rewards (yellow). The Slide results are averaged over
five epochs for improved readability. Figure 4: Analysis of the effect of e =
ensemble size in the Push task. Performance is highest with an ensemble of
five models as useful intrinsic rewards are generated and the policy cannot
easily overfit to model inaccuracies. With an ‘ensemble’ of one model,
however, no intrinsic rewards are generated (ensemble variance is always
zero), and the policy significantly overfits to model inaccuracies when old
imaginary experiences are not regenerated. Note, variance in results is
omitted for improved readability.

###  V-C Ablation Study

#### Intrinsic rewards

The intrinsic rewards prove very useful in Push and PickAndPlace (see Fig. 3),
serving to encourage to interaction with the cube, whose dynamics are complex
and difficult to model. Without curiosity, the agent struggles to learn to
pickup the cube in PickAndPlace. Here, when faced with a goal it was unable to
reach, it was observed that the curious agent would continue to ‘play’ with
the cube while the non-curious agent would remain relatively inactive. This
‘play’ was crucial for the agent to quickly learn to pick up the cube and move
it to goals above the table.

#### Distinguishing between reality and imagination

Contrary to the intrinsic rewards, distinguishing real and imaginary
experiences only provides notable benefits in Slide (see Fig. 3). This can be
explained as follows. In Push and PickAndPlace, the bottleneck in learning the
task is exploration; once the agent has adequately explored the cube, the
success rate quickly converges to 100%. In Slide, however, highly precise
control is required (as demonstrated by the inability of I-HER or HER to reach
an 100% success rate in Fig. 5), and this becomes the bottleneck rather than
exploration; the Slide policy is still being fine-tuned long after it has
learned to interact with the puck.

Since highly precise control over the dynamics is required, small modelling
errors can hurt policy performance significantly. Distinguishing real and
imaginary data allows the policy to account for any noticeable differences in
the real and imaginary dynamics, and thus learn a set of behaviors fine-tuned
for the real world. This helps minimise the adverse effect of model errors,
providing a significant boost in I-HER’s final Slide performance.

#### Regenerating imaginary experiences

The results of Fig. 3 suggest that the regeneration of imaginary experiences
only provides minor benefits in the final version of I-HER (with an ensemble
of five models). However, an analysis of the effect of the ensemble (see
Figure 4) demonstrates that this regeneration step is highly beneficial when
only a single dynamics model is employed.

With a single model, model inaccuracies can be easily exploited in the
imaginary environment. For example, at one stage the policy was observed
capable of ‘magically’ moving the imaginary cube without touching it, due to a
poor dynamics model. Such inaccuracies are generally corrected once the policy
collects further data in real environment, however, if the inaccurate
imaginary data remains in the buffer, the policy will continue to try to
exploit it. Thus, regenerating the imaginary experience in the buffer is
crucial when only a single dynamics model is employed. With an ensemble of
five models, the policy cannot overfit to model inaccuracies as easily and
maintaining older imaginary experiences in the buffer is not as detrimental to
performance.

##  VI Challenges in Model-Based RL

The diversity of the Fetch tasks expose two modes in which model-based RL
methods can struggle: (a) where significant exploratory barriers must be
passed; (b) where highly precise control over complex dynamics is necessary.
These are now discussed in the context of the I-HER results.

#### Exploration

In model-based RL, the policy and model are co-dependant. The model depends on
the policy to provide it with data, while the policy requires an adequately
accurate model to aid its learning of the task. When exploration is difficult,
this codependency becomes an issue. This is evident in PickAndPlace, where
learning to pick up the cube is a major exploratory barrier. While the
intrinsic rewards are highly beneficial here, I-HER still struggles to breach
this barrier; as demonstrated by the learning plateaus around the 50-65%
success rate in the PickAndPlace results of Fig. 3 (half of the goals are on
the table while the remainder are above). Analysis found that before and
during this plateau it was impossible to properly pick up the cube in the
imaginary environment; the model had not yet seen any cube-picking data, so
its predictions were biased towards the cube falling back to the table. Hence,
the policy struggled to learn to pick up the cube as it was impossible to do
so in imaginary environment at this stage. This in turn prevented the policy
from providing the model with the cube-picking data it needed to improve its
predictions, which in turn continued to hurt the policies ability to learn
cube-picking, and so on. This unfortunate cycle that was difficult to break
without the use of intrinsic rewards.

#### Precision

Slide requires highly precise control over complex dynamics. The ‘slide’ must
be perfectly weighted and directed, and once the puck moves out of reach any
misjudgements cannot be corrected. Thus, small biases in the imaginary
environment can hurt policy performance substantially. For example, if the
imagined friction is slightly too low, the policy will tend to leave its shots
too short in the real environment. Ultimately, the requirement for high
precision in both the dynamical modelling and the control policy leads to
suboptimal I-HER results in Slide.

These exploration and precision issues are resolved to some extent by I-HER’s
intrinsic rewards and distinguishing of real and imaginary experiences,
respectively. However, there is room for improvement. An exploration mechanism
better able to identify completely unvisited areas of the state space would be
beneficial; in PickAndPlace the policy can consistently receive easy intrinsic
rewards by aggressively fumbling the cube on the table, thus reducing its
incentive to explore picking up the cube. Learning a separate Q-function for
intrinsic and extrinsic rewards, similar to [44], could be beneficial.
Enhancing model capacity to improve predictions would likely help with both
issues. This could potentially be achieved by increasing the model size, using
probabilistic dynamics models [19] (particularly useful if dealing with the
stochasticity of the real world), or by introducing a multi-step prediction
objective [18].

##  VII Conclusion

Solving difficult robotic tasks via reinforcement learning can require the
collection of excessive amounts of data, while designing and implementing a
suitably shaped reward function can require expert knowledge and complex
engineering. To address these issues, this paper proposes Imaginary Hindsight
Experience Replay (I-HER); a model-based method specifically tailored for
sparse-reward multi-goal tasks. Empirical results demonstrate that this method
significantly reduces sample complexity versus the state-of-the-art model-free
method in challenging robotic tasks, matching its asymptotic performance in
all but one task.

An important direction for future work will be to extend this approach to
visual observations; acquiring state observations in the real world is
challenging and does not scale well to increasingly unstructured settings.
This would require the adaption of the concept of HER to handle image
observations, as attempted in [45, 46]. Finally, the benefits of I-HER should
be verified via its deployment in a real-world robotics scenarios, further
testing the limits of model-based reinforcement learning.

## APPENDIX

#### Dynamics model

An ensemble of 5 models is maintained. Each model is a feed-forward neural
network with 2 hidden layers of size 512 and ReLU nonlinearities. Models are
trained with the Adam optimizer [47] with a learning rate of 0.001 and batch
size of 512 (each MPI worker samples its own batch of size 512 and gradients
are averaged for the update step). Each epoch, each model is trained for K
steps, jumpstarted off the previous epochs model. In the first epoch, the
model is trained for 5K steps.

To attempt to improve the efficiency of the update steps, a bias is introduced
when sampling from $\mathcal{D}_{real}$ to update the dynamics models. Each
transition $i$ is sampled with probability $p(i)=\frac{w(i)}{\sum{w(i)}}$,
where $w(i)=\frac{E(i)}{E}(b-1)$, $E(i)$ is the epoch in which $i$ was
collected, $E$ is the current epoch, and $b\geqslant 1$ decides how much to
bias towards newer transitions. In all experiments, $b=2$ is used.

#### Hyperparameters

The following hyperparameters are equivalent across all tasks:
$\mathcal{D}_{real}$, $\mathcal{D}_{imag}$ buffer sizes = $10^{6}$; imaginary
rollouts per MPI worker (I) = 2; batches per cycle (N) = 40; intrinsic reward
scale ($\nu$) = 0.5; intrinsic reward clip ($\eta$) = 0.8. The following are
different across $\langle$Reach, Push, PickAndPlace, Slide$\rangle$; MPI
workers = $\langle$1, 8, 8, 8$\rangle$; dynamics update steps (K) =
$\langle$50, 1250, 2500, 2500$\rangle$; real rollouts per MPI worker (R) =
$\langle$2, 32, 64, 64$\rangle$; cycles (C) = $\langle$50, 250, 250,
250$\rangle$.

#### Refilling imaginary replay buffer (line 7 of Algorithm 1)

Iterating through the policies stored in $\mathcal{D}_{\pi}$ from newest to
oldest, each policy collects $C\times I$ imaginary rollouts which are added to
$\mathcal{D}_{imag}$. This process stops once $\mathcal{D}_{imag}$ is refilled
to its previous level. A copy of $\pi_{\phi}$ is added to $\mathcal{D}_{\pi}$
every 50 cycles (C).

#### Distinguishing between real and imaginary data

During updates, the policy is always truthfully told whether a transition is
real or imaginary (via the binary variable env_is_real). However, when
collecting experience, 10% of real rollouts are collected with env_is_real = 0
as input, and 10% of imaginary rollouts are collected with env_is_real = 1 as
input. This mixing is to (i) encourage the real and imaginary behaviours to
learn from each other (beyond their shared weights), and (ii) to ensure
exploiting imaginary behaviours are corrected by testing them in the real
environment.

#### HER experiments

1 MPI worker is used in Reach and 8 are used in the remaining tasks666Our HER
implementation is taken from https://github.com/TianhongDai/hindsight-
experience-replay. Besides this, all relevant hyperparameters are kept
equivalent to [16].

## References

  * [1] Volodymyr Mnih et al.  “Human-level control through deep reinforcement learning”  In _nature_ 518.7540 Nature Publishing Group, 2015, pp. 529–533
  * [2] Christopher Berner et al.  “Dota 2 with large scale deep reinforcement learning”  In _arXiv preprint arXiv:1912.06680_ , 2019
  * [3] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan and Chrisina Jayne  “Imitation learning: A survey of learning methods”  In _ACM Computing Surveys (CSUR)_ 50.2 ACM New York, NY, USA, 2017, pp. 1–35
  * [4] OpenAI: Marcin Andrychowicz et al.  “Learning dexterous in-hand manipulation”  In _The International Journal of Robotics Research_ 39.1 SAGE Publications Sage UK: London, England, 2020, pp. 3–20
  * [5] Michael Janner, Justin Fu, Marvin Zhang and Sergey Levine  “When to trust your model: Model-based policy optimization”  In _arXiv preprint arXiv:1906.08253_ , 2019
  * [6] Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi and Jimmy Ba  “Mastering atari with discrete world models”  In _arXiv preprint arXiv:2010.02193_ , 2020
  * [7] Julian Schrittwieser et al.  “Mastering atari, go, chess and shogi by planning with a learned model”  In _Nature_ 588.7839 Nature Publishing Group, 2020, pp. 604–609
  * [8] Ivaylo Popov et al.  “Data-efficient deep reinforcement learning for dexterous manipulation”  In _arXiv preprint arXiv:1704.03073_ , 2017
  * [9] Marcin Andrychowicz et al.  “Hindsight experience replay”  In _arXiv preprint arXiv:1707.01495_ , 2017
  * [10] Anusha Nagabandi, Kurt Konolige, Sergey Levine and Vikash Kumar  “Deep dynamics models for learning dexterous manipulation”  In _Conference on Robot Learning_ , 2020, pp. 1101–1112  PMLR
  * [11] Deepak Pathak, Pulkit Agrawal, Alexei A Efros and Trevor Darrell  “Curiosity-driven exploration by self-supervised prediction”  In _International Conference on Machine Learning_ , 2017, pp. 2778–2787  PMLR
  * [12] Yuri Burda, Harrison Edwards, Amos Storkey and Oleg Klimov  “Exploration by random network distillation”  In _arXiv preprint arXiv:1810.12894_ , 2018
  * [13] Thanard Kurutach et al.  “Model-ensemble trust-region policy optimization”  In _arXiv preprint arXiv:1802.10592_ , 2018
  * [14] Deepak Pathak, Dhiraj Gandhi and Abhinav Gupta  “Self-supervised exploration via disagreement”  In _International Conference on Machine Learning_ , 2019, pp. 5062–5071  PMLR
  * [15] Greg Brockman et al.  “Openai gym”  In _arXiv preprint arXiv:1606.01540_ , 2016
  * [16] Matthias Plappert et al.  “Multi-goal reinforcement learning: Challenging robotics environments and request for research”  In _arXiv preprint arXiv:1802.09464_ , 2018
  * [17] Anusha Nagabandi, Gregory Kahn, Ronald S Fearing and Sergey Levine  “Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 7559–7566  IEEE
  * [18] Danijar Hafner et al.  “Learning latent dynamics for planning from pixels”  In _International Conference on Machine Learning_ , 2019, pp. 2555–2565  PMLR
  * [19] Kurtland Chua, Roberto Calandra, Rowan McAllister and Sergey Levine  “Deep reinforcement learning in a handful of trials using probabilistic dynamics models”  In _arXiv preprint arXiv:1805.12114_ , 2018
  * [20] Vladimir Feinberg et al.  “Model-based value estimation for efficient model-free reinforcement learning”  In _arXiv preprint arXiv:1803.00101_ , 2018
  * [21] Jacob Buckman et al.  “Sample-efficient reinforcement learning with stochastic ensemble value expansion”  In _arXiv preprint arXiv:1807.01675_ , 2018
  * [22] Richard S Sutton  “Integrated architectures for learning, planning, and reacting based on approximating dynamic programming”  In _Machine learning proceedings 1990_ Elsevier, 1990, pp. 216–224
  * [23] Shixiang Gu, Timothy Lillicrap, Ilya Sutskever and Sergey Levine  “Continuous deep q-learning with model-based acceleration”  In _International Conference on Machine Learning_ , 2016, pp. 2829–2838  PMLR
  * [24] Lukasz Kaiser et al.  “Model-based reinforcement learning for atari”  In _arXiv preprint arXiv:1903.00374_ , 2019
  * [25] Gabriel Kalweit and Joschka Boedecker  “Uncertainty-driven imagination for continuous deep reinforcement learning”  In _Conference on Robot Learning_ , 2017, pp. 195–206  PMLR
  * [26] Ignasi Clavera et al.  “Model-based reinforcement learning via meta-policy optimization”  In _Conference on Robot Learning_ , 2018, pp. 617–629  PMLR
  * [27] Yuping Luo et al.  “Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees”  In _arXiv preprint arXiv:1807.03858_ , 2018
  * [28] David Ha and Jürgen Schmidhuber  “World models”  In _arXiv preprint arXiv:1803.10122_ , 2018
  * [29] Yuri Burda et al.  “Large-scale study of curiosity-driven learning”  In _arXiv preprint arXiv:1808.04355_ , 2018
  * [30] Boyao Li et al.  “Curiosity-driven exploration for off-policy reinforcement learning methods”  In _2019 IEEE International Conference on Robotics and Biomimetics (ROBIO)_ , 2019, pp. 1109–1114  IEEE
  * [31] Jürgen Schmidhuber  “Formal theory of creativity, fun, and intrinsic motivation (1990–2010)”  In _IEEE Transactions on Autonomous Mental Development_ 2.3 Ieee, 2010, pp. 230–247
  * [32] Mikael Henaff  “Explicit explore-exploit algorithms in continuous state spaces”  In _arXiv preprint arXiv:1911.00617_ , 2019
  * [33] Rui Yang et al.  “MHER: Model-based Hindsight Experience Replay”  In _arXiv preprint arXiv:2107.00306_ , 2021
  * [34] Rituraj Kaushik, Konstantinos Chatzilygeroudis and Jean-Baptiste Mouret  “Multi-objective model-based policy search for data-efficient learning with sparse rewards”  In _Conference on Robot Learning_ , 2018, pp. 839–855  PMLR
  * [35] Pranav Shyam, Wojciech Jaśkowski and Faustino Gomez  “Model-based active exploration”  In _International conference on machine learning_ , 2019, pp. 5779–5788  PMLR
  * [36] Ramanan Sekar et al.  “Planning to explore via self-supervised world models”  In _International Conference on Machine Learning_ , 2020, pp. 8583–8592  PMLR
  * [37] Henry Charlesworth and Giovanni Montana  “Plangan: Model-based planning with sparse rewards and multiple goals”  In _arXiv preprint arXiv:2006.00900_ , 2020
  * [38] Tianhong Dai, Hengyan Liu and Anil Anthony Bharath  “Episodic Self-Imitation Learning with Hindsight”  In _Electronics_ 9.10 Multidisciplinary Digital Publishing Institute, 2020, pp. 1742
  * [39] Timothy P Lillicrap et al.  “Continuous control with deep reinforcement learning”  In _arXiv preprint arXiv:1509.02971_ , 2015
  * [40] William Fedus et al.  “Revisiting fundamentals of experience replay”  In _International Conference on Machine Learning_ , 2020, pp. 3061–3071  PMLR
  * [41] Shangtong Zhang and Richard S Sutton  “A deeper look at experience replay”  In _arXiv preprint arXiv:1712.01275_ , 2017
  * [42] Qiwei He, Liansheng Zhuang and Houqiang Li  “Soft hindsight experience replay”  In _arXiv preprint arXiv:2002.02089_ , 2020
  * [43] Meng Fang et al.  “Curriculum-guided hindsight experience replay”, 2019
  * [44] John Banister Lanier  “Curiosity-driven multi-criteria hindsight experience replay”  University of California, Irvine, 2019
  * [45] Ashvin Nair et al.  “Visual reinforcement learning with imagined goals”  In _arXiv preprint arXiv:1807.04742_ , 2018
  * [46] Himanshu Sahni, Toby Buckley, Pieter Abbeel and Ilya Kuzovkin  “Addressing Sample Complexity in Visual Tasks Using HER and Hallucinatory GANs”  In _arXiv preprint arXiv:1901.11529_ , 2019
  * [47] Diederik P Kingma and Jimmy Ba  “Adam: A method for stochastic optimization”  In _arXiv preprint arXiv:1412.6980_ , 2014


# Phase Retrieval for Rydberg Quantum Arrays

###### Abstract

In this paper we derive a novel phase retrieval algorithm for use in phased
array or synthetic aperture applications where only measurements of electric
field intensity are possible at each spatial sample. Such array configurations
exist if a Rydberg atom probe is used in place of an antenna. We present
outcomes of numerical experiments showing the effectiveness of the proposed
algorithm.

Index Terms—  phase retrieval, quantum synthetic apertures, Rydberg sensing

##  1 Introduction

Phase retrieval is the process of recovering a signal from the magnitude of
its Fourier transform. This problem has been studied in a number of
applications, such as optics [1], X-ray crystallography [2, 3], speech
recognition [4], blind channel estimation [5], and astronomy [6]. The interest
in phase retrieval is largely due to the inability of a sensing device to
measure the phase of a received signal. For one-dimensional (1D) signals, the
problem is ill-posed, meaning more than one signal with different phase can
map to the same magnitude; the only exceptions being a minimum-phase signal
[7] or a sparse signal with structured support [8]. Substantial work has been
done and is still ongoing to overcome the ill-posedness of the problem, and
the literature is too large to summarize here (e.g. see [9, 10] for
contemporary surveys, and references therein). Overall, two major approaches
have emerged: the first harnesses prior knowledge of the signal structure,
such as sparse support [11, 12, 13], while the other exploits technology to
make additional measurements of the magnitude via, for example, masks [14] and
short-time Fourier transform (STFT) [15, 16, 17].

Recent work at the National Institute of Standards and Technology (NIST) and
many other institutions has demonstrated the feasibility of using Rydberg atom
probes to provide traceable measurements of electric field intensity [18].
These quantum probes have many unique features that set them apart from
traditional antennas including exactly defined response functions based on
quantum mechanics allowing for direct field strength traceability to the
International System of Units (SI) [19, 20, 21, 22, 23], a highly transparent
probe head to RF fields made only of dielectric materials [24, 25], direct
down-conversion of the RF field by the atoms reducing the need for back-end
electronics [26], and intrinsic ultra-wideband tunability from kilohertz
frequencies to terahertz frequencies using a single probe and a tunable laser
[21, 27]. These unique features have attracted a large number of RF
scientists, engineers, and developers to these Rydberg atom probes; however,
one limitation of these probes is in the detection of phase. Typically, in
order to resolve the phase of the incident RF field, a second RF field acting
as a local-oscillator (LO) is radiated onto the Rydberg atom probe [28, 29].
The radiated LO has implications for the transparency of the probe because it
requires an additional antenna as well as impacting the usefulness of these
probes in stealth applications. One alternative phase sensing approach has
been put forward [30] wherein an optical LO is used rather than a radiated RF
LO; however, this scheme is limited to sensing only a single quadrant of phase
space because it is essentially still an intensity detector. Thus, processing
schemes that recover phase information from a purely intensity set of
measurements are of interest for applications that use the Rydberg atom probes
in a synthetic aperture measurement (e.g. [24, 31]).

The algorithm described in this paper is intended to retrieve an estimate of
signal phase from intensity-only measurements such that the Rydberg atom probe
can be used to construct a synthetic aperture. Synthetic apertures increase
the effective area, and thereby angular resolution, of an imaging system
beyond the physical limits of a hardware antenna. A synthetic aperture is
constructed by using a mechanical positioner to move a measurement probe or
antenna through space. The probe measures at discrete locations the RF fields
propagating across the aperture and if the spatial RF samples maintain phase
coherence then they can be combined in post-processing to create high-
resolution snapshots of the ambient scattering.

##  2 Measurement Model

The Rydberg atom probe measurement utilizes sensitive Rydberg states (high
energy states where the electron of an alkali atom is far from and weakly
coupled to the nucleus) to detect RF electric fields. These Rydberg states are
reached using a set of lasers. A near-infrared probe laser couples to the
ground state of the atoms and is detected with a photodetector after passing
through the atomic vapor cell, as depicted in Fig. 1. Because the probe laser
is resonant with ground states of the atoms, this laser is preferentially
absorbed and minimal light reaches the photodetector. Then, a counter-
propagating visible coupling laser excites the atoms to a Rydberg state where
the principle quantum number is large and the energy separation between
adjacent Rydberg states is small (e.g. on the order of RF frequencies). The
combination of these two lasers produces an effect known as
electromagnetically induced transparency (EIT), wherein the atoms become
transparent to the probe laser over a narrow frequency range and the light
reaching the detector increases, see Fig. 2. When the atoms are then radiated
by an RF field that is resonant with a second Rydberg state, the narrow EIT
line splits known as Autler-Townes (AT) splitting. The frequency difference of
this AT line splitting, $\Delta_{f}$, is proportional to the absolute value of
the electric field of the RF, $|E_{RF}|$:

$\Delta_{f}=\frac{\wp}{h}|E_{RF}|,$ (1)

where $\wp$ is the dipole moment of the Rydberg transition, which is
accurately calculated [32], and $h=6.62607015\times
10^{-34}\leavevmode\nobreak\ \mathrm{J\leavevmode\nobreak\ Hz^{-1}}$ is
Planck’s constant, a fundamental unit in the new SI. Once the on-resonance
(detuning of 0 MHz in Fig. 2) change in probe laser transmittance with RF
field strength is calibrated, both laser frequencies can be locked and fast
sensing of the RF field can be achieved. Our paper considers this Rydberg atom
RF field measurement system when the sensor, the vapor cell and lasers packed
with fiber-coupling [24], is scanned over a spatial plane defining a synthetic
aperture and measurements of the RF electric field intensity are captured at
each location. Due to the absolute value in Eq. 1, no phase information of the
RF field is captured and must be retrieved using the processing described
below.

Fig. 1: Diagram of the Rydberg atom electric field measurement setup. PD is a
photodetector and DM is a dichroic mirror. This same set of hardware can be
used to detect RF electric fields from kHz to THz. Fig. 2: Example Rydberg
measurement when a 26.105 GHz RF field radiates 85Rb atoms excited to the
$50D_{5/2}$ Rydberg state by the probe laser (resonant with the $6S_{1/2}$ and
$6P_{3/2}$ states) and the coupling laser (resonant with the $6P_{3/2}$ and
$50D_{5/2}$ states). When the coupling laser frequency is detuned, the EIT
line of the probe transmittance appears (blue line). When the coupling laser
is far detuned in the negative frequency direction, another weaker EIT line
appears from the $50D_{3/2}$ state, which is used to calibrate the frequency
axis of the spectrum. Once the resonant RF field is turned on, the main EIT
line splits (green line) and the frequency difference, $\Delta_{f}$, is
directly proportional to the field strength of the RF and the dipole moment of
the $50D_{5/2}$ to $48F_{7/2}$ transition, Eq. 1.

A common approximation, also used here, is that signal sources in the scene
are far enough away from the observation plane of the aperture that the
impinging phase front is nearly planar. An additional assumption is that the
signal is narrowband, e.g. a sinusoidal tone. With these conditions satisfied,
a linear signal model describes the signals propagating across an array of
${N}$ elements as a sum of plane waves, or
${\mathbf{A}\mathbf{s}=\mathbf{b}}$.

The columns of ${\mathbf{A}}$ are steering vectors sampled on a discrete
search grid of ${K}$ angles ${(u_{i},v_{i})}$ defined in sine space
coordinates,

$\mathbf{A}=\left[\begin{array}[]{ccc}\mathbf{a}(u_{0},v_{0})&\ldots&\mathbf{a}(u_{K-1},v_{K-1})\end{array}\right]$
(2)

with

$\mathbf{a}(u_{i},v_{i})=\left[\begin{array}[]{ccc}e^{\mathrm{j}k({x_{0}}{u_{i}}+y_{0}{v_{i}})}&\ldots&e^{\mathrm{j}k({x_{N-1}}{u_{i}}+y_{N-1}{u_{i}})}\end{array}\right]^{T}.$
(3)

The vector ${\mathbf{s}}$ contains the (unknown) complex signal source
amplitudes and the vector ${\mathbf{b}}$ represents the complex output of the
array or synthetic aperture. The probe measures intensity or
${|[\mathbf{b}]_{k}|^{2}}$ at the ${k}$th spatial sample in the aperture. The
${N}$ rows of ${\mathbf{A}}$ are electrical angle vectors that specify all
possible phases at each array element over all angles in the search grid,

$\mathbf{A}=\left[\begin{array}[]{ccc}\mathbf{f}_{1}^{T}&\ldots&\mathbf{f}_{N-1}^{T}\end{array}\right]^{T}.$
(4)

For example,

$\mathbf{f}_{i}=\left[\begin{array}[]{ccc}e^{\mathrm{j}k({x_{i}}{u_{0}}+y_{i}{v_{0}})}&\ldots&e^{\mathrm{j}k({x_{i}}{u_{K-1}}+y_{i}{v_{K-1}})}\end{array}\right]^{T}.$
(5)

Since the probe measures electric field intensity at each array element,

$|[\mathbf{b}]_{k}|^{2}\equiv
b_{k}^{2}=\big{|}<\mathbf{f}_{k},\mathbf{s}>\big{|}^{2}=\big{|}\mathbf{f}_{k}^{H}\mathbf{s}\big{|}^{2}.$
(6)

The objective function to optimize is the maximum error between the intensity
measurements and the magnitude squared signal model,

$\underset{{}}{\textrm{minimize}}\max_{k}\big{|}\thinspace\big{|}[\mathbf{b}]_{k}\big{|}^{2}-\big{|}[\mathbf{As}]_{k}\big{|}^{2}\thinspace\big{|}$
(7)

or equivalently,

$\underset{{}}{\textrm{minimize}}\max_{k}\big{|}b_{k}^{2}-\big{|}\mathbf{f}_{k}^{H}\mathbf{s}\big{|}^{2}\big{|}$
(8)

for ${k=0,\ldots,N-1}$. This objective can be restated as

minimize $\displaystyle\delta$ (9) subject to
$\displaystyle\big{|}\thinspace|\mathbf{f}_{0}^{H}\mathbf{s}|^{2}-b_{0}^{2}\thinspace\big{|}\leq\delta$
$\displaystyle\quad\vdots$
$\displaystyle\big{|}\thinspace|\mathbf{f}_{N-1}^{H}\mathbf{s}|^{2}-b_{N-1}^{2}\thinspace\big{|}\leq\delta$
$\displaystyle\delta\geq 0.$

Rearranging and expanding terms yields,

$\displaystyle\textrm{minimize}\quad\delta$ (10) $\displaystyle\textrm{subject
to}\quad\mathbf{s}^{H}\mathbf{f}_{0}\mathbf{f}_{0}^{H}\mathbf{s}-\delta\leq
b_{0}^{2},\quad-\mathbf{s}^{H}\mathbf{f}_{0}\mathbf{f}_{0}^{H}\mathbf{s}-\delta\leq-
b_{0}^{2},$ $\displaystyle\qquad\vdots$
$\displaystyle\mathbf{s}^{H}\mathbf{f}_{N-1}\mathbf{f}_{N-1}^{H}\mathbf{s}-\delta\leq
b_{N-1}^{2},\thinspace-\mathbf{s}^{H}\mathbf{f}_{N-1}\mathbf{f}_{N-1}^{H}\mathbf{s}-\delta\leq-
b_{N-1}^{2},$ $\displaystyle\delta\geq 0.$

Using the matrix trace results in,

$\displaystyle\textrm{minimize}\quad\delta\quad\textrm{subject to}$ (11)
$\displaystyle\operatorname{Tr}{\big{[}\mathbf{f}_{0}\mathbf{f}_{0}^{H}\mathbf{s}\mathbf{s}^{H}\big{]}}-\delta\leq
b_{0}^{2},\quad-\operatorname{Tr}{\big{[}\mathbf{f}_{0}\mathbf{f}_{0}^{H}\mathbf{s}\mathbf{s}^{H}\big{]}}-\delta\leq-
b_{0}^{2},$ $\displaystyle\qquad\vdots$
$\displaystyle\operatorname{Tr}{\big{[}\mathbf{f}_{N-1}\mathbf{f}_{N-1}^{H}\mathbf{s}\mathbf{s}^{H}\big{]}}-\delta\leq
b_{N-1}^{2},$ $\displaystyle\hskip
8.5359pt-\operatorname{Tr}{\big{[}\mathbf{f}_{N-1}\mathbf{f}_{N-1}^{H}\mathbf{s}\mathbf{s}^{H}\big{]}}-\delta\leq-
b_{N-1}^{2},$ $\displaystyle\delta\geq 0.$

To relax the constraints replace the rank-one matrix ${\mathbf{ss}^{H}}$ with
the matrix ${\mathbf{S}}$ of general rank. Applying this change and using the
substitution
${\operatorname{Tr}{\big{[}\mathbf{A}^{H}\mathbf{B}\big{]}}=\operatorname{vec}{\big{(}\mathbf{A}\big{)}^{H}}\operatorname{vec}{\big{(}\mathbf{B}}\big{)}}$
yields,

minimize $\displaystyle\delta$ (12) subject to
$\displaystyle\big{[}\widehat{\mathbf{A}}\widehat{\mathbf{s}}\big{]}_{k}-\delta\leq
b_{k}^{2}\quad k=0,\ldots,N-1$
$\displaystyle-\big{[}\widehat{\mathbf{A}}\widehat{\mathbf{s}}\big{]}_{k}-\delta\leq-
b_{k}^{2}\quad k=0,\ldots,N-1$ $\displaystyle\delta\geq 0$

where ${\widehat{\mathbf{s}}=\operatorname{vec}{\big{(}\mathbf{S}\big{)}}}$
and

$\displaystyle\widehat{\mathbf{A}}=\begin{bmatrix}\operatorname{vec}{\big{(}\mathbf{f}_{0}\mathbf{f}_{0}^{H}}\big{)}^{T}&&\ldots&\operatorname{vec}{\big{(}\mathbf{f}_{N-1}\mathbf{f}_{N-1}^{H}}\big{)}^{T}\end{bmatrix}^{T}.$
(13)

We can write this objective in the form of a standard linear program as

minimize
$\displaystyle\left[\begin{array}[]{cccc}0&0&\dots&1\end{array}\right]\left[\begin{array}[]{c}\widehat{\mathbf{s}}\\\
\delta\end{array}\right]$ (17) subject to
$\displaystyle\left[\begin{array}[]{cc}\widehat{\mathbf{A}}&-1\\\
-\widehat{\mathbf{A}}&-1\end{array}\right]\left[\begin{array}[]{c}\widehat{\mathbf{s}}\\\
\delta\end{array}\right]\leq\left[\begin{array}[]{c}\mathbf{b}^{2}\\\
-\mathbf{b}^{2}\end{array}\right]$ (24) $\displaystyle\delta\geq 0.$

Define the matrices

$\displaystyle\overline{\mathbf{A}}=\left[\begin{array}[]{cc}\Re{\big{(}\widehat{\mathbf{A}}\big{)}}&-\Im{\big{(}\widehat{\mathbf{A}}\big{)}}\\\
\Im{\big{(}\widehat{\mathbf{A}}\big{)}}&\Re{\big{(}\widehat{\mathbf{A}}\big{)}}\end{array}\right],$
(27)
$\displaystyle\overline{\mathbf{b}}=\left[\begin{array}[]{c}\mathbf{b}^{2}\\\
\mathbf{0}\end{array}\right],\quad\overline{\mathbf{s}}=\left[\begin{array}[]{c}\Re{\big{(}\widehat{\mathbf{s}}\big{)}}\\\
\Im{\big{(}\widehat{\mathbf{s}}\big{)}}\end{array}\right].$

Then the final optimization program takes the form,

minimize
$\displaystyle\left[\begin{array}[]{cccc}0&0&\dots&1\end{array}\right]\left[\begin{array}[]{c}\overline{\mathbf{s}}\\\
\delta\end{array}\right]$ (34) subject to
$\displaystyle\left[\begin{array}[]{cc}\overline{\mathbf{A}}&-1\\\
-\overline{\mathbf{A}}&-1\end{array}\right]\left[\begin{array}[]{c}\overline{\mathbf{s}}\\\
\delta\end{array}\right]\leq\left[\begin{array}[]{c}\overline{\mathbf{b}}\\\
-\overline{\mathbf{b}}\end{array}\right]$ (41) $\displaystyle\delta\geq 0.$

A regularization term to encourage a sparse solution can be added to the
objective function as in

minimize
$\displaystyle\left[\begin{array}[]{cccc}0&0&\dots&1\end{array}\right]\left[\begin{array}[]{c}\overline{\mathbf{s}}\\\
\delta\end{array}\right]+\left\lVert\left[\begin{array}[]{c}\overline{\mathbf{s}}\\\
\delta\end{array}\right]\right\rVert_{1}$ (47) subject to
$\displaystyle\left[\begin{array}[]{cc}\overline{\mathbf{A}}&-1\\\
-\overline{\mathbf{A}}&-1\end{array}\right]\left[\begin{array}[]{c}\overline{\mathbf{s}}\\\
\delta\end{array}\right]\leq\left[\begin{array}[]{c}\overline{\mathbf{b}}\\\
-\overline{\mathbf{b}}\end{array}\right]$ (54) $\displaystyle\delta\geq 0.$

This program can be solved using interior point methods and optimization
toolboxes such as CVX. Once the complex matrix ${\mathbf{S}}$ has been
reconstructed from the solution ${\overline{\mathbf{s}}}$, it is still
necessary to determine the best rank-one approximation ${\mathbf{s}_{opt}}$ to
${\mathbf{S}}$. One can refer to [33] for a proof that

$\mathbf{s}_{opt}=(\mathbf{v}^{H}\mathbf{S}\mathbf{v})\mathbf{vv}^{H}$ (55)

where ${\mathbf{v}}$ is the eigenvector of ${\mathbf{S}}$ corresponding to the
largest eigenvalue. Next, the final vector of complex signal source
coefficients is used to determine the corresponding array output vector
${\mathbf{b}_{est}=\mathbf{A}\mathbf{s}_{opt}}$. In the second and third
stages of the algorithm described in Section 3, alternating projections
between the sets of feasible signal sources and array output magnitudes are
used to search for ${e^{-\textrm{j}\angle{\mathbf{b}}}}$.

##  3 Alternating Projections

The second stage of the alternating projections algorithm is a projection into
the set of feasible signal sources. To implement this projection it is
necessary to estimate or know the number of signal sources ${\widehat{K}}$
apriori. Define the matrices,

$\displaystyle\widetilde{\mathbf{A}}=\left[\begin{array}[]{cc}\Re{\big{(}{\mathbf{A}}\big{)}}&-\Im{\big{(}{\mathbf{A}}\big{)}}\\\
\Im{\big{(}{\mathbf{A}}\big{)}}&\Re{\big{(}{\mathbf{A}}\big{)}}\end{array}\right],$
(58)
$\displaystyle\widetilde{\mathbf{b}}=\left[\begin{array}[]{c}\Re{\big{(}\mathbf{b}_{est}\big{)}}\\\
\Im{\big{(}\mathbf{b}_{est}\big{)}}\end{array}\right],\quad\widetilde{\mathbf{s}}=\left[\begin{array}[]{c}\Re{\big{(}{\mathbf{s}}\big{)}}\\\
\Im{\big{(}{\mathbf{s}}\big{)}}\end{array}\right].$ (63)

We can solve the lower-dimension linear program for
${\widetilde{\mathbf{s}}}$,

minimize
$\displaystyle\left[\begin{array}[]{cccc}0&0&\dots&1\end{array}\right]\left[\begin{array}[]{c}\widetilde{\mathbf{s}}\\\
\delta\end{array}\right]$ (67) subject to
$\displaystyle\left[\begin{array}[]{cc}\widetilde{\mathbf{A}}&-1\\\
-\widetilde{\mathbf{A}}&-1\end{array}\right]\left[\begin{array}[]{c}\widetilde{\mathbf{s}}\\\
\delta\end{array}\right]\leq\left[\begin{array}[]{c}\widetilde{\mathbf{b}}\\\
-\widetilde{\mathbf{b}}\end{array}\right],\quad\delta\geq 0.$ (74)

The projection onto the set of feasible signal sources is implemented by
retaining the ${\widehat{K}}$ largest coefficients
${\\{\widetilde{s}_{0},\widetilde{s}_{1},\ldots,\widetilde{s}_{\widehat{K}-1}\\}}$
of ${\widetilde{\mathbf{s}}}$ as in

$\mathbf{s}_{proj}=\left[\begin{array}[]{ccccccc}\widetilde{s}_{0}&\widetilde{s}_{1}&\ldots&\widetilde{s}_{\widehat{K}-1}&0&\ldots&0\end{array}\right]^{T}.$
(75)

A new estimate of the array output vector is

$\mathbf{b}_{1}=\left|\mathbf{b}\right|\odot
e^{-\textrm{j}\angle\mathbf{A}\mathbf{s}_{proj}}.$ (76)

1:

2:Construct a dense uniformly sampled grid of ${L}$ angles;
${\\{(u_{l},v_{l})\text{ }|\text{ }l=0,\ldots,L-1\\}}$. Use this grid to
create the array manifold matrix ${\mathbf{A}}$ given in (2).

3:Solve the optimization program in (47) for ${\overline{\mathbf{s}}}$ and
rearrange the result to obtain ${\mathbf{S}}$. Use (55) to obtain
${\mathbf{s}_{opt}}$ from ${\mathbf{S}}$.

4:Compute the corresponding array output vector
${\mathbf{b}_{est}=\mathbf{A}\mathbf{s}_{opt}}$.

5:Set the initial complex array output estimate to
${\mathbf{b}_{0}^{\dagger}=|\mathbf{b}|\odot
e^{-\textrm{j}\angle\mathbf{b}_{est}}}$. This step effectively projects the
vector ${\mathbf{b}_{est}}$ onto the set of feasible array output magnitudes.

6:

1:Use ${\mathbf{b}_{est}}$ to create the vector ${\widetilde{\mathbf{b}}}$ in
(58).

2:Solve the linear program in (67) for ${\widetilde{\mathbf{s}}}$ and retain
the ${\widehat{K}}$-largest coefficients as in (75) to create the vector
${\mathbf{s}_{proj}}$.

3:Update the complex array output estimate to
${\mathbf{b}_{1}^{\dagger}=\mathbf{b}_{0}^{\dagger}\odot
e^{-\textrm{j}\angle\mathbf{A}\mathbf{s}_{proj}}.}$

4:

1:Create the matrix
${\mathbf{R}=\mathbf{b}_{1}^{\dagger}\mathbf{b}_{1}^{\dagger{H}}+\nu\mathbf{I}}$
for small ${\nu>0}$.

2:Initialize the conjugate gradient algorithm with
${\mathbf{w}_{0}=e^{-\mathrm{j}\angle\mathbf{A}\mathbf{s}_{proj}}.}$

3:Set ${\mathbf{g}_{0}=\mathbf{h}_{0}=\nabla{P}(\mathbf{w}_{0}).}$

4:For ${k=0,1,\ldots,J-1}$ iterations compute ${t_{k}}$ such that
${P\left(e^{-\mathrm{j}t_{k}\text{Diag}[\mathbf{h}_{k}]}\mathbf{w}_{k}\right)\geq
P\left(e^{-\mathrm{j}t\text{Diag}[\mathbf{h}_{k}]}\mathbf{w}_{k}\right)\forall
t\geq 0}$.

5:Set
${\mathbf{w}_{k+1}=e^{-\mathrm{j}t_{k}\text{Diag}[\mathbf{h}_{k}]}\mathbf{w}_{k}}$.

6:Set ${\mathbf{g}_{k+1}=\nabla{P}(\mathbf{w}_{k+1})}$.

7:Set
${\mathbf{h}_{k+1}=\mathbf{g}_{k+1}+\gamma_{k}\mathbf{h}_{k},\quad\gamma_{k}=\frac{\left(\mathbf{g}_{k+1}-\mathbf{g}_{k}\right)^{T}\mathbf{g}_{k+1}}{\lVert\mathbf{g}_{k}\rVert^{2}}.}$

8:After ${N_{cg}}$ gradient iterations set
${\mathbf{b}_{est}=\mathbf{b}_{0}^{\dagger}=|\mathbf{b}|{\odot}e^{-\mathrm{j}\angle\mathbf{w}_{cg}}}$
and return to Step 1 in Stage 2 for the next linear program iteration.

9:After executing Stage 2 and Stage 3 for ${N_{ap}}$ iterations the final
solution is ${|\mathbf{b}|{\odot}e^{-\mathrm{j}\angle\mathbf{w}_{ap}}}$.

Algorithm 1 Three-Stage Phase Retrieval Algorithm

We now seek to maximize the array output power,

$P(\mathbf{w})=|\mathbf{w}^{H}\mathbf{b}_{1}|^{2}=\mathbf{w}^{H}\mathbf{b_{1}b_{1}}^{H}\mathbf{w}\equiv\mathbf{w}^{H}\mathbf{Rw}$
(77)

where ${\mathbf{w}}$ is a vector of phase weights,

$\mathbf{w}=\left[\begin{array}[]{cccc}e^{j\phi_{0}}&e^{j\phi_{1}}&\ldots&e^{j\phi{N-1}}\end{array}\right]^{T}.$
(78)

The gradient vector ${\nabla{P}(\mathbf{w})}$ can be computed using an
approach described in [34] as,

$\nabla{P}(\mathbf{w})=\frac{1}{\mathbf{w}^{H}\mathbf{w}}\Im\big{(}\text{diag}\big{[}\mathbf{R}-P_{s}(\mathbf{w})\mathbf{I},\mathbf{ww}^{H}\big{]}\big{)},$
(79)

where ${[\mathbf{A},\mathbf{B}]=\mathbf{AB}-\mathbf{BA}}$. Conjugate gradient
iterations can now be used to find an optimal solution for ${\mathbf{w}}$.

In the algorithm, notice that the input to Stage 2 from Stage 3,
${|\mathbf{b}|{\odot}e^{-\mathrm{j}\angle\mathbf{w}_{cg}}}$, is a projection
onto the set of feasible array output magnitudes. Likewise, the pruning
operation in Step 2 of Stage 2 to create ${\mathbf{s}_{proj}}$ is a projection
onto the set of feasible signal sources. Thus the algorithm alternates
projections between two constraint sets. The primary purpose of Stage 1 is to
provide a reliable and consistent starting point for the alternating
projections.

Fig. 3: Beamformed array output with known phase information

Fig. 4: Beamformed array output after estimating phase

Fig. 5: True versus estimated phase across aperture

Fig. 6: Decreasing upper bound on maximum magnitude error (denoted as
${\delta}$ in the text)

##  4 Numerical Experiments

This section provides simulation results that demonstrate the phase retrieval
algorithm. Two signal sources of 10 and 8 dB are located at coordinates
(-1663, 3155, 2870) and (1301, 2878, 0), in units of mm, with respect to a
planar 7-by-7 synthetic aperture centered at the origin. The spacing between
spatial samples of the synthetic aperture is ${\lambda/2}$ at 40 GHz. The
total signal received at the ${m}$th spatial sample of the synthetic aperture
is given by

$y_{m}=\sum_{k=0}^{\widehat{K}-1}\sqrt{P_{k}}e^{\textrm{j}(2\pi/{\lambda})d_{km}}$
(80)

where ${P_{k}}$ is the signal power and ${d_{km}}$ is the distance from the
${k}$th signal source to the ${m}$th spatial sample in the aperture. In this
scenario, the number of signal sources ${\widehat{K}=2}$. Fig. 6 illustrates
the ground truth beamformed output created with perfect knowledge of the
phase. Fig. 6 illustrates the beamformed output when electric field intensity
is measured and the phase is estimated. The results show agreement between the
estimated and true signal sources within one beamwidth. Fig.6 illustrates the
true and estimated unwrapped phase modulo ${2\pi}$ across the spatial samples
of the synthetic aperture. Fig. 6 illustrates the progression of the scalar
${\delta}$ from (67).

##  5 Summary

In this paper a novel alternating projections algorithm is derived for
estimating signal phase at each spatial sample of a synthetic aperture where
the receive probe is a Rydberg quantum sensor that measures electric field
intensity.

## References

  * [1] A. Walther, “The question of phase retrieval in optics,” _Journal of Modern Optics_ , vol. 10, no. 1, pp. 41–49, 1963.
  * [2] R. W. Harrison, “Phase problem in crystallography,” _Journal of the Optical Society of America_ , vol. 10, no. 5, pp. 1046–1055, 1993.
  * [3] R. P. Millane, “Phase retrieval in crystallography and optics,” _Journal of the Optical Society of America_ , vol. 7, no. 3, pp. 394–411, 1990.
  * [4] L. Rabiner and B.-H. Juang, _Fundamentals of speech recognition_. Prentice Hall, 1993.
  * [5] B. Baykal, “Blind channel estimation via combining autocorrelation and blind phase estimation,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 51, no. 6, pp. 1125–1131, 2004.
  * [6] J. C. Fienup and J. R. Dainty, “Phase retrieval and image reconstruction for astronomy,” in _Image Recovery: Theory and Application_ , H. Stark, Ed. Academic Press, 1987, pp. 231–275.
  * [7] K. Huang, Y. C. Eldar, and N. D. Sidiropoulos, “Phase retrieval from 1d fourier measurements: Convexity, uniqueness, and algorithms,” _arXiv preprint arXiv:1603.05215_ , 2016.
  * [8] J. Ranieri, A. Chebira, Y. M. Lu, and M. Vetterli, “Phase retrieval for sparse signals: Uniqueness conditions,” _arXiv preprint arXiv:1308.3058_ , 2013\.
  * [9] Y. Shechtman, Y. C. Eldar, O. Cohen, H. N. Chapman, J. Miao, and M. Segev, “Phase retrieval with application to optical imaging: A contemporary overview,” _IEEE Signal Processing Magazine_ , vol. 32, no. 3, pp. 87–109, 2015.
  * [10] K. Jaganathan, Y. C. Eldar, and B. Hassibi, “Phase retrieval: An overview of recent developments,” _arXiv preprint arXiv:1510.07713_ , 2015.
  * [11] Y. Shechtman, Y. C. Eldar, A. Szameit, and M. Segev, “Sparsity based sub-wavelength imaging with partially incoherent light via quadratic compressed sensing,” _Optics Express_ , vol. 19, no. 16, pp. 14 807–14 822, 2011.
  * [12] K. Jaganathan, S. Oymak, and B. Hassibi, “Sparse phase retrieval: Uniqueness guarantees and recovery algorithms,” _arXiv preprint arXiv:1311.2745_ , 2013\.
  * [13] Y. Shechtman, A. Beck, and Y. C. Eldar, “GESPAR: Efficient phase retrieval of sparse signals,” _IEEE Transactions on Signal Processing_ , vol. 62, no. 4, pp. 928–938, 2014.
  * [14] E. J. Candès, X. Li, and M. Soltanolkotabi, “Phase retrieval from coded diffraction patterns,” _Applied and Computational Harmonic Analysis_ , vol. 39, no. 2, pp. 277–299, 2015.
  * [15] Y. C. Eldar, P. Sidorenko, D. G. Mixon, S. Barel, and O. Cohen, “Sparse phase retrieval from short-time Fourier measurements,” _IEEE Signal Processing Letters_ , vol. 22, no. 5, pp. 638–642, 2015.
  * [16] K. Jaganathan, Y. C. Eldar, and B. Hassibi, “STFT phase retrieval: Uniqueness guarantees and recovery algorithms,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 10, no. 4, pp. 770–781, 2016.
  * [17] T. Bendory and Y. C. Eldar, “Non-convex phase retrieval from stft measurements,” _arXiv preprint arXiv:1607.08218_ , 2016.
  * [18] A. Artusio-Glimpse, M. T. Simons, N. Prajapati, and C. L. Holloway, “Modern rf measurements with hot atoms: A technology review of rydberg atom-based radio frequency field sensors,” _IEEE Microwave Magazine_ , vol. 23, no. 5, pp. 44–56, 2022.
  * [19] J. A. Gordon, C. L. Holloway, S. Jefferts, and T. Heavner, “Quantum-based si traceable electric-field probe,” in _2010 IEEE International Symposium on Electromagnetic Compatibility_ , 2010, pp. 321–324.
  * [20] J. A. Sedlacek, A. Schwettmann, H. Kübler, R. Löw, T. Pfau, and J. P. Shaffer, “Microwave electrometry with Rydberg atoms in a vapour cell using bright atomic resonances,” _Nature Physics_ , vol. 8, no. 11, pp. 819–824, Nov. 2012.
  * [21] C. L. Holloway, J. A. Gordon, S. Jefferts, A. Schwarzkopf, D. A. Anderson, S. A. Miller, N. Thaicharoen, and G. Raithel, “Broadband rydberg atom-based electric-field probe for si-traceable, self-calibrated measurements,” _IEEE Transactions on Antennas and Propagation_ , vol. 62, no. 12, pp. 6169–6182, 2014.
  * [22] J. A. Gordon, C. L. Holloway, A. Schwarzkopf, D. A. Anderson, S. Miller, N. Thaicharoen, and G. Raithel, “Millimeter wave detection via autler-townes splitting in rubidium rydberg atoms,” _Applied Physics Letters_ , vol. 105, no. 2, p. 024104, 2014. [Online]. Available: https://doi.org/10.1063/1.4890094
  * [23] D. A. Anderson, R. E. Sapiro, and G. Raithel, “A self-calibrated si-traceable rydberg atom-based radio frequency electric field probe and measurement instrument,” _IEEE Transactions on Antennas and Propagation_ , vol. 69, no. 9, pp. 5931–5941, 2021.
  * [24] M. T. Simons, J. A. Gordon, and C. L. Holloway, “Fiber-coupled vapor cell for a portable rydberg atom-based radio frequency electric field sensor,” _Appl. Opt._ , vol. 57, no. 22, pp. 6456–6460, Aug 2018. [Online]. Available: http://www.osapublishing.org/ao/abstract.cfm?URI=ao-57-22-6456
  * [25] D. A. Anderson, G. A. Raithel, E. G. Paradis, and R. E. Sapiro, “Atom-based electromagnetic field sensing element and measurement system,” Application 17/838 954, 2022.
  * [26] C. L. Holloway, M. T. Simons, A. H. Haddab, C. J. Williams, and M. W. Holloway, “A “real-time” guitar recording using rydberg atoms and electromagnetically induced transparency: Quantum physics meets music,” _AIP Advances_ , vol. 9, no. 6, p. 065110, 2019. [Online]. Available: https://doi.org/10.1063/1.5099036
  * [27] D. H. Meyer, P. D. Kunz, and K. C. Cox, “Waveguide-coupled rydberg spectrum analyzer from 0 to 20 ghz,” _Phys. Rev. Applied_ , vol. 15, p. 014053, Jan 2021. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevApplied.15.014053
  * [28] M. T. Simons, A. H. Haddab, J. A. Gordon, and C. L. Holloway, “A rydberg atom-based mixer: Measuring the phase of a radio frequency wave,” _Applied Physics Letters_ , vol. 114, no. 11, p. 114101, 2019.
  * [29] M. Jing, Y. Hu, J. Ma, H. Zhang, L. Zhang, L. Xiao, and S. Jia, “Atomic superheterodyne receiver based on microwave-dressed Rydberg spectroscopy,” _Nature Physics_ , vol. 16, no. 9, pp. 911–915, Jun. 2020\.
  * [30] D. Anderson, R. Sapiro, L. Gonçalves, R. Cardman, and G. Raithel, “Optical radio-frequency phase measurement with an internal-state rydberg atom interferometer,” _Phys. Rev. Applied_ , vol. 17, p. 044020, Apr 2022. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevApplied.17.044020
  * [31] R. Cardman, L. F. Gonçalves, R. E. Sapiro, G. Raithel, and D. A. Anderson, “Atomic 2d electric field imaging of a yagi–uda antenna near-field using a portable rydberg-atom probe and measurement instrument,” _Advanced Optical Technologies_ , vol. 9, no. 5, 2020.
  * [32] M. T. Simons, J. A. Gordon, and C. L. Holloway, “Simultaneous use of cs and rb rydberg atoms for dipole moment assessment and rf electric field measurements via electromagnetically induced transparency,” _Journal of Applied Physics_ , vol. 120, no. 12, p. 123103, 2016.
  * [33] C. R. J. Roger A. Horn, _Matrix Analysis_. Cambridge University Press, 1999.
  * [34] S. T. Smith, “Optimum phase-only adaptive nulling,” _IEEE Transactions on Signal Processing_ , vol. 47, no. 7, pp. 1835–1843, 1999.