#  Learning feedback control strategies for quantum metrology

Alessio Fallani  Dipartimento di Fisica “Aldo Pontremoli”, Università degli
Studi di Milano, I-20133 Milano, Italy  Matteo A. C. Rossi  InstituteQ - the
Finnish Quantum Institute, Aalto University, Finland  QTF Centre of
Excellence, Department of Applied Physics, Aalto University, FI-00076 Aalto,
Finland  Algorithmiq Ltd., Kanavakatu 3C, FI-00160 Helsinki, Finland  Dario
Tamascelli  Dipartimento di Fisica “Aldo Pontremoli”, Università degli Studi
di Milano, I-20133 Milano, Italy  Marco G. Genoni  Dipartimento di Fisica
“Aldo Pontremoli”, Università degli Studi di Milano, I-20133 Milano, Italy

###### Abstract

We consider the problem of frequency estimation for a single bosonic field
evolving under a squeezing Hamiltonian and continuously monitored via homodyne
detection. In particular, we exploit reinforcement learning techniques to
devise feedback control strategies achieving increased estimation precision.
We show that the feedback control determined by the neural network greatly
surpasses in the long-time limit the performances of both the “no-control”
strategy and the standard “open-loop control” strategy, which we considered as
benchmarks. We indeed observe how the devised strategy is able to optimize the
nontrivial estimation problem by preparing a large fraction of trajectories
corresponding to more sensitive quantum conditional states.

##  I Introduction

The goal of quantum metrology is to devise strategies able to exploit purely
quantum properties, such as entanglement and squeezing, in order to estimate
parameters with a precision beyond the one obtainable via classical means [1,
2]. In the classical domain it is usual to study estimation strategies based
on the continuous monitoring of a system, leading to sensors that have
applications ranging from engineering to medicine.

This kind of approach is particular interesting in the context of quantum
metrology with continuously monitored quantum systems [3, 4]. The role of
continuous measurements is indeed twofold: on the one hand, as it happens
classically, the measurement output is exploited to acquire information on the
parameters characterizing the system; on the other, the act of measuring
alters the state of the system itself, thus opening the possibility of
dynamically prepare more sensitive quantum probes. Several works have been
proposed in the literature, both discussing the fundamental statistical tools
to assess the precision achievable in this framework [5, 6, 7, 8, 9, 10, 11,
12], and presenting practical estimation strategies [13, 14, 15, 16, 17, 18,
19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30].

Moreover, in the context of continuously monitored quantum systems, it is also
natural to study strategies able to exploit this information in order to steer
the evolution towards a desired quantum state via feedback control [3, 31].
Much effort has been devoted to the design of strategies able to generate
metrologically relevant quantum states, such as squeezed states [32, 33, 34,
35, 36, 37, 38, 39, 40, 41], or to cool optomechanical systems towards their
ground state, with the outstanding experimental results recently observed in
Refs. [42, 43, 44].

Reinforcement learning (RL) is one of the main paradigms of machine learning,
together with supervised and unsupervised learning. In RL an agent learns how
to perform a task by acting on a system and updating its policy through a
reward/punishment mechanism [45]. The introduction of deep neural networks in
RL has led to formidable results: machines have, e.g., learned how to play
video games [46] or how to beat expert human players at complex board games
like Go [47].

RL has been recently applied in the context of quantum information, and more
in general to quantum technology, to find optimal strategies for some
designated tasks [48], ranging from optimizing feedback for quantum error
correction [49] to quantum control strategies [50, 51], and from optimizing
quantum transport [52, 53] to quantum compiling [54], or even to solve the
Rubik’s cube by exploiting quantum mechanics [55]. Recently, RL has been also
employed to optimize feedback control protocols in continuously monitored
quantum systems [56, 57, 58], with a main focus on quantum state engineering.

Discovering feedback strategies, where decisions are based on previously
observed measurement results, is indeed a challenging task. The stochastic
nature of the problem, together with the presence of feedback mechanisms,
leads to a doubly-exponential growth of the space of possible strategies with
respect to the number of time steps. Such a task falls therefore beyond the
scope of standard optimal control, and also supervised learning, techniques
[57]. On the other end, it suits the RL paradigm: the agent explores the
problem space by performing random experiments on the system while learning,
at the same time, an action policy.

In this work we exploit RL to design a feedback strategy optimizing a given
non trivial metrological problem. In particular, we consider the estimation of
the frequency of a harmonic oscillator subjected to a squeezing Hamiltonian
and undergoing a continuous homodyne detection. Differently form previous work
[56, 57, 58], where the goal was the preparation of a given target state to be
exploited in a selected quantum information protocol, in this paper we aim to
optimize real-time feedback for quantum metrology purposes, as to attain a
high precision in parameter estimation without targeting the preparation of a
precise quantum state.

We show that the feedback strategy determined by RL provides a high precision
in parameter estimation, and overcomes the performance of some benchmark
approaches. Interestingly enough, the feedback protocol determined by the
agent optimizes the interplay between the squeezing direction and the
displacement. Given the stochastic nature of the dynamics induced by the
measurement back-action, such a strategy is highly non-trivial and can not be
easily obtained with standard optimal control techniques or supervised
learning techniques.

The manuscript is organized as follows: In Sec. II we present the physical
model and the estimation problem; in Sec. III we introduce the figures of
merit that we will employ to assess the protocols and we discuss the role of
squeezing and feedback in the estimation procedure. In Sec. IV we show how we
apply RL to our problem and in Sec. V we present our main results. We conclude
our manuscript in Sec. VI with a brief discussion and some outlooks.

##  II The estimation problem

We consider a single bosonic mode described by the quadrature operators
$(\hat{q},\hat{p})$ satisfying the canonical commutation relation
$[\hat{q},\hat{p}]=i\mathbbm{1}$ [59]. The evolution of the mode is determined
by the Hamiltonian

$\displaystyle\hat{H}_{0}=\omega\,\hat{a}^{\dagger}\hat{a}+\chi(\hat{a}^{2}+\hat{a}^{{\dagger}2})\,$
(1)

with $\hat{a}=(\hat{q}+i\hat{p})/\sqrt{2}$ denoting the annihilation operator.
The first term simply corresponds to the usual free quantum oscillator
Hamiltonian, characterized by a frequency $\omega$; the second is a single-
mode squeezing term able to generate, for $\omega=0$ and $\chi>0$, squeezing
in the $\hat{q}$ quadrature. We remind here that a quantum state is said to be
squeezed if it presents fluctuations of a quadrature operator below the vacuum
shot-noise. The amount of squeezing of a quantum state $\varrho$, for example
for the $\hat{q}$ quadrature, is typically evaluated in dB according to the
formula

$\displaystyle\xi_{\sf
dB}=-10\log_{10}(\langle\Delta\hat{q}^{2}\rangle/\langle\Delta\hat{q}^{2}\rangle_{0})\,,$
(2)

where we have denoted with $\langle\Delta\hat{q}^{2}\rangle$ and
$\langle\Delta\hat{q}^{2}\rangle_{0}=1/2$ the variance of $\hat{q}$ evaluated
respectively for the quantum state $\varrho$ and for the vacuum state
$|0\rangle$. More in general, the maximum amount of squeezing of a single-mode
quantum state along a generic quadrature operator can be evaluated as
$\xi_{\sf dB}=-10\log_{10}(\lambda_{-})$, where $\lambda_{-}$ denotes the
minimum eigenvalue of its covariance matrix $\bm{\sigma}$ (see more details on
covariance matrices and Gaussian formalism in Appendix B).

Physically, the Hamiltonian (1) describes an optical parametric oscillator
(OPO) that is a cavity mode with resonance frequency $\omega_{c}$ interacting
with a nonlinear crystal and driven by a laser with frequency $\omega_{l}$. It
can be indeed obtained by going to a frame rotating at the laser frequency,
with $\omega=\omega_{c}-\omega_{l}$ denoting the detuning between cavity
resonance and laser. In what follows we focus on the problem of estimating the
fixed, but unknown, value of this detuning parameter $\omega$. This kind of
estimation problem has been recently discussed in the standard open-system
scenario for a circuit-QED implementation, by also considering the usefulness
of an extra Kerr-type nonlinearity in [60]. We remark that in the continuous-
variable scenario one typically considers the estimation of an optical phase
accumulated during a finite time evolution [61, 62]. Phase estimation and
frequency estimation are, however, fundamentally equivalent and we will focus
on the latter as in our setup we have to deal with with a time-continuous
evolution. While we phrase our results in terms of a quantum optical scenario,
we expect that our findings can be extended to other physical platforms where
frequency estimation is at the basis of quantum enhanced atomic clocks [63,
64] and quantum magnetometry [65].

In our setting, the cavity mode is subjected to loss at rate $\kappa$. The
output (leaking field) signal is then measured by means of a continuous
homodyne measurement, performed with efficiency $\eta$ (this parameter
$\eta=\eta_{D}\eta_{L}$ takes into account both the homodyne detector
efficiency $\eta_{D}$ and the fraction of output field $\eta_{L}$ that is not
collected by the detector). The corresponding continuous measurement outcome
can be written as

$\displaystyle
dy_{t}=\sqrt{\eta\kappa}\langle\hat{a}+\hat{a}^{\dagger}\rangle_{c}\,dt+dw_{t},$
(3)

where $\langle\cdot\rangle_{c}=\Tr[\varrho_{c}\cdot]$ denotes the expectation
over the conditional state $\varrho_{c}$, and $dw_{t}$ is a Wiener increment,
characterized by $\mathbbm{E}[dw_{t}]=0$ and $\mathbbm{E}[dw_{t}^{2}]=dt$.

Under these assumptions the evolution of the conditional state is governed by
the stochastic master equation [3]

$\displaystyle d\varrho_{c}$
$\displaystyle=-i[\hat{H}_{0},\varrho_{c}]\,dt+\kappa\mathcal{D}[\hat{a}]\varrho_{c}\,dt$
$\displaystyle+\sqrt{\eta\kappa}\mathcal{H}[\hat{a}]\varrho_{c}\,dw_{t}\,,$
(4)

where

$\displaystyle\mathcal{D}[\hat{a}]\varrho_{c}$
$\displaystyle=\hat{a}\varrho_{c}\hat{a}^{\dagger}-\frac{\hat{a}^{\dagger}\hat{a}\varrho_{c}+\varrho_{c}\hat{a}^{\dagger}\hat{a}}{2}\,,$
(5) $\displaystyle\mathcal{H}[\hat{a}]\varrho_{c}$
$\displaystyle=\hat{a}\varrho_{c}+\varrho_{c}\hat{a}^{\dagger}-\langle\hat{a}+\hat{a}^{\dagger}\rangle_{c}\,\varrho_{c}\,.$
(6)

Notice that the sequence of measures $\tilde{y}_{t}=\\{dy_{s}\\}_{s=0}^{t},\
0\leq s\leq t$ determines the trajectory followed by the conditional state
$\varrho_{c}$ up to time $t,$ and that the value of $\omega$ determines the
conditional joint probability density $p(\tilde{y}_{t}|\omega)$.

In particular, the stochastic master equation (4) for the conditional state
$\varrho_{c}$ is completely equivalent to the equations for its first moments
vector $\bar{\bf r}_{c}$ and covariance matrix $\bm{\sigma}_{c}$ [66, 67, 59]

$\displaystyle d\bar{\bf r}_{c}$ $\displaystyle=A\bar{\bf
r}_{c}\,dt+(E-\bm{\sigma}_{c}B)\frac{{\bf dw}_{t}}{\sqrt{2}}\,,$ (7)
$\displaystyle\frac{d\bm{\sigma}_{c}}{dt}$
$\displaystyle=A\bm{\sigma}_{c}+\bm{\sigma}_{c}A^{\sf
T}+D-(E-\bm{\sigma}_{c}B)(E-\bm{\sigma}_{c}B)^{\sf T}.$ (8)

The continuous measurement outcome (3) can be written in vectorial form as
${\bf dy}_{t}=-\sqrt{2}B^{\sf T}\bar{\bf r}_{c}\,dt+{\bf dw}_{t}$ with ${\bf
dw}_{t}$ the vector of uncorrelated Wiener increments entering also in Eq.
(7). We refer the reader to Appendix B for details on the matrices entering in
Eqs. (7) and (8).

It is important to remark here that the dynamics determined by the above
equations is stable, i.e. leads to a steady-state, if and only if the Hurwitz
condition ${\rm Re}[{\rm eigs}(A)]<0$ is satisfied, that is if the the real
part of the eigenvalues of the drift matrix $A$ is strictly smaller than zero.
In our case it corresponds to the inequality $\chi<|\kappa/2|$ and we will
always assume that this condition is fulfilled.

As we pointed out before, the Hamiltonian in Eq. (1) is able to generate
squeezing. If we focus on the unmonitored (unconditional) dynamics (i.e. for
$\eta=0$), the maximum squeezing at steady-state is obtained in the case of
$\omega=0$, leading to a steady-state variance of the $\hat{q}$ quadrature
$\langle\Delta\hat{q}^{2}\rangle_{\sf unc}=\kappa/(2\kappa+4\chi)$, that is
indeed below the vacuum limit $\langle\Delta\hat{q}^{2}\rangle_{0}=1/2$ for
$0<\chi<\kappa/2$ (for negative values of $\chi$ one would obtain squeezing
along the $\hat{p}$ quadrature). We observe, in particular, that the squeezing
increases approaching instability and that for $\chi\approx\kappa/2$ the well
known limit of $\xi_{\sf dB}=3$dB of squeezing is saturated [68, 69]. The
Riccati equation (8) can be analytically solved in this case and its solution
shows that continuous monitoring allows indeed to greatly enhance the
squeezing generation for the conditional states $\varrho_{c}$. In particular,
at steady-state and for $\eta=1$, a variance
$\langle\Delta\hat{q}^{2}\rangle_{\rm c}=(\kappa-2\chi)/(2\kappa)$ is
obtained, thus approaching infinite squeezing near criticality.

For $\omega\neq 0$, no analytical solution is available, but we find by
numerical means that a smaller, but still beyond the $3$dB limit, amount of
squeezing can be obtained at steady-state; in this case, moreover, the maximum
value of the squeezing corresponds, in general, to quadratures different from
$\hat{q}$ and $\hat{p}$.

##  III Frequency estimation, squeezing and feedback optimization

Our goal is to devise a protocol able to estimate the frequency parameter
$\omega$ with high precision. In particular we will compare the performance
achieved by our proposal with those of different alternative strategies that
will be detailed later in the manuscript. In all these strategies, information
on the unknown parameter is going to be obtained form two sources: the
continuous measurement outcome $y_{t}$ and a final strong measurement on the
corresponding conditional states $\varrho_{c}$.

The observation above is made rigorous by observing the form of the
corresponding quantum Cramér-Rao bound that applies in this scenario. As
customary in the context of frequency estimation, we will consider the total
time of the experiment $T$, divided in $M$ single runs of duration $t=T/M$ as
a fixed resource [70]. Under this assumption, one proves that the precision
$\delta\omega$ of any possible unbiased estimator is lower bounded as [28]

$\displaystyle\delta\omega\,\sqrt{T}\geq\frac{1}{\sqrt{\mathcal{Q}_{\sf
eff}/t}}\,,$ (9)

where we have defined the effective quantum Fisher information [12, 28]

$\displaystyle\mathcal{Q}_{\sf eff}=\mathcal{F}_{\sf
hom}+\bar{\mathcal{Q}}_{c}\,.$ (10)

The first term, defined as $\mathcal{F}_{\sf
hom}=\mathcal{F}[p(\tilde{y}_{t}|\omega)]$, does indeed correspond to the
classical Fisher information of the conditional probability of observing a
trajectory given the value of the parameter $\omega$, and thus to the
information obtainable via the continuous homodyne detection [71, 11]. The
second term, that we define as $\bar{\mathcal{Q}}_{c}=\mathbbm{E}_{\sf
traj}\left[\mathcal{Q}[\varrho_{c}]\right]$, is the average of the quantum
Fisher information (QFI) $\mathcal{Q}[\varrho_{c}]$ of the different
conditional states generated by the measurement: it thus quantifies the
average information obtainable via a final measurement on the different
trajectory-dependent $\varrho_{c}$ (we have introduced the notation
$\mathbbm{E}_{\sf traj}[\cdot]$ to denote the average over the conditional
distribution $p(\tilde{y}_{t}|\omega)$ of the different trajectories defined
by the stream of measurement outcomes $\tilde{y}_{t}$). Both these quantities
can be numerically obtained via the evolution of the first and second moments
of the conditional states ${\bf r}_{c}$, $\bm{\sigma}_{c}$, via their
derivatives with respect to the parameter, i.e. $\partial_{\omega}{\bf r}_{c}$
and $\partial_{\omega}\bm{\sigma}_{c}$, and by performing a Monte Carlo
average of the trajectories (see Appendix A for more details). According to
Eq. (9), the quantity $\mathcal{Q}_{\sf eff}/t$ will thus act as our figure of
merit to assess the different estimation protocols that we will discuss in the
next sections.

The feedback strategy we are going to consider later on exploits the
information obtained from the continuous measurement output $dy_{t}$ to
perform a unitary feedback operation [3] via the Hamiltonian $\hat{H}_{\sf
fb}=\omega_{\sf fb}(t)\,\hat{a}^{\dagger}\hat{a}$, that is by either changing
the laser or the cavity resonance frequency via the (possibly time-dependent)
parameter $\omega_{\sf fb}(t)$.

In order to better understand the motivation of a machine learning approach
for the optimization of such feedback strategy, it is expedient to discuss the
peculiar features of the estimation problem we are considering. As we have
discussed in Sec. II, via Hamiltonian $\hat{H}_{0}$ in Eq. (1), by fixing
$\omega=0$ and by assuming a positive coupling $\chi>0$, we know that
unconditional squeezing is generated for the quadrature $\hat{q}$, and that a
maximum of $3$dB can be obtained at steady-state near instability (that is for
$\chi\approx\kappa/2$) [68, 69]. However, if we also include a continuous
homodyne detection, as the one described by the stochastic master equation
(4), the squeezing of the conditional states can be greatly enhanced, going
well beyond the $3$dB limit.

The continuous monitoring has however also another effect on the conditional
state, that is, it gives a stochastic nonzero value for the first moments as
described in Eq. (7). Squeezing and nonzero first moments are the relevant
figures of merit for the estimation problem we are indeed considering.
Squeezing by itself is typically the most important resource for frequency
estimation (or analogously for phase estimation [61, 62]). However its
interplay with nonzero first moments may play a crucial role in determining
the estimation precision. A heuristic representation of this fact is given in
Fig. 1: we observe that squeezing could further enhance the estimation if the
squeezed quadrature is orthogonal to the direction of the first moment vector
$\bar{\bf r}_{c}$ in phase space. Remarkably, if $|\bar{\bf r}_{c}|$ is large
enough, squeezing for the quadrature parallel to the direction of $\bar{\bf
r}_{c}$ is going to be detrimental for the estimation of $\omega$. We remark
that in fact there is a non-trivial trade-off between the amount of squeezing
and $|\bar{\bf r}_{c}|$, as for small enough $|\bar{\bf r}_{c}|$, squeezing
along the wrong direction is still going to be a useful resource for
estimation. We thus expect that the RL agent will be able to optimize such
nontrivial problem, by devising feedback strategies able not only to generate
large squeezing but also to generate non-zero first moments, and, more
importantly to adjust their relative directions in phase space. We remark that
in general a final non-Gaussian measurement on the conditional states may be
needed in order to saturate the corresponding quantum Cramér-Rao bound.
However, as demonstrated in phase-estimation protocols with Gaussian states
[61, 72] and as we will describe in Appendix E for our results, a final
homodyne detection is going to extract in general a fair amount of the maximum
amount of information, being nearly optimal for pure Gaussian states.

Figure 1: We show a heuristic representation in phase-space of frequency
(phase) estimation via displaced squeezed state. The estimation precision on
the parameter $\omega$ can indeed be understood both qualitatively and
quantitatively [73] in terms of the distinguishability of quantum states
$\varrho_{\omega}$ and $\varrho_{\omega+\delta\omega}$ differing by an
infinitesimal value $\delta\omega$ of the parameter. In panel (a) we observe
that, if the state is displaced along the $\hat{p}$ axis of the phase space
and it is squeezed along $\hat{q}$, the two states are highly distinguishable,
and thus $\omega$ can be measured with high precision. However, as one can see
in panel (b), the quantum states become very indistinguishable in the opposite
case, that is for both displacement and squeezing along $\hat{q}$. Clearly the
situation is much worse for large values of $|\bar{\bf r}_{c}|$, while for
small first moments, the benefit of squeezing may still yield a high
estimation precision.

##  IV Applying Reinforcement Learning

As we mentioned in the Introduction, RL deals with a reward-based learning
paradigm. An agent learns how to achieve a certain goal by performing actions
on an environment, obtaining complete or partial information on its state, and
a reward, specifically designed for the goal.

In this framework the agent is trained over a number of simulations with
finite duration called episodes. At each timestep $dt$, we give the agent
access to all the possible information of the conditional state and on its
dependence on the parameter $\omega$, that is by considering as observations
the set of parameters ${\bf obs}=\left(\bar{\bf
r}_{c},\bm{\sigma}_{c},\partial_{\omega}\bar{\bf
r}_{c},\partial_{\omega}\bm{\sigma}_{c},{\bf dy}_{t}\right)$. We remark that
all these quantities can be updated at each time, according to Eqs. (7), (8),
(48), (49) once the continuous measurement result ${\bf dy}_{t}$ is obtained.
The agent then performs an action on the environment, which in our case
consists directly in the choice of a real value for the feedback parameter
$\omega_{\sf fb}$.

One of the most important steps in defining a RL problem is to identify the
correct reward function. As discussed in the previous section, we will assess
our feedback strategies via the effective QFI per time $\mathcal{Q}_{\sf
eff}/t$. We first observe that the Fisher information corresponding to the
continuous homodyne detection can be written as (more details in Appendix B)

$\displaystyle\mathcal{F}_{\sf hom}=\mathbbm{E}_{\sf traj}\left[2\int
dt\,(\partial_{\omega}\bar{\bf r}_{c})^{\sf T}BB^{\sf
T}(\partial_{\omega}\bar{\bf r}_{c})\right]\,.$ (11)

As a consequence we may write

$\displaystyle\frac{\mathcal{Q}_{\sf eff}}{t}=\mathbbm{E}_{\sf
traj}\left[\mathcal{R}\right]$ (12)

where we have defined a (positive) trajectory dependent quantity

$\displaystyle\mathcal{R}=\frac{2\int dt\,(\partial_{\omega}\bar{\bf
r}_{c})^{\sf T}BB^{\sf T}(\partial_{\omega}\bar{\bf
r}_{c})+\mathcal{Q}[\varrho_{c}]}{t}\,.$ (13)

This observation allows us to state that the maximization of $\mathcal{Q}_{\sf
eff}/t$ corresponds to the trajectory-wise maximization of $\mathcal{R}$ that
will thus act as our reward function (we remind the fact that, as described in
Appendix B, $\mathcal{Q}[\varrho_{c}]$ can be easily evaluated from the
properties of the Gaussian conditional state $\varrho_{c}$).

We here used the algorithm Proximal Policy Optimization (PPO) [74], a state-
of-the-art actor-critic algorithm where the agent is a neural network
optimizing both its evaluation of the future reward (critic) and its reward
maximization strategy (actor). We exploited the implementation of PPO
available in the package stable-baselines [75]. For this algorithm the
strategy, also called policy, is a stochastic one, meaning that the action of
the agent is extracted from a Gaussian distribution.

The agent we trained is a neural network with a feed-forward and fully
connected architecture, composed by an input layer of the size of the
observations connected to two distinct $64\times 64$ networks (one for the
actor and one for the critic). The network is trained using a gradient descent
method with linearly decreasing learning rate starting from a value of
$2.5\cdot 10^{-4}$ and an entropy coefficient of 0.001 and a discount factor
$\gamma=0.99$. At every step of training the loss function is evaluated on on
batches of 512 elements given by the experience of four parallel workers over
a time horizon of 128 timesteps. The total number of timesteps included in the
training is $30\cdot 10^{6}$ composed by consecutive simulations (episodes)
with finite duration of $10^{5}$ steps. At the beginning of each episode the
initial condition for the system is set randomly. More specifically, both
components of $\textbf{r}_{c}$ are set to be extracted from a uniform
distribution on the interval $[-3,3]$, while the number of initial thermal
excitations in the system is extracted from a uniform distribution on the
interval $[0,5]$.

##  V Results

In the following we will fix the unknown, but fixed, frequency, the squeezing
rate and the efficiency of the homodyne measurement respectively to
$\omega=0.1\kappa$, $\chi=0.49\kappa$ and $\eta=0.9$, with $\kappa$ the cavity
loss rate (see Eq. (3)). Our simulations show, however, that the agent is able
to devise optimized feedback strategies in different regimes; in Appendix D we
exemplify such flexibility of the proposed method by showing the results
obtained for different values of the monitoring efficiency $\eta$.

We will denote our figure of merit, that is the effective QFI in Eq. (10), as
$\mathcal{Q}_{\sf eff}^{\sf(RL)}$. We will compare and contrast the results
obtained by means of RL to two benchmark strategies: the one where no control
is applied, quantified by the figure of merit $\mathcal{Q}_{\sf
eff}^{\sf(0)}$, and the strategy where, thanks to some a-priori information on
the parameter $\omega$ (a typical assumption in the context of local quantum
estimation theory [73]), a deterministic value of the control frequency is
fixed as $\omega_{\sf fb}=-\omega$. Notice that in this latter case the
control is deterministic and thus it does not correspond to a feedback, but
rather to an open-loop (OL) control strategy, yielding the largest amount of
conditional squeezing along the quadrature $\hat{q}$. The continuous
monitoring, on the other hand, will yield a non-zero (but typically small)
stochastic contribution on the $\hat{q}$ axis of phase-space. As a consequence
the directions of squeezing and first moments will not be optimized. We will
denote the figure of merit for this open-loop control strategy as
$\mathcal{Q}_{\sf eff}^{\sf(OL)}$.

The main result of this work is presented in Fig. 2: We considered as initial
state a thermal state with $n_{\sf th}=5$ thermal excitations, and a first
moment vector $\bar{\bf r}_{c}=(0,0)$. We show that, apart from an initial
transient time where $\mathcal{Q}_{\sf eff}^{\sf(RL)}\lesssim\mathcal{Q}_{\sf
eff}^{\sf(OL)}$, the feedback protocol yields a much larger effective QFI than
the benchmark strategies considered. In particular, by looking at the
behaviour of the two terms entering in Eq. (10), we can make two main
observations: i) as regards the average QFIs of the conditional states, that
in Fig. 2 correspond to the difference between the curves with the same
colours, one finds
$\bar{\mathcal{Q}}_{c}^{\sf(0)}<\bar{\mathcal{Q}}_{c}^{\sf(RL)}<\bar{\mathcal{Q}}_{c}^{\sf(OL)}$,
that is, the feedback protocol is able to generate conditional states that are
on average more sensitive respect to the one generated without feedback, but
much less sensitive to the ones generated via the open-loop control protocol;
ii) the enhancement in the estimation is thus mainly obtained thanks to the
information contained in the continuous measurement outcomes: the monitoring
FI $\mathcal{F}_{\sf hom}^{\sf(RL)}$ greatly overcomes the values of the same
figure of merit for the two other protocols; iii) we observe that while for
the open-loop control protocol $\bar{\mathcal{Q}}_{c}^{\sf(OL)}$ saturates to
a given value once $\bm{\sigma}_{c}$ has reached its deterministic steady-
state, the RL agent seems able to keep $\mathcal{F}_{\sf hom}^{\sf(RL)}$
increasing steadily in time, yielding a large enhancement in the long time
limit.

Moreover, as in Eq. (9) we observe that the relevant figure of merit in
frequency estimation is $\mathcal{Q}_{\sf eff}/t$, if one allows to optimize
over the single experiment monitoring time $t$ at fixed total time of the
experiment $T$, the strategy devised by the agent clearly gives the best
result. Our results hint also to the fact that in this case the optimization
is obtained in the long-time limit, where the whole information is basically
completely contained in the continuous homodyne measurement outcomes and the
strong measurement on the conditional states is almost irrelevant (we however
refer the reader to Appendix E for a discussion on the effectiveness of
homodyne detection as a final strong measurement for the three strategies
considered).

Figure 2:  We compare the performances of the feedback strategy devised by the
neural network with the two benchmark strategies described in the main text as
a function of time. Solid lines correspond to the three different effective
QFI ($\mathcal{Q}_{\sf eff}^{\sf(RL)}$,$\mathcal{Q}_{\sf
eff}^{\sf(0)}$,$\mathcal{Q}_{\sf eff}^{\sf(OL)}$) divided by time, while
dashed correspond to the the continuous monitoring classical FIs:
($\mathcal{F}_{\sf hom}^{\sf(RL)}$,$\mathcal{F}_{\sf
hom}^{\sf(0)}$,$\mathcal{F}_{\sf hom}^{\sf(OL)}$), divided by time. The
average QFI of the conditional states
($\mathcal{\bar{Q}}_{c}^{\sf(RL)}$,$\mathcal{\bar{Q}}_{c}^{\sf(0)}$,$\mathcal{\bar{Q}}_{c}^{\sf(OL)}$)
can be derived as the difference between the two curves above.
The results have been obtained simulating $N=5000$ trajectories with a time-
step $dt=0.001/\kappa$, by fixing the parameters: $\omega=0.1\kappa$,
$\chi=0.49\kappa$, $\eta=0.9$ and by considering as an initial state a thermal
state with $n_{\sf th}=5$ and initial first moment vector $\bar{\bf
r}_{c}(0)=(0,0)$.

In order to better understand these results, it is useful to look at the
evolution of single trajectories and thus at the properties of the conditional
states. As mentioned before, the achievable estimation precision will depend
on the amount of squeezing generated during the dynamics and on its interplay
with the first-moment vector $\bar{\bf r}_{c}$. In Fig. 3 we compare the
values of the magnitude of the first moments averaged over the trajectories
$\mathbbm{E}[|\bar{\bf r}_{c}|]$ for the three protocols. We indeed observe
that the RL agent yields the largest values of $\mathbbm{E}[|\bar{\bf
r}_{c}|]$, while the OL-control protocol yields almost negligible first
moments.

Figure 3: We plot $\mathbbm{E}_{\sf traj}[|\bar{\bf r}_{c}|]$ as a function of
time for the three protocols considered (from top to bottom: RL-feedback, no-
control, open-loop control).
The results have been obtained simulating $N=5000$ trajectories with a time-
step $dt=0.001/\kappa$, by fixing the parameters: $\omega=0.1$,
$\chi=0.49\kappa$, $\eta=0.9$ and by considering as an initial state a thermal
state with $n_{\sf th}=5$ and initial first moment vector $\bar{\bf
r}_{c}(0)=(0,0)$.

We stressed before how squeezing is the main resource for this kind of
estimation. In this respect, we know that the maximum amount of squeezing is
generated deterministically in the OL-control protocol, yielding at steady
state with $\xi^{\sf(OL)}\approx 6.05$ dB of squeezing for the values we
considered in these simulations. However we discussed before that this
squeezing is always parallel to the corresponding vector
$\bar{\mathbf{r}}_{c}^{\sf(OL)}$; despite this fact and thanks to the fact the
first moments are close to zero, this protocol still yields large values of
$\mathcal{Q}[\varrho_{c}^{\sf(OL)}]$, as we indeed observed in Fig. 2.

If we now focus on the squeezing along the quadrature perpendicular to
$\bar{\bf r}_{c}$ and thus possibly enhancing the contribution due to non-zero
first moments in phase-space, we find non-trivial and definitely interesting
results as shown in Fig. 4.

Figure 4:  Histograms of the probability density of the squeezing $\xi$
(expressed in dB) perpendicular to the conditional first moment vector
$\bar{\bf r}_{c}$ for different times and for both the no-control protocol
(orange histograms) and for the RL agent feedback protocol (blue histograms).
The two vertical lines correspond to the two bounds on the amount of squeezing
for the no-control protocol $\xi^{(0)}$ and for the open-loop control protocol
$\xi^{\sf(OL)}$.
The results have been obtained simulating $N=5000$ trajectories with a time-
step $dt=0.001/\kappa$, by fixing the parameters: $\omega=0.1$,
$\chi=0.49\kappa$, $\eta=0.9$ and by considering as an initial state a thermal
state with $n_{\sf th}=5$ and initial first moment vector $\bar{\bf
r}_{c}(0)=(0,0)$.

In this figure we plot the histograms corresponding to the probability density
of squeezing perpendicular to the conditional first moment vector $\bar{\bf
r}_{c}$ for the no-control protocol and for the RL agent based feedback
protocol for different times. As regards the protocol without control, we know
that at steady-state one obtains a deterministic squeezing of
$\xi^{(0)}\approx 5.25$dB, and as a consequence the squeezing along the
quadrature perpendicular to the, stochastically varying, $\bar{\bf r}_{c}$ is
going to be bounded by this value. This behaviour is indeed confirmed by
looking at the orange histograms. If we now finally focus on the results
corresponding to the RL agent based feedback (blue histograms), we can clearly
observe how it is indeed able also to generate a large fraction of
trajectories with squeezing perpendicular to $\bar{\bf r}_{c}$ not only well
beyond the maximum value obtainable without control $\xi^{(0)}$, but also near
to the limit $\xi^{\sf(OL)}$ achieved by the open-loop control discussed
before and that we remind here is however always parallel to the first moment
vector $\bar{\mathbf{r}}_{c}$. In particular we observe that, not only the RL-
strategy is able to generate conditional states with the maximum squeezing
achievable and with the most useful direction, but also the mode of this
perpendicular squeezing distribution quickly saturates towards this limit
$\xi^{\sf(OL)}$. Our results thus suggest how the portion of trajectories
characterized by large first moments and large perpendicular squeezing is
responsible for the enhancement in the frequency estimation precision. We
refer to Appendices C and D for some extra results that we have obtained by
considering different values of the coupling constant $\chi$ and of the
monitoring efficiency $\eta$, and that further confirm our intuitions. For
example when one considers smaller values of $\chi$, and as a consequence
smaller amount of squeezing generated, all the the strategies considered yield
as expected smaller values of the effective QFI. Similarly, we show how for
smaller values of $\eta$, the effect on the squeezing generation is slightly
reduced and that the main contribution to the enhancement is given by the
first moment vector amplitude $|\bar{\bf r}_{c}|$.

It is also interesting to observe the behaviour of the feedback parameter
$\omega_{\sf fb}$ as a function of time, both for a sample trajectory and
averaged over the different trajectories. In Fig. 5 we find that the average
value seems to converge to a value near to $\mathbbm{E}[\omega_{\sf
fb}]\approx-\omega$, that is the one implemented in the open-loop control and
yielding the maximum squeezing. However at the trajectory level the
fluctuations of $\omega_{\sf fb}$ are evident and are thus crucial to increase
$|\bar{\bf r}_{c}|$ and to optimize both the squeezing magnitude and more
importantly its direction.

Figure 5: Feedback parameter $\omega_{\sf fb}$ obtained via the RL agent as a
function of time, both for few samples trajectories (corresponding to blue
lines with different shades of blue) and averaged over 5000 trajectories
(black line). The dashed black lines show the standard deviation. The average
of the RL is close to the value $-\omega$, yielding the maximum squeezing
(dotted black line). The other physical parameters are set as in the previous
figures.

Plainly speaking we can conclude that the feedback devised by the neural
network is able to optimize the non-trivial interplay between first moments
and squeezing and indeed to generate a significant amount of trajectories with
larger amount of perpendicular squeezing. These trajectories are thus
responsible for the enhancement in the estimation precision observed Fig. 2.
Our results show also that this feature is much more relevant for the homodyne
FI $\mathcal{F}_{\sf hom}$, that is indeed responsible to the enhancement
yielded by the feedback strategy. A hint in this direction is already given by
the formula (11) for $\mathcal{F}_{\sf hom}$, that depends directly on the
vector $\partial_{\omega}\bar{\bf r}_{c}$ (however we should remark that the
evolution of $\partial_{\omega}\bar{\bf r}_{c}$ in Eq. (48) depends also on
$\bm{\sigma}_{c}$ and thus on the squeezing properties of the conditional
states).

##  VI Conclusions

In this work we have shown how a RL algorithm has been able to optimize a
feedback strategy able to attain a high precision in frequency estimation. We
have understood the results in terms of the optimization of the interplay
between the amplitude and the squeezing generated by the protocol. This kind
of optimization is highly non-trivial: a simple strategy trying to optimize
this kind of feature at each time $t$ cannot be devised because of the
stochasticity of the subsequent evolution.

As a future outlook we aim to optimize the neural network, in order to be able
to reduce the needed observations ${\bf obs}$. In particular we will look at
strategies able to exploit just the real-time measurement output ${\bf
dy}_{t}$, and thus corresponding to Markovian feedback [32, 33]

We have witnessed a great experimental improvement in the implementation of
FPGA-based real-time state-based feedback, as shown recently in the context of
the cooling of mechanical oscillators [42, 43, 44]. We remark that once a
neural network has been trained, its real-time interrogation is not much more
computationally costly than what has been done in the cited experiments. We
are thus confident that feedback strategies previously trained via RL
algorithms can be efficently implemented in the next future for quantum
metrology purposes as we have described, or for more general quantum
technological tasks.

###### Acknowledgements.

We thank F. Albarelli and M. Paris for helpful discussions. MACR acknowledges
financial support from the Academy of Finland via the Centre of Excellence
program (Project no. 336810). MGG and DT acknowledge support from the Sviluppo
UniMi 2018 initiative. The computer resources of the Finnish IT Center for
Science (CSC) and the FGCI project (Finland) are acknowledged.

## References

  * Giovannetti  _et al._ [2011] V. Giovannetti, S. Lloyd, and L. Maccone, Nat. Photonics 5, 222 (2011), 1102.2318 .
  * Pirandola  _et al._ [2018] S. Pirandola, B. R. Bardhan, T. Gehring, C. Weedbrook, and S. Lloyd, Nature Photonics 12, 724 (2018).
  * Wiseman and Milburn [2010] H. M. Wiseman and G. J. Milburn,  _Quantum Measurement and Control_ (Cambridge University Press, New York, 2010).
  * Jacobs and Steck [2006] K. Jacobs and D. A. Steck, Contemp. Phys. 47, 279 (2006), quant-ph/0611067 .
  * Guţă  _et al._ [2007] M. Guţă, B. Janssens, and J. Kahn, Commun. Math. Phys. 277, 127 (2007), arXiv:0608074v3 [arXiv:quant-ph] .
  * Tsang  _et al._ [2011] M. Tsang, H. M. Wiseman, and C. M. Caves, Phys. Rev. Lett. 106, 090401 (2011).
  * Tsang [2013] M. Tsang, New J. Phys. 15, 73005 (2013), 1301.5733 .
  * Gammelmark and Mølmer [2013a] S. Gammelmark and K. Mølmer, Phys. Rev. A 87, 032115 (2013a), 1212.5700 .
  * Gammelmark and Mølmer [2014] S. Gammelmark and K. Mølmer, Phys. Rev. Lett. 112, 170401 (2014).
  * Guţă and Kiukas [2017] M. Guţă and J. Kiukas, J. Mat. Phys. 58, 052201 (2017).
  * Genoni [2017] M. G. Genoni, Phys. Rev. A 95, 012116 (2017), 1608.08429 .
  * Albarelli  _et al._ [2017] F. Albarelli, M. A. C. Rossi, M. G. A. Paris, and M. G. Genoni, New J. Phys. 19, 123011 (2017), 1706.00485 .
  * Mabuchi [1996] H. Mabuchi, Quant. Semiclass. Opt. 8, 1103 (1996).
  * Gambetta and Wiseman [2001] J. Gambetta and H. M. Wiseman, Phys. Rev. A 64, 042105 (2001).
  * Geremia  _et al._ [2003] J. M. Geremia, J. K. Stockton, A. C. Doherty, and H. Mabuchi, Phys. Rev. Lett. 91, 250801 (2003).
  * Mølmer and Madsen [2004] K. Mølmer and L. B. Madsen, Phys. Rev. A 70, 052102 (2004), quant-ph/0402158 .
  * Madsen and Mølmer [2004] L. B. Madsen and K. Mølmer, Phys. Rev. A 70, 052324 (2004).
  * Stockton  _et al._ [2004] J. K. Stockton, J. M. Geremia, A. C. Doherty, and H. Mabuchi, Phys. Rev. A 69, 032109 (2004), quant-ph/0309101 .
  * Tsang [2010] M. Tsang, Phys. Rev. A 81, 013824 (2010), arXiv:0909.2432 .
  * Wheatley  _et al._ [2010] T. A. Wheatley, D. W. Berry, H. Yonezawa, D. Nakane, H. Arao, D. T. Pope, T. C. Ralph, H. M. Wiseman, A. Furusawa, and E. H. Huntington, Phys. Rev. Lett. 104, 093601 (2010).
  * Yonezawa  _et al._ [2012] H. Yonezawa, D. Nakane, T. A. Wheatley, K. Iwasawa, S. Takeda, H. Arao, K. Ohki, K. Tsumura, D. W. Berry, T. C. Ralph, H. M. Wiseman, E. H. Huntington, and A. Furusawa, Science 337, 1514 (2012).
  * Cook  _et al._ [2014] R. L. Cook, C. A. Riofrío, and I. H. Deutsch, Phys. Rev. A 90, 032113 (2014), arXiv:1406.4482 .
  * Six  _et al._ [2015] P. Six, P. Campagne-Ibarcq, L. Bretheau, B. Huard, and P. Rouchon, in  _2015 54th IEEE Conf. Decis. Control_, Cdc (IEEE, 2015) p. 7742.
  * Kiilerich and Mølmer [2016] A. H. Kiilerich and K. Mølmer, Phys. Rev. A 94, 032103 (2016).
  * Cortez  _et al._ [2017] L. Cortez, A. Chantasri, L. P. García-Pintos, J. Dressel, and A. N. Jordan, Phys. Rev. A 95, 012314 (2017), 1606.01407 .
  * Ralph  _et al._ [2017] J. F. Ralph, S. Maskell, and K. Jacobs, Phys. Rev. A 96, 052306 (2017), 1707.04725 .
  * Atalaya  _et al._ [2018] J. Atalaya, S. Hacohen-Gourgy, L. S. Martin, I. Siddiqi, and A. N. Korotkov, npj Quantum Inf. 4, 41 (2018), arXiv:1702.08077 .
  * Albarelli  _et al._ [2018] F. Albarelli, M. A. C. Rossi, D. Tamascelli, and M. G. Genoni, Quantum 2, 110 (2018).
  * Shankar  _et al._ [2019] A. Shankar, G. P. Greve, B. Wu, J. K. Thompson, and M. Holland, Phys. Rev. Lett. 122, 233602 (2019).
  * Rossi  _et al._ [2020] M. A. C. Rossi, F. Albarelli, D. Tamascelli, and M. G. Genoni, Phys. Rev. Lett. 125, 200505 (2020).
  * Doherty and Jacobs [1999] A. C. Doherty and K. Jacobs, Phys. Rev. A 60, 2700 (1999).
  * Wiseman and Milburn [1993] H. M. Wiseman and G. J. Milburn, Phys. Rev. Lett. 70, 548 (1993).
  * Wiseman and Milburn [1994] H. M. Wiseman and G. J. Milburn, Phys. Rev. A 49, 1350 (1994).
  * Thomsen  _et al._ [2002] L. K. Thomsen, S. Mancini, and H. M. Wiseman, Phys. Rev. A 65, 061801 (2002), quant-ph/0202028 .
  * Serafini and Mancini [2010] A. Serafini and S. Mancini, Phys. Rev. Lett. 104, 220501 (2010).
  * Szorkovszky  _et al._ [2011] A. Szorkovszky, A. C. Doherty, G. I. Harris, and W. P. Bowen, Phys. Rev. Lett. 107, 213603 (2011).
  * Genoni  _et al._ [2013] M. G. Genoni, S. Mancini, and A. Serafini, Phys. Rev. A 87, 042333 (2013).
  * Genoni  _et al._ [2015] M. G. Genoni, J. Zhang, J. Millen, P. F. Barker, and A. Serafini, New Journal of Physics 17, 073019 (2015).
  * Hofer and Hammerer [2015] S. G. Hofer and K. Hammerer, Phys. Rev. A 91, 033822 (2015), 1411.1337 .
  * Brunelli  _et al._ [2019] M. Brunelli, D. Malz, and A. Nunnenkamp, Phys. Rev. Lett. 123, 093602 (2019).
  * Di Giovanni  _et al._ [2021] A. Di Giovanni, M. Brunelli, and M. G. Genoni, Phys. Rev. A 103, 022614 (2021).
  * Rossi  _et al._ [2018] M. Rossi, D. Mason, J. Chen, Y. Tsaturyan, and A. Schliesser, Nature 563, 53 (2018).
  * Magrini  _et al._ [2021] L. Magrini, P. Rosenzweig, C. Bach, A. Deutschmann-Olek, S. G. Hofer, S. Hong, N. Kiesel, A. Kugi, and M. Aspelmeyer, Nature 595, 373 (2021).
  * Tebbenjohanns  _et al._ [2021] F. Tebbenjohanns, M. L. Mattana, M. Rossi, M. Frimmer, and L. Novotny, Nature 595, 378 (2021).
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto,  _Reinforcement learning: An introduction_ (MIT press, 2018).
  * Mnih  _et al._ [2015] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis, Nature 518, 529 (2015).
  * Silver  _et al._ [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap, F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis, Nature 550, 354 (2017).
  * Marquardt [2021] F. Marquardt, SciPost Phys. Lect. Notes , 29 (2021).
  * Fösel  _et al._ [2018] T. Fösel, P. Tighineanu, T. Weiss, and F. Marquardt, Phys. Rev. X 8, 031084 (2018).
  * Mavadia  _et al._ [2017] S. Mavadia, V. Frey, J. Sastrawan, S. Dona, and M. J. Biercuk, Nature Communications 8, 14106 (2017).
  * Niu  _et al._ [2019] M. Y. Niu, S. Boixo, V. N. Smelyanskiy, and H. Neven, npj Quantum Information 5, 33 (2019).
  * Porotti  _et al._ [2019] R. Porotti, D. Tamascelli, M. Restelli, and E. Prati, Communications Physics 2, 61 (2019).
  * Brown  _et al._ [2021] J. Brown, P. Sgroi, L. Giannelli, G. S. Paraoanu, E. Paladino, G. Falci, M. Paternostro, and A. Ferraro, “Reinforcement learning-enhanced protocols for coherent population-transfer in three-level quantum systems,” (2021), arXiv:2109.00973 [quant-ph] .
  * Moro  _et al._ [2021] L. Moro, M. G. A. Paris, M. Restelli, and E. Prati, Communications Physics 4, 178 (2021).
  * Corli  _et al._ [2021] S. Corli, L. Moro, D. E. Galli, and E. Prati, Journal of Physics A: Mathematical and Theoretical 54, 425302 (2021).
  * Borah  _et al._ [2021] S. Borah, B. Sarma, M. Kewming, G. J. Milburn, and J. Twamley, arXiv:2104.11856 [physics, physics:quant-ph] (2021), arXiv: 2104.11856.
  * Porotti  _et al._ [2021] R. Porotti, A. Essig, B. Huard, and F. Marquardt, “Deep reinforcement learning for quantum state preparation with weak nonlinear measurements,” (2021), arXiv:2107.08816 [quant-ph] .
  * Evans  _et al._ [2021] E. N. Evans, Z. Wang, A. G. Frim, M. R. DeWeese, and E. A. Theodorou, “Stochastic optimization for learning quantum state feedback control,” (2021), arXiv:2111.09896 [quant-ph] .
  * Serafini [2017] A. Serafini,  _Quantum Continuous Variables_ (CRC Press, Boca Raton, 2017).
  * Candia  _et al._ [2021] R. D. Candia, F. Minganti, K. V. Petrovnin, G. S. Paraoanu, and S. Felicetti, “Critical parametric quantum sensing,” (2021), arXiv:2107.04503 [quant-ph] .
  * Monras [2006] A. Monras, Phys. Rev. A 73, 033821 (2006).
  * Genoni  _et al._ [2011] M. G. Genoni, S. Olivares, and M. G. A. Paris, Phys. Rev. Lett. 106, 153603 (2011).
  * Schleier-Smith  _et al._ [2010] M. H. Schleier-Smith, I. D. Leroux, and V. Vuletić, Phys. Rev. Lett. 104, 073604 (2010).
  * Leroux  _et al._ [2010] I. D. Leroux, M. H. Schleier-Smith, and V. Vuletić, Phys. Rev. Lett. 104, 073602 (2010).
  * Wasilewski  _et al._ [2010] W. Wasilewski, K. Jensen, H. Krauter, J. J. Renema, M. V. Balabas, and E. S. Polzik, Phys. Rev. Lett. 104, 133601 (2010).
  * Wiseman and Doherty [2005] H. M. Wiseman and A. C. Doherty, Phys. Rev. Lett. 94, 070405 (2005), 0408099v4 [quant-ph] .
  * Genoni  _et al._ [2016] M. G. Genoni, L. Lami, and A. Serafini, Contemp. Phys. 57, 331 (2016).
  * Milburn and Walls [1981] G. Milburn and D. Walls, Optics Communications 39, 401 (1981).
  * Collett and Gardiner [1984] M. J. Collett and C. W. Gardiner, Phys. Rev. A 30, 1386 (1984).
  * Huelga  _et al._ [1997] S. F. Huelga, C. Macchiavello, T. Pellizzari, A. K. Ekert, M. B. Plenio, and J. I. Cirac, Phys. Rev. Lett. 79, 3865 (1997), quant-ph/9707014 .
  * Gammelmark and Mølmer [2013b] S. Gammelmark and K. Mølmer, Phys. Rev. A 87, 032115 (2013b), 1212.5700 .
  * Oh  _et al._ [2019] C. Oh, C. Lee, C. Rockstuhl, H. Jeong, J. Kim, H. Nha, and S.-Y. Lee, npj Quantum Information 5, 10 (2019).
  * Paris [2009] M. G. A. Paris, Int. J. Quant. Inf. 07, 125 (2009).
  * Schulman  _et al._ [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, arXiv preprint arXiv:1707.06347 (2017).
  * Hill  _et al._ [2018] A. Hill, A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, and Y. Wu, “Stable baselines,” https://github.com/hill-a/stable-baselines (2018).
  * Pinel  _et al._ [2013] O. Pinel, P. Jian, N. Treps, C. Fabre, and D. Braun, Phys. Rev. A 88, 040102 (2013), arXiv:1307.4637v1 .

##  Appendix A Quantum metrology with continuously monitored quantum system

We start by giving a basic introduction on quantum estimation theory. Let us
consider a quantum statistical model, that is a family of quantum states
$\varrho_{\omega}$ parametrized by a parameter $\omega$ that we want to
estimate. We now suppose to repeat $M$ times a measurement, corresponding to a
certain POVM $\\{\Pi_{x}\\}$, on the quantum state, and thus collecting a set
of measurement outcomes $\\{x_{j}\\}_{j=1}^{M}$. One can prove that the
precision of any unbiased estimator $\tilde{\omega}$, that is a map from the
measurement outcomes $\\{x_{j}\\}$ to the range of parameters taken by the
$\omega$ is lower bounded according the Cramér-Rao bound

$\displaystyle\delta\omega\geq\frac{1}{\sqrt{M\,\mathcal{F}[p(x|\omega)]}}\,,$
(14)

where we have introduced the classical Fisher information

$\displaystyle\mathcal{F}[p(x|\omega)]$
$\displaystyle=\sum_{x}\frac{\left(\partial_{\omega}p(x|\omega)\right)^{2}}{p(x|\omega)}\,,$
(15)
$\displaystyle=\mathbbm{E}_{p(x|\omega)}\left[\left(\frac{\partial_{\omega}p(x|\omega)}{p(x|\omega)}\right)^{2}\right]\,,$
(16)

and we have denoted with $p(x|\omega)=\Tr[\varrho_{\omega}\Pi_{x}]$ the
probability of obtaining the outcome $x$ from the measurement. One can further
optimize over all the possible measurements (POVM) $\\{\Pi_{x}\\}$ that one
can perform on the quantum state $\varrho_{\omega}$, obtaining the quantum
Cramér-Rao bound

$\displaystyle\delta\omega\geq\frac{1}{\sqrt{M\,\mathcal{F}[p(x|\omega)]}}\geq\frac{1}{\sqrt{M\,\mathcal{Q}[\varrho_{\omega}]}}\,,$
(17)

where we have introduced the quantum Fisher information (QFI)

$\displaystyle\mathcal{Q}[\varrho_{\omega}]$
$\displaystyle=\Tr[\varrho_{\omega}L_{\omega}^{2}]\,,$ (18)

written in terms of the symmetric logarithmic derivative defined via the
Lyapunov equation

$\displaystyle\frac{\partial\varrho_{\omega}}{\partial\omega}$
$\displaystyle=\frac{L_{\omega}\varrho_{\omega}+\varrho_{\omega}L_{\omega}}{2}\,.$
(19)

Several alternative formulas can be derived for the QFI can be derived, based
on the diagonalization of the state $\varrho_{\omega}$ or based on the
fidelity between states characterized by parameters $\omega$ differing by an
infinitesimal value [73].

If we want to estimate a parameter in continuously monitored quantum system,
at each run of the experiment we obtain a a continuous measurement output (for
example in the case of continuous homodyne detection) $\tilde{y}_{t}$ with a
certain probability distribution $p_{\sf hom}=p(\tilde{y}_{t}|\omega)$ and
corresponding to a particular trajectory for the quantum conditional state of
the system $\varrho_{c}$. In this framework one proves that the bound on the
estimation precision can be written as [12]

$\displaystyle\delta\omega\geq\frac{1}{\sqrt{M\left(\mathcal{F}_{\sf
hom}+\mathbbm{E}_{\sf traj}\left[\mathcal{Q}[\varrho_{c}]\right]\right)}}\,.$
(20)

The relevant figure of merit it thus the effective QFI

$\displaystyle\mathcal{Q}_{\sf eff}=\mathcal{F}_{\sf hom}+\mathbbm{E}_{\sf
traj}\left[\mathcal{Q}[\varrho_{c}]\right]\,,$ (21)

corresponding to the sum of the Fisher information quantifying the information
obtainable from the continuous homodyne results, plus the average of the
quantum Fisher information of the conditional states, quantifying the
information obtainable from a final measurement on $\varrho_{c}$.

##  Appendix B Gaussian conditional dynamics and numerical evaluation of the
effective QFI

We here briefly review how to treat the evolution of continuously monitored
quantum Gaussian states by following the approach in [59, 67] and how to
evaluate the different figures of merit relevant for our purposes.

A Gaussian quantum state $\varrho$ of a continuous-variable quantum system is
completely identified by its first moment vector $\bar{\bf
r}=\Tr[\varrho\hat{\bf r}]$, and its covariance matrix
$\bm{\sigma}=\Tr[\varrho_{c}\\{\hat{\bf r}-\bar{\bf r},(\hat{\bf r}-\bar{\bf
r})^{\sf T}\\}]$, where $\\{\hat{\bf a},\hat{\bf b}\\}=\hat{\bf a}\hat{\bf
b}^{\sf T}+(\hat{\bf b}\hat{\bf a}^{\sf T})^{\sf T}$. We remind that with this
definition the covariance matrix for a single-mode system reads

$\displaystyle\bm{\sigma}=2\left(\begin{array}[]{c
c}\langle\Delta\hat{q}^{2}\rangle&\langle\Delta\hat{q}\hat{p}\rangle\\\
\langle\Delta\hat{q}\hat{p}\rangle&\langle\Delta\hat{p}^{2}\rangle\end{array}\right)$
(24)

where

$\displaystyle\langle\Delta\hat{A}\hat{B}\rangle=\Tr\left[\varrho\left(\frac{\hat{A}\hat{B}+\hat{B}\hat{A}}{2}\right)\right]-\Tr[\varrho\hat{A}]\Tr[\varrho\hat{B}]$
(25)

The covariance matrix thus directly contains all the squeezing properties of
the quantum state $\varrho$.

As we mentioned in the main text, the dynamics induced by the stochastic
master equation (4) preserves the Gaussian character of the quantum state, and
the corresponding evolution is described by the equations

$\displaystyle d\bar{\bf r}_{c}$ $\displaystyle=A\bar{\bf
r}_{c}\,dt+(E-\bm{\sigma}_{c}B)\frac{{\bf dw}_{t}}{\sqrt{2}}\,,$ (26)
$\displaystyle\frac{d\bm{\sigma}_{c}}{dt}$
$\displaystyle=A\bm{\sigma}_{c}+\bm{\sigma}_{c}A^{\sf
T}+D-(E-\bm{\sigma}_{c}B)(E-\bm{\sigma}_{c}B)^{\sf T},$ (27)

while the continuous homodyne outcome is written in vectorial form as

$\displaystyle{\bf dy}_{t}=-\sqrt{2}B^{\sf T}\bar{\bf r}_{c}\,dt+{\bf
dw}_{t}\,.$ (28)

Figure 6: Performance of the control strategies as a function of time,
quantified by the different Fisher informations divided by time, and for
different values of $\eta$ (from left to right: $\eta=\\{0.9,0.5,0.1\\}$). All
the other parameters are fixed as in Fig. 2.

The matrices entering in these equations can be derived by following different
approaches [67, 66], and for the physical setup we are interested in reads

$\displaystyle A$ $\displaystyle=\left(\begin{array}[]{c
c}-(\chi+\kappa/2)&\omega\\\ -\omega&\chi-\kappa/2\end{array}\right)\,,$ (31)
$\displaystyle D$ $\displaystyle=\kappa\mathbbm{1}_{2}\,,$ (32) $\displaystyle
B$ $\displaystyle=E=\left(\begin{array}[]{c c}-\sqrt{\eta\kappa}&0\\\
0&0\end{array}\right)\,.$ (35)

We observe that the matrices $B$ and $E$ are singular. The second component of
the Wiener increment ${\bf dw}_{t}$ in (28), therefore, does not play any role
at all, whereas the first component is determined by the homodyne detection
output.

The solution of the Riccati equation for the covariance matrix (8) can be in
general obtained numerically. However an analytical solution can be obtained
for the steady-state covariance matrix for $\omega=0$ and by assuming a stable
dynamics (that is $\chi<\kappa/2$), leading to

$\displaystyle\bm{\sigma}_{c}^{\sf ss}(\eta)=\left(\begin{array}[]{c
c}\frac{\kappa(2\eta-1)-2\chi+\sqrt{\kappa^{2}-4\kappa\chi(2\eta-1)+4\chi^{2}}}{2\eta\kappa}&0\\\
0&\frac{\kappa}{\kappa-2\chi}\end{array}\right)\,.$ (38)

Two opposite regimes can be observed here. By taking the limit for the
efficiency $\eta\to 0$, and assuming a stable dynamics, we indeed obtain the
solution for the unconditional (unmonitored) dynamics

$\displaystyle\bm{\sigma}_{\sf unc}^{\sf ss}=\left(\begin{array}[]{c
c}\frac{\kappa}{\kappa+2\chi}&0\\\
0&\frac{\kappa}{\kappa-2\chi}\end{array}\right)\,.$ (41)

We thus find that for $0<\chi<\kappa/2$, the Hamiltonian squeezes the
$\hat{q}$ quadrature, with a maximum amount of 3dB of squeezing near
instability, that is for $\chi\to\kappa/2$ [68, 69]. In the opposite case of
perfect monitoring, that is for $\eta=1$, we find

$\displaystyle\bm{\sigma}_{c}^{\sf ss}=\left(\begin{array}[]{c
c}\frac{\kappa-2\chi}{\kappa}&0\\\
0&\frac{\kappa}{\kappa-2\chi}\end{array}\right)\,,$ (44)

which in turn, for $0<\chi<\kappa/2$, corresponds to even smaller variances of
the $\hat{q}$ quadrature, and in principle infinite squeezing near
instability. We remark that for $\omega\neq 0$ we numerically find that a
lower amount of squeezing can be generated and that the most squeezed
quadrature depends on the value of $\omega$ itself.

Figure 7:  Scatter plot of the squeezing (expressed in dB) perpendicular to
the conditional first moment vector $\bar{\mathbf{r}}_{c}$ and the absolute
value of the first moment vector $|\bar{\mathbf{r}}_{c}|$, for the three
control strategies at time $\kappa t=180$ and for different values of $\eta$
(from left to right, $\eta=\\{0.9,0.5,0.1\\}$). Figure 8:  Scatter plot
between the absolute value of the first moment vector $|\bar{\mathbf{r}}_{c}|$
and the trajectory-contribution to the homodyne Fisher information
$\log(f_{\sf hom}^{\sf(traj)})$, at time $\kappa t=180$ and for different
values of $\eta$ (from left to right, $\eta=\\{0.9,0.5,0.1\\}$). We plot only
the points corresponding to the agent and the no-control strategy, as $f_{\sf
hom}^{\sf(traj)}=0$ for the OL-strategy.

As we described in the previous section, the perfomance of the metrological
protocol is quantified by the effective QFI defined in Eq. (21). Being the
quantum states Gaussian, also this figure of merit can be derived from the
information contained in first and second moments. In particular, one shows
that the homodyne classical Fisher information can be evaluated as [37]

$\displaystyle\mathcal{F}_{\sf hom}=\mathbbm{E}_{\sf traj}\left[2\int
dt\,(\partial_{\omega}\bar{\bf r}_{c})^{\sf T}BB^{\sf
T}(\partial_{\omega}\bar{\bf r}_{c})\right]\,,$ (45)

while the QFI of the (Gaussian) conditional state is obtained via the formula
[76]

$\displaystyle\mathcal{Q}[\varrho_{c}]$
$\displaystyle=\frac{\Tr\left[\left(\bm{\sigma}_{c}^{-1}(\partial_{\omega}\bm{\sigma}_{c})\right)^{2}\right]}{2(1+\mu^{2})}+\frac{2(\partial_{\omega}\mu)}{1-\mu^{4}}\,$
$\displaystyle\,\,\,\,+2(\partial_{\omega}\bar{\bf r}_{c})^{\sf
T}\bm{\sigma}_{c}^{-1}(\partial_{\omega}\bar{\bf r}_{c})\,,$ (46)

with

$\displaystyle\mu=\Tr[\varrho_{c}^{2}]=1/\sqrt{{\rm Det}[\bm{\sigma}_{c}]}$
(47)

denoting the purity of the conditional quantum state. We thus also need the
evolution of the derivatives of first and second moments respect to the
parameter $\omega$, that can be numerically integrated via the equations [37]

$\displaystyle d(\partial_{\omega}\bar{\bf r}_{c})$
$\displaystyle=(\partial_{\omega}A)\bar{\bf
r}_{c}\,dt+A(\partial_{\omega}\bar{\bf
r}_{c})\,dt-\frac{(\partial_{\omega}\bm{\sigma}_{c})B\,d{\bf
w}_{t}}{\sqrt{2}}+(E-\bm{\sigma}_{c}B)B^{\sf T}(\partial_{\omega}\bar{\bf
r}_{c})\,dt$ (48)
$\displaystyle\frac{d(\partial_{\omega}\bm{\sigma}_{c})}{dt}$
$\displaystyle=(\partial_{\omega}A)\bm{\sigma}_{c}+\bm{\sigma}_{c}(\partial_{\omega}A)^{\sf
T}+A(\partial_{\omega}\bm{\sigma}_{c})+(\partial_{\omega}\bm{\sigma}_{c})A^{\sf
T}+(\partial_{\omega}\bm{\sigma}_{c})B(E-\bm{\sigma}_{c}B)^{\sf
T}+(E-\bm{\sigma}_{c}B)B^{\sf T}(\partial_{\omega}\bm{\sigma}_{c})\,.$ (49)

At each time $t$, also the purity $\mu$ and its derivative
$\partial_{\omega}\mu$ can be directly obtained from $\bm{\sigma}_{c}$ and
$(\partial_{\omega}\bm{\sigma}_{c})$ via the formula in Eq. (47).

We remark that these quantities are also exploited as observations for the
neural network that optimizes the feedback strategy. In order to train our
agent and to assess the performance of the different protocols, we have thus
numerically simulated different trajectories of the quantum states via Eqs.
(26), (27), (48) and (49), and we have performed the numerical integral and
the numerical average in Eqs. (45) and (21).

##  Appendix C The effect of the Hamiltonian coupling constant $\chi$

In this appendix we discuss the role of the Hamiltonian coupling constant
$\chi$ in the learning of the strategy by the agent, and on its performances.
As we highlighted in the text, the parameter $\chi$ is directly responsible
for the generation of squeezing in the conditional states. In fact squeezing
can be observed if and only if $\chi$ is larger than zero, and in particular
near criticality, i.e. for $\chi\approx\kappa/2$ the amount of squeezing
generated is close to infinity in the case of perfect monitoring.
In Fig. 9 we plot our figure of merit $\mathcal{Q}_{\sf eff}/t$, obtained by
agents trained with different values of
$\chi=\\{0,0.35\kappa,0.45\kappa,0.49\kappa\\}$, as a function of time, and
compare it with the other benchmark strategies. We observe how the agent
allows to reach values of QFI larger than those obtained by means of the other
strategies, in particular in the long time limit. Other relevant observations
can be drawn from these plots: i) In general, we find that, for all
strategies, larger values of $\chi$ yield larger values of QFI thus
highlighting, once again, the importance of the squeezing generated during the
dynamics. ii) For values of $\chi$ that are large enough (e.g. for
$\chi=0.45\kappa$ and $\chi=0.49\kappa$) the agent is able to devise a
strategy such that the maximum of $\mathcal{Q}_{\sf eff}/t$ is observed in the
long time limit; for smaller values of $\chi$ (e.g. $\chi=0.35\kappa$) one
observes a maximum at short times, while in the long time limit the ratio
between effective QFI and time tends to a smaller stationary value. iii) We
also find that for $\chi=0.45\kappa$ the maximum of $\mathcal{Q}_{\sf eff}/t$
obtained for the open-loop control strategy is compatible with the value
obtained in the long time limit via the agent’s strategy. We stress, however,
that while for the open-loop strategy one would need to stop the dynamics at a
very specific time, by employing the RL-strategy one has a wide available time
window, in which the dynamics has reached a steady-state behaviour.

To better describe the properties of the strategies devised by the neural
network, we have added an extra curve the in the panels (a) and (d) of Fig. 9:
in panel (a) we have added the values of QFI corresponding to the agent
trained at $\chi=0.49\kappa$ applied to the case $\chi=0$, while in panel (d)
we have added the performance of the agent trained at $\chi=0$ applied to the
case $\chi=0.49\kappa$. In the first scenario we observe that the agent
trained at $\chi=0.49\kappa$ is still able to beat the benchmark strategies
and is only slightly less performing than the properly trained agent; in the
second scenario the situation is completely inverted: the agent trained at
$\chi=0$ is indeed yielding very low values of QFI. Our interpretation of this
result is the following: when the agent is trained near criticality, it learns
how to optimize both first moments and squeezing, and thus is performing well
also when squeezing is absent. On the other hand, at $\chi=0.49\kappa$, that
is when squeezing is playing a major role in the estimation protocol, the
agent trained in the no-squeezing scenario completely fails in enhancing the
estimation precision.

Figure 9: Performance of the control strategies as a function of time,
quantified by the effective quantum Fisher information $\mathcal{Q}_{\sf eff}$
divided by time, and for different values of $\chi$: panel (a): $\chi=0$;
panel (b) $\chi=0.35\kappa$; panel (c): $\chi=0.45\kappa$; panel (d):
$\chi=0.49\kappa$. Notice that in panel (a) the scale of the y-axis is
logarithmic and that the curve corresponding to the open-loop strategy
$\mathcal{Q}_{\sf eff}^{(OL)}$ has not been reported as the corresponding
values are almost negligible. Furthermore in panel (a) and panel (d) we have
added an extra red curve, corresponding to the agent trained at respectively
$\chi=0.49$ and at $\chi=0$.
The results have been obtained simulating $N=1000$ trajectories with a time-
step $dt=0.001/\kappa$, by fixing the other parameters as $\omega=0.1\kappa$,
$\eta=0.9$ and by considering as an initial state a thermal state with $n_{\sf
th}=5$ and initial first moment vector $\bar{\bf r}_{c}(0)=(0,0)$.

##  Appendix D The effect of continuous monitoring efficiency

Here we present some results that we obtained for different values of the
monitoring efficiency, $\eta=\\{0.9,0.5,0.1\\}$. In Fig. 6 we report the
behaviour of the different Fisher information divided by time as in Fig. 2. We
observe how the enhancement is still clearly observed for $\eta=0.5$ and that
even for very small monitoring efficiency ($\eta=0.1$) the agent-feedback is
able to yield a larger estimation precision compared to the other strategies,
in particular in the long time limit.

We have also reported in Fig. 7 scatter plots of the perpendicular squeezing
defined in the main text and the absolute value of the first moment vector
$|\bar{\bf r}_{c}|$ for a fixed time $\kappa t=180$. We observe as remarked
also in Fig. 4, that, with respect to the no-control strategy, for $\eta=0.9$
the agent is able to prepare trajectories with larger perpendicular squeezing.
However in this plot we also observe that the agent is in general able to
prepare conditional states that, for a fixed amount of perpendicular
squeezing, yields larger first moments, and thus leading to an enhanced
estimation. By reducing the monitoring efficiency, we find that the first
effect (trajectories with larger perpendicular squeezing) is basically lost
also for $\eta=0.5$, while the second effect, that is, larger first moments at
fixed squeezing, is still obtained and thus it can be considered as the sole
responsible for the better estimation precision.

The relevance of the first moment vector is also highlighted in Fig. 8,
corresponding to the scatter plot of $|\bar{\bf r}_{c}|$ and $\log(f_{\sf
hom}^{\sf(traj)})$ for the different trajectories, where we have introduced
the quantity

$\displaystyle f_{\sf hom}^{\sf(traj)}=2\int dt\,(\partial_{\omega}\bar{\bf
r}_{c})^{\sf T}BB^{\sf T}(\partial_{\omega}\bar{\bf r}_{c})\,,$ (50)

corresponding to the contribution of each trajectory to the homodyne Fisher
information $\mathcal{F}_{\sf hom}=\mathbbm{E}[f_{\sf hom}^{\sf(traj)}]$ (see
Eq. (11)). In the figure we indeed observe how the two quantities seem to be
correlated for the agent strategy, while they seem uncorrelated for the no-
control strategy (we remind that for the OL-strategy $f_{\sf
hom}^{\sf(traj)}=0$ for each trajectory), highlighting once again the
mechanism behind the strategy devised by the agent.

##  Appendix E On the effectiveness of homodyne detection as a final strong
measurement

In this appendix we discuss the effectiveness of homodyne detection as a final
strong measurement in our protocol, i.e., we calculate the FI of a final
homodyne measurement and we compare it to the QFI $\bar{\mathcal{Q}}_{c}$. We
remark that in general a non-Gaussian measurement may be needed to saturate
the quantum Cramér-Rao bound, that is to obtain a classical FI equal to the
QFI; however we expect that homodyne detection is going to extract a fair
amount of the maximum information achievable, being nearly optimal for pure
Gaussian states (see Refs. [61, 72] for an extensive study of phase estimation
with Gaussian states).

A projective Gaussian measurement can be modeled by the covariance matrix of a
squeezed vacuum state, with squeezing parameter $s$, and a phase rotation of
angle $\theta$:

$\bm{\sigma}_{m}(z,\theta)=R(\theta)\text{diag}(z,1/z)R(\theta)^{\sf T},$ (51)

where $z=\exp(2s)$ and $R(\theta)$ is a rotation matrix. In the limit
$z\rightarrow 0$ we have a homodyne measurement: when $\theta=0$
($\theta=\pi/2)$, the quadrature $\hat{q}$ ($\hat{p}$) is measured.

The measurement outcome probability of such measurement on a state with first
moments $\mathbf{r}_{c}$ and covariance matrix $\bm{\sigma}_{c}$ is a
bidimensional Gaussian distribution $\mathcal{N}(\mathbf{r}_{c},\Sigma)$,
where $\Sigma=(\bm{\sigma}_{c}+\bm{\sigma}_{m})/2$, for which the FI reads

$\mathcal{F}[\mathcal{N}(\mathbf{r}_{c},\Sigma)|\omega]=(\partial_{\omega}\bm{r}_{c})^{\mathsf{T}}\Sigma^{-1}\partial_{\omega}\bm{r}_{c}+\frac{1}{2}\text{Tr}[(\Sigma^{-1}\partial_{\omega}\Sigma)^{2}].$
(52)

For each trajectory, we maximize the FI for a final homodyne measurement over
$\theta\in[-\pi/2,\pi/2]$, $\mathcal{F}_{c}^{(\sf hd)}$. We then calculate the
average $\bar{\mathcal{F}}_{c}^{(\sf hd)}$ and compare it with
$\bar{\mathcal{Q}}_{c}$. The resulting ratio $\bar{\mathcal{F}}_{c}^{(\sf
hd)}/\bar{\mathcal{Q}}_{c}$ is shown in Fig. 10 for the three different
strategies. As can be seen, the optimized homodyne detection, although not
ideal, allows for the extraction of a significant amount of information from
the quantum state. Finding the optimal angle $\theta$ in real-time should be
possible by means, for example, of a FPGA (see discussion on the feedback
real-time implementation in Sec. VI). The dotted lines in the inset of Fig. 10
show that even if a fixed, and suitably a priori chosen, angle $\theta$ is
used, it is anyway possible to extract a significant fraction of the QFI for
the state (we set $\theta=0$ for the RL strategy, corresponding to measuring
$\hat{q}$, as suggested by a direct inspection of the distribution of the
optimal $\theta$ values for the different trajectories). The overall effect on
the effective Fisher information when using homodyne detection as strong final
measurement is shown in the main panel of Fig. 10: in the strategy devised by
the RL agent, where the contribution of the final strong measurement to
$\mathcal{Q}_{\sf eff}$ is small, the ratio between the effective Fisher
information $\mathcal{F}_{\sf eff}$ corresponding to a final homodyne
detection (either optimized or for $\theta=0$) and the corresponding to
effective QFI $\mathcal{Q}_{\sf eff}$ tends rapidly to one, by increasing the
monitoring time $\kappa t$. We also notice that the ratio is reasonably high
also for the other benchmark strategies: the more significant effect can be
seen in the OL scenario, particularly at short times, due to the fact that
$\mathcal{Q}_{\sf eff}=\bar{\mathcal{Q}}_{c}$.

Figure 10: Ratio between the effective FI $\mathcal{F}_{\sf eff}$ for final
homodyne detection and the effective QFI of the conditional state
$\bar{\mathcal{Q}}_{\sf eff}$ as a function of $\kappa t$ for the three
different strategies (the inset shows the ratio between the average FI of the
strong homodyne measurement $\bar{\mathcal{F}}_{c}^{(\sf hd)}$ and the average
QFI $\bar{\mathcal{Q}}_{c}$. The solid lines show the FI
$\bar{\mathcal{F}}_{\sf hd,opt}$ optimized over $\theta$, while the dotted
line shows the FI $\bar{\mathcal{F}}_{\sf hd,\hat{q}}$ obtained with a
homodyne on the quadrature $\hat{q}$ for the RL strategy. In all cases, while
not being optimal, homodyne detection allows to extract a significant fraction
of the total available information on the frequency $\omega$ (parameters
values are fixed as in Fig. 2). Figure 11: $\mathcal{Q}_{\sf eff}/t$ (solid
lines) and $\mathcal{F}_{\sf eff}/t$ for homodyne detection (dashed lines) as
functions of $\kappa t$ for the three different strategies under
consideration. Using homodyne detection as a final strong measurement does not
affect the precision in the estimation significantly for the RL strategy,
given that the biggest contribution to $\mathcal{Q}_{\sf eff}$ comes from the
continuous monitoring. The dotted blue line shows $\mathcal{F}_{\sf eff}/t$
for homodyne detection on the $\hat{q}$ quadrature, for the RL strategy. Even
without optimizing the angle of the final homodyne measurement, the RL-
strategy allows to overcome the performances of the other benchmark strategies
(parameters values are fixed as in Fig. 2).

We finally compare the performance of a final homodyne detection for the three
strategies in Fig. 11, where we indeed observe that the enhancement obtained
via the RL-strategy is still mantained when a final homodyne detection is
performed.